Welcome to Seven's page

hash table and algorithm

Wed, 05 Apr 2017 00:00:00 +0000

hashing function

collision resistance
- attach SHA1/MD5 less than 2^(N/2), N bits of hash output.
uniform distribution

collision resolution

chaining with list/RBT/…
- linkedlist vs RBT
open address (hash(key)+ di) mod m, ith times to collision
- how to avoid cluster?

hash open address application in java api

ThreadLocalMap ```java

private Entry getEntry(ThreadLocal<?> key) {
    int i = key.threadLocalHashCode & (table.length - 1);
    Entry e = table[i];
    if (e != null && e.get() == key)
        return e;
    else
        return getEntryAfterMiss(key, i, e); //
}

  
# hash chaining application in java api 
    * LinkedHashMap
      - double-linked list
      - access order by move access node to last
      - default by insert order
    * Hashtable
      - chaining with single linked list
      - indexing
          - hashtable: int index = (hash & 0x7FFFFFFF) % tab.length;
          - hashmap:   int index = (hash ^ (hash>>>16)) % (tab.length-1);
    * HashMap
      - if binCount <6, untrieefy to list 
      - if binCount >8, (if table.length >64, trieefy to RBT; else resize())
      - if map.size > (threshold=capacity*loadfactor), resize()

```java

     /**
       * The maximum capacity, used if a higher value is implicitly specified
       * by either of the constructors with arguments.
       * MUST be a power of two <= 1<<30.
       */
      static final int MAXIMUM_CAPACITY = 1 << 30;

      /**
       * The load factor used when none specified in constructor.
       */
      static final float DEFAULT_LOAD_FACTOR = 0.75f;

      /**
       * The bin count threshold for using a tree rather than list for a
       * bin.  Bins are converted to trees when adding an element to a
       * bin with at least this many nodes. The value must be greater
       * than 2 and should be at least 8 to mesh with assumptions in
       * tree removal about conversion back to plain bins upon
       * shrinkage.
       */
      static final int TREEIFY_THRESHOLD = 8;

      /**
       * The bin count threshold for untreeifying a (split) bin during a
       * resize operation. Should be less than TREEIFY_THRESHOLD, and at
       * most 6 to mesh with shrinkage detection under removal.
       */
      static final int UNTREEIFY_THRESHOLD = 6;

      /**
       * The smallest table capacity for which bins may be treeified.
       * (Otherwise the table is resized if too many nodes in a bin.)
       * Should be at least 4 * TREEIFY_THRESHOLD to avoid conflicts
       * between resizing and treeification thresholds.
       */
      static final int MIN_TREEIFY_CAPACITY = 64;
      /**
       * The table, initialized on first use, and resized as
       * necessary. When allocated, length is always a power of two.
       * (We also tolerate length zero in some operations to allow
       * bootstrapping mechanics that are currently not needed.)
       * RBT Or LinkedList
       */
      transient Node<K,V>[] table;

      /**
       * The number of key-value mappings contained in this map.
       */
      transient int size;

      /**
       * The number of times this HashMap has been structurally modified
       * Structural modifications are those that change the number of mappings in
       * the HashMap or otherwise modify its internal structure (e.g.,
       * rehash).  This field is used to make iterators on Collection-views of
       * the HashMap fail-fast.  (See ConcurrentModificationException).
       */
      transient int modCount;

      /**
       * The next size value at which to resize (capacity * load factor).
       *
       * @serial
       */
      int threshold;

      /**
       * The load factor for the hash table.
       *
       * @serial
       */
      final float loadFactor;

Java author Joshua J. Bloch

the Java Collections Framework
http://www.javaworld.com/article/2073877/core-java/joshua-bloch–a-conversation-about-design.html
http://www.oracle.com/technetwork/articles/javase/bloch-effective-08-qa-140880.html

Distributed HashTable

DHT need to handle:
- autonomy and decentralization
- fault tolerance
- scalability
- load balance
- data integrity
- performance (routing/storage/retrieval)
Keyspace partitioning Task: removal or addition of one node changes only the set of keys owned by the nodes with adjacent IDs, and leaves all other nodes unaffected.
- Consistent hashing hashing ring + tags
- Rendezvous hashing client side: highest random weight hashing, selected = max((h(s,key) for s in serverlist)
- Locality-preserving hashing efficient range query by ensure that similar keys are assigned to similar objects.

Bloomfilter

used to test whether an element is a member of a set.
False positive matches are possible, but false negatives are not
– in other words, a query returns either “possibly in set” or “definitely not in set”.
Elements can be added to the set, but not removed (though this can be addressed with a “counting” removed_filter, but add removed items back is not supported.);
the more elements that are added to the set, the larger the probability of false positives.

    public boolean add(T object) {
        byte[] state = encode(object);

        int hashIterations = this.hashIterations;
        long size = this.size;
        long[] indexes = hash(state, hashIterations, size);
        ...set bits...
    }

    private long[] hash(byte[] state, int iterations, long size) {
        long hash1 = LongHashFunction.xx_r39().hashBytes(state);
        long hash2 = LongHashFunction.farmUo().hashBytes(state);

        long[] indexes = new long[iterations];
        long hash = hash1;
        for (int i = 0; i < iterations; i++) {
            indexes[i] = (hash & Long.MAX_VALUE) % size;
            if (i % 2 == 0) {
                hash += hash2;
            } else {
                hash += hash1;
            }
        }
        return indexes;
    }

Reference

https://en.wikipedia.org/wiki/Poisson_distribution

famous authors in Java

Wed, 05 Apr 2017 00:00:00 +0000

James Arthur Gosling

Java VM father
Scala advisor

Douglas S. Lee

concurrent data structures
http://gee.cs.oswego.edu/dl/index.html

Joshua J. Bloch

the Java Collections Framework
the java.math package
the assert mechanism
http://www.javaworld.com/article/2073877/core-java/joshua-bloch–a-conversation-about-design.html
http://www.oracle.com/technetwork/articles/javase/bloch-effective-08-qa-140880.html

TODO

LCS algorithm

Wed, 05 Apr 2017 00:00:00 +0000

LCS large common subsequence

In general, LCS algorithms can be solved in O(m∗n)O(m∗n) time and space using basic dynamic programming. Hirschberg’s algorithm reduces the space requirement to O(min(m,n))O(min(m,n)), thus making it feasible for very large substrings (time is usually cheaper than memory)

* DP
input: x[m], y[n]
optimal substructure: f[m-1][n-1]= f[m-2][n-2]+1 if x[m-1]==y[n-1] else max(f[m-1][n-2],f[m-2][n-1])

# A Naive recursive Python implementation of LCS problem
def lcs(X, Y, m, n):
    if m == 0 or n == 0:
       return 0;
    elif X[m-1] == Y[n-1]:
       return 1 + lcs(X, Y, m-1, n-1);
    else:
       return max(lcs(X, Y, m, n-1), lcs(X, Y, m-1, n));

def lcs2(X , Y):
    # find the length of the strings
    m = len(X)
    n = len(Y)
 
    # declaring the array for storing the dp values
    L = [[None]*(n+1) for i in xrange(m+1)]
 
    """Following steps build L[m+1][n+1] in bottom up fashion
    Note: L[i][j] contains length of LCS of X[0..i-1]
    and Y[0..j-1]"""
    for i in range(m+1):
        for j in range(n+1):
            if i == 0 or j == 0 :
                L[i][j] = 0
            elif X[i-1] == Y[j-1]:
                L[i][j] = L[i-1][j-1]+1
            else:
                L[i][j] = max(L[i-1][j] , L[i][j-1])
 
    # L[m][n] contains the length of LCS of X[0..n-1] & Y[0..m-1]
    return L[m][n]
#end of function lcs

LCs large common substring

DP O(nm)
suffix tree time: O(n+m)

application

diff
git merge

Reference

https://en.wikipedia.org/wiki/Hirschberg%27s_algorithm
LUCS https://leetcode.com/problems/longest-uncommon-subsequence-ii/#/description
Reverse Index

count distinct problem

Thu, 23 Mar 2017 00:47:14 +0000

streaming algorithm

HyperLogLog

online algorithm

Ref

HyperLogLog

markdown mode in emacs

Sun, 19 Mar 2017 07:47:56 +0000

h1: markdown mode in emacs

# h1: markdown mode in emacs #

h2 styles

## h2 styles ##

bold

**bold**

list items

a
b
- a
- b

* a
* b
  * a
  * b

quotes

line1…
line1

line2

>line1...<br>
line1

>line2

links
google

[google](http://google.com)

h2 code block

inline code alert(s)

inline code `alert(s)`

python code

```python
def say:
    yield 1
    pass
```

def say:
    yield 1
    pass

redis study notes

Fri, 17 Mar 2017 07:47:56 +0000

redis key notes

redis data types
cluster design
write safety
- Redis cluster protocol
nodes handshake on discovering
fault tolerance
slave election / promotion
- Sentinel
quorum
TILT mode
- keys distribution model
hash slot/hash tag
hash algorithm
consistent hash
- RedLock
algorithm
- Raft
roles
how to elect leader?
how to replicate log entries?
visualization (RaftScope)

redis data types

how to select redis key?
- binary safe
- schema, e.g. “comment:123:reply-to”
- max key size = 512MB
string/list/capped list/set/range/bitmap

cluster design

performance
- async replication
- avoid key-value version conflicts or merge
- scaling readonly reads using slave nodes
scalability
- add/remove nodes
- redirection & resharding
- key-node mapping
avaliability
- failover
- partition
- persistent
  - AOF vs RDB Snapshot
- replicate migration for orphaned masters

write safety

failure cases
- master die without write reaching slaves before slave promoted
- out-of-date routing table in client side

Redis cluster protocol

discover new nodes
promote slave to master
propagate Pub/Sub msg

nodes handshake on discovering

admin command CLUSTER MEET ip port
auto-discover into fully connnected graph

if A know B and B know C, then B send gossip msg to A about C

fault tolerance

heartbeat & gossip msg ping/pong
failure detection PFAIL/FAIL flag

slave election / promotion

epoch - term in Raft
winer slave gain majority votes from masters

Sentinel

quorum

* quorum - number of sentinels agree about master failure
* start failover only when, e.g. quorum=2, total=5
    * 2 agree about one master failure
    * 3+/majority sentinels reachable (=== no failover in minority partition)

TILT mode

stop acting/response failure request until computer time resolved.

if the time difference is negative or unexpectedly big (2 seconds or more) the TILT mode is entered (or if it was already entered the exit from the TILT mode postponed)

keys distribution model

hash slot/hash tag

//HASH_SLOT = CRC16(key) mod 16384
def HASH_SLOT(key)
    s = key.index "{"
    if s
        e = key.index "}",s+1
        if e && e != s+1
            key = key[s+1..e-1]
        end
    end
    crc16(key) % 16384
end

hash algorithm

how to select a hash function?
1. input data
2. probability distribution

consistent hash

* 1-n mapping: node-vnode
hashlist = hash(nodename+vnode[i]) for i in range(n)
rbt.add(hashlist)

* hit cache
h1 = hash(request)
cache = rbt.findCacheNode(h1)
response = cache.handle(request)

RedLock

algorithm

if (cluster.acquireCount(key,val) > N/2+1) and elapsed_time < lock_validity_time:
    return true
else:
    cluster.unlockAll(key,val) 
    return false

Raft

roles

follower
candidate
leader

how to elect leader?

terms
avoid split vote by randomized selection timeout

how to replicate log entries?

commit once log entry has replicated it on a majority of servers

visualization (RaftScope)

Here’s a Raft cluster running in your browser. You can interact with it to see Raft in action. Five servers are shown on the left, and their logs are shown on the right. We hope to create a screencast soon to explain what’s going on.

Redisson

lock implementation

    <T> RFuture<T> tryLockInnerAsync(long leaseTime, TimeUnit unit, long threadId, RedisStrictCommand<T> command) {
        internalLockLeaseTime = unit.toMillis(leaseTime);

        return commandExecutor.evalWriteAsync(getName(), LongCodec.INSTANCE, command,
                  "if (redis.call('exists', KEYS[1]) == 0) then " +
                      "redis.call('hset', KEYS[1], ARGV[2], 1); " +
                      "redis.call('pexpire', KEYS[1], ARGV[1]); " +
                      "return nil; " +
                  "end; " +
                  "if (redis.call('hexists', KEYS[1], ARGV[2]) == 1) then " +
                      "redis.call('hincrby', KEYS[1], ARGV[2], 1); " +
                      "redis.call('pexpire', KEYS[1], ARGV[1]); " +
                      "return nil; " +
                  "end; " +
                  "return redis.call('pttl', KEYS[1]);",
                    Collections.<Object>singletonList(getName()), internalLockLeaseTime, getLockName(threadId));
    }

unlock implementation

public void unlock() {
        Boolean opStatus = commandExecutor.evalWrite(getName(), LongCodec.INSTANCE, RedisCommands.EVAL_BOOLEAN,
                        "if (redis.call('exists', KEYS[1]) == 0) then " +
                            "redis.call('publish', KEYS[2], ARGV[1]); " +
                            "return 1; " +
                        "end;" +
                        "if (redis.call('hexists', KEYS[1], ARGV[3]) == 0) then " +
                            "return nil;" +
                        "end; " +
                        "local counter = redis.call('hincrby', KEYS[1], ARGV[3], -1); " +
                        "if (counter > 0) then " +
                            "redis.call('pexpire', KEYS[1], ARGV[2]); " +
                            "return 0; " +
                        "else " +
                            "redis.call('del', KEYS[1]); " +
                            "redis.call('publish', KEYS[2], ARGV[1]); " +
                            "return 1; "+
                        "end; " +
                        "return nil;",
                        Arrays.<Object>asList(getName(), getChannelName()), LockPubSub.unlockMessage, internalLockLeaseTime, getLockName(Thread.currentThread().getId()));
                    }

Questions:

1. How does redis provide high avaliability and reliability?
   - Redis Sentinel
   - Redis Cluster
     - scale/add node/add replica slave
     - slave replicas
       - replicas auto-migration
       - async-replicate --> weak consistency
     - data sharding/partition
       - hashing(hash slot) vs consistent hashing(hashring+vnode)
   - Raft algorithm

2. How to persistent and recovery data in Redis?
   - AOF
   - RDB

4. why redis is fast? 
   - memory
   - network 
     - no blocking i/o api (epoll/kqueue, edge trigger > level trigger) 
   - scaling reading(more read-only salves)/writing(sharding by more master)

5. user stories
   - DB vs cache
   - message broker
   - RedLock
   - Bloomfilter

Reference:

emacs

Fri, 17 Mar 2017 07:46:26 +0000

emacs configuration

use vi-mode as default

(require 'package) (add-to-list 'package-archives '("melpa" . "http://melpa.org/packages/")) (package-initialize) (require 'evil) (evil-mode 1)

install package

1. install org-mode
`M-x package-install RET org-mode RET`

2. install markdown
`M-x package-install RET markdown-mode RET`

Others

1. input "shell" output
   e.g. C-u M-! date

Java Technical Notes

Sat, 11 Mar 2017 03:24:55 +0000

Java Technical Notes

kafka

used for building real-time data pipelines and streaming apps. scalable/fault-tolerant/fast

ref: kafka

Redis

Active MQ/Kafka

Lucence/ Solr

FastDFS

MyBatis / Hibernate

HashMap history

1. linkedlist -> RBT
2. enhance hash algorithm in string

json

how to parse json faster?
- small memory footprint
- stream API?

front end study notes

Thu, 02 Mar 2017 03:24:55 +0000

1. javascript
2. CSS

javascript

keywords

closure

hide sth/ enable private method
use lexicalEnv as memoizer

“use strict”

- “prototype”

obj.__proto__ function.prototype/function.__proto__
- Object.create(proto, propertiesObject)

obj = new Function;
- obj.__proto__ = Function.prototype
- obj.constructor = Function = Function.prototype.constructor

- constructor

obj.constructor | Object.prototype.constructor alert( rabbit.hasOwnProperty(‘constructor’) ) / false alert( Rabbit.prototype.hasOwnProperty(‘constructor’) ) / true

- LexicalEnvironment | closure | block

- ECMAScript ES3/ES5/ES6

- typeof | instanceof | null | undefined

- compare

stric equal =
abstract equal ==
Object.is(val1,val2)

- this

called as obj.method
called as function
- Window/Global/undefined (“use strict”)
called in new
explicit this func.call(context, args…) func.apply(context, [arg]|args…) func.bind(context, args…)()

Built-in obj

Function
1. arguments arguments.callee
2. static fields: arguments.callee.count

DOM document object model

BOM browser object model

alert/confirm/prompt
url
frame
XMLHttpRequest

FOP

Arrow Functions => (ES6 only, non IE support!)

definition

An arrow function expression has a shorter syntax than a function expression and does not bind its own this, arguments, super, or new.target. These function expressions are best suited for non-method functions, and they cannot be used as constructors.

(param1, param2, …, paramN) => { statements }
(param1, param2, …, paramN) => expression
// equivalent to: (param1, param2, …, paramN) => { return expression; }
    
// Parentheses are optional when there's only one parameter:
(singleParam) => { statements }
singleParam => { statements }
    
// A function with no parameters requires parentheses:
() => { statements }
() => expression // equivalent to: () => { return expression; }
    
// Parenthesize the body to return an object literal expression:
params => ({foo: bar})
    
// Rest parameters and default parameters are supported
(param1, param2, ...rest) => { statements }
(param1 = defaultValue1, param2, …, paramN = defaultValueN) => { statements }
    
// Destructuring within the parameter list is also supported
var f = ([a, b] = [1, 2], {x: c} = {x: a + b}) => a + b + c;
f();  // 6

not bind its own this, arguments, super, or new.target; not as constructor

'use strict';
var obj = {
  i: 10,
  b: () => console.log(this.i, this),
  c: function() {
    console.log(this.i, this);
  }
}
obj.b(); // prints undefined, undefined
obj.c(); // prints 10, Object {...}

forEach/for in/for of

var s = 'a𝓌a';
for(var c of s) console.log(c);

for(var c in s) console.log(s[c]);
Array.prototype.forEach.call(s, c => console.log(c));

every/some

reverse/sort

map/reduce/filter

var materials = ['Hydrogen', 'Helium', 'Lithium', 'Beryllium'];
var result = materials.map(material => material.length).filter(n=>n>7).reduce((a,b)=>a+b);

reduce

[0, 1, 2, 3, 4].reduce( (accumulator, currentValue, currentIndex, array) => accumulator + currentValue,

100 );

generator/yield/*yield

Promise

OOP patterns

pseudo-class

function Hamster() { 
  this.food = []
}
Hamster.prototype = {
  found: function(something) {
    this.food.push(something)
  }
}

speedy = new Hamster()
lazy = new Hamster()

speedy.found("apple")
speedy.found("orange")

alert(speedy.food.length) // 2
alert(lazy.food.length) // 0(!)

Event

Asynchronous Event Queue

Synchronous Event

other source code

// 1. Function declarations are initialized before the code is executed.
// window = { f: function }

// 2. Variables are added as window properties.
// window = { f: function, a: undefined, g: undefined }

var a = 5   // <-- var

function f(arg) { alert('f:'+arg) }

var g = function(arg) { alert('g:'+arg) } // <-- var

//read vs set
function f() {
  x = 5 // writing x puts it into outmost LexicalEnvironment
  console.log(typeof y); //undefined
  console.log(abc); //error
}

Ref

javascript.info
- http://javascript.info/tutorial/inheritance#inheritance-the-proto
google: RegExp mdc/ regexp msdn jscript

CSS

CSS3

SASS

Sass is an extension of CSS3 which adds nested rules, variables, mixins, selector inheritance, and more. Sass generates well formatted CSS and makes your stylesheets easier to organize and maintain.

Compass

Compass is an open-source CSS authoring framework which uses the Sass stylesheet language to make writing stylesheets powerful and easy.

Welcome to personal page powered by Jekyll!

Tue, 28 Feb 2017 03:24:55 +0000

Jekyll

This is my first post with Jekyll. Check out the Jekyll docs for more info on how to get the most out of Jekyll. File all bugs/feature requests at Jekyll’s GitHub repo. If you have questions, you can ask them on Jekyll Talk.

emacs integration

for org-mode user:

write a org file in emacs org-mode
export to markdown
jekyll build

for markdown-mode user:

1. write a md file in emacs markdown-mode
2. jekyll build

Welcome to Seven's page

hash table and algorithm

hashing function

collision resolution

hash open address application in java api

Java author Joshua J. Bloch

Distributed HashTable

Bloomfilter

Reference

famous authors in Java

James Arthur Gosling

Douglas S. Lee

Joshua J. Bloch

TODO

LCS algorithm

LCS large common subsequence

LCs large common substring

application

Reference

count distinct problem

streaming algorithm

HyperLogLog

online algorithm

Ref

markdown mode in emacs

h1: markdown mode in emacs

h2 styles

h2 code block

redis study notes

redis data types

cluster design

write safety

Redis cluster protocol

nodes handshake on discovering

fault tolerance

slave election / promotion

Sentinel

quorum

TILT mode

keys distribution model

hash slot/hash tag

hash algorithm

consistent hash

RedLock

algorithm

Raft

roles

how to elect leader?

how to replicate log entries?

visualization (RaftScope)

Redisson

lock implementation

unlock implementation

Questions:

Reference:

emacs

emacs configuration

use vi-mode as default

install package

Others

Java Technical Notes

Java Technical Notes

kafka

Redis

Active MQ/Kafka

Lucence/ Solr

FastDFS

MyBatis / Hibernate

HashMap history

json

front end study notes

Table of Contents

javascript

keywords

closure

“use strict”

- “prototype”

- constructor

- LexicalEnvironment | closure | block

- ECMAScript ES3/ES5/ES6