README
June 13, 2017 ยท View on GitHub
Redis Sharding
Standard ML (MLton and PolyML) version of https://github.com/kni/redis-sharding-hs-strict
Worked on *BSD and Linux.
Building: make poly or make mlton
For Linux replace kqueue on epoll in redis-sharding.ml* files.
To use multiprocessing you must set argument --N n, where n - count OS process (MLton) or threads (PolyML). If you use Linux and want use multiprocessing, you must have kernel 3.9 and above and set SO_REUSEPORT argument to 1.
Examples: FreeBSD ./redis-sharding-mlton --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084 --N=4 Linux ./redis-sharding-mlton --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084 --N=4 --SO_REUSEPORT=1
Note
Cursor of SCAN command is integer of arbitrary precision.
Benchmark
redis-benchmark -p 8080 -n 100000 -c 10 -q -t set,get,mset -P N
Heap size is 20M
cpuset -l 0 time ./redis-sharding-haskell --port=8080 --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084 +RTS -N1 -H20M -M20M -qa
cpuset -l 0 time ./redis-sharding-mlton @MLton fixed-heap 20M -- --port=8080 --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084
cpuset -l 0 time ./redis-sharding-poly --minheap 20M --maxheap 20M --port=8080 --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084
Thousands of requests per second
Haskell MLton PolyML Redis direct
-P 1; SET 33 (32 - 34) 52 (51 - 55) 44 (42 - 46) 56 (53 - 58)
-P 1; GET 34 (33 - 35) 51 (48 - 52) 40 (37 - 45) 59 (57 - 61)
-P 1; MSET 14 (14 - 15) 30 (27 - 31) 21 (20 - 21) 39 (37 - 41)
-P 2; SET 34 (33 - 35) 54 (52 - 56) 46 (44 - 47) 57 (55 - 59)
-P 2; GET 34 (33 - 34) 56 (54 - 57) 39 (37 - 40) 59 (58 - 60)
-P 2; MSET 15 (14 - 15) 31 (29 - 33) 20 (19 - 21) 40 (39 - 41)
-P 10; SET 46 (44 - 47) 66 (61 - 68) 63 (60 - 67) 67 (66 - 69)
-P 10; GET 44 (43 - 45) 69 (67 - 70) 67 (60 - 70) 71 (66 - 74)
-P 10; MSET 6 (6 - 6) 11 (11 - 11) 16 (16 - 17) 44 (43 - 45)
-P 100; SET 44 (43 - 44) 62 (61 - 64) 56 (52 - 59) 57 (54 - 59)
-P 100; GET 41 (40 - 43) 58 (54 - 63) 55 (49 - 62) 61 (58 - 62)
-P 100; MSET 13 (12 - 13) 23 (21 - 24) 17 (17 - 17) 40 (37 - 41)
-P 200; SET 42 (41 - 44) 59 (56 - 61) 57 (54 - 62) 59 (56 - 63)
-P 200; GET 41 (40 - 41) 60 (55 - 63) 54 (50 - 59) 61 (59 - 62)
-P 200; MSET 12 (12 - 13) 23 (22 - 23) 17 (16 - 17) 41 (40 - 41)
-P 500; SET 39 (37 - 41) 55 (53 - 59) 54 (50 - 58) 60 (56 - 62)
-P 500; GET 40 (38 - 41) 62 (58 - 64) 58 (55 - 62) 63 (57 - 67)
-P 500; MSET 12 (12 - 12) 21 (20 - 23) 17 (17 - 17) 41 (40 - 42)
-P 1000; SET 36 (35 - 37) 57 (53 - 61) 53 (51 - 54) 58 (56 - 60)
-P 1000; GET 36 (35 - 38) 57 (54 - 60) 57 (49 - 61) 61 (59 - 63)
-P 1000; MSET 11 (11 - 11) 20 (19 - 22) 15 (13 - 16) 40 (39 - 42)
Default heap size
cpuset -l 0 time ./redis-sharding-haskell --port=8080 --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084 +RTS -N1 -qa cpuset -l 0 time ./redis-sharding-haskell --port=8080 --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084 +RTS -N1 -A10M -qa cpuset -l 0 time ./redis-sharding-mlton --port=8080 --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084 cpuset -l 0 time ./redis-sharding-poly --port=8080 --nodes=127.0.0.1:8081,127.0.0.1:8082,127.0.0.1:8083,127.0.0.1:8084
Thousands of requests per second
Haskell -A10M MLton PolyML Redis direct
-P 1; SET 33 (31 - 34) 34 53 (51 - 54) 39 (31 - 45) 56 (53 - 58) -P 1; GET 33 (32 - 34) 34 52 (50 - 53) 35 (34 - 37) 59 (57 - 61) -P 1; MSET 11 (11 - 11) 14 25 (24 - 26) 21 (18 - 22) 39 (37 - 41)
-P 2; SET 33 (33 - 34) 34 55 (53 - 57) 43 (36 - 48) 57 (55 - 59) -P 2; GET 33 (32 - 34) 35 55 (51 - 57) 37 (35 - 39) 59 (58 - 60) -P 2; MSET 11 (11 - 11) 15 25 (24 - 25) 21 (17 - 23) 40 (39 - 41)
-P 10; SET 39 (37 - 40) 45 65 (61 - 68) 63 (60 - 66) 67 (66 - 69) -P 10; GET 38 (37 - 39) 42 67 (65 - 70) 58 (49 - 67) 71 (66 - 74) -P 10; MSET 5 ( 5 - 5) 6 17 (16 - 18) 16 (16 - 17) 44 (43 - 45)
-P 100; SET 36 (36 - 38) 40 58 (57 - 60) 56 (53 - 62) 57 (54 - 59) -P 100; GET 36 (35 - 37) 42 56 (53 - 61) 59 (54 - 62) 61 (58 - 62) -P 100; MSET 10 ( 9 - 10) 13 18 (17 - 19) 23 (22 - 24) 40 (37 - 41)
-P 200; SET 36 (35 - 38) 43 60 (56 - 62) 55 (47 - 63) 59 (56 - 63) -P 200; GET 36 (36 - 37) 41 58 (55 - 61) 57 (48 - 62) 61 (59 - 62) -P 200; MSET 10 ( 9 - 10) 12 20 (19 - 21) 21 (20 - 23) 41 (40 - 41)
-P 500; SET 35 (35 - 36) 41 56 (50 - 62) 47 (43 - 53) 60 (56 - 62) -P 500; GET 36 (35 - 37) 39 58 (55 - 62) 60 (53 - 63) 63 (57 - 67) -P 500; MSET 9 ( 9 - 10) 12 19 (17 - 21) 21 (20 - 22) 41 (40 - 42)
-P 1000; SET 34 (33 - 36) 37 53 (51 - 56) 48 (43 - 55) 58 (56 - 60) -P 1000; GET 36 (35 - 37) 39 55 (52 - 58) 54 (52 - 56) 61 (59 - 63) -P 1000; MSET 9 ( 9 - 9) 11 23 (21 - 24) 22 (21 - 23) 40 (39 - 42)