[hsynz]

July 28, 2025 · View on GitHub

release license PRs Welcome +issue Welcome
Build Status

english | 中文版

hsynz is a library for delta update using sync algorithm, like zsync.
rsync over http(s); implement the sync algorithm on client side, and server side only need http(s) cdn. support compressor zstd & libdeflate & zlib, support large file & directory(folder), support multi-thread.

hsynz defines its own file format (.hsyni and .hsynz files), and this library is also compatible with the file format of zsync (including apply and create .zsync and their .gz files).

Recommended scenarios: Very large number of older versions or where older versions are not available (not saved or modified, etc.) so that all deltas cannot be calculated in advance.

The server uses hsync_make to process the latest version of the data once, generating a summary info file(hsyni) of the new version of the data in chunks, and optionally compressing the new version of the data in chunks to get the release file(hsynz), which would be the hsynz equivalent if the new version of the original file were not compressed.

The client first downloads the hsyni file from the server or another user's share, calculates the updated blocks it needs to download based on its old version, and learns the location of these blocks in hsynz based on the information in hsyni, selects a communication method to download them on demand from the server's hsynz file, and merges the downloaded blocks with the existing data locally to get the latest version of the data.

hsync_demo provides a test client demo for local file testing.
hsync_http provides a download client demo with http(s) support for sync update from a server that provides an http(s) file download service(e.g CDN, support HTTP/1.1 multi range Requests).
Tip: You can also customise other communication methods for sync.

additional, if you have the new file locally & not the old file, but can get a hash certificate file(.hsyni) of the old file, you can also create a hpatchz format patch file(usage scenario like [rsync]); see the demo cmdline hsign_diff.

Compare with zsync

  • In addition to supporting source and target as files, support is also provided for directories(folders).
  • In addition to supporting compressed release package by zlib; also supported libdeflate & zstd compressor, providing better compression ratio, i.e. smaller downloaded patch package.
  • The server-side make support multi-threaded parallel acceleration.
  • The client-side diff speed has been optimized, and also support multi-threaded parallel acceleration.

Releases/Binaries

Download from latest release : Command line app for Windows, Linux, MacOS; and .so lib for Android.
( release files build by projects in path hsynz/builds )

Build it yourself

Linux or MacOS X

$ cd <dir>
$ git clone --recursive https://github.com/sisong/hsynz.git
$ cd hsynz
$ make

Windows

$ cd <dir>
$ git clone --recursive https://github.com/sisong/hsynz.git

build hsynz/builds/vc/hsynz.sln with Visual Studio

libhsynz.so for Android

  • install Android NDK
  • $ cd <dir>/hsynz/builds/android_ndk_jni_mk
  • $ build_libs_static.sh (or $ build_libs_static.bat on windows, then got *.so files)
  • import file com/github/sisong/hsynz.java (from hsynz/builds/android_ndk_jni_mk/java/) & .so files, java code can call the sync patch function in libhsynz.so

hsync_make command line usage:

hsync_make: [options] newDataPath out_hsyni_file [out_hsynz_file]
  newDataPath can be file or directory(folder),
  if newDataPath is a file & no -c-... option, out_hsynz_file can empty.
options:
  -s-matchBlockSize
      matchBlockSize>=128, DEFAULT -s-2k, recommended 1024,4k,...
  -b-safeBit
      set allow patch fail hash clash probability: 1/2^safeBit;
      safeBit>=14, DEFAULT -b-24, recommended 20,32...
  -p-parallelThreadNumber
    DEFAULT -p-4;
    if parallelThreadNumber>1 then open multi-thread Parallel mode;
  -c-compressType[-compressLevel]
      set out_hsynz_file Compress type & level, DEFAULT uncompress;
      support compress type & level:
        -c-zlib[-{1..9}[-dictBits]]     DEFAULT level 9
            dictBits can 9--15, DEFAULT 15.
        -c-gzip[-{1..9}[-dictBits]]     DEFAULT level 9
            dictBits can 9--15, DEFAULT 15.
            compress by zlib, out_hsynz_file is .gz file format.
        -c-ldef[-{1..12}]           DEFAULT level 12 (dictBits always 15).
            compress by libdeflate, compatible with zlib's deflate encoding.
        -c-lgzip[-{1..12}]          DEFAULT level 12 (dictBits always 15)
            compress by libdeflate, out_hsynz_file is .gz file format.
        -c-zstd[-{10..22}[-dictBits]]   DEFAULT level 21
            dictBits can 15--30, DEFAULT 24.
  -C-checksumType
      set strong Checksum type for block data, DEFAULT -C-xxh128;
      support checksum type:
        -C-xxh128
        -C-md5
        -C-sha512
        -C-sha256
        -C-crc32
            WARNING: crc32 is not strong & secure enough!
  -zsync[#KeY#=...#ValuE#=...[#KeY#=...#ValuE#=...]]
      create out_hsyni_file(.zsync file format) or out_hsynz_file(.gz file
         format) compatible with zsync.
      checksum default used sha1 & md4, not need set -C
      -s-matchBlockSize size must 2^N; if used -c-gzip or -c-lgzip (out
         .gz file), recommend set 4k at most, >=8k may fail.
      key-value string pairs will be write in out_hsyni_file; if needed,
         you can set Filename,Z-Filename,URL,Z-URL,MTime,Recompress,...
      zsync project https://zsync.moria.org.uk
  -n-maxOpenFileNumber
      limit Number of open files at same time when newDataPath is directory;
      maxOpenFileNumber>=8, DEFAULT -n-48, the best limit value by different
        operating system.
  -g#ignorePath[#ignorePath#...]
      set iGnore path list in newDataPath directory; ignore path list such as:
        #.DS_Store#desktop.ini#*thumbs*.db#.git*#.svn/#cache_*/00*11/*.tmp
      # means separator between names; (if char # in name, need write #: )
      * means can match any chars in name; (if char * in name, need write *: );
      / at the end of name means must match directory;
  -f  Force overwrite, ignore write path already exists;
      DEFAULT (no -f) not overwrite and then return error;
      if used -f and write path is exist directory, will always return error.
  --patch
      swap to hsync_demo mode.
  -v  output Version info.
  -h or -?
      output Help info (this usage).

hsync_http command line usage:

download   : [options] -dl#hsyni_file_url hsyni_file
local  diff: [options] oldPath hsyni_file hsynz_file_url -diff#outDiffFile
local patch: [options] oldPath hsyni_file -patch#diffFile outNewPath
sync  infos: [options] oldPath hsyni_file [-diffi#cacheTempFile]
sync  patch: [options] oldPath [-dl#hsyni_file_url] hsyni_file hsynz_file_url [-diffi#cacheTempFile] outNewPath
  oldPath can be file or directory(folder),
  if oldPath is empty input parameter ""
options:
  -dl#hsyni_file_url
    download hsyni_file from hsyni_file_url befor sync patch;
  -diff#outDiffFile
    create diffFile from ranges of hsynz_file_url befor local patch;
  -diffi#cacheTempFile
    saving diffInfo to cache file for optimize speed when continue sync patch;
  -patch#diffFile
    local patch(oldPath+diffFile) to outNewPath;
  -U
    set hsynz_file_url is the original file before compress, please ignore the
      compress info in hsyni_file and directly access the data of hsynz file
      without decompress.
  -cdl-{0|1}        or  -cdl-{off|on}
    continue download data from breakpoint;
    DEFAULT -cdl-1 opened, need set -cdl-0 or -cdl-off to close continue mode;
  -rdl-retryDownloadNumber
    number of auto retry connection, when network disconnected while downloading;
    DEFAULT -rdl-0 retry closed; recommended 5,1k,1g,...
  -r-stepRangeNumber
    DEFAULT -r-32, recommended 16,20,...
    limit the maximum number of .hsynz data ranges that can be downloaded
    in a single request step;
    if http(s) server not support multi-ranges request, must set -r-1
  -p-parallelThreadNumber
    DEFAULT -p-4;
    if parallelThreadNumber>1 then open multi-thread Parallel mode;
    NOTE: now download data always used single-thread.
  -n-maxOpenFileNumber
      limit Number of open files at same time when oldPath is directory;
      maxOpenFileNumber>=8, DEFAULT -n-24, the best limit value by different
        operating system.
  -g#ignorePath[#ignorePath#...]
      set iGnore path list in oldPath directory; ignore path list such as:
        #.DS_Store#desktop.ini#*thumbs*.db#.git*#.svn/#cache_*/00*11/*.tmp
      # means separator between names; (if char # in name, need write #: )
      * means can match any chars in name; (if char * in name, need write *: );
      / at the end of name means must match directory;
  -f  Force overwrite, ignore write path already exists;
      DEFAULT (no -f) not overwrite and then return error;
      not support oldPath outNewPath same path!
      if used -f and outNewPath is exist file:
        if patch output file, will overwrite;
        if patch output directory, will always return error;
      if used -f and outNewPath is exist directory:
        if patch output file, will always return error;
        if patch output directory, will overwrite, but not delete
          needless existing files in directory.
  -v  output Version info.
  -h or -?
      output Help info (this usage).

hsync_demo command line usage:

This cmdline is used for local sync tests, replacing the actual URL remote file with local file, see the hsync_http usage.

hsynz vs zsync:

case list(download from OneDrive):

newFile <-- oldFilenewSizeoldSize
17-Zip_22.01.win.tar <-- 7-Zip_21.07.win.tar59089925748224
2Chrome_107.0.5304.122-x64-Stable.win.tar <-- 106.0.5249.119278658560273026560
3cpu-z_2.03-en.win.tar <-- cpu-z_2.02-en.win.tar87183368643072
4curl_7.86.0.src.tar <-- curl_7.85.0.src.tar2627584026030080
5douyin_1.5.1.mac.tar <-- douyin_1.4.2.mac.tar407940608407642624
6Emacs_28.2-universal.mac.tar <-- Emacs_27.2-3-universal.mac.tar196380160257496064
7FFmpeg-n_5.1.2.src.tar <-- FFmpeg-n_4.4.3.src.tar8052736076154880
8gcc_12.2.0.src.tar <-- gcc_11.3.0.src.tar865884160824309760
9git_2.33.0-intel-universal-mavericks.mac.tar <-- 2.31.07330252870990848
10go_1.19.3.linux-amd64.tar <-- go_1.19.2.linux-amd64.tar468835840468796416
11jdk_x64_mac_openj9_16.0.1_9_openj9-0.26.0.tar <-- 9_15.0.2_7-0.24.0363765760327188480
12jre_1.8.0_351-linux-x64.tar <-- jre_1.8.0_311-linux-x64.tar267796480257996800
13linux_5.19.9.src.tar <-- linux_5.15.80.src.tar12696371201138933760
14Minecraft_175.win.tar <-- Minecraft_172.win.tar166643200180084736
15OpenOffice_4.1.13.mac.tar <-- OpenOffice_4.1.10.mac.tar408364032408336896
16postgresql_15.1.src.tar <-- postgresql_14.6.src.tar151787520147660800
17QQ_9.6.9.win.tar <-- QQ_9.6.8.win.tar465045504464837120
18tensorflow_2.10.1.src.tar <-- tensorflow_2.8.4.src.tar275548160259246080
19VSCode-win32-x64_1.73.1.tar <-- VSCode-win32-x64_1.69.2.tar364025856340256768
20WeChat_3.8.0.41.win.tar <-- WeChat_3.8.0.33.win.tar505876992505018368

test PC: Windows11, CPU R9-7945HX, SSD PCIe4.0x4 4T, DDR5 5200MHz 32Gx2
Program version: hsynz 1.3.0, zsync 0.6.3
test Program:
zsync run make with zsyncmake -b 2048 -o {out_newi} {new},
client sync diff&patch by zsync -i {old} -o {out_new} {newi} (all files are local)
zsync -z run make with zsyncmake -b 2048 -z -u {new.gz} -o {out_newi} {new}
hsynz run make with hsync_make -s-2k {new} {out_newi} [{-c-?} {out_newz}],
run make with -zsync means create compatible zsync's file format, client sync diff&patch by hsync_demo {old} {newi} {newz} {out_new} (all files are local)
hsynz p1 run make without compressor & out_newz , add -p-1
hsynz p8 run make without compressor & out_newz , add -p-8
hsynz p1 zlib run make with -p-1 -c-zlib-9 (run hsync_demo with -p-1)
hsynz p8 zlib run make with -p-8 -c-zlib-9 (run hsync_demo with -p-8)
hsynz p1 gz run make with -p-1 -c-gzip-9 (run hsync_demo with -p-1)
hsynz p8 gz run make with -p-8 -c-gzip-9 (run hsync_demo with -p-8)
hsynz p1 ldef run make with -p-1 -c-ldef-12 (run hsync_demo with -p-1)
hsynz p8 ldef run make with -p-8 -c-ldef-12 (run hsync_demo with -p-8)
hsynz p1 lgz run make with -p-1 -c-lgzip-12 (run hsync_demo with -p-1)
hsynz p8 lgz run make with -p-8 -c-lgzip-12 (run hsync_demo with -p-8)
hsynz p1 zstd run make with -p-1 -c-zstd-21-24 (run hsync_demo with -p-1)
hsynz p8 zstd run make with -p-8 -c-zstd-21-24 (run hsync_demo with -p-8)
additional tests added hsign_diff, which can create hpatchz-compatible patch files using only old data's .hsyni file and the new data.

test result average:

makememspeedpatchcompressmemmax memspeed
zsyncmake1M381.6MB/shsynz p145.74%6M18M181MB/s
zsyncmake1M381.6MB/shsynz p845.74%13M27M276MB/s
zsyncmake1M383.4MB/szsync45.75%9M38M122MB/s
zsyncmake -z1M14.8MB/shsynz p116.56%6M21M169MB/s
zsyncmake -z1M14.8MB/shsynz p816.56%13M29M245MB/s
zsyncmake -z1M14.9MB/szsync16.58%13M52M71MB/s
hsync_make p1 -zsync4M430.9MB/shsynz p145.39%6M19M187MB/s
hsync_make p8 -zsync14M591.0MB/shsynz p845.39%13M27M275MB/s
hsync_make p8 -zsync14M590.9MB/szsync45.39%9M37M119MB/s
hsync_make p1 gz -zsync6M16.9MB/shsynz p116.59%6M21M174MB/s
hsync_make p8 gz -zsync29M101.8MB/shsynz p816.59%13M29M252MB/s
hsync_make p8 gz -zsync29M101.8MB/szsync16.60%14M52M70MB/s
hsync_make p1 lgz -zsync14M7.8MB/shsynz p116.21%6M21M176MB/s
hsync_make p8 lgz -zsync95M40.3MB/shsynz p816.21%14M29M252MB/s
hsync_make p8 lgz -zsync95M40.1MB/szsync16.22%14M52M70MB/s
hsync_make p15M2179.7MB/shsynz p144.57%5M19M301MB/s
hsync_make p812M3709.2MB/shsynz p844.57%13M27M478MB/s
hsync_make p1 zlib7M17.5MB/shsynz p116.05%6M22M270MB/s
hsync_make p8 zlib30M105.0MB/shsynz p816.05%13M29M407MB/s
hsync_make p1 ldef15M8.0MB/shsynz p115.68%6M22M270MB/s
hsync_make p8 ldef96M40.6MB/shsynz p815.68%13M29M405MB/s
hsync_make p1 gz7M17.6MB/shsynz p116.12%6M22M271MB/s
hsync_make p8 gz30M104.9MB/shsynz p816.12%13M29M408MB/s
hsync_make p1 lgz15M7.9MB/shsynz p115.74%6M22M270MB/s
hsync_make p8 lgz96M40.6MB/shsynz p815.74%13M29M405MB/s
hsync_make p1 zstd532M1.8MB/shsynz p112.54%24M34M263MB/s
hsync_make p8 zstd3353M8.1MB/shsynz p812.54%23M34M384MB/s
hsign_diff p14M494.7MB/shpatchz43.13%3M4M2409MB/s
hsign_diff p811M1412.3MB/shpatchz43.13%3M4M2422MB/s
hsign_diff p1 zlib4M42.9MB/shpatchz14.78%4M4M926MB/s
hsign_diff p8 zlib13M246.3MB/shpatchz14.79%3M4M927MB/s
hsign_diff p1 ldef16M20.1MB/shpatchz14.38%3M4M920MB/s
hsign_diff p8 ldef115M104.4MB/shpatchz14.38%3M4M917MB/s
hsign_diff p1 zstd205M8.3MB/shpatchz11.17%17M21M1369MB/s
hsign_diff p8 zstd1348M17.2MB/shpatchz11.17%18M21M1316MB/s

input Apk Files for test:

case list:

appnewFile <-- oldFilenewSizeoldSize
1cn.wps.moffice_eng_13.30.0.apk <-- 13.29.09590491894914262
2com.achievo.vipshop_7.80.2.apk <-- 7.79.9127395632120237937
3com.adobe.reader_22.9.0.24118.apk <-- 22.8.1.235872735143727087718
4com.alibaba.android.rimet_6.5.50.apk <-- 6.5.45195314449193489159
5com.amazon.mShop.android.shopping_24.18.2.apk <-- 24.18.07632885876287423
6com.baidu.BaiduMap_16.5.0.apk <-- 16.4.5131382821132308374
7com.dragon.read_5.5.3.33.apk <-- 5.5.1.324511265843518658
8com.ebay.mobile_6.80.0.1.apk <-- 6.79.0.16120258761123285
9com.eg.android.AlipayGphone_10.3.0.apk <-- 10.2.96122073135119046208
10com.google.android.apps.translate_6.46.0.apk <-- 6.45.04889296748843378
11com.google.android.googlequicksearchbox_13.38.11.apk <-- 13.37.10190539272189493966
12com.jingdong.app.mall_11.3.2.apk <-- 11.3.0101098430100750191
13com.netease.cloudmusic_8.8.45.apk <-- 8.8.40181914846181909451
14com.reddit.frontpage_2022.36.0.apk <-- 2022.34.05020511947854461
15com.sankuai.meituan.takeoutnew_7.94.3.apk <-- 7.92.27496589374833926
16com.sankuai.meituan_12.4.207.apk <-- 12.4.2059361373293605911
17com.sina.weibo_12.10.0.apk <-- 12.9.5156881776156617913
18com.smile.gifmaker_10.8.40.27845.apk <-- 10.8.30.27728102403847101520138
19com.ss.android.article.news_9.0.7.apk <-- 9.0.65444400353947221
20com.ss.android.ugc.aweme_22.6.0.apk <-- 22.5.0171683897171353597
21com.taobao.taobao_10.18.10.apk <-- 10.17.0117218670117111874
22com.tencent.mm_8.0.28.apk <-- 8.0.27266691829276603782
23com.tencent.mobileqq_8.9.15.apk <-- 8.9.13311322716310529631
24com.tencent.mtt_13.2.0.0103.apk <-- 13.2.0.00459734274797296757
25com.tripadvisor.tripadvisor_49.5.apk <-- 49.32874449828695346
26com.twitter.android_9.61.0.apk <-- 9.58.23614184035575484
27com.ubercab_4.442.10002.apk <-- 4.439.100026992323264284150
28com.ximalaya.ting.android_9.0.66.3.apk <-- 9.0.62.3115804845113564876
29com.xunmeng.pinduoduo_6.30.0.apk <-- 6.29.13089683330951567
30com.youdao.dict_9.2.29.apk <-- 9.2.28110624682110628778
31org.mozilla.firefox_105.2.0.apk <-- 105.1.08307846483086656
32tv.danmaku.bili_7.1.0.apk <-- 7.0.0104774723104727005

changed test Program:
zsync ... make -b 2048 changed to -b 1024
hsynz ... make -s-2k changed to -s-1k

test result average:

makememspeedpatchcompressmemmax memspeed
zsyncmake1M352.4MB/shsynz p160.20%4M11M163MB/s
zsyncmake1M352.4MB/shsynz p860.20%12M19M228MB/s
zsyncmake1M353.6MB/szsync60.20%6M18M107MB/s
zsyncmake -z1M20.0MB/shsynz p157.16%4M12M170MB/s
zsyncmake -z1M20.0MB/shsynz p857.16%12M20M240MB/s
zsyncmake -z1M20.0MB/szsync57.16%9M25M79MB/s
hsync_make p1 -zsync4M412.6MB/shsynz p160.25%4M11M165MB/s
hsync_make p8 -zsync14M569.4MB/shsynz p860.25%12M19M231MB/s
hsync_make p8 -zsync14M568.6MB/szsync60.25%6M18M106MB/s
hsync_make p1 gz -zsync5M22.2MB/shsynz p157.24%4M12M169MB/s
hsync_make p8 gz -zsync29M119.3MB/shsynz p857.24%12M20M240MB/s
hsync_make p8 gz -zsync29M119.7MB/szsync57.25%9M25M79MB/s
hsync_make p1 lgz -zsync13M23.2MB/shsynz p157.18%4M12M168MB/s
hsync_make p8 lgz -zsync96M111.5MB/shsynz p857.18%12M20M235MB/s
hsync_make p8 lgz -zsync96M111.6MB/szsync57.19%9M25M77MB/s
hsync_make p14M1517.4MB/shsynz p159.82%4M10M229MB/s
hsync_make p810M2200.1MB/shsynz p859.82%12M18M355MB/s
hsync_make p1 zlib5M22.9MB/shsynz p156.17%4M11M242MB/s
hsync_make p8 zlib29M121.8MB/shsynz p856.17%12M19M383MB/s
hsync_make p1 ldef14M24.1MB/shsynz p156.11%4M11M242MB/s
hsync_make p8 ldef96M112.0MB/shsynz p856.11%12M19M382MB/s
hsync_make p1 gz5M23.1MB/shsynz p156.44%4M11M242MB/s
hsync_make p8 gz29M122.3MB/shsynz p856.44%12M19M382MB/s
hsync_make p1 lgz14M24.0MB/shsynz p156.38%4M11M240MB/s
hsync_make p8 lgz96M111.7MB/shsynz p856.38%12M19M380MB/s
hsync_make p1 zstd534M2.6MB/shsynz p155.22%24M28M234MB/s
hsync_make p8 zstd3443M10.0MB/shsynz p855.22%24M28M361MB/s
hsign_diff p13M392.8MB/shpatchz58.92%3M4M2134MB/s
hsign_diff p811M1043.7MB/shpatchz58.92%3M4M2152MB/s
hsign_diff p1 zlib3M54.0MB/shpatchz54.77%4M4M978MB/s
hsign_diff p8 zlib11M265.7MB/shpatchz54.77%4M4M972MB/s
hsign_diff p1 ldef15M53.4MB/shpatchz54.64%4M4M667MB/s
hsign_diff p8 ldef119M212.4MB/shpatchz54.64%4M4M661MB/s
hsign_diff p1 zstd213M11.2MB/shpatchz53.89%20M20M1461MB/s
hsign_diff p8 zstd1087M13.3MB/shpatchz53.89%20M20M1459MB/s

Contact

housisong@hotmail.com