Zstandard 是facebook厂商提供的一种高压缩率的实时压缩算法,他在压缩比和压缩速度上做了很好的平衡,同时还提供高速的解码器,针对小数据,也提供了一种叫做字典压缩算法,从样本中可以创建出字典集合数据,Zstandard库以BSD协议方式开源。
该算法是由Facebook的Yann Collet开发的一个无损数据压缩算法。Zstandard在设计上与DEFLATE(.zip、gzip)算法有着差不多的压缩比,但有更高的压缩和解压缩速度。
Zstandard使用字典算法(LZ77)结合熵编码法的有限状态熵(tANS)。—–Wikipedia
Zlib 就不介绍了,大家都很熟悉了,一套古老的压缩算法。性能对比,两年前做过简单对比,Zstandard (zstd)压缩算法在JAVA上的使用
这次再做一次深入的评测和对比,其中,实验样本用的是我们游戏协议的文本样本,测试结果见文章结尾:
和zlib对比
1.在小数据量下(256),level=3,相对zlib,!!#ff0000 压缩效率大幅提高!!(94230.272 -> 153682 , 163.09%),表现更加稳定(± 7958.782 ops/s–>±1801.146 ops/s),解压缩效率小幅降低(608231.351 -> 565699.876 , -7%),并且表现更加稳定± 312460.294 ops/s -->± 12919.602 ops/s), 但压缩比会小幅降低(18.0%–>17.97%)
2.在大数据量下(8192),level=3,相对zlib!!#ff0000 ,压缩效率大幅提高!!(3939.309-> 29897.221 ,758.99%),稳定性小幅降低(45.24/3939.309=1.14%->513.773/ 29897.221 =1.71%),解压缩效率大幅提高(30828.328 -> 82746.561 , 268.41%),稳定性小幅提高((4388.578/30828.328=14.23%–>8866.313/ 82746.561=10.71%), 压缩比基本持平( 63.5%–>63.48%)
总结 1,从数据上看出,zstandard比zlib,同样的压缩比情况下,速度高于zlib,同样的速度情况下,压缩比要高于zlib
自身不同level 对比
3.小数据量下(256 bytes),随着level增大,压缩率会小幅提高,提升不明显,在level>=2 下 ,!!#ff0000 压缩效率会大幅降低,!!解压缩效率也会小幅降低。
压缩(level=3)
其中,ratio=1-compress/size
Benchmark (level) (size) Mode Cnt Score Error Units compress ratio Compress.ZstdLevelBenchMark.zstdCompress 1 256 thrpt 10 158247.321 2539.721 ops/s 219 14.45% Compress.ZstdLevelBenchMark.zstdCompress 2 256 thrpt 10 160513.122 1055.204 ops/s 217 15.23% Compress.ZstdLevelBenchMark.zstdCompress 3 256 thrpt 10 153682.210 1801.146 ops/s 210 17.97% Compress.ZstdLevelBenchMark.zstdCompress 4 256 thrpt 10 133600.466 1974.326 ops/s 206 19.53% Compress.ZstdLevelBenchMark.zstdCompress 5 256 thrpt 10 124519.999 3317.794 ops/s 204 20.31% Compress.ZstdLevelBenchMark.zstdCompress 6 256 thrpt 10 122500.287 1030.539 ops/s 204 20.31%
解压(level=3)
Benchmark (level) (size) Mode Cnt Score Error Units Compress.ZstdLevelBenchMark.zstdUncompress 1 256 thrpt 10 582084.385 7778.949 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 2 256 thrpt 10 580312.371 6809.382 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 3 256 thrpt 10 565699.876 12919.602 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 4 256 thrpt 10 562818.616 5036.532 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 5 256 thrpt 10 501616.311 5596.259 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 6 256 thrpt 10 494291.843 7102.794 ops/s
4.大数据量下(8192 bytes),随着level增大,压缩率会小幅提高,提升不明显,!!#ff0000 压缩效率会大幅降低!!,解压缩效率会小幅提高
压缩(level=6)
Benchmark (level) (size) Mode Cnt Score Error Units compress ratio Compress.ZstdLevelBenchMark.zstdCompress 1 8192 thrpt 10 34586.557 302.868 ops/s 3091 62.27% Compress.ZstdLevelBenchMark.zstdCompress 2 8192 thrpt 10 34113.274 243.294 ops/s 3035 62.95% Compress.ZstdLevelBenchMark.zstdCompress 3 8192 thrpt 10 29897.221 513.773 ops/s 2992 63.48% Compress.ZstdLevelBenchMark.zstdCompress 4 8192 thrpt 10 18699.637 176.416 ops/s 2948 64.01% Compress.ZstdLevelBenchMark.zstdCompress 5 8192 thrpt 10 13744.149 76.323 ops/s 2885 64.78% Compress.ZstdLevelBenchMark.zstdCompress 6 8192 thrpt 10 9931.221 79.205 ops/s 2886 64.77%
解压(level=6)
Benchmark (level) (size) Mode Cnt Score Error Units Compress.ZstdLevelBenchMark.zstdUncompress 1 8192 thrpt 10 83840.509 9960.244 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 2 8192 thrpt 10 83274.771 6297.771 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 3 8192 thrpt 10 82746.561 8866.313 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 4 8192 thrpt 10 84652.976 6150.251 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 5 8192 thrpt 10 84665.619 6465.654 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 6 8192 thrpt 10 85139.016 5582.793 ops/s
总结 2,从以上3,4点对比,默认值3 ,或者 2 可以认为是相对较优解。
综上所述,
在协议这块,服务器用压缩API的频率远大于解压缩API频率, 在由zlib替换了level=3的Zstd后,压缩效率会大幅提高,表现也更加稳定,压缩率会有小幅降低。
在level的调整方面,随着level增大,压缩率会有小幅提高,但压缩效率会大幅降低,得不偿失,所以保持默认值3或者2是比较合适的。如果追求压缩/解压缩效率则选择2,追求压缩比,则选择3
##测试结果
zlib
Benchmark (size) Mode Cnt Score Error Units compress ratio Compress.ZstdLevelBenchMark.zlibCompress 256 thrpt 10 94230.272 ± 7958.782 ops/s 197 18.0% Compress.ZstdLevelBenchMark.zlibCompress 512 thrpt 10 67053.071 ± 5606.840 ops/s 308 39.3% Compress.ZstdLevelBenchMark.zlibCompress 1024 thrpt 10 32361.968 ± 1426.980 ops/s 448 58.5% Compress.ZstdLevelBenchMark.zlibCompress 2048 thrpt 10 15301.493 ± 354.073 ops/s 733 64.3% Compress.ZstdLevelBenchMark.zlibCompress 4096 thrpt 10 6656.437 ± 96.692 ops/s 1359 68.4% Compress.ZstdLevelBenchMark.zlibCompress 8192 thrpt 10 3939.309 ± 45.240 ops/s 3126 63.5% Compress.ZstdLevelBenchMark.zlibUncompress 256 thrpt 10 608231.351 ± 312460.294 ops/s Compress.ZstdLevelBenchMark.zlibUncompress 512 thrpt 10 460449.016 ± 206413.902 ops/s Compress.ZstdLevelBenchMark.zlibUncompress 1024 thrpt 10 169738.557 ± 33999.682 ops/s Compress.ZstdLevelBenchMark.zlibUncompress 2048 thrpt 10 110027.019 ± 17135.330 ops/s Compress.ZstdLevelBenchMark.zlibUncompress 4096 thrpt 10 65892.491 ± 11272.236 ops/s Compress.ZstdLevelBenchMark.zlibUncompress 8192 thrpt 10 30828.328 ± 4388.578 ops/s
Zstd
Benchmark (level) (size) Mode Cnt Score Error Units compress ratio Compress.ZstdLevelBenchMark.zstdCompress 1 256 thrpt 10 158247.321 2539.721 ops/s 219 14.45% Compress.ZstdLevelBenchMark.zstdCompress 1 512 thrpt 10 138946.669 707.322 ops/s 314 38.67% Compress.ZstdLevelBenchMark.zstdCompress 1 1024 thrpt 10 116642.039 1013.260 ops/s 443 56.74% Compress.ZstdLevelBenchMark.zstdCompress 1 2048 thrpt 10 84307.054 1845.538 ops/s 738 63.96% Compress.ZstdLevelBenchMark.zstdCompress 1 4096 thrpt 10 57226.220 277.580 ops/s 1304 68.16% Compress.ZstdLevelBenchMark.zstdCompress 1 8192 thrpt 10 34586.557 302.868 ops/s 3091 62.27% Compress.ZstdLevelBenchMark.zstdCompress 2 256 thrpt 10 160513.122 1055.204 ops/s 217 15.23% Compress.ZstdLevelBenchMark.zstdCompress 2 512 thrpt 10 139264.767 1479.608 ops/s 311 39.26% Compress.ZstdLevelBenchMark.zstdCompress 2 1024 thrpt 10 114781.873 1121.017 ops/s 411 59.86% Compress.ZstdLevelBenchMark.zstdCompress 2 2048 thrpt 10 82449.922 681.039 ops/s 704 65.63% Compress.ZstdLevelBenchMark.zstdCompress 2 4096 thrpt 10 55644.063 749.733 ops/s 1329 67.55% Compress.ZstdLevelBenchMark.zstdCompress 2 8192 thrpt 10 34113.274 243.294 ops/s 3035 62.95% Compress.ZstdLevelBenchMark.zstdCompress 3 256 thrpt 10 153682.210 1801.146 ops/s 210 17.97% Compress.ZstdLevelBenchMark.zstdCompress 3 512 thrpt 10 132749.271 1685.915 ops/s 311 39.26% Compress.ZstdLevelBenchMark.zstdCompress 3 1024 thrpt 10 109401.965 1214.010 ops/s 425 58.50% Compress.ZstdLevelBenchMark.zstdCompress 3 2048 thrpt 10 81395.267 1234.244 ops/s 732 64.26% Compress.ZstdLevelBenchMark.zstdCompress 3 4096 thrpt 10 52683.335 485.770 ops/s 1295 68.38% Compress.ZstdLevelBenchMark.zstdCompress 3 8192 thrpt 10 29897.221 513.773 ops/s 2992 63.48% Compress.ZstdLevelBenchMark.zstdCompress 4 256 thrpt 10 133600.466 1974.326 ops/s 206 19.53% Compress.ZstdLevelBenchMark.zstdCompress 4 512 thrpt 10 112088.964 4874.136 ops/s 307 40.04% Compress.ZstdLevelBenchMark.zstdCompress 4 1024 thrpt 10 88978.838 1492.176 ops/s 420 58.98% Compress.ZstdLevelBenchMark.zstdCompress 4 2048 thrpt 10 62289.020 1805.546 ops/s 724 64.65% Compress.ZstdLevelBenchMark.zstdCompress 4 4096 thrpt 10 38117.885 496.076 ops/s 1288 68.55% Compress.ZstdLevelBenchMark.zstdCompress 4 8192 thrpt 10 18699.637 176.416 ops/s 2948 64.01% Compress.ZstdLevelBenchMark.zstdCompress 5 256 thrpt 10 124519.999 3317.794 ops/s 204 20.31% Compress.ZstdLevelBenchMark.zstdCompress 5 512 thrpt 10 101343.380 1078.978 ops/s 293 42.77% Compress.ZstdLevelBenchMark.zstdCompress 5 1024 thrpt 10 75868.554 405.721 ops/s 420 58.98% Compress.ZstdLevelBenchMark.zstdCompress 5 2048 thrpt 10 50464.417 497.478 ops/s 714 65.14% Compress.ZstdLevelBenchMark.zstdCompress 5 4096 thrpt 10 29152.233 320.349 ops/s 1243 69.65% Compress.ZstdLevelBenchMark.zstdCompress 5 8192 thrpt 10 13744.149 76.323 ops/s 2885 64.78% Compress.ZstdLevelBenchMark.zstdCompress 6 256 thrpt 10 122500.287 1030.539 ops/s 204 20.31% Compress.ZstdLevelBenchMark.zstdCompress 6 512 thrpt 10 96711.036 965.084 ops/s 293 42.77% Compress.ZstdLevelBenchMark.zstdCompress 6 1024 thrpt 10 63150.268 2078.721 ops/s 420 58.98% Compress.ZstdLevelBenchMark.zstdCompress 6 2048 thrpt 10 38628.981 669.073 ops/s 714 65.14% Compress.ZstdLevelBenchMark.zstdCompress 6 4096 thrpt 10 19968.703 271.908 ops/s 1240 69.73% Compress.ZstdLevelBenchMark.zstdCompress 6 8192 thrpt 10 9931.221 79.205 ops/s 2886 64.77% Compress.ZstdLevelBenchMark.zstdUncompress 1 256 thrpt 10 582084.385 7778.949 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 1 512 thrpt 10 480672.486 2697.970 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 1 1024 thrpt 10 361403.668 3443.690 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 1 2048 thrpt 10 189317.885 1558.473 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 1 4096 thrpt 10 134702.624 1711.808 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 1 8192 thrpt 10 83840.509 9960.244 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 2 256 thrpt 10 580312.371 6809.382 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 2 512 thrpt 10 487137.103 3394.255 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 2 1024 thrpt 10 364243.783 3039.647 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 2 2048 thrpt 10 266417.312 11819.619 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 2 4096 thrpt 10 128809.854 5231.345 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 2 8192 thrpt 10 83274.771 6297.771 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 3 256 thrpt 10 565699.876 12919.602 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 3 512 thrpt 10 484201.159 8496.238 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 3 1024 thrpt 10 351868.427 6034.207 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 3 2048 thrpt 10 264361.085 13951.503 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 3 4096 thrpt 10 134640.265 1068.485 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 3 8192 thrpt 10 82746.561 8866.313 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 4 256 thrpt 10 562818.616 5036.532 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 4 512 thrpt 10 461458.897 5798.547 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 4 1024 thrpt 10 349583.096 2818.184 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 4 2048 thrpt 10 265175.187 10967.808 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 4 4096 thrpt 10 132574.256 1409.214 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 4 8192 thrpt 10 84652.976 6150.251 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 5 256 thrpt 10 501616.311 5596.259 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 5 512 thrpt 10 419099.670 1799.619 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 5 1024 thrpt 10 353087.876 6711.421 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 5 2048 thrpt 10 264624.591 2530.357 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 5 4096 thrpt 10 133355.669 1524.036 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 5 8192 thrpt 10 84665.619 6465.654 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 6 256 thrpt 10 494291.843 7102.794 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 6 512 thrpt 10 418350.498 2466.316 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 6 1024 thrpt 10 355302.788 2142.318 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 6 2048 thrpt 10 265958.624 11417.774 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 6 4096 thrpt 10 131764.090 6460.557 ops/s Compress.ZstdLevelBenchMark.zstdUncompress 6 8192 thrpt 10 85139.016 5582.793 ops/s