Comparison of compression tools for taking snapshots of Radix database

RadixPool is planning to host snapshots of the Radix node database on a Content Delivery Network (CDN) to allow Radix validator nodes to quickly recover from a corrupted database.

We compared a number of compression tools on a 43Gb copy of the Radix database and recorded the following results:

Algorithm Tool Options Compress Size (bytes) Decompress 10Mbits/s 50Mbit/s 100Mbit/s
- - - - 45,652,470,885 - 10h08m 2h01m 1h00m
gzip gzip 19m35s 25,009,428,317 3m50s 5h34m 1h07m 0h34m
gzip pigz 2m32s 24,962,921,223 1m58s 5h33m 1h07m 0h34m
gzip pigz --best 6m37s 24,932,450,560 2m10s 5h32m 1h07m 0h34m
xz pixz 21m33s 21,702,409,016 1m38s 4h49m 0h57m 0h28m
xz pixz -9 -p8 42m58s 21,412,039,372 1m39s 4h45m 0h57m 0h28m
7z 7z 20m50s 21,683,532,361 16m15s 4h49m 0h57m 0h28m
7z 7z -m0=lzma2 -mx 35m57s 21,367,916,757 16m01s 4h44m 0h56m 0h28m
zstd zstd --fast=1 1m26s 24,062,687,524 1m13s 5h20m 1h04m 0h32m
zstd zstd 2m53s 23,032,636,843 1m23s 5h07m 1h01m 0h30m
zstd zstd --long 4m53s 22,743,480,552 1m26s 5h03m 1h00m 0h30m
zstd zstd -19 -T8 39m25s 21,999,437,864 1m32s 4h53m 0h58m 0h29m


The xz algorithm was selected for Radix database compression for it’s high compression ratio and fast decompression speed. xz has binaries for Linux and Windows.

Not Tested:

Further Reading: