RadixPool is planning to host snapshots of the Radix node database on a Content Delivery Network (CDN) to allow Radix validator nodes to quickly recover from a corrupted database.
We compared a number of compression tools on a 43Gb copy of the Radix database and recorded the following results:
| Algorithm | Tool | Options | Compress | Size (bytes) | Decompress | 10Mbits/s | 50Mbit/s | 100Mbit/s |
|---|---|---|---|---|---|---|---|---|
| - | - | - | - | 45,652,470,885 | - | 10h08m | 2h01m | 1h00m |
| gzip | gzip | 19m35s | 25,009,428,317 | 3m50s | 5h34m | 1h07m | 0h34m | |
| gzip | pigz | 2m32s | 24,962,921,223 | 1m58s | 5h33m | 1h07m | 0h34m | |
| gzip | pigz | --best |
6m37s | 24,932,450,560 | 2m10s | 5h32m | 1h07m | 0h34m |
| xz | pixz | 21m33s | 21,702,409,016 | 1m38s | 4h49m | 0h57m | 0h28m | |
| xz | pixz | -9 -p8 |
42m58s | 21,412,039,372 | 1m39s | 4h45m | 0h57m | 0h28m |
| 7z | 7z | 20m50s | 21,683,532,361 | 16m15s | 4h49m | 0h57m | 0h28m | |
| 7z | 7z | -m0=lzma2 -mx |
35m57s | 21,367,916,757 | 16m01s | 4h44m | 0h56m | 0h28m |
| zstd | zstd | --fast=1 |
1m26s | 24,062,687,524 | 1m13s | 5h20m | 1h04m | 0h32m |
| zstd | zstd | 2m53s | 23,032,636,843 | 1m23s | 5h07m | 1h01m | 0h30m | |
| zstd | zstd | --long |
4m53s | 22,743,480,552 | 1m26s | 5h03m | 1h00m | 0h30m |
| zstd | zstd | -19 -T8 |
39m25s | 21,999,437,864 | 1m32s | 4h53m | 0h58m | 0h29m |
Results
The xz algorithm was selected for Radix database compression for it’s high compression ratio and fast decompression speed. xz has binaries for Linux and Windows.