(DE)COMP TIME is the benchmarked time of how long the compressor took to compress/decompress the file.FSIZE refers to the size of the resulting compressed file.REDUCTION refers to how many megabytes that the original file was reduced by, after compression.

The lower the number, the better - a lower number means the compressor compresses data faster than others.

SECS PER MB SAVED is a statistic that measures how many seconds on average that the compressor takes to compress 1 single megabyte ( 1024 * 1024 * 1024 = 1,073,741,824 bytes).

RAM: 64GB DDR3 ECC (1333 MHz) - 8x 8GB DIMMsįilesystem/RAID: ZFS with SSDs in ZFS Mirror, ashift 12, all pools filesystem compressed with lz4 Compression tools which support threading ( xz and lbzip2) were ran with 10 threads in these benchmarks.īenchmark System: Distro: Ubuntu Linux (Server) 20.04 (Focal Fossa).YMMV, depending on what you're (de-)compressing, different compression algorithms may work better or worse.These stats are from a single run at level 1 and level 9, so there may be discrepancies.

The compression benchmark file is a 619 MB uncompressed Tar file containing a debootstrap'd Ubuntu 20.04 (Focal Fossa) root filesystem, which contains a large mixture of different filetypes, some which compress well, and some which don't compress at all due to already being compressed.

However, after seeing these benchmarks and our summary, you should be able to make an educated decision on which algorithm to use per file/folder you're compressing. Spoiler: There isn't a 'best' compression algorithm, because every tool / algorithm has tradeoffs, whether it's speed, RAM usage, CPU usage, faster/slow decompression than compression, among other tradeoffs. In this article, we'll be showing compress + decompress benchmarks for 4 of the most popular Linux compressionĪlgorithms: gzip, bzip2 (using lbzip2), xz, and lz4. Maybe I need to prefer fsarchiver or something like them instead of dd, but it is less commonly used and not released as v1.x yet.Which Linux/UNIX compression algorithm is best? Intro The setuid, setgid, and sticky bits are not copied to the target file.ĭo I need to worry about command options like -xattrs, -acls in the case when I use a dd | tar pipe?įinally, do I have to always specify the bs attribute of the dd command if I backup entire /dev/sdX and use a compressed file as an intermediate backup location if I want to get the most identical copy of the original device as a result of recovery? In this case, which bs size to prefer? Since xz 5.2.6, this option also makes xz compress or decompress even if the input is a symbolic link to a regular file, has more than one hard link, or has the setuid, setgid, or sticky bit set. I'm not sure that i don't in the view of this quote from xz's manual: Is the sudo dd if=/dev/sdX | xz -9 -threads=0 -keep -v > pipe enough to save all specified data? And a pipe like xz -dck | dd of=/dev/sdX status=progress to restore the backup? Do I need to worry about metadata (attributes, ACLs, etc)? xz doesn't support copying other metadata like access control lists or extended attributes yet.ĭoes it matter in case I use it in a pipe with dd which copies blocks not files?

If copying the group fails, the permissions are modified so that the target file doesn't become accessible to users who didn't have permission to access the source file. But man saysĪfter successfully compressing or decompressing the file, xz copies the owner, group, permissions, access time, and modification time from the source file to the target file. Other option is to use the most modern, efficient and multi-threaded compression tools like xz. Вesides, I'm not sure gzip can offer the most efficient compression mechanism. I see some options like to use dd | gzip or dd | tar pipe but I'm not sure that this will save all the necessary data in view of this answer. The easiest way is to use dd or even dump utility but we will get big uncompressed file (.iso. What is the most complete way to backup/restore entire /dev/sdX (to/from a file) including device partition table and partition flags, all possible metadata, owners, file permissions, ACLs, creation/access/change times, file attributes (like append only (a) or immutable (i)), xattrs, access rights flags (like setuid, setgid, sticky bit), filesystem-specific attributes, etc if I want to get compressed backup file? /dev/sdX's partitions can be formatted in ext4, fat, exfat, ntfs, different file system for each partition or not.

Speed up xz compression