Compressing data

When dealing with large quantities of data (size and/or number of files), standard tools like gzip, tar, etc can take a long time. We would encourage use of utilities that support parallel compression which can be much faster compressing/archiving large directories of files.

Please do not run any parallel utilities on the login nodes. You must submit these as jobs via Slurm (example job scripts below), otherwise the processes will be terminated (and would likely run slower than without parallelism in the interim, due to restrictions on login node process parallelism).

Important

Please be responsible with the parallelism used and avoid excessive load on the filesystem.

Parallel compression programs

pigz

pigz compresses using threads to make use of multiple processors and cores. The input is broken up into 128 KB chunks with each compressed in parallel.

Example usage

tar --use-compress-program="pigz -p $SLURM_CPUS_PER_TASK" -cf directory.tar.gz /path/to/directory

Example job script

#!/usr/bin/env bash
#----------------------------- Slurm directives ------------------------------#
#SBATCH --job-name=compress_example     # Job name
#SBATCH --nodes=1                       # Run on a single node
#SBATCH --cpus-per-task=8               # Number of CPU cores (threads) to use
#SBATCH --time=0-01:00:00               # Time limit (DD-HH:MM:SS)
#SBATCH --account=dept-proj-year        # Project account to use
#SBATCH --output=%x-%j.log              # Standard output log

# Abort if any command fails
set -e

# Purge any existing modules
module purge

# Load modules
module load pigz/2.8-GCCcore-12.3.0

# Directory to archive and the resulting archive name
SOURCE_DIR="/path/to/directory"
ARCHIVE="directory.tar.gz"

# Create a compressed tar archive using all requested cores.
tar --use-compress-program="pigz -p $SLURM_CPUS_PER_TASK" -cf "$ARCHIVE" "$SOURCE_DIR"

zstd

zstd is a fast lossless compression algorithm and data compression tool, with command line syntax similar to gzip and xz.

Example usage

tar --use-compress-program="zstd -T$SLURM_CPUS_PER_TASK" -cf directory.tar.zst /path/to/directory

Example job script

#!/usr/bin/env bash
#----------------------------- Slurm directives ------------------------------#
#SBATCH --job-name=compress_example     # Job name
#SBATCH --nodes=1                       # Run on a single node
#SBATCH --cpus-per-task=8               # Number of CPU cores (threads) to use
#SBATCH --time=0-01:00:00               # Time limit (DD-HH:MM:SS)
#SBATCH --account=dept-proj-year        # Project account to use
#SBATCH --output=%x-%j.log              # Standard output log

# Abort if any command fails
set -e

# Purge any existing modules
module purge

# Load modules
module load zstd/1.5.7-GCCcore-14.3.0

# Directory to archive and the resulting archive name
SOURCE_DIR="/path/to/directory"
ARCHIVE="directory.tar.zst"

# Create a compressed tar archive using all requested cores.
tar --use-compress-program="zstd -T$SLURM_CPUS_PER_TASK" -cf "$ARCHIVE" "$SOURCE_DIR"

dtar and dbz2 (from mpifileutils)

These are MPI-parallelised utilities for creating tar/bz2 archives respectively.

dtar writes archives in pax file format. In addition to the archive file, dtar creates an index to record the number of items and the starting byte offset of each entry within the archive. This index enables faster parallel extraction. When compressing using dbz2, a new file will be created with a .dbz2 extension.

Note

The target items to be archived must be under the current working directory where dtar is running, so you must cd to the parent directory first.

Example usage

module load mpifileutils/0.12-gompi-2025b
cd /path/to/parent
mpirun -np $SLURM_NTASKS dtar -c -f directory.tar directory
mpirun -np $SLURM_NTASKS dbz2 --compress directory.tar

Example job script

#!/usr/bin/env bash
#----------------------------- Slurm directives ------------------------------#
#SBATCH --job-name=compress_example     # Job name
#SBATCH --ntasks=8                      # Number of MPI tasks to request
#SBATCH --nodes=1                       # Run on a single node
#SBATCH --cpus-per-task=1               # Number of CPU cores (threads) to use
#SBATCH --time=0-01:00:00               # Time limit (DD-HH:MM:SS)
#SBATCH --account=dept-proj-year        # Project account to use
#SBATCH --output=%x-%j.log              # Standard output log

# Abort if any command fails
set -e

# Purge any existing modules
module purge

# Load modules
module load mpifileutils/0.12-gompi-2025b

# Directory to archive and the resulting archive name
PARENT_DIR="/path/to/parent"
SOURCE_DIR="directory"
ARCHIVE="directory.tar"

# Navigate to target directory's parent (dtar requirement)
cd "$PARENT_DIR"
# Create a compressed tar archive using all requested ntasks
mpirun -np $SLURM_NTASKS dtar -c -f "$ARCHIVE" "$SOURCE_DIR"
# Create a compressed dbz2 archive using all requested ntasks
mpirun -np $SLURM_NTASKS dbz2 --compress "$ARCHIVE"