.. raw:: html

   <style>
   .wy-nav-content { max-width: 860px !important; }
   </style>

Compressing data
================

When dealing with large quantities of data (size and/or number of files), standard tools like ``gzip``, ``tar``, etc can take a long time. We would encourage use of utilities that support parallel compression which can be much faster compressing/archiving large directories of files.

Please **do not run any parallel utilities on the login nodes**. You must submit these as jobs via :doc:`Slurm <../using_viking/submitting_jobs>` (example job scripts below), otherwise the processes will be terminated (and would likely run slower than without parallelism in the interim, due to restrictions on login node process parallelism).

.. important::

    Please be responsible with the parallelism used and avoid excessive load on the filesystem.


Parallel compression programs
-----------------------------

pigz
^^^^

``pigz`` compresses using threads to make use of multiple processors and cores. The input is broken up into 128 KB chunks with each compressed in parallel.

Example usage
"""""""""""""

.. code-block:: bash
    
    tar --use-compress-program="pigz -p $SLURM_CPUS_PER_TASK" -cf directory.tar.gz /path/to/directory


Example job script
""""""""""""""""""

.. code-block:: bash

    #!/usr/bin/env bash
    #----------------------------- Slurm directives ------------------------------#
    #SBATCH --job-name=compress_example     # Job name
    #SBATCH --nodes=1                       # Run on a single node
    #SBATCH --cpus-per-task=8               # Number of CPU cores (threads) to use
    #SBATCH --time=0-01:00:00               # Time limit (DD-HH:MM:SS)
    #SBATCH --account=dept-proj-year        # Project account to use
    #SBATCH --output=%x-%j.log              # Standard output log

    # Abort if any command fails
    set -e

    # Purge any existing modules
    module purge

    # Load modules
    module load {MOD_PIGZ}

    # Directory to archive and the resulting archive name
    SOURCE_DIR="/path/to/directory"
    ARCHIVE="directory.tar.gz"

    # Create a compressed tar archive using all requested cores.
    tar --use-compress-program="pigz -p $SLURM_CPUS_PER_TASK" -cf "$ARCHIVE" "$SOURCE_DIR"


zstd
^^^^

``zstd`` is a fast lossless compression algorithm and data compression tool, with command line syntax similar to ``gzip`` and ``xz``.


Example usage
"""""""""""""

.. code-block:: bash

    tar --use-compress-program="zstd -T$SLURM_CPUS_PER_TASK" -cf directory.tar.zst /path/to/directory


Example job script
""""""""""""""""""

.. code-block:: bash

    #!/usr/bin/env bash
    #----------------------------- Slurm directives ------------------------------#
    #SBATCH --job-name=compress_example     # Job name
    #SBATCH --nodes=1                       # Run on a single node
    #SBATCH --cpus-per-task=8               # Number of CPU cores (threads) to use
    #SBATCH --time=0-01:00:00               # Time limit (DD-HH:MM:SS)
    #SBATCH --account=dept-proj-year        # Project account to use
    #SBATCH --output=%x-%j.log              # Standard output log

    # Abort if any command fails
    set -e

    # Purge any existing modules
    module purge

    # Load modules
    module load {MOD_ZSTD}

    # Directory to archive and the resulting archive name
    SOURCE_DIR="/path/to/directory"
    ARCHIVE="directory.tar.zst"

    # Create a compressed tar archive using all requested cores.
    tar --use-compress-program="zstd -T$SLURM_CPUS_PER_TASK" -cf "$ARCHIVE" "$SOURCE_DIR"


dtar and dbz2 (from mpifileutils)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

These are MPI-parallelised utilities for creating tar/bz2 archives respectively. 

``dtar`` writes archives in pax file format. In addition to the archive file, ``dtar`` creates an index to record the number of items and the starting byte offset of each entry within the archive. This index enables faster parallel extraction. When compressing using ``dbz2``, a new file will be created with a ``.dbz2`` extension.

.. note::

    The target items to be archived must be under the `current working directory <https://mpifileutils.readthedocs.io/en/0.12-rtd/dtar.1.html#limitations>`_ where ``dtar`` is running, so you must ``cd`` to the parent directory first.


Example usage
"""""""""""""

.. code-block:: bash
    
    module load {MOD_MPIFILEUTILS}
    cd /path/to/parent
    mpirun -np $SLURM_NTASKS dtar -c -f directory.tar directory
    mpirun -np $SLURM_NTASKS dbz2 --compress directory.tar



Example job script
""""""""""""""""""

.. code-block:: bash

    #!/usr/bin/env bash
    #----------------------------- Slurm directives ------------------------------#
    #SBATCH --job-name=compress_example     # Job name
    #SBATCH --ntasks=8                      # Number of MPI tasks to request
    #SBATCH --nodes=1                       # Run on a single node
    #SBATCH --cpus-per-task=1               # Number of CPU cores (threads) to use
    #SBATCH --time=0-01:00:00               # Time limit (DD-HH:MM:SS)
    #SBATCH --account=dept-proj-year        # Project account to use
    #SBATCH --output=%x-%j.log              # Standard output log

    # Abort if any command fails
    set -e

    # Purge any existing modules
    module purge

    # Load modules
    module load {MOD_MPIFILEUTILS}

    # Directory to archive and the resulting archive name
    PARENT_DIR="/path/to/parent"
    SOURCE_DIR="directory"
    ARCHIVE="directory.tar"

    # Navigate to target directory's parent (dtar requirement)
    cd "$PARENT_DIR"
    # Create a compressed tar archive using all requested ntasks
    mpirun -np $SLURM_NTASKS dtar -c -f "$ARCHIVE" "$SOURCE_DIR"
    # Create a compressed dbz2 archive using all requested ntasks
    mpirun -np $SLURM_NTASKS dbz2 --compress "$ARCHIVE"