Automating transfers in Slurm jobs
==================================

The Globus command line interface (CLI) lets a Slurm job run Globus transfers. For example, a job can move its
outputs to Longship or Storage once it has finished.

This page covers:

1. `Loading the Globus CLI`_ on Viking
2. `Creating a service account <Creating a service account_>`_ for your jobs to log in to Globus with
3. `Giving the service account access <Giving the service account access_>`_ to your project folders
4. `An example job script <Example job script_>`_ that runs on Scratch and moves its outputs to Longship via Globus
5. `Checking the transfer and clean-up`_ after the job has finished


Loading the Globus CLI
----------------------

The `Globus CLI <https://docs.globus.org/cli/>`_ is available as a module on Viking. To check that it works, load it
on a login node:

.. code-block:: console

    $ module load {MOD_GLOBUS_CLI}
    $ globus version


Creating a service account
--------------------------

To use Globus without you being logged in, your jobs need a **service account**. This is a Globus identity for an
application rather than a person. It authenticates with a client ID and secret, so your jobs keep working if you log
out of Globus or your session expires.

1.  In the `Globus web app <https://app.globus.org>`_, go to
    `Settings > Developers <https://app.globus.org/settings/developers>`_ and click
    :guilabel:`Register a service account or application credential for automation`.
2.  Choose or create a project, give the service account a name (for example, ``viking-jobs``) and click
    :guilabel:`Register App`.
3.  Make a note of the **Client UUID**. Then click :guilabel:`Add Client Secret`, give the secret a name and copy
    it.
4.  On Viking, create the file ``~/.globus.env`` with the client ID and secret:

    .. code-block:: bash

        export GLOBUS_CLI_CLIENT_ID="<client-uuid>"
        export GLOBUS_CLI_CLIENT_SECRET="<client-secret>"

5.  Make the file readable only by you, then check that the Globus CLI can log in:

    .. code-block:: console

        $ chmod 600 ~/.globus.env
        $ module load {MOD_GLOBUS_CLI}
        $ source ~/.globus.env
        $ globus whoami
        <client-uuid>@clients.auth.globus.org

.. warning::
    Anyone with the client secret can access everything the service account can. Never share it or add it to a
    git repository. If the secret is exposed, delete it in the Globus web app and create a new one.


Giving the service account access
---------------------------------

The service account can't use the Viking collections directly. Instead, :ref:`create a guest collection
<create-guest-collection>` on your project folder in each of these collections:

* `UoY - Viking2 Longship Projects SE <https://app.globus.org/file-manager/collections/60dc7322-2782-40b5-8095-93f8de251f8c/shares>`_,
  on the folder ``/dept-proj-year/``
* `UoY - Viking2 Scratch Projects SE <https://app.globus.org/file-manager/collections/29929ac0-0e89-435f-abae-bbfe422d7768/shares>`_,
  on the folder ``/dept-proj-year/``

Here, ``dept-proj-year`` is your project code. These folders are ``/mnt/longship/projects/dept-proj-year`` and
``/mnt/scratch/projects/dept-proj-year`` on Viking.

On each guest collection's :guilabel:`Permissions` tab, click :guilabel:`Add Permissions - Share With`, choose
**user**, enter the identity of the service account you created above (``<client-uuid>@clients.auth.globus.org``) and
grant **read-write** access. Make a note of each guest collection's UUID, which is on its :guilabel:`Overview` tab.

You can check that the service account has access from a login node:

.. code-block:: console

    $ globus ls <longship-guest-collection-uuid>:/

If your data is in your personal folder rather than a project folder, create the guest collections on the folder
``/abc123/`` (your username) in
`UoY - Viking2 Longship Users SE <https://app.globus.org/file-manager/collections/2031b65b-9469-45c5-b203-061cb224a43d/shares>`_
and `UoY - Viking2 Scratch Users SE <https://app.globus.org/file-manager/collections/ccd1e1d8-9a3c-49b6-9e01-7c1fd863b5ef/shares>`_.
In the job script, set ``LONGSHIP_GUEST_ROOT`` to ``/mnt/longship/users/abc123`` and ``SCRATCH_GUEST_ROOT`` to
``/mnt/scratch/users/abc123``.


Example job script
------------------

The script below runs a program on an input file in your project folder on Longship. The program writes its outputs
to a temporary folder in your project folder on Scratch. The example program just counts the lines in the input file.
Replace it with your own.

At the end of the job, the script starts the Globus
`Move (copy and delete) files <https://docs.globus.org/api/flows/examples/move_flow/>`_ flow. The flow copies the
temporary folder to an ``output`` folder next to the input, then deletes it from Scratch. The delete only runs if the
copy succeeds. The job finishes as soon as the flow has started, so it doesn't hold on to a compute node while the data
is moved.

**Before you use the script, change:**

* ``--account`` and ``--mail-user``
* ``PROJECT``, to your project code
* ``LONGSHIP_COLLECTION`` and ``SCRATCH_COLLECTION``, to your guest collection UUIDs
* ``LONGSHIP_GUEST_ROOT`` and ``SCRATCH_GUEST_ROOT``, if you created your guest collections on different folders
* ``MY_GLOBUS_ID``, to your own Globus identity ID. To find it, run ``globus get-identities abc123@york.ac.uk`` on a
  login node.
* the program in the "Run program" section

.. code-block:: bash
    :linenos:

    {SHEBANG}
    #----------------------------- Slurm directives ------------------------------#
    #SBATCH --job-name=scratch_to_longship    # Job name
    #SBATCH --ntasks=1                        # Number of tasks
    #SBATCH --cpus-per-task=1                 # Number of CPU cores per task
    #SBATCH --mem=1G                          # Total memory to request
    #SBATCH --time=0-00:15:00                 # Time limit (DD-HH:MM:SS)
    #SBATCH --account=dept-proj-year          # Project account to use
    #SBATCH --mail-type=END,FAIL              # Mail events
    #SBATCH --mail-user=abc123@york.ac.uk     # Where to send mail
    # Longship is read only on the compute nodes, so the logs go to your home folder.
    # The logs folder must exist before you submit the job.
    #SBATCH --output=/users/%u/logs/%x-%j.log
    #SBATCH --error=/users/%u/logs/%x-%j.err

    # Abort if any command fails
    set -euo pipefail

    #------------------------------- Configuration -------------------------------#
    PROJECT="dept-proj-year"

    # Your guest collections, and the folder on Viking each one was created on
    LONGSHIP_COLLECTION="LONGSHIP-GUEST-COLLECTION-UUID"
    LONGSHIP_GUEST_ROOT="/mnt/longship/projects/${PROJECT}"
    SCRATCH_COLLECTION="SCRATCH-GUEST-COLLECTION-UUID"
    SCRATCH_GUEST_ROOT="/mnt/scratch/projects/${PROJECT}"

    # Your Globus identity ID, so that you can follow the flow in the Globus web app
    MY_GLOBUS_ID="YOUR-GLOBUS-IDENTITY-ID"

    # Globus's "Move (copy and delete) files" flow
    FLOW_ID="6336492e-e308-4a67-b78e-13684c747472"

    # Read input from Longship
    INPUT_FILE="${SLURM_SUBMIT_DIR}/input.txt"
    # Your program writes to Scratch
    WORK_DIR="${SCRATCH_GUEST_ROOT}/${USER}/${SLURM_JOB_ID}"
    # Once it completes, your outputs are moved to Longship via Globus
    RESULTS_DIR="${SLURM_SUBMIT_DIR}/output"

    # Turn a path on Viking into a path on a guest collection.
    to_globus_path() {
        local path root
        path=$(realpath -m "$1")
        root=$(realpath -m "$2")
        if [[ "${path}" != "${root}" && "${path}" != "${root}"/* ]]; then
            echo "Error: $1 is not inside $2" >&2
            return 1
        fi
        echo "${path#"${root}"}/"
    }

    SRC_PATH=$(to_globus_path "${WORK_DIR}" "${SCRATCH_GUEST_ROOT}")
    DST_PATH=$(to_globus_path "${RESULTS_DIR}" "${LONGSHIP_GUEST_ROOT}")

    #------------------------------- Globus set-up -------------------------------#
    module purge
    module load {MOD_GLOBUS_CLI}
    source ~/.globus.env                      # Service account ID and secret
    globus whoami                             # Stop early if Globus can't be reached

    #-------------------------------- Run program --------------------------------#
    mkdir -p "${WORK_DIR}"
    cd "${WORK_DIR}"

    # Replace this with your own program. The exit code is saved so that the
    # outputs are moved to Longship even if the program fails.
    set +e
    wc -l "${INPUT_FILE}" > "${WORK_DIR}/result.txt"
    EXIT_CODE=$?
    set -e

    #------------------------ Move the outputs to Longship ------------------------#
    LABEL="slurm-${SLURM_JOB_ID}"
    FLOW_INPUT=$(cat <<EOF
    {
      "source": {"id": "${SCRATCH_COLLECTION}", "path": "${SRC_PATH}"},
      "destination": {"id": "${LONGSHIP_COLLECTION}", "path": "${DST_PATH}"}
    }
    EOF
    )

    RUN_ID=$(globus flows start "${FLOW_ID}" \
        --input "${FLOW_INPUT}" --label "${LABEL}" \
        --manager "urn:globus:auth:identity:${MY_GLOBUS_ID}" \
        --jmespath run_id --format unix)

    echo "Started Globus flow run ${RUN_ID} to move ${WORK_DIR} to ${RESULTS_DIR}"
    echo "Check its progress with: globus flows run show ${RUN_ID}"

    exit "${EXIT_CODE}"

Save the script as ``scratch_to_longship.sh`` in a folder in your project on Longship. The example program reads
``input.txt`` from that folder. To try it out, create a small test file with the numbers 1 to 10, one per line, and
submit the script from the same folder:

.. code-block:: console

    $ mkdir -p ~/logs
    $ cd /mnt/longship/projects/dept-proj-year/my-analysis
    $ seq 1 10 > input.txt
    $ sbatch scratch_to_longship.sh

Once the flow has finished, ``output/result.txt`` in the same folder contains the number of lines in ``input.txt``
(10), followed by the path of the file. Outputs from an earlier run are overwritten.

Slurm emails you when the job ends, which is before the flow has finished moving the outputs. To follow the flow, see
`Checking the transfer and clean-up`_.

.. tip::
    Reading input straight from Longship should work well for most jobs. If your program reads many small files, or
    reads the same data many times, copy the input to Scratch with ``cp`` before running the program, and delete it
    with ``rm`` before the outputs are moved. See :ref:`Longship <longship>` for more on this.


Checking the transfer and clean-up
----------------------------------

The flow carries on after the job has finished. The last lines of the job log
(``~/logs/scratch_to_longship-<job-id>.log``) give the ID of the flow run. Every run is labelled ``slurm-<job-id>``.

In the Globus web app
^^^^^^^^^^^^^^^^^^^^^

The job makes you a manager of the run, so you can follow it on the
`Runs page <https://app.globus.org/runs?filter_owner=>`_ of the Globus web app. Click a run to see its status and its
event log, which lists each step as it starts and finishes. You can also cancel the run from there.

.. note::
    Your service account starts the run, not you. On the :guilabel:`Runs` page, untick the
    :guilabel:`STARTED BY ME` filter, otherwise the run won't be listed. The link above opens the page with this
    filter already removed.

From a login node
^^^^^^^^^^^^^^^^^

The flow runs as your service account, so load its credentials first:

.. code-block:: console

    $ module load {MOD_GLOBUS_CLI}
    $ source ~/.globus.env

Then check the status of the run:

.. code-block:: console

    $ globus flows run show <run-id>

The status is one of:

* ``ACTIVE``: the flow is still moving the outputs.
* ``SUCCEEDED``: the outputs are on Longship and the temporary folder on Scratch has been deleted.
* ``FAILED``: one of the steps failed. If it was the transfer, the temporary folder on Scratch has been kept.

To see which step the run is on, list its event log. Add ``--details`` for more information on each step:

.. code-block:: console

    $ globus flows run show-logs <run-id>

To see how far the transfer has got, list your service account's latest transfer and delete tasks (newest first) and
show the one you're interested in. ``globus task show`` gives the number of files and bytes transferred so far, and
any errors:

.. code-block:: console

    $ globus task list --limit 10
    $ globus task show <task-id>

To list your recent runs:

.. code-block:: console

    $ globus flows run list --filter-flow-id 6336492e-e308-4a67-b78e-13684c747472

If the transfer fails, nothing is lost. The outputs are still in the temporary folder on Scratch,
``/mnt/scratch/projects/dept-proj-year/abc123/<job-id>``. Fix the problem, move the outputs yourself (for
example in the Globus web app) and then delete the folder.


Troubleshooting
---------------

The job fails straight away and there is no log
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Slurm couldn't create the log file because the logs folder doesn't exist. Create it, or change ``--output`` and
``--error`` in the script:

.. code-block:: console

    $ mkdir -p ~/logs

The job fails at ``globus whoami``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The Globus CLI couldn't log in as your service account. Check that ``~/.globus.env`` contains the right client ID
and secret, then test them on a login node:

.. code-block:: console

    $ module load {MOD_GLOBUS_CLI}
    $ source ~/.globus.env
    $ globus whoami

The job fails at ``globus flows start``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Check ``FLOW_ID`` and ``MY_GLOBUS_ID`` in the script. The error log (``~/logs/scratch_to_longship-<job-id>.err``)
usually says which one is wrong.


Further assistance
------------------

For more on what the Globus CLI can do, see the `Globus CLI documentation <https://docs.globus.org/cli/>`_. For more
on monitoring flow runs, see `Monitoring a run <https://docs.globus.org/api/flows/getting-started/monitor-a-run/>`_.
For how the flow works, see `Move (copy and delete) files <https://docs.globus.org/api/flows/examples/move_flow/>`_.