Automating transfers in Slurm jobs

The Globus command line interface (CLI) lets a Slurm job run Globus transfers. For example, a job can move its outputs to Longship or Storage once it has finished.

This page covers:

  1. Loading the Globus CLI on Viking

  2. Creating a service account for your jobs to log in to Globus with

  3. Giving the service account access to your project folders

  4. An example job script that runs on Scratch and moves its outputs to Longship via Globus

  5. Checking the transfer and clean-up after the job has finished

Loading the Globus CLI

The Globus CLI is available as a module on Viking. To check that it works, load it on a login node:

$ module load Globus-CLI/3.42.0-GCCcore-14.3.0
$ globus version

Creating a service account

To use Globus without you being logged in, your jobs need a service account. This is a Globus identity for an application rather than a person. It authenticates with a client ID and secret, so your jobs keep working if you log out of Globus or your session expires.

  1. In the Globus web app, go to Settings > Developers and click Register a service account or application credential for automation.

  2. Choose or create a project, give the service account a name (for example, viking-jobs) and click Register App.

  3. Make a note of the Client UUID. Then click Add Client Secret, give the secret a name and copy it.

  4. On Viking, create the file ~/.globus.env with the client ID and secret:

    export GLOBUS_CLI_CLIENT_ID="<client-uuid>"
export GLOBUS_CLI_CLIENT_SECRET="<client-secret>"

  5. Make the file readable only by you, then check that the Globus CLI can log in:

    $ chmod 600 ~/.globus.env
$ module load Globus-CLI/3.42.0-GCCcore-14.3.0
$ source ~/.globus.env
$ globus whoami
<client-uuid>@clients.auth.globus.org

Warning

Anyone with the client secret can access everything the service account can. Never share it or add it to a git repository. If the secret is exposed, delete it in the Globus web app and create a new one.

Giving the service account access

The service account can’t use the Viking collections directly. Instead, create a guest collection on your project folder in each of these collections:

Here, dept-proj-year is your project code. These folders are /mnt/longship/projects/dept-proj-year and /mnt/scratch/projects/dept-proj-year on Viking.

On each guest collection’s Permissions tab, click Add Permissions - Share With, choose user, enter the identity of the service account you created above (<client-uuid>@clients.auth.globus.org) and grant read-write access. Make a note of each guest collection’s UUID, which is on its Overview tab.

You can check that the service account has access from a login node:

$ globus ls <longship-guest-collection-uuid>:/

If your data is in your personal folder rather than a project folder, create the guest collections on the folder /abc123/ (your username) in UoY - Viking2 Longship Users SE and UoY - Viking2 Scratch Users SE. In the job script, set LONGSHIP_GUEST_ROOT to /mnt/longship/users/abc123 and SCRATCH_GUEST_ROOT to /mnt/scratch/users/abc123.

Example job script

The script below runs a program on an input file in your project folder on Longship. The program writes its outputs to a temporary folder in your project folder on Scratch. The example program just counts the lines in the input file. Replace it with your own.

At the end of the job, the script starts the Globus Move (copy and delete) files flow. The flow copies the temporary folder to an output folder next to the input, then deletes it from Scratch. The delete only runs if the copy succeeds. The job finishes as soon as the flow has started, so it doesn’t hold on to a compute node while the data is moved.

Before you use the script, change:

  • --account and --mail-user

  • PROJECT, to your project code

  • LONGSHIP_COLLECTION and SCRATCH_COLLECTION, to your guest collection UUIDs

  • LONGSHIP_GUEST_ROOT and SCRATCH_GUEST_ROOT, if you created your guest collections on different folders

  • MY_GLOBUS_ID, to your own Globus identity ID. To find it, run globus get-identities abc123@york.ac.uk on a login node.

  • the program in the “Run program” section

 1#!/usr/bin/env bash
 2#----------------------------- Slurm directives ------------------------------#
 3#SBATCH --job-name=scratch_to_longship    # Job name
 4#SBATCH --ntasks=1                        # Number of tasks
 5#SBATCH --cpus-per-task=1                 # Number of CPU cores per task
 6#SBATCH --mem=1G                          # Total memory to request
 7#SBATCH --time=0-00:15:00                 # Time limit (DD-HH:MM:SS)
 8#SBATCH --account=dept-proj-year          # Project account to use
 9#SBATCH --mail-type=END,FAIL              # Mail events
10#SBATCH --mail-user=abc123@york.ac.uk     # Where to send mail
11# Longship is read only on the compute nodes, so the logs go to your home folder.
12# The logs folder must exist before you submit the job.
13#SBATCH --output=/users/%u/logs/%x-%j.log
14#SBATCH --error=/users/%u/logs/%x-%j.err
15
16# Abort if any command fails
17set -euo pipefail
18
19#------------------------------- Configuration -------------------------------#
20PROJECT="dept-proj-year"
21
22# Your guest collections, and the folder on Viking each one was created on
23LONGSHIP_COLLECTION="LONGSHIP-GUEST-COLLECTION-UUID"
24LONGSHIP_GUEST_ROOT="/mnt/longship/projects/${PROJECT}"
25SCRATCH_COLLECTION="SCRATCH-GUEST-COLLECTION-UUID"
26SCRATCH_GUEST_ROOT="/mnt/scratch/projects/${PROJECT}"
27
28# Your Globus identity ID, so that you can follow the flow in the Globus web app
29MY_GLOBUS_ID="YOUR-GLOBUS-IDENTITY-ID"
30
31# Globus's "Move (copy and delete) files" flow
32FLOW_ID="6336492e-e308-4a67-b78e-13684c747472"
33
34# Read input from Longship
35INPUT_FILE="${SLURM_SUBMIT_DIR}/input.txt"
36# Your program writes to Scratch
37WORK_DIR="${SCRATCH_GUEST_ROOT}/${USER}/${SLURM_JOB_ID}"
38# Once it completes, your outputs are moved to Longship via Globus
39RESULTS_DIR="${SLURM_SUBMIT_DIR}/output"
40
41# Turn a path on Viking into a path on a guest collection.
42to_globus_path() {
43    local path root
44    path=$(realpath -m "$1")
45    root=$(realpath -m "$2")
46    if [[ "${path}" != "${root}" && "${path}" != "${root}"/* ]]; then
47        echo "Error: $1 is not inside $2" >&2
48        return 1
49    fi
50    echo "${path#"${root}"}/"
51}
52
53SRC_PATH=$(to_globus_path "${WORK_DIR}" "${SCRATCH_GUEST_ROOT}")
54DST_PATH=$(to_globus_path "${RESULTS_DIR}" "${LONGSHIP_GUEST_ROOT}")
55
56#------------------------------- Globus set-up -------------------------------#
57module purge
58module load Globus-CLI/3.42.0-GCCcore-14.3.0
59source ~/.globus.env                      # Service account ID and secret
60globus whoami                             # Stop early if Globus can't be reached
61
62#-------------------------------- Run program --------------------------------#
63mkdir -p "${WORK_DIR}"
64cd "${WORK_DIR}"
65
66# Replace this with your own program. The exit code is saved so that the
67# outputs are moved to Longship even if the program fails.
68set +e
69wc -l "${INPUT_FILE}" > "${WORK_DIR}/result.txt"
70EXIT_CODE=$?
71set -e
72
73#------------------------ Move the outputs to Longship ------------------------#
74LABEL="slurm-${SLURM_JOB_ID}"
75FLOW_INPUT=$(cat <<EOF
76{
77  "source": {"id": "${SCRATCH_COLLECTION}", "path": "${SRC_PATH}"},
78  "destination": {"id": "${LONGSHIP_COLLECTION}", "path": "${DST_PATH}"}
79}
80EOF
81)
82
83RUN_ID=$(globus flows start "${FLOW_ID}" \
84    --input "${FLOW_INPUT}" --label "${LABEL}" \
85    --manager "urn:globus:auth:identity:${MY_GLOBUS_ID}" \
86    --jmespath run_id --format unix)
87
88echo "Started Globus flow run ${RUN_ID} to move ${WORK_DIR} to ${RESULTS_DIR}"
89echo "Check its progress with: globus flows run show ${RUN_ID}"
90
91exit "${EXIT_CODE}"

Save the script as scratch_to_longship.sh in a folder in your project on Longship. The example program reads input.txt from that folder. To try it out, create a small test file with the numbers 1 to 10, one per line, and submit the script from the same folder:

$ mkdir -p ~/logs
$ cd /mnt/longship/projects/dept-proj-year/my-analysis
$ seq 1 10 > input.txt
$ sbatch scratch_to_longship.sh

Once the flow has finished, output/result.txt in the same folder contains the number of lines in input.txt (10), followed by the path of the file. Outputs from an earlier run are overwritten.

Slurm emails you when the job ends, which is before the flow has finished moving the outputs. To follow the flow, see Checking the transfer and clean-up.

Tip

Reading input straight from Longship should work well for most jobs. If your program reads many small files, or reads the same data many times, copy the input to Scratch with cp before running the program, and delete it with rm before the outputs are moved. See Longship for more on this.

Checking the transfer and clean-up

The flow carries on after the job has finished. The last lines of the job log (~/logs/scratch_to_longship-<job-id>.log) give the ID of the flow run. Every run is labelled slurm-<job-id>.

In the Globus web app

The job makes you a manager of the run, so you can follow it on the Runs page of the Globus web app. Click a run to see its status and its event log, which lists each step as it starts and finishes. You can also cancel the run from there.

Note

Your service account starts the run, not you. On the Runs page, untick the STARTED BY ME filter, otherwise the run won’t be listed. The link above opens the page with this filter already removed.

From a login node

The flow runs as your service account, so load its credentials first:

$ module load Globus-CLI/3.42.0-GCCcore-14.3.0
$ source ~/.globus.env

Then check the status of the run:

$ globus flows run show <run-id>

The status is one of:

  • ACTIVE: the flow is still moving the outputs.

  • SUCCEEDED: the outputs are on Longship and the temporary folder on Scratch has been deleted.

  • FAILED: one of the steps failed. If it was the transfer, the temporary folder on Scratch has been kept.

To see which step the run is on, list its event log. Add --details for more information on each step:

$ globus flows run show-logs <run-id>

To see how far the transfer has got, list your service account’s latest transfer and delete tasks (newest first) and show the one you’re interested in. globus task show gives the number of files and bytes transferred so far, and any errors:

$ globus task list --limit 10
$ globus task show <task-id>

To list your recent runs:

$ globus flows run list --filter-flow-id 6336492e-e308-4a67-b78e-13684c747472

If the transfer fails, nothing is lost. The outputs are still in the temporary folder on Scratch, /mnt/scratch/projects/dept-proj-year/abc123/<job-id>. Fix the problem, move the outputs yourself (for example in the Globus web app) and then delete the folder.

Troubleshooting

The job fails straight away and there is no log

Slurm couldn’t create the log file because the logs folder doesn’t exist. Create it, or change --output and --error in the script:

$ mkdir -p ~/logs

The job fails at globus whoami

The Globus CLI couldn’t log in as your service account. Check that ~/.globus.env contains the right client ID and secret, then test them on a login node:

$ module load Globus-CLI/3.42.0-GCCcore-14.3.0
$ source ~/.globus.env
$ globus whoami

The job fails at globus flows start

Check FLOW_ID and MY_GLOBUS_ID in the script. The error log (~/logs/scratch_to_longship-<job-id>.err) usually says which one is wrong.

Further assistance

For more on what the Globus CLI can do, see the Globus CLI documentation. For more on monitoring flow runs, see Monitoring a run. For how the flow works, see Move (copy and delete) files.