SLURM Configuration Guide

This guide covers how the BIOMERO Python client reads slurm-config.ini, which settings affect runtime behaviour, and when environment variables should override the file.

For a full lookup table of all supported configuration environment variables and their slurm-config.ini counterparts, see Configuration Reference.

How BIOMERO Finds Configuration

SlurmClient.from_config() reads configuration from these locations, in this order:

1. /etc/slurm-config.ini

2. /OMERO/slurm-config.ini

3. ~/slurm-config.ini

4. the explicit configfile=... path passed by the caller

Later files override earlier files.

For deployments where an administration UI owns the complete configuration, set BIOMERO_SLURM_CONFIG_FILE to the path of that file. This enables authoritative-file mode: BIOMERO reads only that file and does not merge the default paths or an explicit configfile=... path. Removing a setting from the authoritative file therefore cannot reveal a lower-precedence file value. BIOMERO logs the files it read, and warns when an explicit configfile is ignored or the authoritative file is missing (the defaults and environment variables then apply). If the variable is unset or empty, the layered behavior above is unchanged.

For the settings documented on this page, environment variables then override the resolved slurm-config.ini value when supported.

Configuration Sections

BIOMERO uses these sections in slurm-config.ini:

• [SSH] for the Fabric SSH host alias

• [SLURM] for shared paths and client-level runtime behaviour

• [ANALYTICS] for BIOMERO 2.x workflow tracking and provenance (database integration)

• [CONVERTERS] for external converter container images

• [WORKFLOWS] (or [MODELS] for legacy configs) for workflow repositories, job scripts, and per-workflow sbatch overrides

The [ANALYTICS] section enables BIOMERO 2.x provenance and workflow tracking. See the Analytics and Provenance Settings section below for details.

See Eventsourcing and Views (Developer) for the event model and view table details.

Runtime Processed Data Folder

PROCESSED_DATA_FOLDER is a runtime environment variable consumed by the importer library, with no slurm-config.ini key. It defaults to .processed when unset. Set it on the importer container to change preprocessing output.

If BIOMERO_SHALLOW_ZARR=true and you use a custom processed folder, set the same PROCESSED_DATA_FOLDER value on biomeroworker: Image Transfer calls the importer library there to create new canonical Zarr copies. NL-BIOMERO’s processor forwards this name automatically from biomero.constants.slurm_env. Both libraries in the worker must include support for this setting. With shallow-Zarr disabled, no worker-side setting is needed.

For example, supply PROCESSED_DATA_FOLDER=.import in the relevant container environments, then recreate those containers. Adding a value only to Compose’s .env file does not pass it into a service unless its environment or env_file configuration includes it. Existing canonical paths remain stored in metadata and are not migrated; keep the original data paths accessible.

Minimal Working Example

[SSH]
host=slurm

[SLURM]
slurm_data_path=my-scratch/data
slurm_images_path=my-scratch/singularity_images/workflows
slurm_converters_path=my-scratch/singularity_images/converters
slurm_script_path=my-scratch/slurm-scripts

[WORKFLOWS]
cellpose=cellpose
cellpose_repo=https://github.com/TorecLuik/W_NucleiSegmentation-Cellpose/tree/v1.2.7
cellpose_job=jobs/cellpose.sh

Core Path Settings

These [SLURM] settings define where BIOMERO stores and looks for runtime data:

Core paths

Setting	Purpose	Runtime impact
slurm_data_path	Base directory for transferred input data and produced output data	Used to build DATA_PATH for workflow and conversion jobs
slurm_images_path	Base directory for workflow containers on the HPC	Used to build IMAGE_PATH and locate workflow .sif files
slurm_converters_path	Base directory for converter containers on the HPC	Used for conversion workflows and converter setup
slurm_script_path	Directory containing generated or cloned Slurm job scripts	Used when submitting workflow jobs and conversion jobs

Relative paths are typically interpreted relative to the remote user’s home directory. Absolute paths are also supported if that better matches your cluster.

Operational Settings

Container bind path

Use slurm_data_bind_path when your Apptainer or Singularity jobs need an explicit APPTAINER_BINDPATH so containers can see the transferred data.

Impact:

• If set, BIOMERO injects APPTAINER_BINDPATH into workflow and conversion job environments.

• If unset, BIOMERO relies on the cluster’s default container bind behaviour.

Use this when:

• your HPC administrator told you to set APPTAINER_BINDPATH

• containers cannot see files under slurm_data_path

• jobs fail with missing input or missing output path errors inside the container

Conversion partition

Use slurm_conversion_partition when conversion jobs need an explicit partition. The data conversion job is always submitted via sbatch, so it honours the same generic sbatch settings as workflow jobs.

Impact:

• If set, BIOMERO injects --partition=<value> as a real flag on the conversion sbatch command (and also exports CONVERSION_PARTITION to the conversion environment for custom scripts that read it).

• Precedence: slurm_conversion_partition wins over the generic slurm_default_partition fallback for conversion jobs.

• If unset, conversion falls back to slurm_default_partition (if configured), otherwise the cluster default partitioning behaviour.

Default (fallback) partition

Use slurm_default_partition on clusters that have no usable system default partition, so you do not have to hard-code --partition into every workflow’s [WORKFLOWS] job parameters. It applies to both workflow and conversion jobs.

Impact:

• If set, BIOMERO appends --partition=<value> to a workflow or conversion submission only when the job does not already carry a --partition directive.

• Precedence: for workflows, a per-workflow --partition in [WORKFLOWS] wins, and the GPU partition (from inject_gpu_flag or a per-workflow _use_gpu) wins. For conversion, slurm_conversion_partition wins. The default partition is a last-resort fallback in both cases.

• If unset (default), no --partition is injected and the cluster default is used, so existing deployments are unaffected.

Overridable via the BIOMERO_DEFAULT_PARTITION environment variable (precedence: code default None → slurm_default_partition in slurm-config.ini → BIOMERO_DEFAULT_PARTITION).

Sacct history window

The client uses sacct to list historical jobs. Two settings control how far back it looks:

• sacct_start_time: absolute date, format YYYY-MM-DD

• sacct_days_ago: relative rolling window in days

Effective precedence when BIOMERO chooses a default start time:

1. built-in fallback: 2023-01-01

2. resolved sacct_start_time

3. resolved sacct_days_ago

That means sacct_days_ago wins over sacct_start_time when both are set.

Environment-based overrides follow the same logic:

• BIOMERO_SACCT_START_TIME overrides the ini absolute date

• BIOMERO_SACCT_START_DAYS_AGO overrides both the ini rolling window and the absolute date when valid

• if BIOMERO_SACCT_START_DAYS_AGO is invalid, BIOMERO falls back to the resolved absolute date instead of failing hard

Processing settings

env_file_submission

Impact:

• When false: BIOMERO submits jobs by passing environment variables directly through the SSH/Fabric execution path.

• When true: BIOMERO writes a per-job shell file with exports and passes that file as $1 to the job script.

• Generated job scripts also receive a small sourcing block so they can read that env file.

Use this when:

• SSH session environment propagation into sbatch is unreliable on your cluster

• AcceptEnv or related SSH policy prevents the expected workflow parameters from arriving in jobs

inject_gpu_flag

This setting controls two related but distinct behaviours:

1. ``–nv`` in the generated script — when false, --nv is baked directly into the generated job script at script-generation time (singularity run --nv ...). The script always passes --nv to the container runtime regardless of what happens at submission time. When true, the generated script instead contains a shell variable reference $GPU_FLAG, and BIOMERO sets that variable to --nv or an empty string at submission time based on use_gpu.

2. Runtime toggling — when false, whether GPU sbatch resource params (--partition, --gres / --gpus) are added is decided at config time via <name>_use_gpu=true in [WORKFLOWS]. This cannot be changed per-run. When true, the caller can pass use_gpu=true or use_gpu=false at submission time to switch between GPU and CPU mode on a per-run basis.

In short: without inject_gpu_flag, GPU is either always on (hardcoded --nv) or never injected — the two are controlled independently at script generation and submission time. With inject_gpu_flag, one script covers both modes and the caller decides per run.

Impact:

• When false: --nv is hardcoded in the generated script. <name>_use_gpu=true in [WORKFLOWS] adds GPU sbatch resource params at submission, but the container flag is already fixed in the script. No runtime override is possible.

• When true: the script contains $GPU_FLAG. BIOMERO sets it to --nv (GPU) or empty string (CPU) at submission time based on the resolved use_gpu value.

Use this when:

• you want one generated job script to run in both CPU and GPU modes without maintaining two separate scripts

• you need to submit the same workflow to GPU or CPU resources depending on the dataset or queue availability

• your generated scripts use the standard singularity run $GPU_FLAG or apptainer run $GPU_FLAG pattern

Image pull execution mode

slurm_image_pull_via_sbatch

Impact:

• When false: BIOMERO starts image pulls/builds via the existing remote shell path.

• When true: BIOMERO submits one combined workflow-and-converter Slurm job array with one task per missing or invalid image.

• Valid existing versioned SIFs are inspected and omitted on rerun.

Use this when:

• container pulls/builds are too heavy for the login node

• your cluster requires build work to run as scheduled jobs

image_pull_cpus, image_pull_mem, image_pull_time, image_pull_concurrency, and image_pull_partition

Impact:

• Control resources and the %N concurrency throttle when slurm_image_pull_via_sbatch=true.

• Global sbatch_* parameters, including sbatch_time, apply to the array. Dedicated image_pull_* resource values override matching global flags.

• Blank image_pull_cpus and image_pull_mem values inherit sbatch_cpus-per-task and sbatch_mem respectively. When neither level is configured, Slurm uses its scheduler defaults; SquashFS falls back to one worker if the allocation does not expose SLURM_CPUS_PER_TASK.

• Every submission writes its status files and job logs to its own folder under <slurm_script_path>/image-pulls. After submitting, BIOMERO removes the folders of earlier submissions that are older than seven days (never the current one); if that cleanup fails it only logs a warning.

• The compatibility default concurrency is 1; 2 to 4 is recommended to limit metadata and I/O pressure on shared storage.

• Each compute task auto-detects apptainer first and then singularity. The selected runtime resolves the OCI manifest as the first phase of its native build, so no separate registry utility is required.

• Permanent manifest, authentication, and reference errors fail immediately; classified transient registry and network errors are retried.

• Each task writes its own pull-image-%A_%a.log and structured status.

Apptainer temp and cache directories

Use apptainer_tmpdir and apptainer_cachedir when the default temporary or cache location is too small for large image pulls/builds.

Impact:

• Array tasks prefer SLURM_TMPDIR for OCI extraction, SquashFS creation, and a task-private cache. Configured paths are fallbacks when it is absent.

• Temporary data and partial shared SIFs are removed after success or failure.

• A completed SIF is published only after local and shared-copy inspection, using an atomic rename in the destination directory.

Default GPU fallback settings

gpu_partition, gpu_gres, and gpu_gpus are shared client defaults used when adding GPU sbatch params to a workflow submission. They are relevant in both GPU code paths:

• Static path (inject_gpu_flag=false, <name>_use_gpu=true in [WORKFLOWS]): BIOMERO always adds these params for that workflow. The caller cannot override it at submission time.

• Dynamic path (inject_gpu_flag=true): BIOMERO adds these params only when use_gpu resolves to true — either because the caller passed use_gpu=true, or because <name>_use_gpu=true is set in [WORKFLOWS] and no explicit use_gpu argument was given.

gpu_gres and gpu_gpus are mutually exclusive. Set one or the other:

• gpu_gres — BIOMERO appends --gres=<value> to GPU workflow submissions. Use this when your cluster schedules GPUs via Generic RESources (--gres).

• gpu_gpus — BIOMERO appends --gpus=<value> to GPU workflow submissions. Use this when your cluster uses the --gpus flag (common on newer Slurm versions).

Setting both raises a ValueError at startup.

Impact:

• If a workflow already has per-workflow sbatch overrides such as cellpose_job_partition, cellpose_job_gres, or cellpose_job_gpus, those per-workflow settings take precedence over these shared defaults.

• If no per-workflow --partition, --gres, or --gpus is already present, BIOMERO appends these defaults to the submission command.

Full precedence order for GPU resource params:

1. explicit per-workflow [WORKFLOWS] sbatch overrides (e.g. cellpose_job_gres)

2. shared gpu_partition and gpu_gres / gpu_gpus defaults

3. nothing — BIOMERO adds no GPU resource arguments

Global sbatch parameters

Any [SLURM] key that starts with sbatch_ is treated as a global sbatch parameter that BIOMERO adds to workflow, conversion, and scheduler-native image pull submissions.

The key pattern is sbatch_<flag>=<value>, which produces --<flag>=<value> on the sbatch command line.

Examples:

[SLURM]
sbatch_reservation=biomero
sbatch_nice=1

This appends --reservation=biomero and --nice=1 to workflow, conversion, and scheduled image initialization jobs.

Impact:

• Global params are applied after per-workflow [WORKFLOWS] sbatch overrides.

• If a per-workflow override already sets the same flag (e.g. cellpose_job_reservation), the global default for that flag is skipped.

• For conversion jobs, a --partition set via slurm_conversion_partition (or the slurm_default_partition fallback) takes precedence over a global sbatch_partition.

• For image arrays, dedicated image_pull_cpus, image_pull_mem, image_pull_time, and image_pull_partition values take precedence over matching global flags. Blank dedicated values inherit their matching global flag; if that is also unset, the scheduler default applies. The slurm_default_partition applies to image arrays as it does to workflow jobs, ahead of a global sbatch_partition. Image arrays have no built-in CPU or memory request: BIOMERO does not guess a size for your cluster, and logs the resources it submits at INFO.

• Global params with empty values are ignored.

• There is no environment variable override for these — they are intentionally admin-only at config time.

Use this when:

• you want to apply a cluster-specific constraint (e.g. a reservation or QOS) to all workflow jobs

• you need a cluster-wide --nice or --account that applies unless overridden per workflow

ZIP command

slurm_zip_cmd controls which archive command BIOMERO uses when it creates and extracts ZIP files on the cluster. It can also be overridden by BIOMERO_SLURM_ZIP_CMD.

Impact:

• If unset, BIOMERO defaults to $(command -v 7z || command -v 7za).

• 7z and 7za use the existing 7-Zip command syntax.

• zip uses Info-ZIP syntax for creation and unzip for extraction. An explicit path ending in /zip, such as /usr/bin/zip, uses the matching sibling path (/usr/bin/unzip) for extraction.

• Other configured values retain the legacy 7-Zip syntax for backward compatibility.

• Info-ZIP extraction with a file filter also needs mktemp and a find that supports -cnewer and -print -quit (GNU findutils) on the cluster: it uses them to tell a partial filter match from an empty extraction.

Use this when:

• your cluster exposes only 7za or only 7z

• your cluster provides the Info-ZIP zip and unzip commands instead of 7-Zip; set slurm_zip_cmd=zip

• auto-detection is not reliable in your environment

Lock command

slurm_lock_cmd is the command BIOMERO uses to lock a file on the cluster before it runs a script that must not run twice at once: submitting a job, extracting an input archive, creating a result archive and updating the job scripts. It can also be overridden by BIOMERO_SLURM_LOCK_CMD. See Recovering Interrupted SSH Operations for why these operations are locked.

It is a template. BIOMERO fills in {wait_seconds} (how long to wait for the lock), {busy_exit_code} (the exit code that means “locked by someone else”) and {lock_file} (the quoted path of the lock file) and appends the script. {lock_file} and {busy_exit_code} are required, because the exit code is how BIOMERO knows the lock was busy; {wait_seconds} is optional, for tools without a wait option.

Impact:

• If unset or empty, BIOMERO uses flock -w {wait_seconds} -E {busy_exit_code} {lock_file}, which needs flock from util-linux 2.25 or newer.

• Any command that takes the lock, runs the script given as its remaining arguments, and exits with the busy exit code when the lock is not obtained in time can replace flock (for example a small wrapper script around the locking tool of your cluster).

• BIOMERO checks during Slurm setup (SlurmClient.setup_slurm) that the lock command and sacct exist on the cluster, and logs a WARNING for each that is missing. The setup continues.

• A template with an unknown placeholder, or without {lock_file} or {busy_exit_code}, is rejected when the client is created.

• If the lock command is missing when an operation runs (shell exit code 127), the operation stops with an error that names slurm_lock_cmd.

Analytics and Provenance Settings

The [ANALYTICS] section in slurm-config.ini controls BIOMERO 2.x provenance and workflow tracking. When enabled, BIOMERO records every workflow and job event into a PostgreSQL event store (via eventsourcing_sqlalchemy). Lightweight view tables are then derived from those events and exposed to Metabase dashboards in OMERO.biomero — showing live workflow status, per-user job accounting (who ran what, useful for Slurm resource accounting), timing, and failures across the full OMERO → Slurm → OMERO lifecycle.

This is on by default since BIOMERO 2.x. The individual listeners can be enabled or disabled independently. Turn off track_workflows=False only if you are running a basic 1.x-style deployment without a PostgreSQL analytics database, in which case none of the Metabase dashboard views will be populated.

Set the database connection with sqlalchemy_url in [ANALYTICS]. The SQLALCHEMY_URL environment variable has higher precedence and overrides the ini value when set.

See Eventsourcing and Views (Developer) for the event model and view table details.

Analytics view table rebuild window

Warning

This is an advanced opt-in feature. Enabling it means jobs outside the cutoff window will not appear in analytics views. Only use it when full rebuilds are genuinely too slow for your installation.

When the SLURM Init script resets analytics view tables, BIOMERO replays the full event history to rebuild the views. On large installations with thousands of events this can take several minutes. By default, BIOMERO always does a full rebuild.

Two settings allow capping how far back the replay goes. They are configured under [ANALYTICS], not [SLURM]:

• analytics_rebuild_start_time: absolute cutoff date, format YYYY-MM-DD

• analytics_rebuild_days_ago: rolling window in days

Effective precedence:

1. built-in default: full rebuild from event ID 1

2. resolved analytics_rebuild_start_time

3. resolved analytics_rebuild_days_ago (overrides absolute date when both are set)

4. values set via environment variables (highest priority)

Environment variable overrides:

• BIOMERO_ANALYTICS_REBUILD_START_TIME

• BIOMERO_ANALYTICS_REBUILD_DAYS_AGO (takes highest priority; falls back gracefully on invalid input)

These settings can also be overridden for a single run via the Rebuild From Days Ago and Rebuild From Date inputs on the SLURM Init OMERO script.

Note

These settings only affect view table rebuilds during reset_tables=True init runs. They do not affect sacct history queries or what jobs appear in the Monitor. View tables built from a partial event history will not include older jobs.

Workflow And Model Configuration

For most installations, we recommend leaving slurm_script_repo empty and using BIOMERO’s generated job scripts.

Why this is the safer default:

• generated scripts are updated together with BIOMERO and can be redeployed via init_slurm=True or script regeneration

• we do not maintain a separate always-up-to-date public scripts repository for operators to track

• an external scripts repository can drift from the BIOMERO version you are running and break newer features or assumptions between minor releases

• adding or evolving workflows is usually simpler through descriptor-driven generation than through maintaining a custom scripts repository

Use slurm_script_repo only when you explicitly need custom hand-maintained job scripts and are prepared to keep them aligned with the BIOMERO version in production.

The [WORKFLOWS] section (also accepted as [MODELS] for backward compatibility with existing configurations) defines each workflow and optionally adds per-workflow sbatch overrides. When both sections are present in the same file their entries are merged; [WORKFLOWS] entries take precedence on key collisions.

Example:

[WORKFLOWS]
cellpose=cellpose
cellpose_repo=https://github.com/TorecLuik/W_NucleiSegmentation-Cellpose/tree/v1.2.7
cellpose_job=jobs/cellpose.sh
cellpose_job_gres=gpu:1g.10gb:1
cellpose_job_partition=gpu
cellpose_job_mem=16GB
cellpose_use_gpu=true

How BIOMERO interprets these keys:

• cellpose defines the subdirectory under slurm_images_path

• cellpose_repo points to the workflow repository and descriptor metadata

• cellpose_job points to the job script inside slurm_script_path or the cloned scripts repository

• any cellpose_job_<name>=<value> entry is translated to `` –<name>=<value>`` on the sbatch command line

• cellpose_use_gpu=true marks this workflow as GPU-enabled by default, so BIOMERO activates GPU handling even without an explicit use_gpu argument at submission time

Direct descriptor URL (optional): <key>_repo may point directly at a specific descriptor file inside the repository instead of the repository root. BIOMERO will fetch that exact file rather than auto-discovering one:

[WORKFLOWS]
# Explicit bilayers config.yaml (repo also has descriptor.json — pick one)
mito_bilayers=mito_bilayers
mito_bilayers_repo=https://github.com/org/W_MitoSeg/tree/v0.0.3/config.yaml
mito_bilayers_job=jobs/mito_bilayers.sh

# Same repository but using the BIAflows interface
mito_biaflows=mito_biaflows
mito_biaflows_repo=https://github.com/org/W_MitoSeg/tree/v0.0.3/descriptor.json
mito_biaflows_job=jobs/mito_biaflows.sh

This lets you register the same container under two workflow names with different parameter interfaces. Omitting the file suffix (plain /tree/vX.Y.Z) keeps the current auto-discovery behaviour (descriptor.json → descriptor.yaml → config.yaml).

Example slurm-config.ini Notes

The example file in resources/slurm-config.ini documents the newer client-level options introduced, including:

• sacct_start_time

• sacct_days_ago

• env_file_submission

• inject_gpu_flag

• gpu_partition

• gpu_gres

• gpu_gpus

• sbatch_<key> (global sbatch params pattern)

• slurm_image_pull_via_sbatch

• image_pull_cpus

• image_pull_mem

• image_pull_time

• image_pull_concurrency

• image_pull_partition

• apptainer_tmpdir

• apptainer_cachedir

• slurm_zip_cmd

• analytics_rebuild_start_time

• analytics_rebuild_days_ago

Troubleshooting

Common Issues and Solutions

Container Access Errors

Problem: Workflows fail with “file not found” or permission errors.

Solutions:

1. Check bind paths: Configure slurm_data_bind_path if required.

2. Verify permissions: Ensure SLURM user can access data directories.

3. Check container binding: Verify Singularity or Apptainer can access the required paths.

# Add explicit binding if needed
slurm_data_bind_path=/data/your-scratch/data

SSH Connection Issues

Problem: Cannot connect to the SLURM cluster.

Solutions:

1. SSH config: Verify SSH configuration for the BIOMERO SSH host alias.

2. Authentication: Check SSH keys and authentication methods.

3. Network: Confirm network connectivity to the cluster.

# Test SSH connection manually
ssh your-slurm-host

# Check SSH config
ssh -F ~/.ssh/config your-slurm-host

Job Submission Failures

Problem: Jobs fail to submit or execute.

Solutions:

1. Partition access: Check whether the specified partition is available.

2. Resource limits: Verify memory, CPU, and GPU requests are within limits.

3. Queue policies: Check SLURM queue policies and restrictions.

# Use appropriate partition
cellpose_job_partition=gpu-partition

# Adjust resource requests
cellpose_job_mem=8GB
cellpose_job_gres=gpu:1

Path Configuration Issues

Problem: Containers or scripts not found.

Solutions:

1. Absolute vs relative paths: Use the path format that matches your cluster setup.

2. Directory existence: Verify directories exist on the SLURM cluster.

3. Path permissions: Check read and write permissions.

# Relative to home directory
slurm_data_path=my-scratch/data

# Or absolute path
slurm_data_path=/data/users/username/my-scratch/data

Jobs do not see workflow parameters

Problem: Jobs start, but workflow parameters or paths appear empty inside the job environment.

What to check:

1. Enable env_file_submission=true if the cluster does not reliably propagate SSH session environment variables into sbatch jobs.

2. Verify that the generated or installed job script accepts the env-file argument as $1 when env-file submission is enabled.

3. Review cluster SSH policy if AcceptEnv or similar controls are known to be restrictive.

Containers cannot access transferred data

Problem: The BIOMERO transfer succeeds, but the container still cannot see the files.

What to check:

1. Set slurm_data_bind_path if the container runtime requires an explicit bind path.

2. Confirm that the configured bind path includes the effective slurm_data_path location.

3. Compare the cluster’s default bind policy with the paths used by BIOMERO.

GPU jobs land on the wrong resources

Problem: GPU jobs run on the wrong partition or with the wrong GRES settings.

What to check:

1. per-workflow *_job_partition and *_job_gres values in [WORKFLOWS]

2. shared gpu_partition and gpu_gres values

3. whether inject_gpu_flag is enabled

4. whether the workflow was actually submitted with use_gpu=true

Historical job listing returns too much or too little data

Problem: Historical job lookups return too much or too little data.

What to check:

1. Adjust sacct_start_time if you want a fixed absolute boundary.

2. Prefer sacct_days_ago if you want a rolling window.

3. Use the corresponding environment variables only when you need runtime-specific overrides.

FAQ

Q: Should I use relative or absolute paths?

A: Use relative paths if your SLURM setup expects paths relative to the user home directory. Use absolute paths if you need to specify exact filesystem locations.

Q: When do I need to set ``slurm_data_bind_path``?

A: Set this when your HPC administrator tells you to configure APPTAINER_BINDPATH, or when containers cannot access your data directories.

Q: How do I know which partition to use?

A: Check with your HPC documentation or administrator. Common partitions include cpu, gpu, short, and long. Leave the setting empty to use the default.

Q: Can I override job parameters for specific workflows?

A: Yes. Add workflowname_job_parameter=value entries in the [WORKFLOWS] section to override default SLURM job parameters.

Q: How do I debug workflow execution issues?

A: Check SLURM job logs, verify container access to data, and ensure all required directories exist with proper permissions.

Q: When should I enable ``env_file_submission``?

A: Enable it when workflow jobs start correctly but do not receive the expected environment variables, especially on clusters where SSH environment forwarding into sbatch jobs is limited or disabled.

Q: Why do ``gpu_partition`` and ``gpu_gres`` seem to do nothing?

A: There are two conditions that trigger GPU sbatch param injection:

• inject_gpu_flag=true and the workflow is submitted with use_gpu=true (or <name>_use_gpu=true in [WORKFLOWS] with no explicit use_gpu argument)

• inject_gpu_flag=false but <name>_use_gpu=true is set in [WORKFLOWS] — the static path

In both cases, per-workflow *_job_partition, *_job_gres, and *_job_gpus settings in [WORKFLOWS] take precedence and will suppress the shared defaults for those flags.

Q: Should I prefer ``sacct_start_time`` or ``sacct_days_ago``?

A: Use sacct_start_time for a fixed absolute boundary and sacct_days_ago for a rolling window. If both are set, sacct_days_ago takes precedence.

Further Reading

• Configuration Reference

• SLURM Documentation

• Apptainer Documentation

• Singularity User Guide

Optional result shallower

The administrator-only CPU helper runs before result ZIP creation. All options are in [SLURM]; environment values override ini values. Remote shallowing is off unless you set remote_shallow_zarr=true (or BIOMERO_REMOTE_SHALLOW_ZARR=true); a missing or empty value keeps the local importer shallowing. Shallow Zarr itself must be enabled separately. The helper requests no GPU; workers set CPUs per task. Shallowing and recovery share conversion’s resource merging: the helper partition overrides slurm_default_partition, which overrides global sbatch_partition. Otherwise Slurm chooses its default. Global sbatch_* options (including account, reservation, QoS and constraint) are inherited, except GPU resources, multi-task/node allocation and flags owned by the helper (array, command, export and log paths). Optional helper memory and time limits override their global counterparts; otherwise global values apply. Image pulls use the normal image-pull resource configuration.

The import script owns session keepalive and supplies a heartbeat callback to the shared SlurmJob monitor for shallowing and recovery. Callback failures propagate unchanged. This applies to inline and detached workflows alike. An existing job ID is adopted without resubmission; unavailable status pauses retrieval rather than triggering recovery. The helper task is completed only after its output report has been validated.

Only SLURM_Init_environment acquires the shallower image. Verify setup with SLURM_check_setup before running workflows. In that script integration, acquisition requires IMPORTER_ENABLED=true, BIOMERO_SHALLOW_ZARR=true and the remote preference enabled. Runtime shallowing validates the installed image and raises a setup error if it is missing or invalid; it never downloads an image or silently falls back because setup is incomplete.

Shallowing and recovery have distinct submission identities. Recorded job IDs remain authoritative when resuming older tasks; an unresolved pre-upgrade recovery intent may require administrator reconciliation, rather than automatic resubmission.

Canonical input manifests are published atomically and verified on resume, never overwritten. Missing manifests after submission or changed inputs stop retrieval. New tasks record an input fingerprint and the helper SIF path; existing tasks retain their recorded image and tool version for recovery and receipt validation even when deployment defaults change. Keep the original SIF available until those tasks finish. Resource settings remain configurable.

Result shallower configuration

Ini option	Default	Environment variable
remote_shallow_zarr	false (shallow Zarr must be enabled separately)	BIOMERO_REMOTE_SHALLOW_ZARR
remote_shallower_image	cellularimagingcf/biomero-shallower:latest; prefer an ini pin	BIOMERO_REMOTE_SHALLOWER_IMAGE
remote_shallower_version	Unset; read the installed image’s OCI version label	BIOMERO_REMOTE_SHALLOWER_VERSION
remote_shallower_workers	1	BIOMERO_REMOTE_SHALLOWER_WORKERS
remote_shallower_partition	Unset (inherit generic partition)	BIOMERO_REMOTE_SHALLOWER_PARTITION
remote_shallower_mem	Unset (inherit global memory)	BIOMERO_REMOTE_SHALLOWER_MEM
remote_shallower_time	Unset (inherit global time limit)	BIOMERO_REMOTE_SHALLOWER_TIME

Prefer a versioned tag or immutable digest and matching importer/schema/shallower versions. Importer enablement, existing shallow capability, and workflow tracking are required. Safe failures fall back to full transfer and local import; unresolved recovery preserves results and pauses retrieval. These options are administrator settings, not scientific workflow parameters. A compatible OMERO.biomero admin interface exposes them when shallow storage is enabled; environment overrides still take precedence over values saved to the ini file.

Core falls back to our helper’s :latest tag. Prefer an image pin in [SLURM]; refer to resources/slurm-config.ini for the maintained release selection. When remote_shallower_version is unset, core reads the installed SIF’s OCI version label and records it before submitting a new task. An explicit value overrides discovery and must match the Python package version written into receipts, including any normalized prerelease suffix. Existing tasks retain their recorded version for recovery.

An explicit :latest image tag is accepted, as with configured converters, but is not a reproducible release pin. Initialization reuses an already valid SIF; it does not refresh that file merely because the registry tag moved. Keep any explicit expected tool version consistent with the installed image. Initialization acquires the configured or default image. Missing version labels require an explicit tool-version setting; receipt validation is not disabled. Importer trust configuration must still match the selected image and actual tool version.

For recovery behavior and component responsibilities, see Execution, storage and recovery.

Recovering Interrupted SSH Operations

Set ssh_recovery_timeout in [SLURM] to a nonnegative recovery window in seconds. The default is 900 (15 minutes), including when the setting is absent or empty. Recovery applies to inline and detached workflows. The window only sets how long an interrupted operation is retried: the same code runs for every value, and 0 means no retries, so an interrupted operation fails with its own error. The environment override is BIOMERO_SSH_RECOVERY_TIMEOUT. This runtime setting applies to new clients; Slurm Init is not needed just to change the window. (It lives in [SLURM] because it also governs the job submission, extraction and archive operations on the cluster, not only the connection.)

The window starts at the first failed attempt of an operation and nothing else moves it: not a successful nested step, and not a later retry by an outer operation. A repeating failure therefore ends at the window instead of retrying forever. Healthy queueing, computation and helper runtime do not consume it, and each new operation (the next status poll, the next transfer) gets a new window. Caller-supplied heartbeat callbacks remain active during nested recovery and backoff, including shallower polling. Retries wait 1, 2, 4 … up to 30 seconds, with a little random jitter so that clients do not retry in lock step.

Detection precedes that window. Within a recovery operation, native connection stages and channel opening use at most 30 seconds, or the configured recovery window when smaller; stricter caller settings remain effective. Short read-only commands (status queries, progress and result-location reads) use the same default detection limit. The 30-second limit follows the existing banner, authentication and SFTP-read bounds, rather than Paramiko’s hour-long channel default. A positive explicit timeout on run_read_only or run_commands_read_only permits a longer legitimate read. Once an outage is active, native timeouts and read timeouts are capped by its remaining budget; an expired budget never starts another attempt. Temporary native settings are restored afterward, including on a reconnected transport.

Heartbeat and cancellation callbacks run on the caller thread before attempts and during backoff, and regain control when a bounded call returns. They do not run concurrently inside a blocking dependency call. Detection can therefore add time before recovery starts, and callbacks can wait for the current native timeout. Native bounds apply per connection stage, not to DNS or the whole call. Paramiko’s exec/subsystem acknowledgement waits are not covered by these native settings. They require a lower-layer fix; this is not a universal SSH deadline. Healthy long durable commands and complete transfers have no new runtime cap; SFTP bounds individual reads instead. Arbitrary commands retain their original timeout and single-attempt behavior. With 0, no native timeouts or keepalive are set and nothing is retried; a busy lock or an unresolved submission still stops with the pending error.

With recovery enabled, BIOMERO closes a stale SSH transport and retries explicit read-only operations with exponential backoff, capped at 30 seconds. The activity log reports Waiting for SSH connection while the last known Slurm job state is retained. Authentication, host-key, permission, configuration and remote-command failures are not treated as transient transport loss. Cancellation and deadline expiry stop recovery with an explicit connection diagnostic, leaving remote jobs and recovery records intact. Exhaustion remains terminal Failed in both inline and detached execution; retaining records does not queue automatic recovery later.

A channel that closes without an exit status (Fabric exit code -1) is a transport failure, including when Fabric is configured with warn=True. Real remote nonzero exit codes retain their usual failure behavior. Temporary lock contention waits for the existing durable operation; it does not authorize another submission or classify a rejected Slurm job as a connection problem. While the cluster still runs the operation (a long extraction or archive) or a submitted job is not yet visible in Slurm, the connection is healthy: BIOMERO waits within its own window and then stops with a message that says the remote operation is still pending and that nothing was repeated, rather than reporting an SSH outage. Ask again later; the operation continues on the cluster.

Workflow, conversion and shallower-helper submissions all use one submit-exactly-once routine, SlurmClient.submit_job_once. It works with the tracked task identity, receipt files in <slurm_data_path>/.biomero-recovery and the Slurm job name biomero-<task id>. The receipt files are a lock file, a hash of the command, the intent (written before sbatch runs), the job ID and, if sbatch refuses the job, the refusal. After a lost response, BIOMERO adopts the recorded job or finds its job by exact name through squeue and sacct. Unresolved submission intent never authorizes a second sbatch. Once the job ID is stored on the task, the receipt files are removed. A refusal by sbatch is replayed to the caller; correcting the submission command replaces it. A command that differs from the one recorded for a possibly submitted job is rejected.

A task identifies its own submission, and a run is continued only while its task is unfinished. After a crash, BIOMERO continues the task while its job is running, and also while its job COMPLETED but the outputs were not processed yet, so the analysis does not run a second time. It never re-attaches to a job that failed, was cancelled or timed out, nor to a task that was completed or whose submission failed: the next run of that workflow step gets a new task with its own job name and receipt files. A new workflow run always has new tasks.

If a submission never resolves, the error names the intent file (<slurm_data_path>/.biomero-recovery/<task id>.intent), the time the submission started, and what to check. Look in sacct and squeue for a job named biomero-<task id> (for the shallower helper: biomero-shallower-<task id> or biomero-recovery-<task id>). If one exists, wait for accounting to show it. Only if no such job exists, delete all receipt files with that prefix (<task id>.*, or shallow.* / recovery.* in the helper’s .biomero-shallower/<task id> directory) to allow a fresh submission. If sbatch refused a job, the answer is replayed with a note; deleting the same receipt files submits that task again, and a corrected command replaces the recorded refusal by itself. Receipts do not expire by age.

Tracking must remain enabled for resumable submission; arbitrary run and run_commands calls are deliberately not retried. Untracked submissions are attempted once because they lack a durable task identity.

Result archive creation is serialized and published through a temporary path. Completed archives are reused on retry, unless a result file changed after the archive was made. An input archive that was already extracted is extracted again only when it was uploaded again after the extraction. Downloads and input archives are copied to partial paths, checked for completeness and published atomically. A retry restarts an interrupted copy from the beginning, rather than requiring rsync or SSHFS. Existing local final files remain available until replacement succeeds. Normal ZIP extraction still validates member CRCs before import. The first healthy copy does not consume a recovery window. Once interrupted, whole-file retries share the window until the complete copy succeeds; repeated interruptions during lengthy retries can exhaust it. There is no ranged or progress-aware transfer resume.

Native connection timeouts and keepalive are set only when the window is above 0, so installs that set ssh_recovery_timeout=0 keep their own timeouts for slow login or jump hosts.

Recovery requires the lock command (slurm_lock_cmd, flock by default), Slurm accounting (sacct) and job-name lookup, writable shared HPC storage for submission and archive records, and atomic SFTP rename support for input and generated-script publication. setup_slurm checks that the lock command and sacct exist and logs a WARNING if not. Keep these records and result files while recovery is pending. Do not delete an unresolved intent to force a retry; reconcile the job first. Marker files of finished runs (<name>.zip.biomero-*) are removed with the run’s other files by the normal cleanup. A processor restart can reuse the recorded task and job identity, but recovery does not make arbitrary remote commands or OMERO postprocessing exactly-once. Use matching core and scripts for the complete pipeline.