📋 Documentation Version Notice

You're reading the documentation for a development version (master). For the latest released version, please have a look at v1.7.

Optional remote Zarr shallowing

BIOMERO can shallow eligible workflow OME-Zarr output on Slurm before ZIP creation and transfer. The helper retains new/changed labels and references verified canonical pixels already available on the OMERO side. The importer validates the helper receipt and registers the result without repeating pixel hashing or shallowing. Other workflow files follow the existing import path.

The NL-BIOMERO demo enables shallow Zarr. Other deployments opt in with BIOMERO_SHALLOW_ZARR=true. Remote processing is then the default; set BIOMERO_REMOTE_SHALLOW_ZARR=false for local importer processing instead. Without shallow Zarr enabled, the remote setting has no effect.

The demonstration stack shares web/slurm-config.ini between OMERO.biomero and the workflow worker. SLURM_CONFIG_HOST_PATH selects this canonical configuration in the root Compose files; deployment-scenario files mount the same file. The INIs under biomeroworker/ are alternative examples, not additional active configurations. Shallow storage and detached execution are enabled through the deployment environment, not through INI feature flags.

Lifecycle overview

Only derived workflow results are shallowed. The managed source remains unchanged and must remain available. This diagram shows the Zarr workflow path; non-Zarr results continue through their usual import path.

        flowchart TD
    A["Existing managed Zarr"] --> C["Full source Zarr<br/>Retained unchanged"]
    B["Non-Zarr input"] -->|"Create reusable Zarr"| C
    C --> I["Record or reuse source<br/>ISCC-BIO pixel identities"]
    I --> T["Transfer full Zarr to HPC"]
    T --> H["Workflow produces images and labels"]
    H --> Q{"Where is eligibility checked?"}
    Q -->|"Remote"| R["On HPC, before ZIP and transfer"]
    Q -->|"Local"| L["In importer, after full transfer"]
    R --> V{"Pixels match source<br/>and result is eligible?"}
    L --> V
    I -. "Recorded identities" .-> V
    V -->|"Changed or uncertain"| F["Keep full result"]
    V -->|"Verified unchanged"| S["Shallow result<br/>Remove duplicate result arrays<br/>Keep new labels and managed references"]
    F --> O["Register results in OMERO"]
    S --> O
    C -. "PixelBuffer reads source pixels" .-> O
    S -. "Label views read retained labels" .-> O
    S --> M["Reconstruct on demand"]
    C --> M
    M --> Z["Full, self-contained OME-Zarr<br/>Source pixels and result labels"]
    Z -->|"Next Zarr workflow"| T
    

In remote mode, transfer follows the eligibility decision: eligible results travel shallow; other results travel in full. New or changed labels are retained; unchanged inherited labels may also be referenced instead of duplicated.

A shallow result is a BIOMERO-managed representation, not a self-contained OME-Zarr for generic readers. OMERO’s registered pixel paths reference the managed source or retained labels; OMERO does not invent missing pixels. Image Transfer reconstructs a full Zarr for subsequent Zarr workflows. For a standalone copy on disk, see Reconstruct a standalone Zarr on disk.

Choosing local or remote shallowing

Illustrative storage and processing trade-offs

Mode	Result disk space	Time and compute cost
Full results	No deduplication savings	No shallowing work; full results transferred and stored.
Local shallow Zarr	Approximately 90% saved in measured examples	Extra importer work: 63 minutes for the 846-image Plate.
Remote shallow Zarr	Preserves shallow-storage savings	18-image comparison: 57% less time in measured return stages and 88% fewer transfer bytes; extra HPC CPU job (25 seconds).

Results depend on data, storage and cluster queues; HPC charges may apply. The full-Plate remote speedup has not yet been measured. See Experimental BIOMERO shallow OME-Zarr storage for the complete timings and limitations.

Requirements and enablement

Install compatible BIOMERO core, scripts, importer, shallower and schema packages. The Python package installation does not acquire the helper image on Slurm. Supply a helper image in a registry accessible to the cluster; a pinned tag or immutable @sha256:... reference is recommended.

Set these deployment environment values:

IMPORTER_ENABLED=true
BIOMERO_SHALLOW_ZARR=true
BIOMERO_REMOTE_SHALLOW_ZARR=true

Configure helper runtime settings under [SLURM] in the shared web/slurm-config.ini or through OMERO.biomero’s admin settings. Do not supply worker runtime overrides in Compose: they take precedence over saved INI values.

Prefer a pinned remote_shallower_image under [SLURM] in the worker’s slurm-config.ini. The fallback is cellularimagingcf/biomero-shallower:latest; the BIOMERO core sample resources/slurm-config.ini contains a maintained release selection. When remote_shallower_version is unset, core reads the installed image’s OCI tool-version label before submitting a new helper task. An explicit value must match the version written into receipts, including any normalized prerelease suffix. Existing tasks retain their recorded version for recovery.

Set BIOMERO_REMOTE_SHALLOWER_IMAGE and BIOMERO_REMOTE_SHALLOWER_VERSION on the importer to the same selected values for receipt validation. The importer has its own settings and does not read the Slurm INI. The demonstration supplies these trust values only to the importer, not to web or the workflow worker. Keep them aligned when changing the helper image through the admin interface.

The worker loads helper runtime settings from the shared Slurm INI. Compose supplies matching trust settings to the importer. An empty helper partition inherits the generic configured partition, then the scheduler default; administrators can choose their CPU partition explicitly. The helper requests no GPU. It uses its worker count as CPUs per task and inherits global memory, time, account, reservation, and QoS settings. Image acquisition uses the established image-pull resource settings. Run SLURM_Init_environment before running analyses and verify image availability with SLURM_check_setup. Runtime shallowing never downloads images. A missing or invalid image stops result retrieval with a setup error identifying the required image; initialize and verify it before retrying.

Equivalent [SLURM] options are remote_shallow_zarr, remote_shallower_image, remote_shallower_version, remote_shallower_workers, and remote_shallower_partition. Environment values override ini values when explicitly supplied. The demonstration keeps runtime settings in the INI and supplies only feature flags to the worker through the deployment environment.

Admin settings

When shallow Zarr is enabled, OMERO.biomero’s admin settings show a Shallow Zarr section. The demonstration configuration enables remote shallowing; switching it off hides the helper fields without removing their saved values. When enabled, the section provides image, tool version, worker count, partition, memory and time settings. These fields save the worker’s Slurm configuration, not the importer’s separate settings. Keep the importer trust settings aligned. Run Slurm Init after changing the helper image.

Failure and observability

Unsupported results use full transfer and local importer handling. A failed shallowing job runs a recovery job: interrupted stores roll back; completed shallow stores retain their verified receipts. Retrieval pauses if submission state or rollback cannot be resolved safely. Results and journals remain available for recovery. keep-full is the only failure policy.

Inspect the shallower task in workflow provenance, its Slurm job IDs, the .biomero-shallower/<task-id>/canonical.json.<job-id>.log files beside the workflow data, and the .biomero-shallow-report.json inside shallow stores. Image-pull status remains under <slurm_script_path>/image-pulls. Helper task events do not replace the main analysis progress.

After a detached restart BIOMERO adopts submitted jobs or completed reports. For an unresolved submission intent, reconcile the unique biomero-shallower-<task-id> job in Slurm accounting and retry retrieval. Do not remove prune journals; they can contain arrays needed for rollback. Keep the original image configured until outstanding recovery is complete.

The helper uses the trusted BIOMERO Slurm account, image, event store, and import-order writers. Canonical managed identifiers are resolved through the importer’s existing authoritative local group mappings. The canonical store is not mounted into the helper. ZIP remains the archive format.