Skip to content

Architecture, transactions, and recovery

biomero-schema owns only portable contracts. biomero-shallower owns NGFF discovery, ISCC-BIO filesystem identity and transactional shallowing. biomero-importer.utils.result_zarr exports the shared implementation for existing callers; its identity subclass keeps the OMERO reader adapter in the importer. Registration planning and managed-source resolution retain their established semantics.

biomero acquires a versioned SIF through its image runner and starts a CPU job. biomero-scripts selects the stage only when shallow storage and compatible importer capabilities are enabled. Within that enabled feature, remote shallowing is preferred unless the administrator selects the local path with BIOMERO_REMOTE_SHALLOW_ZARR=false. It runs before ZIP creation. The import order carries receipts read from completed event-sourced shallower tasks, rather than trusting a batch file found in a workflow archive.

Transaction

  1. Acquire a kernel file lock for the returned store; recover any earlier intent.
  2. Discover nodes and generate exact image and label identities using the existing ISCC-BIO/Imagewalk adapter and semantic guards.
  3. Save original image attributes and every planned move in a durable sibling .STORE.biomero-prune-TOKEN/journal.json before moving arrays.
  4. Move duplicate image arrays and inherited label trees into that sibling journal on the same filesystem. Retained trees are never copied.
  5. Write shallow metadata and the operation report, and validate the complete reference inventory and retained label structure.
  6. Persist the committed journal state, then unlink verified duplicates.

An exception before commit restores arrays and original attributes. A killed process leaves durable intent: on resume, uncommitted intent is rolled back; committed intent only finishes cleanup. A valid committed report can be reused without generating identities again. The canonical store is never mutated or mounted by the Slurm helper. Returned symlink trees are rejected.

The completed workflow must have stopped writing its output before this stage. POSIX locks, atomic rename, and fsync on the mounted filesystem are required. The helper does not claim durability on filesystems that ignore these primitives. Never remove a prune journal to free space: it may contain the only remaining copy of a pre-commit array.

Slurm events and detached resume

The _SLURM_Remote_Shallower task records its configured image/version and input directory. TaskCreated/TaskAdded/TaskStarted precede submission; JobIdAdded is saved immediately after the remote submission ledger returns. TaskCompleted stores the validated batch report. Workflow progress projections ignore this internal task; analytics still retain its provenance.

The remote submission ledger sits outside archived output under .biomero-shallower/TASK/. flock serializes submission. A saved job ID is adopted. An intent without an ID is reconciled through the unique task job name in sacct; an empty or ambiguous accounting result blocks retrieval rather than submitting another job. Retry once accounting is available. Operators can reconcile a proven job ID into the ledger if accounting retention has expired.

A valid batch report is adopted even if the orchestration process died before recording TaskCompleted. A failed helper launches a CPU recovery job with the same image: interrupted stores roll back and completed stores retain receipts. Only after recovery completes can the mixed full/shallow output be archived. Unresolved submission, unknown scheduler state, or failed rollback preserves the output and stops retrieval for safe operator recovery.

Image acquisition failures precede all result mutation and safely fall back to the existing full-result transfer. The only supported policy is keep-full; there is no new strict scientific-result policy.

Trust and portability

Managed storage root IDs, relative paths, generations, and canonical pixel identities travel unchanged from workflow input tracking. OMERO-side registration resolves them using its authoritative local storage mappings. Slurm paths are only operational job parameters, never portable source references.

The importer requires administrator enablement and matching configured image and package versions. It checks the receipt's report SHA-256, canonical snapshot, task/job, manifest equality, reference membership, and retained structure. User renaming after transfer can bind through the unchanged report checksum. It does not hash pixel chunks again. SHA-256 here binds small metadata; it is not a cryptographic signature or a pixel integrity scan.

This assumes trusted BIOMERO orchestration, its event store and import-order writers, configured container images, and the Slurm account/filesystem. Workflows must not forge reserved .biomero-* metadata. It is not an attestation protocol against a malicious workflow sharing that account. Deployments requiring that threat model need isolated ownership and signed receipts before enabling it.

The report formats and receipt compatibility rules are documented in BIOMERO Schema. For deployment flags and image initialization, use the NL-BIOMERO administration guide.

The archive extension point is after shallower completion and before zip_data_on_slurm_server; a later archive adapter can replace ZIP independently.

Adding a contract adapter

Add versioned models to biomero-schema, then a filesystem adapter with parity, rollback, and import-validation tests. Dispatch it from operations.ADAPTERS and expose its version in the CLI. The current adapter targets NGFF 0.4 / Zarr v2; the schema contract documentation explains why its portable graph is not currently written as RFC-8 metadata.

Prerelease schema-1 migration

Schema-1 shallow stores are upgraded only through the explicit one-time command; normal readers do not silently reinterpret them:

biomero-shallower migrate-v1 --returned-zarr /results/result.ome.zarr

The migration validates the complete old manifest, creates deterministic graph node IDs, separates graph nodes from storage bindings, rewrites the embedded operation report and image-node metadata, and keeps an adjacent rollback copy. It rejects older manifests that lack the label-component identities required by schema 2. Because the report checksum changes, migrate only settled data, not an artifact covered by an in-flight remote receipt.

The filesystem-only command intentionally has no OMERO dependency. Complete BIOMERO deployments provide the administrator script BIOMERO Migrate Shallow Storage, which discovers schema-1 MapAnnotations, validates their storage targets, upgrades each store once, and updates all linked OMERO projections. It defaults to a dry run and writes recovery snapshots when applied.

Python API

These filesystem interfaces are shared with BIOMERO.importer. They do not accept OMERO connections.

biomero_shallower.operations

Version-dispatched normalization and cheap receipt validation.

normalize(root, manifest, *, contract=1, identity_workers=1, failure_policy='keep-full', image=None, task_id=None, identity_provider=None, measure_bytes=False)

validate_report(root, manifest, *, image=None, tool_version=None, artifact=None)