Skip to content

BIOMERO.shallower

BIOMERO.shallower is the shared filesystem-only implementation of shallow Zarr in BIOMERO 2.0. It compares returned OME-Zarr images and labels with canonical workflow inputs, references reusable source pixels, and retains new or changed arrays. The importer runs it locally, or BIOMERO runs its CPU-only container on Slurm before transferring results. It requires no OMERO connection or credentials.

The current adapter writes the experimental BIOMERO shallow-manifest schema 2 for NGFF 0.4 / Zarr v2 and 0.5 / Zarr v3 Images and Plates. This private format is not RFC-8. Deduplication never suppresses registration of the result in OMERO, and installing this package does not enable the feature.

For feature flags, deployment, and image acquisition, see the NL-BIOMERO administration guide.

Installation

Python 3.11 or newer is required. Install the identity extra for real pixel hashing:

pip install 'biomero-shallower[identity]'
biomero-shallower --version
biomero-shallower health

The package requires published receipt-capable schema contracts, biomero-schema>=0.3.0b3,<0.4. The container pins the complete CPU identity runtime in requirements.lock.

Shallow one result

The canonical-input manifest is the trusted snapshot recorded for the workflow, not a replacement generated from its returned metadata.

biomero-shallower inspect --returned-zarr /results/result.zarr
biomero-shallower normalize \
  --returned-zarr /results/result.zarr \
  --canonical-inputs /inputs/canonical.json \
  --report /results/report.json --contract-version 1 \
  --identity-workers 4 --failure-policy keep-full \
  --image cellularimagingcf/biomero-shallower:0.1.0
biomero-shallower verify \
  --returned-zarr /results/result.zarr \
  --canonical-inputs /inputs/canonical.json \
  --report /results/verification.json --contract-version 1 \
  --image cellularimagingcf/biomero-shallower:0.1.0

The normalize command omits returned originals according to the recorded decision, never the canonical input. New or changed labels remain in the store. Unknown contracts are rejected; unsupported or ineligible full artifacts are retained. Four identity workers are an explicit example, not the default: select parallelism for the allocated CPU and filesystem resources.

verify validates the committed report and collection without rehashing omitted pixels. It is not a checksum scan of retained pixel data and does not make an untrusted sidecar authoritative. Remote imports also require the orchestration receipt described in Architecture and recovery.

Exit 0 means a safe terminal result: normalized, kept full, or skipped. Exit 2 means invalid input, an unsupported contract, or unresolved recovery. Inspect the report's result and reason rather than interpreting exit 0 as proof that all artifacts became shallow.

Shallow a result directory

normalize-tree is the orchestration interface. It discovers outermost Zarr stores and writes .biomero-shallow-batch.json with receipts for shallowed artifacts. Ordinary non-Zarr outputs remain in place. Each artifact's durable report is .biomero-shallow-report.json; --report writes an additional copy. Receipts also cover kept-full results whose source fingerprint was refreshed. See the report contracts.

Output handling and comparison

Shallow storage must be enabled by the calling deployment. The disabled route keeps full results and does no deduplication-only hashing or pruning.

For enabled storage, usable RO-Crate label output links automatically select the known run input as image background. The supported graph connects a CreateAction.object input, its label-level result, the output's exampleOfWork parameter and the crate's hasPart. A root-level Zarr pointer alone is insufficient. Missing or unusable declarations fall through to the independent workflow setting; this requires an unambiguous transferred-input binding. Neither path calculates otherwise unused original-pixel fingerprints. New/changed labels and sidecars remain; other derived stores are retained. RO-Crate metadata is read locally with rocrate, including its standard @context and metadata descriptor; referenced URLs are not fetched. If a crate cannot be read unambiguously or explicitly inventories arrays that pruning would remove, the artifact stays full so its file references remain valid. The original crate is retained; committed BIOMERO storage manifests record source relationships.

With neither shortcut, the pixels-only ISCC-BIO API fingerprints the coarsest compatible stored pyramid level while retaining all channels/Z/timepoints and checking full-image semantics. Labels still use their full-resolution pixels. Equal selected-level bytes are exact; inferring full-resolution agreement from them is BIOMERO policy, not a full-resolution equality proof. No raw-store checksum or rendered thumbnail is calculated. See the shared comparison profile.

Old or absent source fingerprints are refreshed lazily from the same backing Zarr. Local callers pass managed storage roots. The remote helper uses a current cached fingerprint; it does not treat a post-analysis input copy as the authoritative source. Missing/stale fingerprints keep that remote result full, then the importer compares the actual managed backing store and refreshes its cache. A converted TIFF input can still support a declared source-linked output through recorded image metadata. Refreshed records are returned to the consumer for cache persistence, including failed result matches.

Container

docker run --rm --network none cellularimagingcf/biomero-shallower:0.1.0 health

The image runs as UID/GID 10001:10001. Bind the returned result directory writable and the canonical-input manifest read-only. The helper does not need access to transferred input copies or original managed canonical stores. Writable results require POSIX locks, atomic rename, and fsync. Do not shallow output while a workflow is still writing it.

For Slurm execution and SIF acquisition, follow the NL-BIOMERO deployment documentation.