Training sets¶
A training set is created or opened as a TrainingSet; everything else (annotating,
adding images, notes) is a method on it.
omero_annotate_ai.trainingset.TrainingSet
¶
TrainingSet(config: AnnotationConfig, conn: Any = None, *, rows: Sequence[Row] = (), image_source: ImageSource | None = None, omero_writes: OmeroWrites | None = None, home: str | Path | None = None, events: EventSink | None = None, cancel: Callable[[], bool] | None = None, bundle: Path | None = None)
A training set: its definition (the config), its rows, folder and OMERO table.
Make one with :meth:create (selects images, plans the units, creates the
tracking table and the folder) or open one with :meth:open (by table ID or
folder; repairs what an interrupted session left first). Annotate with
:meth:annotate, or step by step through :meth:start_session.
The connection is used from one thread at a time.
Use :meth:create or :meth:open; this only wires the parts together.
config
property
¶
A copy of the set's config; changing it does not change the set.
bundle_dir
property
¶
This training set's folder under :attr:home (see :mod:..bundle).
create
classmethod
¶
create(config: AnnotationConfig, conn: Any = None, *, home: str | Path | None = None, image_source: ImageSource | None = None, omero_writes: OmeroWrites | None = None, events: EventSink | None = None, cancel: Callable[[], bool] | None = None, images: Sequence[Any] | None = None) -> TrainingSet
A new training set: select the images, plan the units, start tracking.
images (image handles of the image source) limits the selection to
those images; by default the config's container or OME-Zarr URIs are used.
For storage: omero the tracking table is created (and linked to the
containers) and the config published next to it; storage: local sets
get a local ID. Either way the set's folder holds config.yaml and
tracking_table.csv. config itself is not changed.
Raises:
| Type | Description |
|---|---|
ConfigError
|
|
SelectionError
|
Too few images for the requested counts. |
open
classmethod
¶
open(ref: int | str | Path, conn: Any = None, *, home: str | Path | None = None, image_source: ImageSource | None = None, omero_writes: OmeroWrites | None = None, events: EventSink | None = None, cancel: Callable[[], bool] | None = None, prune_duplicates: bool = False) -> TrainingSet
An existing training set, by tracking-table ID or by its folder.
For OMERO sets the rows come from the tracking table and the set is
repaired (:meth:reconcile) before it is returned. Sessions an earlier
run left open are closed as interrupted.
Raises:
| Type | Description |
|---|---|
ConfigError
|
The set has no config (on OMERO or in its folder). |
TableReadError
|
The tracking table cannot be read. |
extend
¶
extend(*, train_n: int | None = None, validate_n: int | None = None, test_n: int | None = None) -> int
Raise the image counts and add images to reach them.
Existing rows are never removed. Returns the number of images added.
Raises:
| Type | Description |
|---|---|
ConfigError
|
A count is lower than the set already has. |
annotate
¶
annotate(annotator: Annotator | None = None, *, batch_size: int | None = None, model: str | None = None, note: str | None = None) -> int
Annotate the pending units, batch by batch; returns how many were saved.
annotator defaults to micro-SAM in napari. note is stored with the
session (see :meth:set_session_note to add one afterwards).
review
¶
review(annotator: Annotator | None = None, *, ids: Sequence[str] | None = None, batch_size: int | None = None, model: str | None = None, note: str | None = None) -> int
Correct annotated units (all, or ids); returns how many changed.
Each unit opens with its saved mask (downloaded from OMERO if the folder
lacks it). A changed mask replaces the upload; an unchanged one is left
alone. The session is recorded as a review, with note.
start_session
¶
start_session(*, model: str | None = None, batch_size: int | None = None, review: bool = False, ids: Sequence[str] | None = None) -> AnnotationSession
A new session over the pending units, or a review of annotated ones.
See :mod:..session; ids limits a session to those units.
set_session_note
¶
Set (or replace) a session's note, locally and on OMERO.
Works for sessions of other computers too: the note set here wins over their copy on OMERO.
Raises:
| Type | Description |
|---|---|
KeyError
|
The set has no such session. |
reconcile
¶
Repair what an interrupted session left (see :mod:..reconcile).
prepare_external
¶
Save the pending units' images for an external tool; returns how many.
Images go to annotation_input/ (and model_input/ for separate
channels); the tool writes external_output/{annotation_id}_mask.tif.
The units stay pending until :meth:collect_masks finds their masks:
annotation_output/ only ever holds accepted masks, so drafts in
external_output/ are never uploaded by the repair on open.
collect_masks
¶
Save the masks an external tool wrote as {annotation_id}_mask.tif.
folder is relative to the set's folder and may be a glob pattern. Each
mask goes through the same write order as a session (one new session). On
OMERO sets the table is read first and the units claimed: units saved or
claimed on another computer are skipped. Returns the number of units saved.
write_ro_crate
¶
Describe the set's folder as an RO-Crate; returns the metadata file.
Writes ro-crate-metadata.json (see :mod:omero_annotate_ai.crate): the
files with their digests, the source images, and the sessions with their
notes. Call it before sharing or archiving the folder.
omero_annotate_ai.trainingset.list_training_sets
¶
The training sets tracked on a container, newest first (read-only).
omero_annotate_ai.trainingset.summarize
¶
summarize(ref: int | str | Path, conn: Any = None, *, home: str | Path | None = None) -> TrainingSetSummary
Progress and sessions of a set, by table ID or folder, without opening it.
Unlike :meth:TrainingSet.open nothing is repaired or written.
omero_annotate_ai.trainingset.training_set_choices
¶
training_set_choices(conn: Any = None, container_type: str | None = None, container_id: int | None = None, *, home: str | Path | None = None, on_error: Callable[[str], None] | None = None) -> list[tuple[str, int | Path]]
(label, ref) for a picker: the sets on a container, then the local sets.
ref is a table ID (OMERO set) or a folder (local set), as :func:summarize,
:meth:TrainingSet.open and the training export take it. Read-only. With
on_error, an error listing the OMERO sets is reported there and the local sets
are still returned; without it, it is raised.
omero_annotate_ai.trainingset.read_set_config
¶
A set's config, by table ID (from OMERO) or folder, without opening it.
Annotation sessions¶
omero_annotate_ai.session.AnnotationSession
¶
AnnotationSession(training_set: TrainingSet, *, batch_size: int | None = None, model: str | None = None, cancel: Callable[[], bool] | None = None, review: bool = False, ids: Collection[str] | None = None)
Hands out batches of pending units and saves the masks that come back.
Each pending unit is handed out at most once per session. Batches may be
committed in parts and committed again (each unit is saved once); units never
committed are handed back with :meth:release and stay pending for the next
session. With review=True the annotated units (or those in ids) are
handed out instead, each with its saved mask. Not thread-safe: call it from one
thread.
scratch
property
¶
A folder of this session inside the set's folder (the same filesystem as
the masks, so files move into place), removed at :meth:finish.
next_batch
¶
Load the next pending units (pixels, context channels, input images).
Returns None when nothing is left to hand out.
commit
¶
Save the masks in result ({annotation_id: file}), then update the table.
result may cover part of the batch, and a unit already committed is
skipped, so calling this again is safe. A unit this session excluded is
saved as a normal annotation. With publish_config=False the config on
OMERO is only replaced at :meth:finish.
Raises:
| Type | Description |
|---|---|
TableWriteError
|
The tracking table could not be updated (the uploads are saved locally and tagged; resuming repairs the table). |
saved_unit
¶
The image and the saved mask of a unit saved in this session, from the set's folder, e.g. to show it again.
Raises:
| Type | Description |
|---|---|
ValueError
|
The unit was not handed out and saved in this session. |
revise
¶
revise(annotation_id: str, mask: Path, *, publish_config: bool = True) -> Literal['revised', 'unchanged', 'failed', 'conflict']
Replace the saved mask of a unit saved in this session (the annotator went back to it), as a review would: an unchanged mask keeps its upload.
On OMERO sets the row is read again first: if another session changed the
unit since, theirs is kept and mask is kept as a conflict file.
Raises:
| Type | Description |
|---|---|
ValueError
|
The unit was not handed out and saved in this session. |
exclude
¶
exclude(batch: Batch, annotation_ids: Collection[str], *, publish_config: bool = True) -> tuple[str, ...]
Mark units of batch excluded (e.g. out of focus), then update the table.
An excluded unit is done: it has no mask, uploads or claim, is not handed
out again and is not exported. Units not open in the batch (committed,
released, excluded already) are skipped, so calling this again is safe; on
OMERO sets so is a unit another session saved first (its row is taken
over). With publish_config=False the config on OMERO is only replaced
at :meth:finish.
Returns:
| Type | Description |
|---|---|
tuple[str, ...]
|
The units excluded by this call. |
Raises:
| Type | Description |
|---|---|
ValueError
|
This is a review session (its units have uploads). |
TableWriteError
|
The tracking table could not be updated (the exclusion is saved locally; resuming writes it to the table). |
release
¶
Hand back the batch's uncommitted units (they stay pending) and clean up.
finish
¶
End the session: record it (with note) in the config and publish it.
Returns the number of units annotated. Calling it again does nothing.
omero_annotate_ai.session.BatchReport
dataclass
¶
BatchReport(committed: tuple[str, ...], failed: tuple[str, ...], conflicts: tuple[str, ...] = (), unchanged: tuple[str, ...] = ())
What a :meth:AnnotationSession.commit call did with each unit.
committed: saved (in a review: changed and saved); failed: upload
failed (stays as it was); conflicts: another session saved it first (its
upload is kept); unchanged: reviewed without changes.
Content codes (provenance extra)¶
omero_annotate_ai.provenance
¶
ISCC content codes: how similar two images or masks are, not only whether equal.
The SHA-256 pixel digests in the tracking table (:func:~omero_annotate_ai.bundle.
pixel_digest) answer "is this exactly the image that was annotated?". An ISCC
Image-Code (ISO 24138) answers "how much does it differ?": codes of similar
images differ in few of their 64 bits. Together they tell a user whether a mask
that changed after upload was touched up (a near-duplicate) or replaced by
something else.
Codes are computed only when the optional extra is installed
(pip install 'omero-annotate-ai[provenance]', which brings iscc-lib);
without it every function here returns None and nothing else changes.
A plane is normalised before coding: several planes (a volume) are reduced by a maximum projection, the plane is area-averaged to the 32 x 32 grayscale pixels the Image-Code takes (colour: mean of its channels), and intensities are stretched to 0-255 between the full plane's minimum and maximum (masks: background 0, every object 255, so renumbering objects keeps the code). Uniform images all get the same code, so for them only the SHA-256 digest says anything. Codes are comparable between images coded by this package; they are not meant to match codes another tool computed from the image file.
image_code
¶
The ISCC Image-Code of an image (or a label mask).
None without the extra, or for an empty array.
file_code
¶
:func:image_code of a TIFF file; None without the extra (file not read).
distance
¶
Bits in which two Image-Codes differ (0-64).
None without the extra, or when a code cannot be decoded.
describe_change
¶
How pixels whose digest no longer matches relate to the recorded code.
Returns "" when the codes cannot tell (no recorded code, the extra is not
installed, an empty or uniform image, or a damaged code): the SHA-256 digest
alone then says the file changed. Otherwise a short phrase for a message.
RO-Crate¶
omero_annotate_ai.crate
¶
Describe a training set's folder as an RO-Crate (ro-crate-metadata.json).
An RO-Crate <https://w3id.org/ro/crate/1.1>_ is a JSON-LD file next to the data
that says what the folder holds and how it came about, in a form repositories and
workflow tools read. For a training set it lists:
- the dataset (name, description, licence, authors) and the software that made it,
- the files:
config.yaml,tracking_table.csv, each annotated image inannotation_input/and each mask inannotation_output/, with the pixel SHA-256 digest and, with theprovenanceextra, the ISCC code of each, - the source images by URI (
omero:image:<id>or an OME-Zarr URL), which the local images are based on; a local OME-Zarr store gets a#source-<key>id and its name only, so no local path (which may hold a user name) is shared, - every annotation session as an action: who, when, which model, its note, and
the masks it produced (an annotate session is a
CreateAction, a review anUpdateAction).
The crate is written on request (:meth:TrainingSet.write_ro_crate
<omero_annotate_ai.trainingset.TrainingSet.write_ro_crate>), not after every
commit: it describes the folder at the moment you share or archive it.
build_ro_crate
¶
build_ro_crate(config: AnnotationConfig, rows: Sequence[Row], bundle_dir: Path, *, software_version: str = 'unknown') -> dict[str, Any]
The RO-Crate metadata (JSON-LD) of a training set's folder.
Only files that exist are listed; images not saved locally (e.g. never shown) appear as source images only.
write_ro_crate
¶
write_ro_crate(config: AnnotationConfig, rows: Sequence[Row], bundle_dir: Path, *, software_version: str = 'unknown') -> Path
Write ro-crate-metadata.json into bundle_dir (atomically); its path.
Several computers (claims)¶
omero_annotate_ai.claims
¶
Claims: several computers annotating one OMERO training set without collisions.
Before a session shows a unit in napari it claims the unit in the tracking table:
claimed_by gets the session's token (<session>@<user>/<host>) and
claimed_at the time (UTC). Other sessions skip units with a live claim. A claim
is released when the unit is committed or handed back, and expires after
workflow.claim_hours - so a crashed session never blocks a unit for good.
Claims never block their owner: a session of the same user on the same computer
takes units an earlier session left claimed (a dropped connection, an exception),
and opening a set releases such leftovers.
OMERO.tables has no compare-and-swap, so claiming is a read-modify-write that
writes only the claim cells, and only those still holding what was read
(:func:~omero_annotate_ai.omero.tracking_table.update_cells): two sessions
claiming the same unit within the instant of one table update could both win.
The commit check in :mod:omero_annotate_ai.session (re-read the row, adopt an
upload already carrying the unit's tag, never upload over a unit another session
saved) keeps that case from duplicating uploads.
The set's config on OMERO has the same limit: publishing merges what other
computers published (sessions, notes, image counts) just before posting, and the
newest file wins. Two computers publishing within that instant can lose one
session record's latest change; the rows in the table are not affected. A session
number taken by two computers at once is detected after publishing and the later
one renumbered (:meth:~omero_annotate_ai.trainingset.TrainingSet.begin_session).
Local sets have no claims: their folder belongs to one computer.
Rows¶
omero_annotate_ai.rows.Row
pydantic-model
¶
Bases: BaseModel
One annotation unit and its state: what to annotate, and what was saved.
Fields:
-
image_uri(str) -
image_id(int | None) -
image_name(str) -
annotation_id(str) -
level(int) -
well(str) -
field(str) -
well_metadata(dict[str, str]) -
category(Category) -
timepoint(int) -
z_slice(int) -
channel(int) -
is_volumetric(bool) -
z_start(int) -
z_end(int) -
z_length(int) -
is_patch(bool) -
patch_x(int) -
patch_y(int) -
patch_width(int) -
patch_height(int) -
processed(bool) -
annotation_created_at(str | None) -
annotation_updated_at(str | None) -
annotation_type(str) -
session(int) -
roi_id(int | None) -
label_id(int | None) -
annotation_input_id(int | None) -
input_digest(str | None) -
mask_digest(str | None) -
input_iscc(str | None) -
mask_iscc(str | None) -
claimed_by(str) -
claimed_at(str) -
error(str | None)
Validators:
-
_fill_image_uri
input_iscc
pydantic-field
¶
ISCC Image-Code of input (provenance extra)
claimed_by
pydantic-field
¶
Session holding the unit (see omero_annotate_ai.claims)
excluded
property
¶
Whether the annotator excluded the unit: done, with no mask and no uploads.
Excluded units are never handed out again and never exported for training.
from_unit
classmethod
¶
A new, not yet annotated row for a planned unit.
mark_processed
¶
Mark the unit annotated, keeping its first annotation time.
mark_excluded
¶
Mark the unit excluded by session: done, without mask, uploads or claim.
Deprecated one-call pipeline¶
omero_annotate_ai.core.annotation_pipeline.create_pipeline
¶
create_pipeline(config: AnnotationConfig, conn: Any = None, *, annotator: Annotator | None = None, omero_writes: OmeroWrites | None = None, home: str | Path | None = None, events: EventSink | None = None, cancel: Callable[[], bool] | None = None) -> AnnotationPipeline
A one-call workflow over a training set.
Deprecated: use :class:~omero_annotate_ai.trainingset.TrainingSet directly.
omero_annotate_ai.core.annotation_pipeline.AnnotationPipeline
¶
AnnotationPipeline(config: AnnotationConfig, conn: Any = None, *, annotator: Annotator | None = None, omero_writes: OmeroWrites | None = None, home: str | Path | None = None, events: EventSink | None = None, cancel: Callable[[], bool] | None = None, image_source: ImageSource | None = None)
Runs a whole workflow on a training set: create (or open) it, then annotate.
Opens the set config.identity.table_id names (raising its image counts to the
config's), otherwise creates a new one.
run_full_micro_sam_workflow
¶
Annotate the pending units (micro-SAM in napari unless an annotator was
given). Returns (table_id, config); table_id is -1 for local sets.
run_cp_workflow
¶
Save the pending units' images for Cellpose (annotation_input/).
Units stay pending until :meth:collect_annotations_from_disk finds their
masks.
collect_annotations_from_disk
¶
collect_annotations_from_disk(folder_pattern: str = 'external_output') -> tuple[int, AnnotationConfig]
Save the masks an external tool wrote; returns (n_saved, config).