Skip to content

Training data

Preparing training folders

omero_annotate_ai.processing.training_functions.export_training_set

export_training_set(ref: int | str | Path, output_dir: str | Path, conn: Any = None, *, training_channels: list[int] | None = None, label_channel: int | None = None, apply_normalization: bool | None = None, home: str | Path | None = None) -> dict[str, Any]

Write a training set's annotated units into the training layout.

One entry point for the widget, the marimo notebook and scripts. ref is a table ID (the data comes from OMERO, see :func:prepare_training_data_from_table) or a set's folder (offline, see :func:reorganize_local_data_for_training). Channels and normalisation default to the set's config; apply_normalization overrides normalization.apply_at_export.

Channel arguments equal to the set's own are always accepted, so a caller can pass its form values as they are. A local set exports the channels its folder holds, so other channels are refused there; an OMERO export trains on one channel.

Raises:

Type Description
ValueError

Other channels for a local set, more than one training channel, or an output folder inside the set's folder (home is searched for the local folder of an OMERO set).

Returns:

Type Description
dict[str, Any]

The export result: the created folders and stats.

omero_annotate_ai.processing.training_functions.default_export_dir

default_export_dir(ref: int | str | Path, name: str, home: str | Path | None = None) -> Path

Where a set's training export goes by default, under <home>/exports/.

<name>_<table id> for an OMERO set, the set's own folder name (which holds its unique ID) for a local set: one folder per set, never inside the set's folder.

omero_annotate_ai.processing.training_functions.prepare_training_data_from_table

prepare_training_data_from_table(conn: Any, table_id: int, output_dir: str | Path, training_name: str = 'micro_sam_training', clean_existing: bool = True, tmp_dir: str | Path | None = None, verbose: bool = False, label_channel: int | None = None, training_channels: list[int] | None = None, upload_annotation_input: bool = False, normalization: Union[NormalizationConfig, dict[str, Any]] | None = None) -> dict[str, Any]

Prepare training data from OMERO annotation table.

Downloads images and labels from OMERO based on annotation table data and organizes them by each row's category (training, validation, test) into a directory structure suitable for micro-SAM training.

Parameters:

Name Type Description Default
conn Any

OMERO connection object

required
table_id int

ID of the annotation table in OMERO

required
output_dir str | Path

Directory to store training data

required
training_name str

Name for the training session (used in directory naming)

'micro_sam_training'
clean_existing bool

Whether to clean existing output directories

True
tmp_dir str | Path | None

Temporary directory for downloads (optional)

None
verbose bool

If True, show detailed debug information in console output

False
label_channel int | None

Optional channel index for label/segmentation images. If provided and different from training_channels, downloads label channel images to *_annotation_input directories alongside the training data.

None
training_channels list[int] | None

Optional list of channel indices for training input images. If different from label_channel, downloads from these channels for train_input and val_input. Currently uses first channel if multiple specified.

None
upload_annotation_input bool

If True and using separate channels, uploads the annotation-channel images back to OMERO as file annotations. Default is False.

False
normalization Union[NormalizationConfig, dict[str, Any]] | None

Normalisation spec (config.normalization or an equivalent dict). Images are written at native bit depth unless apply_at_export is set, in which case they are written as float32 and the bounds per image are recorded. When given, the spec is saved as normalization.json in output_dir.

None

Returns:

Type Description
dict[str, Any]

Dictionary with paths to created directories:

dict[str, Any]

{ 'base_dir': Path to base output directory, 'train_input': Path to training images, 'train_label': Path to training labels (segmentation masks), 'train_annotation_input': Path to annotation-channel images (separate channels only), 'val_input': Path to validation images, 'val_label': Path to validation labels (segmentation masks), 'val_annotation_input': Path to annotation-channel validation images (separate channels only), 'test_input', 'test_label': Test images and labels (only when the table has test rows), 'stats': Statistics about the prepared data

dict[str, Any]

}

Raises:

Type Description
ValueError

If table not found or invalid parameters

ImportError

If required dependencies missing

omero_annotate_ai.processing.training_functions.reorganize_local_data_for_training

reorganize_local_data_for_training(config: AnnotationConfig, annotation_dir: str | Path, output_dir: str | Path | None = None, file_mode: Literal['copy', 'move', 'symlink'] = 'copy', clean_existing: bool = True, include_test: bool | None = None, verbose: bool = False) -> dict[str, Any]

Reorganize locally-stored annotation data into the training layout.

Works entirely offline - no OMERO connection required. Reads the annotation phase's folders (annotation_input/, model_input/, annotation_output/) and writes the training layout (train_input/, train_label/, val_input/, val_label/).

Parameters:

Name Type Description Default
config AnnotationConfig

The training set's config (channels, normalisation)

required
annotation_dir str | Path

The training set's folder (TrainingSet.bundle_dir); its tracking_table.csv says which units are annotated and their split

required
output_dir str | Path | None

Target directory. Must NOT be inside annotation_dir - cleaning the training layout there would delete the source images. Defaults to the sibling _training/.

None
file_mode Literal['copy', 'move', 'symlink']

"copy" (default), "move", or "symlink". Symlinks fall back to a copy where they are unavailable (e.g. Windows without Developer Mode).

'copy'
clean_existing bool

Remove the training folders before writing

True
include_test bool | None

Write a test split. None (default) auto-detects from the annotations.

None
verbose bool

Show detailed progress

False

Normalisation follows config.normalization: with apply_at_export the model input images are written as float32 (file_mode then applies to labels only) and the bounds per image go to normalization.json; otherwise images keep their native bit depth and normalization.json records what the trainer should apply.

Returns:

Type Description
dict[str, Any]

Dictionary with the created directories and statistics. Keys include base_dir,

dict[str, Any]

train_input, train_label, val_input, val_label and stats.

Raises:

Type Description
ValueError

If config has no processed annotations, or output_dir is inside annotation_dir

FileNotFoundError

If annotation_dir does not exist

micro-SAM training

omero_annotate_ai.processing.training_utils.setup_training

setup_training(training_result: dict[str, Any], model_name: str = '', model_type: str = 'vit_b_lm', epochs: int = 50, n_iterations: int | None = None, batch_size: int = 2, learning_rate: float = 1e-05, patch_shape: tuple[int, int] | tuple[int, int, int] = (512, 512), n_objects_per_batch: int = 25, save_every: int = 1000, validate_every: int = 500, **kwargs) -> dict[str, Any]

Setup training configuration from training_result dict.

Parameters:

Name Type Description Default
training_result dict[str, Any]

Dictionary from prepare_training_data_from_table()

required
model_name str

Name for the training session/model

''
model_type str

SAM model variant ("vit_b", "vit_l", "vit_h")

'vit_b_lm'
epochs int

Number of training epochs

50
n_iterations int | None

Number of training iterations; when given it overrides epochs (micro-SAM's train_sam semantics), e.g. for a short test run

None
batch_size int

Training batch size

2
learning_rate float

Learning rate for training

1e-05
patch_shape tuple[int, int] | tuple[int, int, int]

Input patch dimensions (height, width) or (slices, height, width)

(512, 512)
n_objects_per_batch int

Number of objects per batch for sampling

25
save_every int

Save checkpoint every N iterations

1000
validate_every int

Run validation every N iterations

500
**kwargs

Framework-specific parameters

{}

Returns:

Type Description
dict[str, Any]

Dictionary containing all training configuration and paths

Raises:

Type Description
ValueError

If training_result is missing required keys

FileNotFoundError

If training directories don't exist

omero_annotate_ai.processing.training_utils.run_training

run_training(training_config: dict[str, Any], framework: str = 'microsam') -> dict[str, Any]

Execute training with framework-specific implementation.

Parameters:

Name Type Description Default
training_config dict[str, Any]

Configuration dictionary from setup_training()

required
framework str

Training framework to use ("microsam", future: "cellpose", etc.)

'microsam'

Returns:

Type Description
dict[str, Any]

Dictionary containing training results and model paths

Raises:

Type Description
ValueError

If framework is not supported

ImportError

If required framework packages are not available

BioImage Archive export

omero_annotate_ai.processing.bia_data.prepare_bia_data_from_table

prepare_bia_data_from_table(conn: Any, table_id: int, output_dir: str | Path, clean_existing: bool = False) -> dict[str, Any]

Download the data referenced by an OMERO annotation table for a BIA export.

Loads the table's rows and writes, for every row that is processed and has a label_id:

  • {output_dir}/annotation_input/{annotation_id}.tif - the image plane, patch or volume
  • {output_dir}/annotation_output/{annotation_id}_mask.tif - the mask FileAnnotation

which is the layout :func:~omero_annotate_ai.core.mifa_export.save_bia_package copies from: pass it the returned rows and output_dir.

Images are written at their native dtype - no 8-bit rescaling.

Rows that are unprocessed, or processed but carry no mask, are counted and skipped with a warning; excluded rows are skipped silently. Individual fetch failures are warned about and skipped too; only a missing/empty table is fatal.

The returned rows hold only the exported records - the ones whose image and mask both reached the disk - so the BIA file lists never name a file that was not downloaded.

Parameters:

Name Type Description Default
conn Any

OMERO connection (BlitzGateway).

required
table_id int

OMERO ID of the annotation table (OriginalFile).

required
output_dir str | Path

Directory to write the annotation_input and annotation_output trees into.

required
clean_existing bool

Remove existing annotation_input/annotation_output directories first.

False

Returns:

Type Description
dict[str, Any]

Dict with output_dir, rows (the exported rows), n_images,

dict[str, Any]

n_masks, skipped_unprocessed and skipped_no_mask.

Raises:

Type Description
ValueError

If the table cannot be loaded or is empty.

IntegrityError

A downloaded mask no longer matches the mask_digest recorded when it was annotated (it was changed on OMERO since).

omero_annotate_ai.core.mifa_export.save_bia_package

save_bia_package(config, rows: Sequence[Row], data_dir: str | Path, dest_dir: str | Path, *, accession: str | None = None, funding_statement: str | None = None, move_data: bool = False) -> dict[str, Any]

Assemble a self-contained BioImage Archive submission bundle in dest_dir.

Writes metadata/ (the three MIFA YAMLs), file_list_images.tsv and file_list_annotations.tsv, and transfers every referenced image/mask out of data_dir (a training set's folder) into the bundle, from its on-disk layout (:func:_source_paths) to the bundle's own (:func:_bundle_paths):

  • annotation_input/{id}.tif -> images/{id}.tif
  • annotation_output/{id}_mask.tif -> annotations/{id}_mask.tif

Returns a summary dict (paths + counts). Missing source files are skipped with a warning rather than failing the whole bundle.

Parameters:

Name Type Description Default
config

The annotation config describing the submission.

required
rows Sequence[Row]

The rows to submit (e.g. TrainingSet.units()); units not annotated yet are left out.

required
data_dir str | Path

The folder holding the files (TrainingSet.bundle_dir).

required
dest_dir str | Path

Where to build the bundle.

required
accession str | None

Accession used in the MIFA filenames.

None
funding_statement str | None

Optional override for the Study funding statement.

None
move_data bool

Moves the data into the bundle instead of copying it, leaving data_dir empty of the transferred files. Only use this when that directory is a disposable staging area (e.g. one just filled by :func:~omero_annotate_ai.processing.bia_data.prepare_bia_data_from_table). It is destructive against a real annotation run's output directory, which is why it defaults to copying.

False