Training data¶
Preparing training folders¶
omero_annotate_ai.processing.training_functions.export_training_set
¶
export_training_set(ref: int | str | Path, output_dir: str | Path, conn: Any = None, *, training_channels: list[int] | None = None, label_channel: int | None = None, apply_normalization: bool | None = None, home: str | Path | None = None) -> dict[str, Any]
Write a training set's annotated units into the training layout.
One entry point for the widget, the marimo notebook and scripts. ref is a
table ID (the data comes from OMERO, see :func:prepare_training_data_from_table)
or a set's folder (offline, see :func:reorganize_local_data_for_training).
Channels and normalisation default to the set's config; apply_normalization
overrides normalization.apply_at_export.
Channel arguments equal to the set's own are always accepted, so a caller can pass its form values as they are. A local set exports the channels its folder holds, so other channels are refused there; an OMERO export trains on one channel.
Raises:
| Type | Description |
|---|---|
ValueError
|
Other channels for a local set, more than one training channel,
or an output folder inside the set's folder ( |
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
The export result: the created folders and |
omero_annotate_ai.processing.training_functions.default_export_dir
¶
Where a set's training export goes by default, under <home>/exports/.
<name>_<table id> for an OMERO set, the set's own folder name (which holds its
unique ID) for a local set: one folder per set, never inside the set's folder.
omero_annotate_ai.processing.training_functions.prepare_training_data_from_table
¶
prepare_training_data_from_table(conn: Any, table_id: int, output_dir: str | Path, training_name: str = 'micro_sam_training', clean_existing: bool = True, tmp_dir: str | Path | None = None, verbose: bool = False, label_channel: int | None = None, training_channels: list[int] | None = None, upload_annotation_input: bool = False, normalization: Union[NormalizationConfig, dict[str, Any]] | None = None) -> dict[str, Any]
Prepare training data from OMERO annotation table.
Downloads images and labels from OMERO based on annotation table data and
organizes them by each row's category (training, validation, test) into a
directory structure suitable for micro-SAM training.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conn
|
Any
|
OMERO connection object |
required |
table_id
|
int
|
ID of the annotation table in OMERO |
required |
output_dir
|
str | Path
|
Directory to store training data |
required |
training_name
|
str
|
Name for the training session (used in directory naming) |
'micro_sam_training'
|
clean_existing
|
bool
|
Whether to clean existing output directories |
True
|
tmp_dir
|
str | Path | None
|
Temporary directory for downloads (optional) |
None
|
verbose
|
bool
|
If True, show detailed debug information in console output |
False
|
label_channel
|
int | None
|
Optional channel index for label/segmentation images. If provided and different from training_channels, downloads label channel images to *_annotation_input directories alongside the training data. |
None
|
training_channels
|
list[int] | None
|
Optional list of channel indices for training input images. If different from label_channel, downloads from these channels for train_input and val_input. Currently uses first channel if multiple specified. |
None
|
upload_annotation_input
|
bool
|
If True and using separate channels, uploads the annotation-channel images back to OMERO as file annotations. Default is False. |
False
|
normalization
|
Union[NormalizationConfig, dict[str, Any]] | None
|
Normalisation spec ( |
None
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Dictionary with paths to created directories: |
dict[str, Any]
|
{ 'base_dir': Path to base output directory, 'train_input': Path to training images, 'train_label': Path to training labels (segmentation masks), 'train_annotation_input': Path to annotation-channel images (separate channels only), 'val_input': Path to validation images, 'val_label': Path to validation labels (segmentation masks), 'val_annotation_input': Path to annotation-channel validation images (separate channels only), 'test_input', 'test_label': Test images and labels (only when the table has test rows), 'stats': Statistics about the prepared data |
dict[str, Any]
|
} |
Raises:
| Type | Description |
|---|---|
ValueError
|
If table not found or invalid parameters |
ImportError
|
If required dependencies missing |
omero_annotate_ai.processing.training_functions.reorganize_local_data_for_training
¶
reorganize_local_data_for_training(config: AnnotationConfig, annotation_dir: str | Path, output_dir: str | Path | None = None, file_mode: Literal['copy', 'move', 'symlink'] = 'copy', clean_existing: bool = True, include_test: bool | None = None, verbose: bool = False) -> dict[str, Any]
Reorganize locally-stored annotation data into the training layout.
Works entirely offline - no OMERO connection required. Reads the annotation phase's folders (annotation_input/, model_input/, annotation_output/) and writes the training layout (train_input/, train_label/, val_input/, val_label/).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
AnnotationConfig
|
The training set's config (channels, normalisation) |
required |
annotation_dir
|
str | Path
|
The training set's folder ( |
required |
output_dir
|
str | Path | None
|
Target directory. Must NOT be inside annotation_dir - cleaning the
training layout there would delete the source images. Defaults to the sibling
|
None
|
file_mode
|
Literal['copy', 'move', 'symlink']
|
"copy" (default), "move", or "symlink". Symlinks fall back to a copy where they are unavailable (e.g. Windows without Developer Mode). |
'copy'
|
clean_existing
|
bool
|
Remove the training folders before writing |
True
|
include_test
|
bool | None
|
Write a test split. None (default) auto-detects from the annotations. |
None
|
verbose
|
bool
|
Show detailed progress |
False
|
Normalisation follows config.normalization: with apply_at_export the model
input images are written as float32 (file_mode then applies to labels only) and the
bounds per image go to normalization.json; otherwise images keep their native bit
depth and normalization.json records what the trainer should apply.
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Dictionary with the created directories and statistics. Keys include base_dir, |
dict[str, Any]
|
train_input, train_label, val_input, val_label and stats. |
Raises:
| Type | Description |
|---|---|
ValueError
|
If config has no processed annotations, or output_dir is inside annotation_dir |
FileNotFoundError
|
If annotation_dir does not exist |
micro-SAM training¶
omero_annotate_ai.processing.training_utils.setup_training
¶
setup_training(training_result: dict[str, Any], model_name: str = '', model_type: str = 'vit_b_lm', epochs: int = 50, n_iterations: int | None = None, batch_size: int = 2, learning_rate: float = 1e-05, patch_shape: tuple[int, int] | tuple[int, int, int] = (512, 512), n_objects_per_batch: int = 25, save_every: int = 1000, validate_every: int = 500, **kwargs) -> dict[str, Any]
Setup training configuration from training_result dict.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
training_result
|
dict[str, Any]
|
Dictionary from prepare_training_data_from_table() |
required |
model_name
|
str
|
Name for the training session/model |
''
|
model_type
|
str
|
SAM model variant ("vit_b", "vit_l", "vit_h") |
'vit_b_lm'
|
epochs
|
int
|
Number of training epochs |
50
|
n_iterations
|
int | None
|
Number of training iterations; when given it overrides
|
None
|
batch_size
|
int
|
Training batch size |
2
|
learning_rate
|
float
|
Learning rate for training |
1e-05
|
patch_shape
|
tuple[int, int] | tuple[int, int, int]
|
Input patch dimensions (height, width) or (slices, height, width) |
(512, 512)
|
n_objects_per_batch
|
int
|
Number of objects per batch for sampling |
25
|
save_every
|
int
|
Save checkpoint every N iterations |
1000
|
validate_every
|
int
|
Run validation every N iterations |
500
|
**kwargs
|
Framework-specific parameters |
{}
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Dictionary containing all training configuration and paths |
Raises:
| Type | Description |
|---|---|
ValueError
|
If training_result is missing required keys |
FileNotFoundError
|
If training directories don't exist |
omero_annotate_ai.processing.training_utils.run_training
¶
Execute training with framework-specific implementation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
training_config
|
dict[str, Any]
|
Configuration dictionary from setup_training() |
required |
framework
|
str
|
Training framework to use ("microsam", future: "cellpose", etc.) |
'microsam'
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Dictionary containing training results and model paths |
Raises:
| Type | Description |
|---|---|
ValueError
|
If framework is not supported |
ImportError
|
If required framework packages are not available |
BioImage Archive export¶
omero_annotate_ai.processing.bia_data.prepare_bia_data_from_table
¶
prepare_bia_data_from_table(conn: Any, table_id: int, output_dir: str | Path, clean_existing: bool = False) -> dict[str, Any]
Download the data referenced by an OMERO annotation table for a BIA export.
Loads the table's rows and writes, for every row that is processed and has a
label_id:
{output_dir}/annotation_input/{annotation_id}.tif- the image plane, patch or volume{output_dir}/annotation_output/{annotation_id}_mask.tif- the mask FileAnnotation
which is the layout :func:~omero_annotate_ai.core.mifa_export.save_bia_package
copies from: pass it the returned rows and output_dir.
Images are written at their native dtype - no 8-bit rescaling.
Rows that are unprocessed, or processed but carry no mask, are counted and skipped with a warning; excluded rows are skipped silently. Individual fetch failures are warned about and skipped too; only a missing/empty table is fatal.
The returned rows hold only the exported records - the ones whose image and
mask both reached the disk - so the BIA file lists never name a file that was not
downloaded.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
conn
|
Any
|
OMERO connection ( |
required |
table_id
|
int
|
OMERO ID of the annotation table ( |
required |
output_dir
|
str | Path
|
Directory to write the |
required |
clean_existing
|
bool
|
Remove existing |
False
|
Returns:
| Type | Description |
|---|---|
dict[str, Any]
|
Dict with |
dict[str, Any]
|
|
Raises:
| Type | Description |
|---|---|
ValueError
|
If the table cannot be loaded or is empty. |
IntegrityError
|
A downloaded mask no longer matches the |
omero_annotate_ai.core.mifa_export.save_bia_package
¶
save_bia_package(config, rows: Sequence[Row], data_dir: str | Path, dest_dir: str | Path, *, accession: str | None = None, funding_statement: str | None = None, move_data: bool = False) -> dict[str, Any]
Assemble a self-contained BioImage Archive submission bundle in dest_dir.
Writes metadata/ (the three MIFA YAMLs), file_list_images.tsv and
file_list_annotations.tsv, and transfers every referenced image/mask out of
data_dir (a training set's folder) into the bundle, from its on-disk
layout (:func:_source_paths) to the bundle's own (:func:_bundle_paths):
annotation_input/{id}.tif->images/{id}.tifannotation_output/{id}_mask.tif->annotations/{id}_mask.tif
Returns a summary dict (paths + counts). Missing source files are skipped with a warning rather than failing the whole bundle.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
The annotation config describing the submission. |
required | |
rows
|
Sequence[Row]
|
The rows to submit (e.g. |
required |
data_dir
|
str | Path
|
The folder holding the files ( |
required |
dest_dir
|
str | Path
|
Where to build the bundle. |
required |
accession
|
str | None
|
Accession used in the MIFA filenames. |
None
|
funding_statement
|
str | None
|
Optional override for the Study funding statement. |
None
|
move_data
|
bool
|
Moves the data into the bundle instead of copying it, leaving
|
False
|