Cutout writer zarr
cutana.cutout_writer_zarr
¶
Zarr cutout writer module for Cutana - handles zarr archive output using images-to-zarr.
This module provides static functions for: - Direct memory-to-zarr conversion using images-to-zarr convert function - Incremental sub-batch writing with append mode - Smart chunking to stay below 2GB limit - Compression and chunking strategies - No temporary files - writes directly from memory to zarr
generate_process_subfolder(process_id)
¶
Generate a unique subfolder name for a specific process.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
process_id
|
str
|
Unique process identifier (e.g., "cutout_process_000_unique_id") |
required |
Returns:
| Type | Description |
|---|---|
str
|
Subfolder name (e.g., "batch_cutout_process_000_unique_id") |
calculate_optimal_chunk_shape(n_sources, height, width, n_channels, dtype, max_chunk_size_gb=1.8)
¶
Calculate optimal chunk shape to stay below 2GB limit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n_sources
|
int
|
Number of sources in the batch |
required |
height
|
int
|
Image height |
required |
width
|
int
|
Image width |
required |
n_channels
|
int
|
Number of channels |
required |
dtype
|
dtype
|
Data type |
required |
max_chunk_size_gb
|
float
|
Maximum chunk size in GB (default 1.8 to stay safely below 2GB) |
1.8
|
Returns:
| Type | Description |
|---|---|
Tuple[int, int, int, int]
|
Tuple of (chunk_n_sources, chunk_height, chunk_width, chunk_channels) |
prepare_cutouts_for_zarr(batch_data)
¶
Prepare cutout data for zarr conversion by organizing into 4D NCHW format. Handles single batch result from sub-batch processing.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
batch_data
|
Dict[str, Any]
|
Single batch result containing cutouts tensor and metadata |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Tuple of (images_array, metadata_list) |
List[Dict[str, Any]]
|
|
create_zarr_from_memory(images, metadata, output_path, config, append=False)
¶
Create or append to zarr archive directly from images in memory using images_to_zarr.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
images
|
ndarray
|
4D numpy array in NCHW format (samples, channels, height, width) |
required |
metadata
|
List[Dict[str, Any]]
|
List of original metadata for each image |
required |
output_path
|
str
|
Full path to zarr archive (including images.zarr) |
required |
config
|
DotMap
|
Configuration DotMap |
required |
append
|
bool
|
If True, append to existing zarr archive |
False
|
Returns:
| Type | Description |
|---|---|
str
|
Path to created/updated zarr archive |
create_process_zarr_archive_initial(batch_data, output_path, config)
¶
Create initial zarr archive for the first sub-batch.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
batch_data
|
Dict[str, Any]
|
First sub-batch cutout data dictionary |
required |
output_path
|
str
|
Full path to zarr archive |
required |
config
|
DotMap
|
Configuration DotMap |
required |
Returns:
| Type | Description |
|---|---|
Optional[str]
|
Path to created zarr archive, or None if failed |
append_to_zarr_archive(batch_data, output_path, config)
¶
Append a sub-batch to an existing zarr archive.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
batch_data
|
Dict[str, Any]
|
Sub-batch cutout data dictionary |
required |
output_path
|
str
|
Full path to existing zarr archive |
required |
config
|
DotMap
|
Configuration DotMap |
required |
Returns:
| Type | Description |
|---|---|
Optional[str]
|
Path to updated zarr archive, or None if failed |