Skip to content

source_validation

anomaly_match.data_io.source_validation

Validate that labeled filenames/IDs exist in data sources.

Provides efficient validation for container sources (Zarr, Cutana) without building full O(N) indexes. For each labeled item, produces a location tuple that downstream code can use for targeted extraction.

ValidationResult(found=list(), missing=list(), found_locations=dict(), partial=False) dataclass

Result of validating labeled files against a data source.

Attributes:

Name Type Description
found list[str]

Filenames/IDs that exist in the source.
missing list[str]

Filenames/IDs that were not found.
found_locations dict[str, tuple]

Map from filename to source-specific location info. - Image folder: {filename: filepath} - Zarr: {filename: (store_path_str, local_index)} - Cutana: {source_id: (catalogue_path_str, row_index)}
partial bool

True when the validator bailed out before scanning every catalogue (e.g. because stale_check reported the request was superseded). When True, found/missing/ found_locations reflect only the catalogues processed so far and must not be cached or used to build a labeled cache.

validate_labeled_files(source_dir, label_df, source_type, *, stale_check=None)

Check which labeled files/IDs exist in the data source.

Dispatches to source-specific validation. Never builds a full index of all N items in the source -- only checks the labeled subset.

Parameters:

Name Type Description Default
source_dir str

Path to the data directory or file.

 required
label_df DataFrame

DataFrame with id and label columns.

 required
source_type str | DataSourceType

A :class:DataSourceType value (or its string equivalent).

 required
stale_check

Optional zero-arg callable returning True when this validation request has been superseded. Only the Cutana path inspects it (the others are fast).

 None

Returns:

Type Description
ValidationResult

ValidationResult with found/missing lists and location info.

Raises:

Type Description
ValueError

If source_type is not recognised.
Source code in anomaly_match/data_io/source_validation.py 
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
def validate_labeled_files(
    source_dir: str,
    label_df: pd.DataFrame,
    source_type: str | DataSourceType,
    *,
    stale_check=None,
) -> ValidationResult:
    """Check which labeled files/IDs exist in the data source.

    Dispatches to source-specific validation. Never builds a full index
    of all N items in the source -- only checks the labeled subset.

    Args:
        source_dir: Path to the data directory or file.
        label_df: DataFrame with ``id`` and ``label`` columns.
        source_type: A :class:`DataSourceType` value (or its string equivalent).
        stale_check: Optional zero-arg callable returning ``True`` when
            this validation request has been superseded.  Only the
            Cutana path inspects it (the others are fast).

    Returns:
        ValidationResult with found/missing lists and location info.

    Raises:
        ValueError: If *source_type* is not recognised.
    """
    if source_type == DataSourceType.IMAGE_FOLDER:
        return _validate_image_folder(source_dir, label_df)
    elif source_type == DataSourceType.ZARR:
        return _validate_zarr(source_dir, label_df)
    elif source_type == DataSourceType.CUTANA:
        return _validate_cutana(source_dir, label_df, stale_check=stale_check)
    else:
        raise ValueError(f"Unknown source type: {source_type!r}")