Catalogue preprocessor
cutana.catalogue_preprocessor
¶
Catalogue preprocessing and validation functions for Cutana.
Provides functionality to validate, preprocess, and analyze source catalogues, including comprehensive data validation, FITS file checking, and metadata extraction.
CatalogueValidationError
¶
Bases: Exception
Exception raised when catalogue validation fails.
extract_fits_sets(fits_files, filters=None)
¶
Extract FITS sets from a list of FITS files and determine resolution ratios.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fits_files
|
List[str]
|
List of FITS file paths |
required |
filters
|
List[str]
|
Optional list of filter names for resolution checking |
None
|
Returns:
| Type | Description |
|---|---|
Dict[tuple, List[str]]
|
Tuple of (fits_set_dict, resolution_ratios) where: |
Dict[str, float]
|
|
Tuple[Dict[tuple, List[str]], Dict[str, float]]
|
|
extract_filter_name(filename)
¶
Extract filter name from FITS filename. This is a Euclid specific function, based on the namings of the extensions.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filename
|
str
|
FITS file path or name |
required |
Returns:
| Type | Description |
|---|---|
str
|
Filter name (e.g., 'VIS', 'NIR-Y', 'NIR-H'), or 'UNKNOWN' when no Euclid pattern |
str
|
matches. |
str
|
'UNKNOWN' is deliberately a constant. |
str
|
whole run, so a channel label has to mean the same thing in every row; anything |
str
|
derived from the filename of an unrecognised tile is a tile identity and |
str
|
changes row to row, which makes the catalogue's rows disagree about their own |
str
|
channels. A band token is the only per-file thing that is stable across a survey, |
str
|
and recognising one is exactly what this function does or fails to do. |
str
|
One 'UNKNOWN' channel in a row is workable: it is a single channel, so |
str
|
|
str
|
collapse onto the same label and are refused, because they are genuinely |
str
|
indistinguishable to weights and to the WCS lookup. |
analyze_fits_file(fits_path)
¶
Analyze a FITS file and return extension information.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fits_path
|
str
|
Path to FITS file |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dictionary with extension information |
validate_catalogue_columns(catalogue_df)
¶
Validate that required columns exist and have correct types.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_df
|
DataFrame
|
DataFrame to validate |
required |
Returns:
| Type | Description |
|---|---|
List[str]
|
List of validation errors (empty if valid) |
validate_coordinate_ranges(catalogue_df)
¶
Validate RA, Dec, and size values are in expected ranges.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_df
|
DataFrame
|
DataFrame to validate |
required |
Returns:
| Type | Description |
|---|---|
List[str]
|
List of validation errors (empty if valid) |
validate_resolution_ratios(catalogue_df)
¶
Validate that if diameter_pixel is used with multiple filters, resolution ratios are acceptable.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_df
|
DataFrame
|
DataFrame to validate |
required |
Returns:
| Type | Description |
|---|---|
List[str]
|
List of validation errors (empty if valid) |
check_fits_files_exist(catalogue_df)
¶
Check if FITS files exist. Smart checking based on number of unique files.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_df
|
DataFrame
|
DataFrame with fits_file_paths column |
required |
Returns:
| Type | Description |
|---|---|
Tuple[List[str], List[str]]
|
Tuple of (errors, warnings) lists |
preprocess_catalogue(catalogue_df)
¶
Preprocess catalogue by resetting index and any other required operations. Ensures SourceID column is converted to string type.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_df
|
DataFrame
|
Input DataFrame |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
Preprocessed DataFrame with reset index and string SourceID |
Raises:
| Type | Description |
|---|---|
CatalogueValidationError
|
If rows share both SourceID and position, leaving no way to key them apart during cutout extraction. |
load_catalogue(catalogue_path)
¶
Load catalogue from file without validation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_path
|
str
|
Path to catalogue file (CSV, FITS, or parquet) |
required |
Returns:
| Type | Description |
|---|---|
DataFrame
|
DataFrame |
Raises:
| Type | Description |
|---|---|
ValueError
|
If file format is unsupported |
NotImplementedError
|
If file format is not yet implemented (parquet) |
stream_catalogue_chunks(path, batch_size=100000, columns=None)
¶
Stream catalogue in chunks for memory-efficient processing.
Works with both CSV and Parquet formats. For parquet, uses pyarrow's iter_batches for true streaming. For CSV, uses pandas chunksize.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
Path to catalogue file (CSV or Parquet) |
required |
batch_size
|
int
|
Number of rows per chunk |
100000
|
columns
|
Optional[List[str]]
|
Optional list of columns to load (None = all columns) |
None
|
Yields:
| Type | Description |
|---|---|
DataFrame
|
DataFrame chunks with '_row_idx' column added for tracking |
Raises:
| Type | Description |
|---|---|
ValueError
|
If file format is unsupported |
validate_catalogue_sample(path, sample_size=10000, skip_fits_check=False)
¶
Validate a sample from the catalogue without loading it fully.
Streams through the catalogue and validates column types, coordinate ranges, and optionally FITS file existence on a sample.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
path
|
str
|
Path to catalogue file |
required |
sample_size
|
int
|
Number of rows to sample for validation |
10000
|
skip_fits_check
|
bool
|
Skip FITS file existence checking |
False
|
Returns:
| Type | Description |
|---|---|
List[str]
|
List of validation errors (empty if valid) |
load_and_validate_catalogue(catalogue_path, skip_fits_check=False)
¶
Load catalogue from file and perform comprehensive validation.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_path
|
str
|
Path to catalogue file (CSV or FITS) |
required |
skip_fits_check
|
bool
|
Skip FITS file existence checking (for testing) |
False
|
Returns:
| Type | Description |
|---|---|
DataFrame
|
Validated and preprocessed DataFrame |
Raises:
| Type | Description |
|---|---|
CatalogueValidationError
|
If validation fails |
analyse_source_catalogue(catalogue_path)
¶
Analyze a source catalogue and return comprehensive metadata. This function combines validation and analysis functionality.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_path
|
str
|
Path to catalogue file (CSV or FITS) |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dictionary containing analysis results |
Raises:
| Type | Description |
|---|---|
CatalogueValidationError
|
If validation fails |