Orchestrator
cutana.orchestrator
¶
Orchestrator module for Cutana - manages process spawning and delegation.
This module handles: - Delegation of sourceIDs/fitstile to cutout processes - Spawning processes to create cutouts in the background - Respecting system memory limitations and CPU cores - Progress tracking and status reporting - Workflow resumption capability
Orchestrator(config, status_panel=None)
¶
Main orchestrator for managing cutout creation workflows.
Handles process spawning, resource management, and progress tracking while respecting system limitations.
Initialize the orchestrator with configuration.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
config
|
DotMap
|
Configuration DotMap containing workflow parameters |
required |
status_panel
|
Optional reference to UI status panel for direct updates |
None
|
start_processing(catalogue_path)
¶
Start the main cutout processing workflow using streaming catalogue loading.
Uses memory-efficient streaming for catalogues of any size (supports 10M+ sources): 1. Builds a lightweight index (FITS set to row indices mapping) 2. Validates a sample of the catalogue 3. Reads only the specific rows needed for each batch on-demand
Memory usage: O(index_size) + O(batch_size) instead of O(catalogue_size)
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
catalogue_path
|
str
|
Path to catalogue file (CSV or Parquet) |
required |
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dictionary containing workflow results and status |
get_progress()
¶
Get current progress and status information.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dictionary containing progress information |
get_progress_for_ui(completed_sources=None)
¶
Get progress information optimized for UI display.
Returns a clean ProgressReport dataclass with all relevant information for the status panel to display, including LoadBalancer resource information.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
completed_sources
|
int
|
Use specific completed_sources value instead of recalculating |
None
|
Returns:
| Type | Description |
|---|---|
ProgressReport
|
ProgressReport containing UI-relevant progress information |
stop_processing()
¶
Stop all active subprocesses gracefully.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dictionary containing stop operation results |
run()
¶
Run the orchestrator main loop. Meant for backend usage to be called after orchestrator creation.
Uses streaming catalogue loading for memory-efficient processing of large catalogues.
Returns:
| Type | Description |
|---|---|
Dict[str, Any]
|
Dict[str, Any]: The final status report after running the orchestrator. |
cleanup()
¶
Clean up resources.
__del__()
¶
Ensure cleanup when object is destroyed.
Destructors must never raise (Python will print a warning and swallow it anyway), but we surface the failure via the logger so the root cause is visible rather than silently discarded.