Skip to content

db_location

anomaly_match.prediction.db_location

Resolve where the live predictions.db lives and keep the session durable.

WAL-mode SQLite always keeps its -wal/-shm sidecars next to the main .db file, so the whole database must sit on fast local storage to avoid the WAL bloat a network filesystem (NFS) suffers under a continuous UI reader (a slow read pins a reader snapshot, no checkpoint can advance past it, and the WAL grows without bound — it reached 11 GB in practice).

This module is the single source of truth for the live DB path. When output_dir is on a network filesystem it transparently relocates the DB to a local-disk scratch directory; :func:prediction_db_path is a pure function of the config, so every process (the prediction subprocess, the UI poller, the orchestrating Session) resolves the same location regardless of call order. :func:prepare_local_db and :func:snapshot_db_to_session keep the portable session directory seeded and durable around a relocated run.

is_network_filesystem(path)

Report whether path is backed by a known network filesystem.

Parameters:

Name Type Description Default
path str

Filesystem path to classify.

required

Returns:

Type Description
bool

True for NFS/CIFS/Lustre/etc., False for local storage or when

bool

the type cannot be determined (e.g. non-Linux platforms).

Source code in anomaly_match/prediction/db_location.py
def is_network_filesystem(path: str) -> bool:
    """Report whether *path* is backed by a known network filesystem.

    Args:
        path: Filesystem path to classify.

    Returns:
        ``True`` for NFS/CIFS/Lustre/etc., ``False`` for local storage or when
        the type cannot be determined (e.g. non-Linux platforms).
    """
    return _filesystem_type(path) in _NETWORK_FSTYPES

prediction_db_path(cfg, output_dir=None)

Return the path to the live predictions.db for cfg.

Pure function of the config — all writers and readers call this so they agree on one location. See the module docstring for the relocation policy.

Parameters:

Name Type Description Default
cfg

Session configuration (DotMap).

required
output_dir str | None

Override for the session dir (e.g. a UI chooser selection); defaults to cfg.output_dir.

None

Returns:

Type Description
str

Absolute-or-relative path to predictions.db.

Source code in anomaly_match/prediction/db_location.py
def prediction_db_path(cfg, output_dir: str | None = None) -> str:
    """Return the path to the live ``predictions.db`` for *cfg*.

    Pure function of the config — all writers and readers call this so they
    agree on one location. See the module docstring for the relocation policy.

    Args:
        cfg: Session configuration (DotMap).
        output_dir: Override for the session dir (e.g. a UI chooser selection);
            defaults to ``cfg.output_dir``.

    Returns:
        Absolute-or-relative path to ``predictions.db``.
    """
    return os.path.join(_effective_db_dir(cfg, output_dir), _DB_FILENAME)

is_relocated(cfg, output_dir=None)

Whether the live DB lives outside the session dir (in local scratch).

Returns:

Type Description
bool

True when relocated to local scratch, else False.

Source code in anomaly_match/prediction/db_location.py
def is_relocated(cfg, output_dir: str | None = None) -> bool:
    """Whether the live DB lives outside the session dir (in local scratch).

    Returns:
        ``True`` when relocated to local scratch, else ``False``.
    """
    session_dir = output_dir if output_dir else cfg.output_dir
    return os.path.normpath(_effective_db_dir(cfg, output_dir)) != os.path.normpath(session_dir)

session_db_path(cfg, output_dir=None)

Path to the durable predictions.db snapshot in the session dir.

This is the portable copy that survives a relocated run (see :func:snapshot_db_to_session); it equals :func:prediction_db_path when the DB is not relocated. Centralises the filename so callers don't hard-code it.

Parameters:

Name Type Description Default
cfg

Session configuration (DotMap).

required
output_dir str | None

Override for the session dir; defaults to cfg.output_dir.

None

Returns:

Type Description
str

Path to predictions.db inside the session directory.

Source code in anomaly_match/prediction/db_location.py
def session_db_path(cfg, output_dir: str | None = None) -> str:
    """Path to the durable ``predictions.db`` snapshot in the session dir.

    This is the portable copy that survives a relocated run (see
    :func:`snapshot_db_to_session`); it equals :func:`prediction_db_path` when
    the DB is not relocated. Centralises the filename so callers don't hard-code
    it.

    Args:
        cfg: Session configuration (DotMap).
        output_dir: Override for the session dir; defaults to ``cfg.output_dir``.

    Returns:
        Path to ``predictions.db`` inside the session directory.
    """
    base = output_dir if output_dir else cfg.output_dir
    return os.path.join(base, _DB_FILENAME)

prepare_local_db(cfg)

Create the local scratch dir and seed it from any session snapshot.

Call once before launching prediction subprocesses. No-op when the DB is not relocated. The seed lets a fresh pod resume a run whose durable copy lives in the (networked) session directory.

Parameters:

Name Type Description Default
cfg

Session configuration; cfg.output_dir is the session dir.

required
Source code in anomaly_match/prediction/db_location.py
def prepare_local_db(cfg) -> None:
    """Create the local scratch dir and seed it from any session snapshot.

    Call once before launching prediction subprocesses. No-op when the DB is not
    relocated. The seed lets a fresh pod resume a run whose durable copy lives in
    the (networked) session directory.

    Args:
        cfg: Session configuration; ``cfg.output_dir`` is the session dir.
    """
    if not is_relocated(cfg):
        logger.debug("Prediction DB not relocated; no local scratch to prepare.")
        return
    local = prediction_db_path(cfg)
    os.makedirs(os.path.dirname(local), exist_ok=True)
    logger.info(
        "Live predictions.db relocated to local scratch {} (session dir {} is on a "
        "network filesystem); snapshotting back to the session after each chunk.",
        os.path.dirname(local),
        cfg.output_dir,
    )
    if os.path.isfile(local):
        # Scratch is pod-local and ephemeral, so a present copy is this pod's own
        # in-progress DB — authoritative over the session snapshot. Don't reseed.
        logger.debug("Local prediction DB already present at {}; not reseeding.", local)
        return
    session = session_db_path(cfg)
    if os.path.isfile(session):
        _copy_db_files(session, local)
        logger.info("Seeded local prediction DB from session snapshot {}.", session)

snapshot_db_to_session(cfg)

Copy the live local DB back into the session dir for durability.

No-op when the DB is not relocated (it already lives in the session). Safe to call between chunks: the chunk subprocess has exited, so the local DB has no open writer.

Parameters:

Name Type Description Default
cfg

Session configuration; the DB is copied into cfg.output_dir.

required
Source code in anomaly_match/prediction/db_location.py
def snapshot_db_to_session(cfg) -> None:
    """Copy the live local DB back into the session dir for durability.

    No-op when the DB is not relocated (it already lives in the session). Safe to
    call between chunks: the chunk subprocess has exited, so the local DB has no
    open writer.

    Args:
        cfg: Session configuration; the DB is copied into ``cfg.output_dir``.
    """
    if not is_relocated(cfg):
        logger.debug("Prediction DB not relocated; session copy is already live.")
        return
    local = prediction_db_path(cfg)
    if not os.path.isfile(local):
        # No local DB yet (no chunk has written one) — nothing to snapshot.
        logger.debug("No local prediction DB at {} to snapshot.", local)
        return
    session = session_db_path(cfg)
    # Best-effort durability: a failed snapshot (e.g. a read-only session dir or
    # a transient NFS error) must not abort the run — the authoritative data is
    # still in the local DB and the next chunk will retry the snapshot. A *run*
    # of failures is escalated to an error so a persistently undurable session is
    # not lost in debug noise.
    global _consecutive_snapshot_failures
    try:
        _copy_db_files(local, session)
        _consecutive_snapshot_failures = 0
        logger.debug("Snapshotted prediction DB to session {}.", session)
    except OSError as exc:
        _consecutive_snapshot_failures += 1
        if _consecutive_snapshot_failures >= _SNAPSHOT_FAILURE_ERROR_THRESHOLD:
            logger.error(
                "Prediction DB snapshot to session {} has failed {} times in a row "
                "(latest: {}). The durable session copy is stale; recent scores live "
                "only on local scratch and would be lost if this pod is reclaimed.",
                session,
                _consecutive_snapshot_failures,
                exc,
            )
        else:
            logger.warning("Could not snapshot prediction DB to session {}: {}", session, exc)