Stale .ccd_codes_cache: components added under an existing letter dir are silently invisible

[ssiddhantsharma]

Summary

get_available_ccd_codes_in_mirror can return a stale .ccd_codes_cache, so a CCD component that is added (or edited) in the mirror after the cache was written is silently invisible — with no error. Any code path that resolves that component then falls back as if it did not exist (e.g. a downstream tool that reads the mirror gets no reference conformer for it).

Version: atomworks==2.2.0.

Root cause

The freshness check compares the cache-file mtime against the mtime of the mirror root only:

https://github.com/RosettaCommons/atomworks/blob/main/src/atomworks/io/utils/ccd.py — get_available_ccd_codes_in_mirror:

cache_file = os.path.join(root, ".ccd_codes_cache")
if os.path.exists(cache_file):
    cache_mtime = os.path.getmtime(cache_file)
    dir_mtime   = os.path.getmtime(root)      # <-- only the mirror ROOT
    if cache_mtime > dir_mtime:
        ... return cached codes ...

Two filesystem facts make this incorrect:

1. Adding a component at root/<letter>/<code>/<code>.cif updates the mtime of root/<letter>/ (and root/<letter>/<code>/), but not root/ itself. So dir_mtime does not change when a component is added under an existing first-letter directory. (root/ changes only when a brand-new first-letter directory is created.)
2. Rewriting root/.ccd_codes_cache in place updates the file's mtime but not root/'s mtime (the directory entry is unchanged). So after the cache has been written more than once, cache_mtime > dir_mtime holds permanently.

Together, once the cache has been (re)written, every component later added under an existing letter directory — and every in-place edit of an existing component — is missed until .ccd_codes_cache is deleted by hand.

Minimal repro

import os, sys, time, tempfile, shutil
from atomworks.io.utils.ccd import get_available_ccd_codes_in_mirror as codes

def add_component(root, code):
    d = os.path.join(root, code[0], code); os.makedirs(d, exist_ok=True)
    open(os.path.join(d, f"{code}.cif"), "w").write(f"data_{code}\n")

def build(root):
    codes.cache_clear()                 # simulate a fresh process; the file cache is the cross-process layer
    return codes(root)

root = tempfile.mkdtemp(prefix="ccd_mirror_"); cache = os.path.join(root, ".ccd_codes_cache")
try:
    add_component(root, "AAA")          # creates root/A/  (bumps root mtime)
    time.sleep(1.1); build(root)        # 1st build: CREATES the cache file
    time.sleep(1.1); build(root)        # 2nd build: OVERWRITES cache in place -> cache_mtime now > root mtime
    add_component(root, "AAB")          # 2nd component under the EXISTING root/A/  -> root mtime unchanged
    second = build(root)
    print("codes with stale cache present:", sorted(second))
    os.remove(cache); third = build(root)
    print("codes after removing the cache :", sorted(third))
    assert "AAB" not in second and "AAB" in third, "no bug observed"
    print("BUG: AAB was invisible until .ccd_codes_cache was deleted")
finally:
    shutil.rmtree(root, ignore_errors=True)

Output on atomworks==2.2.0:

codes with stale cache present: ['AAA']
codes after removing the cache : ['AAA', 'AAB']
BUG: AAB was invisible until .ccd_codes_cache was deleted

Impact

This is a silent-correctness bug, not a crash. A workflow that registers a custom component in a mirror (via CCD_MIRROR_PATH) in one process and reads it back in a later process will intermittently not see the component, depending on prior cache state — and the failure surfaces far downstream (e.g. a missing/zeroed reference conformer), not here.

Suggested fixes (in rough order of cost)

1. Cheap, catches add/remove: compare against the newest mtime of the root and its first-letter subdirectories, e.g. dir_mtime = max([os.path.getmtime(root)] + [os.path.getmtime(d) for d in root_path.iterdir() if d.is_dir() and len(d.name) == 1]). Detects any added/removed component (the common case); still misses an in-place edit of an existing <code>.cif.
2. Correct, ~scan-cost: invalidate on a directory signature computed during the scan the function already does — e.g. store the sorted (code) set plus a hash of the component-file mtimes/sizes, and cheaply recompute it. (A full os.walk max-mtime is O(N), i.e. the same order as the scan the cache is meant to avoid.)
3. Simplest & safe: given get_available_ccd_codes_in_mirror is already @functools.cached per process, consider dropping the on-disk cache (a two-level iterdir is fast) or gating it behind an opt-in / an env flag, so it can't go stale silently.