#440 Datastore: lazy hydration with metadata-only setup and transparent content download

Problem

When a new team member clones a repo with an S3/GCS datastore, or CI checks out a fresh copy, the first sync downloads the entire datastore contents — every model's data, outputs, workflow runs, audit logs, telemetry. If the repo has hundreds of models and gigabytes of data, this takes minutes even when the user only needs to run one model.

Key Insight

The data catalog (_catalog.db) backfill reads ONLY metadata.yaml files — never raw content files. Downloading metadata without content gives full catalog visibility (data list, data query, CEL expressions all work) while skipping the expensive bulk content download.

Data file structure per version:

data/{type}/{model-id}/{data-name}/{version}/
  metadata.yaml   ← ~200 bytes, contains all tags/owner/lifecycle info
  raw             ← variable size, the actual content (could be megabytes)
data/{type}/{model-id}/{data-name}/latest  ← few bytes, version pointer

Proposed Solution

Selective pull on setup

When hydrationStrategy: "lazy" is configured, the setup/initial pull downloads only:

• All metadata.yaml files (catalog needs them)
• All latest marker files (catalog needs them for is_latest flag)
• All partition index files (_index/)
• Skips all raw content files

The catalog rebuilds normally from the downloaded metadata. data list, data query, and CEL expressions work immediately.

Transparent content hydration

When a command needs a raw file that wasn't downloaded:

• Model runs / workflow runs: acquireModelLocks → scoped pull → partition file lists all files → pull sees raw missing locally → downloads automatically. Already works with Phase 1+2 machinery, no new code needed.
• data get (read-only, no sync): A hydrateFile hook on UnifiedDataRepository — when getContent() can't find the raw file, calls the hook to download just that file from the remote, then retries the read. Same injection pattern as markDirty.

Framework changes

1. SyncCapabilities gains lazyHydration?: boolean
2. DatastoreSyncService gains optional hydrateFile?(relativePath: string, options?: DatastoreSyncOptions): Promise<boolean>
3. CustomDatastoreConfig gains hydrationStrategy?: "full" | "lazy" (defaults to "full")
4. UnifiedDataRepository gains optional HydrateFileHook constructor parameter — called in getContent() when raw file is missing
5. repo_context.ts wires the hook from sync service to data repository (same pattern as buildMarkDirtyHook)

Extension changes (S3 + GCS)

1. pullChanged with lazy strategy: filter files by suffix — download metadata.yaml and latest, skip raw, create directory structure for skipped files
2. hydrateFile implementation: download a single file from remote by key
3. capabilities() returns { scopedSync: true, lazyHydration: true }

Backward Compatibility

• hydrationStrategy defaults to "full" — zero change for existing users
• "lazy" is opt-in via .swamp.yaml config
• hydrateFile is optional on DatastoreSyncService — old extensions unaffected
• Read-only commands on full-hydration repos: hook never fires (file always exists)

Benchmarking Requirements

The following benchmarks must be run before and after the change to validate performance and correctness.

Setup benchmarks

• Full setup (baseline): total wall time, file count downloaded
• Lazy setup: total wall time, file count downloaded (should be dramatically faster — metadata only)

Data command benchmarks (run AFTER setup, before any model runs)

• data list after lazy setup: wall time, result count (must match full setup count)
• data list after full setup (baseline): wall time, result count
• data query --where "tags.type == 'state'" after lazy setup: wall time, result count (must match full setup)
• data get @type/model dataName after lazy setup (first access): wall time (includes transparent hydration)
• data get @type/model dataName after lazy setup (second access): wall time (cached, should match full setup)
• data get after full setup (baseline): wall time

Model run benchmarks (CI pipeline scenario)

• Full setup → model run: setup + first run total time (baseline)
• Lazy setup → model run (same model): setup + first run total time (should be faster overall)
• Lazy setup → second different model run: run time (slight overhead on first run of each new model)

Cross-repo benchmarks (team collaboration)

• Writer pushes 50 models, reader does full setup + data list: total time (baseline)
• Writer pushes 50 models, reader does lazy setup + data list: total time (faster setup, same results)
• Writer pushes 50 models, reader does lazy setup + runs 1 model: total time (much faster — only 1 model hydrated)

Scale parameters

• 50 models, ~20 data artifacts each (1000 total)
• Each raw file: ~5KB
• Each metadata.yaml: ~200 bytes
• Total raw content: ~5MB, total metadata: ~200KB
• Lazy setup downloads ~200KB instead of ~5.2MB

Dependencies

• Phase 1 (#378) — framework contracts (shipped)
• Phase 2 (#379, #434) — S3/GCS extension overhaul with partitioned index and scoped sync (shipped)
• #436 — dirty-only partition writes (shipped)

Supersedes #380.