Problem
Observability surfaces are still wired to legacy pipeline_runs data, but the scheduler now tracks active runs in scheduler-state.json and agentd logs.
Current mismatch:
/pipeline web API/dashboard reads pipeline_runs from state.dbpipeline digest email also reads pipeline_runs- scheduler run loop does not insert/update
pipeline_runs
Observed symptom:
- dashboard reports stale active run(s) from historical records
- digest metrics remain tied to old data instead of current scheduler activity
Impact
- Operators receive incorrect health signals
- "active runs" can be ghost entries
- daily digest can report misleading shipped/failure counts
Proposed fix
- Define a single source of truth for scheduler run state
- Either:
1) write scheduler lifecycle events into pipeline_runs again, or
2) migrate web/digest to scheduler-native state + agentd run history
- Add cleanup/migration for stale legacy
running rows
Acceptance criteria
- Dashboard/digest reflect current scheduler activity
- No stale/ghost active runs after migration