Updated 31 Oct 06:57:32 BKK by kookwatch. Alert sender (leader): kookwatch. Open problems that every layer failed to fix: 5
- kookwatch: machine - disk 87% used, 4301 MB RAM free, load 6.25 (bad 147.6 h; no automatic fix exists)
- kookwatch: M runtrack snapshots freshness (Test Manager, from super) - 311.9 h old (max 75 min) (bad 310.7 h; 1812 fix(es), last: restart runtrack.service (rc 0))
- kookwatch: M 10 s holder and buy/sell tape freshness (Test Manager, from super) - 296.6 h old (max 30 min) (bad 296.1 h; 1727 fix(es), last: restart rhfast.service (rc 0))
- kookwatch: M Picker retrain freshness (Test Manager, from super) - 374.0 h old (max 36.0 h) (bad 338.0 h; 1971 fix(es), last: restart hist-retrain.service (rc 0))
- kookwatch: M Launch catch-up (reconcile) freshness (Test Manager, from super) - 336.1 h old (max 10 min) (bad 335.9 h; 1959 fix(es), last: restart reconcile.timer (rc 0))
Layer 1 = each box's own guard (twice a minute). Layer 2 = the other boxes call its entry over SSH every minute (revive guard, start dead units).
Telegram only when both fail. To stop a unit on purpose: systemctl disable --now X or touch /opt/guard/hold/X.
| firstmin-trader - local guard 4 s ago | |||
| AI pick feed reaches this box | OK | 0 s old | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 2 min old | |
| L live-change log cron job | OK | 1 jobs present | |
| LIVE bot fmmc5 (real money) | OK | running, heartbeat 2 s | |
| S stop rule: 10 losses in a row (paper/live test 2 only; Ben 8 Oct: none for fmmc5) | OK | 1 jobs present | |
| dashboard pnl.html (live) | OK | 22 s old | |
| dashboard pnl2.html | OK | 25 s old | |
| dashboard pnl2v.html | OK | 25 s old | |
| dashboard tests.json | OK | 26 s old | |
| gmgnd (live sells) running | OK | running | |
| gmgnd answers | OK | answers ping {"warmOk": 34704, "warmErr": 7, "lastWarmMs": 216} | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 38% used, 269 MB RAM free, load 2.85 | |
| outside-money card | OK | 2 min old | |
| paper test 1 fml2 | OK | running, heartbeat 1 s | |
| paper test 1b fml2v | OK | running, heartbeat 1 s | |
| safety + dashboard cron jobs | OK | 12 jobs present | |
| kookwatch - local guard 29 s ago | |||
| 60 s AI picker (feeds paper tests) | OK | running, heartbeat 0 s | |
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 39 min old | |
| L ROT chains.json from rot-box (Manager) | OK | 21 min old | |
| L ROT closes.json from rot-box (Manager) | OK | 51 min old | |
| L ROT map-data.json from rot-box (Manager) | OK | 5 min old | |
| L ROT rh_drill.json from rot-box (Manager) | OK | 41 min old | |
| L ROT rotation.json from rot-box (Manager) | OK | 35 min old | |
| L ROT rotmap.json from rot-box (Manager) | OK | 3.5 h old | |
| L ROT sync main<->rot-box (Manager) | OK | 54 s old | |
| L judge guard ok (PP Data) | OK | 17 s old | |
| L lane check ok (PP Data) | OK | 5 s old | |
| L lane price tracker (PP Data) | OK | 3 min old | |
| L lane stats page (PP Data) | OK | 6 min old | |
| L live-change log cron job | OK | 1 jobs present | |
| L nightly review cron job | OK | 1 jobs present | |
| L nightly review page | OK | 23.3 h old | |
| M 10 s holder and buy/sell tape freshness (Test Manager, from super) | BAD | 296.6 h old (max 30 min) | 1727 fix(es): restart rhfast.service (rc 0) |
| M AI picker round 5 (paper) freshness (Test Manager, from super) | OK | 0 s old | |
| M Ben Telegram buttons (benbot) | OK | running | |
| M Ben Telegram line in cron | OK | 2 jobs present | |
| M Launch catch-up (reconcile) freshness (Test Manager, from super) | BAD | 336.1 h old (max 10 min) | 1959 fix(es): restart reconcile.timer (rc 0) |
| M Live job checker (sentinel) freshness (Test Manager, from super) | OK | 29 s old | |
| M PP Feeds layer-2 guard (ppfeedguard2) | OK | running, heartbeat 17 s | |
| M PP data cron lines (PP Data) | OK | 3 jobs present | |
| M PP data liquidity table (liqinc) (PP Data) | OK | 33 min old | |
| M PP data picker-box copy (pbsync pull) (PP Data) | OK | 11 s old | |
| M PP data ponstape (PP Data) | OK | running | |
| M PP data ppexit honest-exit tracker (PP Data) | OK | 1 s old | |
| M PP data venuepick paper picker (PP Data) | OK | 0 s old | |
| M PP feed GMGN board (price-predict-rh) | - | disabled on purpose | |
| M PP feed crewlive (crews) | OK | running, heartbeat 11 s | |
| M PP feed klines | OK | running, heartbeat 2 s | |
| M PP feed ponstape (Pons curve) | OK | running, heartbeat 0 s | |
| M PP feed rhledger | OK | running | |
| M PP feed runtrack (runners) | OK | heartbeat 311.9 h old but log live (not hung) | |
| M Picker guard (pickguard) freshness (Test Manager, from super) | OK | 17 s old | |
| M Picker retrain freshness (Test Manager, from super) | BAD | 374.0 h old (max 36.0 h) | 1971 fix(es): restart hist-retrain.service (rc 0) |
| M SW engine w2coder (paper) | OK | running, heartbeat 14 s | |
| M bot watchdog freshness (Test Manager, from super) | OK | 4 min old | |
| M cleaner freshness (Test Manager, from super) | OK | 6 min old | |
| M daily miss review in cron | OK | 1 jobs present | |
| M gmgn feeds recorder freshness (Test Manager, from super) | OK | 99 s old | |
| M instant miss watch ran | OK | 108 s old | |
| M lane price tracker (lanetrack) freshness (Test Manager, from super) | OK | 3 min old | |
| M lane radar v3 (laneradar3) freshness (Test Manager, from super) | OK | 0 s old | |
| M lane radar v3 beat (PP Data) | OK | 0 s old | |
| M lane stats page (lanestats) freshness (Test Manager, from super) | OK | 6 min old | |
| M meta radar (metaradar) freshness (Test Manager, from super) | OK | 104 min old | |
| M paper test (pp-paper) freshness (Test Manager, from super) | OK | 102 s old | |
| M picker learning loop cron (hourly) | OK | 1 jobs present | |
| M ppinc hourly new pools + supply (PP Data) | OK | 15 min old | |
| M rate-limit review in cron | OK | 2 jobs present | |
| M rate-limit review ran (heartbeat) | OK | 31 min old | |
| M runtrack snapshots freshness (Test Manager, from super) | BAD | 311.9 h old (max 75 min) | 1812 fix(es): restart runtrack.service (rc 0) |
| S AI picker (paper) freshness (Test Manager, from super) | OK | 0 s old | |
| S P3 mirror cross (a1pull@cross) | OK | running | |
| S P3 mirror early (a1pull@early) | OK | running | |
| S backup checker (livewatch2) freshness (Test Manager, from super) | OK | 17 s old | |
| S fast checker (livewatch) freshness (Test Manager, from super) | OK | 1 s old | |
| S fast checker livewatch | OK | running, heartbeat 1 s | |
| S live scanner rhfast (feeds Price Predict) | - | disabled on purpose | |
| S live settings watchdog | OK | 1 jobs present | |
| S live settings watchdog ran | OK | 53 s old | |
| backup picker aipick5b stays off | OK | off (as decided) | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| machine | BAD | disk 87% used, 4301 MB RAM free, load 6.25 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 32 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| pick feed file l2picks.json | OK | 0 s old | |
| pick feed l2feed | OK | running, heartbeat 0 s | |
| published tests dashboard | OK | 51 s old | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |
| testspub cron job | OK | 1 jobs present | |
| picker-box - local guard 5 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 40 min old | |
| M lab paper books | - | on hold (/opt/guard/hold/labbooks) | |
| M shadow picker pbshadow | OK | running | |
| free RPC relay (2nd IP) | OK | running | |
| guard triggers | OK | cron + timer present | |
| machine | OK | disk 34% used, 494 MB RAM free, load 0.40 | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| research-box - local guard 5 s ago | |||
| L Git snapshot cron job | OK | 1 jobs present | |
| L Git snapshot ran | OK | 40 min old | |
| L kitb | OK | 7 min old | |
| L shadow grid report (Test Manager, nightly 02:30) | OK | 3.4 h old | |
| L trending feed | OK | 29 s old | |
| M KITFAST kit feed (PP Feeds) | OK | running, heartbeat 8 s | |
| M PP Feeds layer-1 guard (ppfeedguard) | OK | running, heartbeat 5 s | |
| M auditq ran | OK | 7 min old | |
| M crew tagger pretag | OK | running, heartbeat 23 s | |
| M datajob runner ran | OK | 88 s old | |
| M flowwatch ran | OK | 5 min old | |
| M ideaq ran | OK | 4 min old | |
| M killbot ran | OK | 46 min old | |
| M paper cards ran | OK | 2 min old | |
| M picker learning loop ran (hourtrain, hourly) | OK | 4 min old | |
| M v3 live feed v3live | OK | running, heartbeat 1 s | |
| M v3 mc5 paper book | OK | running, heartbeat 0 s | |
| S A1 early + cross feed (a1feed) | OK | running, heartbeat 0 s | |
| drpc_paid.json is free | OK | points to robinhood.drpc.org | |
| guard triggers | OK | cron + timer present | |
| history rebuild moving (one-time, paid) | OK | last write 27 min ago | |
| machine | OK | disk 88% used, 10937 MB RAM free, load 5.28 | |
| paid key: no stray copies | OK | paid key only in the opt-in files (scanned 53 min ago) | |
| paid key: only allowed processes | OK | only allowed processes use the paid key | |
| rebuild moves to free when done (cron) | OK | 1 jobs present | |
| rpc_keys.json is free | OK | points to robinhood.drpc.org | |
| rpcpool.py healthy | OK | compiles | |