Log learning: find execution, failure, and acceptance in the noise

Local logs are teaching material, but they must be classified first; WARNING counts are not a result

Published on

July 13, 2026

Log inventory

The table below is the file inventory from this round's actual scan; it shows both "there is a lot of material" and that not every log can be stuffed into one prompt.

Source pipeline sources configs main/error logs main/error bytes command/run/replay logs command/run/replay bytes
XARTATOMS 0900170101 3 11 178 8,099,898 119 15,714,109
M31CGM data tree 4 3 16 369,293,176 8 352
M31HotISM project 7 19 12 187,510 6 264
upstream ipynb tree 102 historical files 10 452 107,190,448 170 329,774

These numbers are for selecting learning material, not a science result. The 102 upstream files include historical versions, backups, and notebook artifacts; count must not be read as quality.

First split logs into five classes

Class What to record Example What it cannot imply
launch log env、engine/config hash、command、PID XART clean-shell wrapper products passed
engine log stage、SAS command、region conversion、runtime warnings LOG_esas_playground... every WARNING is a real failure
error log task stderr / exception M31CGM epchain / shell function import the whole run is unusable
status pipeline RC、wrapper RC、state、timestamp XART .status; M31HotISM supervisor status a single RC replaces product QA
qualification FITS/product checks、hash、QPB/response、visual QA XART JSON; M31HotISM 84 gates fitting/physics already holds

Case 1 — XARTATOMS: pipeline RC 0, but wrapper RC 1 (XART-E04 + status)

Representative log head:

4 passed in 0.27s
PILOT_RELEASE_HASH_GATE_PASS
<engine SHA-256>  esas_playground_v1.4.1.py
LAUNCH_REMAINING_SEVEN_MOS2_PIPELINE

engine tail:

PIPELINE_RC 0
STATUS_FILE ...remaining_seven...status

the status actually reads:

pipeline_rc=0
wrapper_rc=1
state=failed

Correct interpretation

  • the engine worked through the regions and returned 0;
  • the wrapper expected 7 output namespaces, of which S8 has only CCD1 and no QPB outer anchor;
  • the wrapper fail-closed with return code 1;
  • subsequent inventory/qualification proved the six new regions usable and S8 unavailable; with the Wind N pilot the final tally is 7/8 qualified.

So: wrapper_rc=1 is not "all extractions failed", and pipeline_rc=0 is not "all regions passed". You must read the region-level qualification.

Logs that could improve

The tail of that same 1.79 MB wrapper log is mostly hardlink size/no-space-saved output; only the last few lines actually decide status. Future logs should split into:

  • launch.log
  • engine.log
  • region_status.jsonl
  • qualification.json

Do you agree the wrapper should return overall RC=1 when some regions are unavailable? Or should it return RC=0 + an explicit PARTIAL state, as long as all available regions are qualified?

Case 2 — M31CGM: 49,609,621-byte single log (CGM-E03)

The log head tells us precisely:

obsID : 0800732701
mode : redu
cifbuild : yes
odfingest : yes
emchain : yes
epchain : yes
2.filter : yes
edetect_stack : yes
3B.background : yes
background : yes

This proves it is a full reduction, not a single-annulus extraction.

But the same log also contains:

  • the complete environment variables;
  • every parameter of every SAS task;
  • repeated module/ml/which shell-function import warnings;
  • large volumes of hardlink size metrics;
  • early/mid-run error content;
  • no independent product qualification marker at the tail.

Learning value

  • can reconstruct the real cluster environment and failures;
  • can show which stages redu ran;
  • serves as the counterexample of "which logs should never be printed in full again";
  • allows picking one golden full-reduction log and one known-failure log.

What not to do

You cannot declare an observation usable by searching for the last ERROR or spotting a few *-back.pi. The log should be distilled into:

{
  "obsid": "0800732701",
  "mode": "redu",
  "stages": {},
  "tasks": [],
  "products": [],
  "fatal_error": null,
  "qualification": "not recorded"
}

Please name one M31CGM log you consider a success and one you consider a failure. Hermes will learn from your labels, not guess from 49 MB of text itself.

Case 3 — M31HotISM: pipeline 0, supervisor bug, verifier rerun PASS (HOTISM-E009-QA)

The real status:

pipeline_exit_status=0
postrun_supervisor_failure=zsh_reserved_status_variable
manual_repair=link_then_verify
manual_verifier_exit_status=1
manual_verifier_rerun_exit_status=0

The final verifier:

status=PASS
checks_total=84
checks_failed=0

Correct interpretation

  • the extraction engine succeeded;
  • the postrun supervisor itself had a shell variable bug;
  • the first verifier had a false negative / link handling issue;
  • after the fix the same real product set passed 84/84;
  • the attempt count must not be read as a science workflow, and this audit history must not be deleted.

Log design worth keeping

  • attempt directories never overwrite;
  • rejected attempts are explicitly quarantined;
  • final verification is an independent JSON;
  • source/QPB/response/hash are recorded together with the exact region.

What should move out of the routine runner

  • compatibility recovery;
  • engine runtime copy;
  • manual link repair;
  • zsh-specific supervisor workaround.

Of M31HotISM's 84 gates, which are your universal contract? If only 15 could be kept, list them by priority.

Case 4 — upstream engine log: clear bootstrap noise first

The v1.4.1 production log still shows at startup:

  • test INFO/ERROR/CRITICAL;
  • touch test.txt;
  • PATH/environment setup expansion;
  • UnicodeEncodeError on Chinese log lines when the locale is ASCII;
  • long final rProfile/sync output.

These do not automatically indicate a science product error, but they bury real failures. The engine logger should emit structured events:

{"stage":"region","camera":"mos2","region":"R4-7_S3","event":"start"}
{"stage":"mos-spectra","camera":"mos2","region":"R4-7_S3","rc":0}
{"stage":"mos_back","camera":"mos2","region":"R4-7_S3","rc":0}
{"stage":"product","camera":"mos2","region":"R4-7_S3","state":"qa_pending"}

How Hermes should read logs

  1. read wrapper/status first to fix run boundaries;
  2. then the config snapshot to fix mode and regions;
  3. grab only each SAS task's command, RC, fatal exception;
  4. build a camera×region ledger;
  5. run the product validator on every output;
  6. only then decide PASS / PARTIAL / REJECT;
  7. keep the original log hashes; never replace evidence with summaries.
WarningCredential hygiene

A hard-coded notification credential exists in the M31CGM historical job script. The site and downloads write only [REDACTED]; any future training set must run a secret scan first.

Downloads

中文