Extraction tutorial thread: every step can be corrected by you

From observation root to no-fit handoff; every step notes its source, gate, and open teaching question

Published on

July 13, 2026

How to use this thread

This is not a "just run it" cookbook. Every thread block has four parts:

What I learned from the real materials;
How I plan to proceed;
When I must stop;
What you should correct.

As you read, you can fill the same RPC / ODF / region into the interactive configurator, and finally give your standard answers on Teach Hermes.

Runtime boundary

This page discusses the legacy mos-spectra / mos_back invoked by the local xmmsas_20211130_0941 exemplar. Current SAS documentation uses mosspectra / mosback; until the user specifies a golden runtime and migration rules, task names are not auto-replaced and current-SAS compatibility is not claimed. Evidence IDs and official links are on the four-material comparison.

Thread 0 — decide "what do I want to do" first, then choose a mode

XMM/ESAS work involves at least three distinct tasks that must not be conflated into "spectrum extraction":

flowchart LR
  A[Raw ODF] -->|once per OBSID| B[Field-ready RPC]
  B -->|per approved region batch| C[Source + QPB + RMF + ARF]
  C -->|after acceptance| D[No-fit QA / fitting handoff]

Task	Input	Output	Representative material	Routine region extraction?
ODF reduction	raw ODF	calibrated/filter events	M31CGM `mode=redu`	No
field preparation	clean events/source list/masks/WCS	frozen field-ready RPC	XARTATOMS prior field state	No; once per OBSID
C-mode extraction	field-ready RPC + approved region	source/QPB/RMF/ARF	XARTATOMS Phase2 / M31HotISM E009	Yes
product release	extracted products	verifier report + no-fit plots + overlay	XARTATOMS qualification / M31HotISM 84 gates	No; acceptance layer

My rule: a new region does not automatically re-run cifbuild → emchain → filter → edetect_stack. Only when the RPC fails the field-ready contract do we go back to the first two rows.

When you actually use esas_playground, how do you name and distinguish these three tasks? What should redu, C, full, resume each be responsible for?

Thread 1 — identify observation root, ODF, and RPC

Directory conventions I learned

The esas_playground config does not take an RPC absolute path directly; it composes:

datafolder / obsID / odffolder
                     rpcfolder

For example, the interactive page derives from the observation root:

<DATA_ROOT>/<OBSID>/odf
<DATA_ROOT>/<OBSID>/rpc

Minimal pre-launch checklist

OBSID is 10 digits;
the ODF directory actually exists;
SAS_ODF points to a unique, non-empty *SUM.SAS, or the engine can resolve it uniquely;
the RPC directory actually exists;
ccf.cif matches this observation;
HEADAS, SAS, SAS_CCFPATH, ESAS_CCFPATH are explicit;
in this shell, mos-spectra / mos_back resolve to the intended programs;
engine and .par record SHA-256.

Reminders from real materials

the XARTATOMS clean-shell wrapper explicitly checks one SUM, tool paths, and hashes;
in the M31CGM historical log the same observation moves between cluster/home/work paths, showing that configs and runtime environments must be audited together;
M31HotISM used a short RPC to work around HEASOFT long-path problems — a one-off engineering recovery, not a science step.

Should the web configurator's first field be datafolder + OBSID, or is entering the observation root directly more natural for you? Should the ODF be a directory or a specific SUM.SAS?

Thread 2 — decide whether the RPC is field-ready

My provisional field-ready contract

For every camera/exposure you plan to use, at minimum these exist and are readable:

clean event file;
corner events / the anchor QPB needs;
object image and mask image;
attitude / WCS support;
ccf.cif and SUM provenance;
source exclusion / region seed;
flare/residual-SP and anomalous-CCD disposition.

To go straight to the XARTATOMS-style 4.background=yes, you must also confirm which layer the 4.background support state comes from (2.filter or 2.1.filter) — never guessed from an implicit copy.

Stop conditions

a camera-region has only CCD1 and no outer-corner QPB anchor;
a CCD you plan to use is excluded by an anomalous state;
the WCS of clean event / mask / obj images disagree;
only parent/other-region QPB exists, with no region-specific QPB generation path.

XARTATOMS S8 is the correct example: unavailable, not a forced spectrum.

Please define the authoritative checklist for a field-ready RPC. Which files will esas_playground fill in itself, and which missing ones must stop the run?

Thread 3 — source exclusion and science region are two things

source exclusion

Purpose: exclude point/extended sources; usually goes into extra_region_file or a camera-specific region mask.

science region

Purpose: bound the annulus/sector/polygon to extract; usually goes into extra_region_list.

[back]
extra_region_file = "<source-exclusion.reg>"
extra_region_list = <science-region-01.reg>,<science-region-02.reg>,
MASK = 1,

Why they must not be mixed

The initial M31HotISM historical mask was one large ROTBOX, not a point-source exclusion mask; it happened to fall inside the science polygon, so old and new spectra were nearly identical. Only after E009 rebuilt a camera-specific point-source mask did BACKSCAL and channel counts change meaningfully (HOTISM-E009 / HOTISM-E009-QA).

What do you now recommend for source exclusion: a direct DS9 exclude region in extra_region_file, or first building a camera-specific XMM region FITS? What exactly is MASK=1 responsible for in your standard config?

Thread 4 — exact DS9 regions: visual approval first, then conversion

Evidence that must be kept

the exact .reg file;
coordinate frame (usually fk5);
region semantics (include/exclude);
camera image + mask + source exclusion overlay;
conversion engine/module/hash;
converted DET expression;
camera×region coverage / active CCD.

Why machine checks alone are not enough

an angle/PA convention can be globally rotated yet still pass closure;
positive/negative wrap can turn a 45° sector into 315°;
an outer annulus can become a huge polygon beyond response support in DET;
file existence alone cannot prove the correct sky region was extracted.

Fixed rule

No spectrum may be extracted from a new spatial transform before visual approval. The page configurator only accepts region strings and produces drafts — it performs no coordinate conversion and claims no geometry approval.

Rejected route, not teaching material

E021's custom FullFOV-clipped DET expression and DETX/DETY visual QA were rejected and retracted by the user. Future inputs must be canonical FK5/RA-Dec regions going through native Mode-C conversion, audited with sky-coordinate region + camera/mask overlays; the late sub-audit's E021 clipping suggestion must not be revived.

Please name the DS9→DET conversion implementation and golden overlay most worth Hermes learning. v1.4.1's ds9_to_xmmexpr, or another validated XARTATOMS converter/replay?

Thread 5 — compile the camera × region matrix

Engines often treat INST × extra_region_list as a Cartesian product. Actual coverage can be:

Region	MOS1	MOS2	Correct batch
R1	READY	READY	dual-MOS batch
R2	SKIP	READY	MOS2-only batch
R3	READY	SKIP	MOS1-only batch

So a single INST=mos1,mos2 config with all regions cannot be used to launch unsupported pairs.

My compilation rules

each camera-region first gets READY / SKIP / BLOCK;
group by camera readiness;
each group produces one homogeneous .par;
BLOCK does not enter extraction;
if any output namespace in the batch already exists, fail closed to avoid overwriting.

How do you usually handle differing camera coverage in esas_playground? Split .par files, let the engine auto-skip, or generate lists after a region-specific CCD preflight?

Thread 6 — routine extraction configs should be short

For a field-ready RPC, the core of the recent XARTATOMS production is just:

[full]
4.background = yes
extra_region = yes
background = yes
4.fitting = no

[back]
INST = mos2,
generate_spectra_and_back = yes
EMINLIST = 700,
EMAXLIST = 1000,
MASK = 1,
extra_region_file = "<approved-exclusion.reg>"
extra_region_list = <approved-region.reg>,

Two observed background-only profiles

XARTATOMS v1.4.1 recent production：

[full]
3B.background = no
4.background = yes
extra_region = yes
background = yes
4.fitting = no

M31HotISM v1.3.2 accepted E009：

[C]
3B.background = yes
4.background = no
extra_region = no
background = yes
4.fitting = no

Both can enter region extraction, but their stage semantics differ. You cannot just swap the mode= name; the interactive configurator generates the whole switch set per evidence-bound profile.

The launch is still a single line:

python3 '<QUALIFIED_ENGINE.py>' '<CONFIG.par>'

What must not leak into the routine core

re-downloading/re-calibrating the ODF;
re-running source detection per annulus;
runtime-copying and patching the engine;
manually copying parent QPB/ARF/RMF;
fitting;
Cloudflare publication;
an independent agent science-explaining every shell line.

For your standard extended-source extraction, how wide should the stored energy range uniformly be? XART R4–7 uses 700–1000 eV as the current special diagnostic; M31HotISM E009 stores 300–7200 eV and separately ignores the MOS contamination band at fitting time. Which strategy should the generic tutorial teach?

Thread 7 — `exit 0` is not product acceptance

Technical products per camera-region

source PHA;
region-specific QPB PHA;
RMF;
ARF;
exact converted region expression;
executed command / config / engine hash;
source image / mask / camera overlay.

Minimal acceptance

FITS readable;
OBS_ID / INSTRUME / EXP_ID correct;
source/QPB channel grids compatible;
exposure, BACKSCAL, AREASCAL finite positive;
source PHA links to camera-matched background/response;
ARF non-zero; RMF energy grid covers the target range;
source/QPB SLCTEXPR contains the same science region;
QPB active CCD / outer-corner anchor sensible;
no unintended camera products exist;
no surrogate / parent-scaled QPB / copied response.

M31HotISM E009's 84 gates (HOTISM-E009-QA) and XARTATOMS Phase2 qualification (XART-E04) provide two implementations. They need not be merged verbatim, but you must decide the minimal common set.

Please pick the 10–20 hard gates you believe every new spectrum must pass; where should the remaining project-specific gates live?

Thread 8 — no-fit handoff precedes fitting

The visual deliverable for every new extraction:

exact DS9 FK5 / RA-Dec region;
source PHA path and camera;
no-fit xsherpa inspect;
region + camera mask + source exclusion overlay;
product manifest and caveats.

Only after both technical products and spatial evidence pass do we proceed to grouping / fitting. For M31HotISM, the MOS 1.4–2.0 keV handling is a fitting contract and must not be disguised as "the stored spectrum lacks these channels".

What should the fixed no-fit inspect figure set be? rate / sb / sb_flux, 1col/2col, linear/log — always generate all, or the full set for pilots and a reduced set for batches?

Thread 9 — scale up only after the pilot

The pilot should cover the riskiest case

the most complex boundary/wrap;
counts are weak but still usable;
camera coverage is most likely to differ;
the QPB outer anchor is most marginal;
representative source exclusion.

Pre-scale-up gate

the pilot release manifest passes;
exact engine/config/region hashes unchanged;
product verifier, no-fit inspect, and spatial overlay pass;
all batch output namespaces are clean;
homogeneous camera groups are generated;
every concurrent task uses an independent working directory / output namespace; multiple ESAS tasks writing the same directory simultaneously are forbidden to avoid temp-file and fixed-filename collisions (HEASARC ESAS warning);
parallelism does not exceed what the environment can bear.

Minimal failure matrix

Change	Re-run field preparation	Re-run extraction
New OBSID	Yes	Yes
Same OBSID, new region	No	new regions only
Native region converter change	No	affected regions
Energy band change	No	Yes
Clean event / GTI / CCF / mask policy change	new field generation	all
Verifier fix	No	no SAS run, re-verify only
Figure-style, report, or web fix	No	No

This table is the core of "reduce re-runs": a verification/publication failure must not force a SAS re-run; when field-owned assets are unchanged, a new region must not trigger ODF/clean/source-detection redo.

The recent XARTATOMS branch is exactly pilot Wind N → remaining batch; the M31CGM historical root script iterates OBSID/qsub directly, lacking an equivalent pilot-product gate.

How do you usually pick a pilot? Hardest region, most typical region, or simplest region? After one spectrum passes, how many should Hermes parallelize to by default?

Thread 10 — user decision received; each run still needs per-run input

The user has explicitly given:

canonical engine: esas_playground_v1.4.1.py;
one program + a per-run .par config, no per-OBSID engine copies;
inventory the ODF/RPC stage state first, filling only missing field stages;
pre-extraction evselect RA/Dec image visual QA;
post-extraction mos-spectra image + exact RA/Dec region overlay + xsherpa inspect;
keep all 11 safety/QA patterns.

The user has not named a golden wrapper, golden .par, or successful/failure log; this tutorial will not write them for him. Before every actual run we must re-ask: band, region, whether to mask point sources, whether to remove extended sources, and obtain exact spatial approval. The original decision is execution_authorized=false.

Continue to the interactive configurator to fill in this run's inputs; when the method needs changing, go to Teach Hermes to export a new decision.