1.0 — ECDFS spec-z sample and golden catalog match

Authors: Xiangchong Li, Tianqing Zhang, Eric Charles, Sam Schmidt LSST Science Pipelines: Weekly 2025_49 Kernel: Python-dp1

Purpose

End-to-end preparation of the DP1 ECDFS spec-z + shape training/test sample, plus quick sky-coverage diagnostics for the other DP1 fields.

Steps

1. Read the spec-z parquet comcam_ecdfs_redshift_catalog_20250618.parquet and the AnaCal shape catalog anacal_catalog_ecdfs.fits.
2. Overlay (RA, Dec) of both as a scatter plot — measurement (blue) vs. spec-z (red) — to show field coverage.
3. Sky-match within 0.75 arcsec using utils.sky_match, append the four spec-z columns (redshift, confidence, type, source) to the measurement table via utils.append_specz_columns.
4. Apply the basic golden selection (utils.golden_selection): per-band magnitude cuts + trace > 0.10 + spec-z confidence > 0.82.
5. Write rail_training/match.fits containing only matched + golden rows (every original AnaCal column + the four appended spec-z columns) — this is the input for the scripts_pz/step2_prepare_train_test.py train/test split.
6. Additional (RA, Dec) scatter plots for the Abell 360 and EDFS measurement catalogs.

Inputs

• specz/comcam_ecdfs_redshift_catalog_20250618.parquet
• catalogs/anacal_catalog_ecdfs.fits
• catalogs/anacal_catalog_a360.fits
• catalogs/anacal_catalog_edfs.fits
• utils/ (local importable package; wraps specz/match.py)

Outputs

• rail_training/match.fits — matched + golden sample.
• Three inline scatter figures (ECDFS, Abell 360, EDFS).

import numpy as np
import tables_io
from astropy.table import Table
import matplotlib.pyplot as plt

specz_path = "/gpfs/mnt/gpfs02/astro/workarea/xli6/work/DP1/specz/comcam_ecdfs_redshift_catalog_20250618.parquet"
meas_path  = "/gpfs/mnt/gpfs02/astro/workarea/xli6/work/DP1/catalogs/anacal_catalog_ecdfs.fits"

specz = tables_io.read(specz_path, tType="astropyTable")
meas  = Table.read(meas_path)
print(f"spec-z catalog       : {len(specz):,} rows")
print(f"measurement catalog  : {len(meas):,} rows")

column_list None

spec-z catalog       : 104,078 rows
measurement catalog  : 128,542 rows

fig, ax = plt.subplots(figsize=(8, 8))
ax.scatter(
    np.asarray(meas["ra"]),  np.asarray(meas["dec"]),
    s=2, alpha=0.25, color="C0",
    label=f"measurement ({len(meas):,})",
)
ax.scatter(
    np.asarray(specz["RA"]), np.asarray(specz["DEC"]),
    s=6, alpha=0.7, color="C3",
    label=f"spec-z ({len(specz):,})",
)
ax.set_xlabel("RA [deg]")
ax.set_ylabel("Dec [deg]")
ax.set_title("ECDFS spec-z sample vs. measurement catalog")
ax.legend(loc="best", frameon=False)
ax.set_aspect("equal")
ax.grid(True, alpha=0.25)
fig.tight_layout()

Match spec-z to measurement catalog

Sky-match within 0.75 arcsec, append the four spec-z columns (redshift, confidence, type, source), apply the basic golden selection (per-band mag cuts + trace + confidence > 0.9), and write a new matched catalog.

from utils import sky_match, append_specz_columns, golden_selection

idx, sepcheck = sky_match(meas, specz)
append_specz_columns(meas, specz, idx)

sel = golden_selection(meas, sepcheck)
print(f"matched: {int(sepcheck.sum()):,} / {len(meas):,}")
print(f"+ basic selection (mag/trace/conf): {int(sel.sum()):,} / {len(meas):,}")

matched: 14,174 / 128,542
+ basic selection (mag/trace/conf): 8,314 / 128,542

import os
# Keep only matched + basic-selection rows; carry every original column plus
# the four new spec-z columns. Lands in rail_training/ alongside the train/test
# split inputs.
matched_dir = "/gpfs/mnt/gpfs02/astro/workarea/xli6/work/DP1/rail_training"
os.makedirs(matched_dir, exist_ok=True)
matched_path = os.path.join(matched_dir, "match.fits")
meas[sel].write(matched_path, overwrite=True)
print(f"wrote {matched_path} ({int(sel.sum()):,} rows)")

Other-field sky coverage

Quick (ra, dec) scatter for the two other measurement catalogs: Abell 360 and EDFS.

a360 = Table.read(
    "/gpfs/mnt/gpfs02/astro/workarea/xli6/work/DP1/catalogs/anacal_catalog_a360.fits"
)
print(f"a360 catalog: {len(a360):,} rows")

fig, ax = plt.subplots(figsize=(8, 8))
ax.scatter(
    np.asarray(a360["ra"]), np.asarray(a360["dec"]),
    s=2, alpha=0.25, color="C2",
    label=f"a360 ({len(a360):,})",
)
ax.set_xlabel("RA [deg]")
ax.set_ylabel("Dec [deg]")
ax.set_title("Abell 360 measurement catalog")
ax.legend(loc="best", frameon=False)
ax.set_aspect("equal")
ax.grid(True, alpha=0.25)
fig.tight_layout()

a360 catalog: 113,945 rows

edfs = Table.read(
    "/gpfs/mnt/gpfs02/astro/workarea/xli6/work/DP1/catalogs/anacal_catalog_edfs.fits"
)
print(f"edfs catalog: {len(edfs):,} rows")

fig, ax = plt.subplots(figsize=(8, 8))
ax.scatter(
    np.asarray(edfs["ra"]), np.asarray(edfs["dec"]),
    s=2, alpha=0.25, color="C4",
    label=f"edfs ({len(edfs):,})",
)
ax.set_xlabel("RA [deg]")
ax.set_ylabel("Dec [deg]")
ax.set_title("EDFS measurement catalog")
ax.legend(loc="best", frameon=False)
ax.set_aspect("equal")
ax.grid(True, alpha=0.25)
fig.tight_layout()

edfs catalog: 89,354 rows