Capstone: One Batch, End to End

📍 Where we are: the last working chapter — we run a complete simulated fed-batch through every layer we built and end with one reviewable, signed batch record.

The simple version

Imagine you have spent the whole book building a factory's plumbing — pipes (sensors), tanks (the historian), labels on every valve (the namespace), a logbook that can't be erased (the audit trail), and a control room full of dashboards. The capstone is the day you finally turn the water on and watch one full batch flow from the inlet to a sealed, signed bottle at the far end. Nothing new gets built. We just press one button per step and prove the pipes actually connect — and that what comes out the other end is something a quality reviewer could sign.

What this chapter covers

This is the moment the platform stops being a pile of services and becomes a system. We take one deterministic, 14-day simulated fed-batch CHO + Protein A monoclonal-antibody (mAb) run and push it through the entire stack we assembled chapter by chapter: simulator → historian → ISA-88/95 batch model → contextualization → lab results → an ALCOA+ audit chain → the Raman→titer soft-sensor → a reviewable, FAIR-aligned dataset. We will run the real command sequence from the companion repo, look at the real SQL and Python that does the joining, and end on the honest question every reader has been waiting for: is this a batch record a regulator would accept? (Short answer: the data is; the validated system around it is the hybrid last mile.)

The running case is the classic one. We use the industry's canonical worked example — an IgG1 mAb produced by fed-batch CHO culture with a Protein A capture step and a Quality-by-Design control strategy linking critical quality attributes to process parameters [8]. Everything below is one batch, BATCH-2026-001.

One command surface: Make as the orchestrator

We never invented a bespoke pipeline engine. The book's whole "every claim is runnable" promise rests on the reader typing the exact commands the book prints, so the orchestrator is the oldest, most boring, most dependable tool we have: GNU Make. Make is a dependency-driven build tool — you declare targets and what they need, and it runs them in order [9]. For a laptop-scale, reproducible, end-to-end run that is exactly enough; for larger, branching scientific pipelines the same idea scales up into Python-based, dependency-aware workflow engines such as Snakemake, which carry the reproducibility story from a workstation to a cluster [10]. We stay with Make because the reader already has it and it keeps the command surface honest.

Here is the real top-level Makefile in examples/Makefile. Notice that every target documents the Docker Compose profile it needs, and that help auto-lists the targets from their ## comments — the command surface is the documentation.

# examples/Makefile
COMPOSE := docker compose -f platform/compose/compose.yaml
PY := sim/.venv/bin/python
export DATABASE_URL ?= postgresql://bioproc:bioproc@localhost:5432/bioproc

venv: ## create the Python env and install the simulator (uv)
	cd sim && uv venv --python 3.12 .venv && uv pip install --python .venv -e . "psycopg[binary]" scikit-learn

up: ## bring up the core stack (postgres+timescale, mosquitto, grafana)
	$(COMPOSE) --profile core up -d
	@echo "waiting for postgres..." && sleep 3
	@until docker exec sensor-to-submission-postgres-1 pg_isready -U bioproc >/dev/null 2>&1; do sleep 2; done
	@echo "core stack up."

seed: ## load the ISA-88/95 reference CHO line into postgres
	docker exec -i -e PGPASSWORD=bioproc sensor-to-submission-postgres-1 \
	  psql -U bioproc -d bioproc -q < platform/db/seed/seed_cho_line.sql

data: ## (re)generate every dataset deterministically + MANIFEST.sha256
	$(PY) -m bioproc_sim.generate --all

load: ## load the datasets into the running stack (historian + lab + genealogy)
	$(PY) tools/load_datasets.py

The end-to-end run is just those targets in order, followed by the contextualization, audit, and analytics targets. This is the literal capstone sequence, copied from the repo's README.md:

make venv          # Python env + the simulator (uv)
make data          # generate every dataset deterministically + MANIFEST.sha256
make up            # bring up the core stack (postgres+timescale, mosquitto, grafana)
make seed          # load the ISA-88/95 reference CHO line
make load          # load the datasets into the historian + lab tables
make contextualize # join time-series to batch phases (Ch 14)
make alcoa         # verify the ALCOA+ audit hash chain (0 = intact)
make soft-sensor   # train the Raman -> titer PLS soft-sensor (Ch 26)
make test          # the whole suite (determinism + db + analytics)

Each line is a layer from the book. Watch the data move.

Step 1 — generate the batch deterministically (make data)

make data runs python -m bioproc_sim.generate --all with the master seed SIM_SEED=2026, so the 14-day trace is byte-for-byte identical on your laptop and on the CI runner. This matters more than it sounds: reproducibility is the difference between "a demo that worked once" and an artifact you can re-derive on demand — the Findable/Accessible/Reusable spine of FAIR data [11]. The generator writes a MANIFEST.sha256 and make test later asserts every file matches it. If a simulator drifts by a single float, CI fails loudly.

This single, deterministic source is also why the whole batch agrees with itself: the in-line titer, the offline assay titer, and the Raman spectra are all drawn from the same underlying kinetic state. That internal consistency is the "Consistent" in ALCOA+ — not a happy accident but a property of the design [3].

Step 2 — stand up the record-of-truth and load it (make up, make seed, make load)

make up boots the core stack — PostgreSQL + TimescaleDB, Mosquitto, Grafana, the simulator — and blocks until Postgres answers pg_isready. PostgreSQL is the relational store that holds the contextualized, signed records and the audit trail; it is the system of record the reviewable batch record is assembled from [12]. make seed applies the ISA-88/95 reference CHO line: the enterprise→site→area→unit hierarchy, the recipe and its phases, and the BR101 production bioreactor.

make load runs the dataset loader in examples/tools/load_datasets.py. This is the one script that does, in a single pass, what Chapters 5–13 build up piece by piece: high-rate sensor readings stream into the historian by bulk COPY, and offline lab results land through the normal INSERT path so the audit trigger actually fires on each one.

# examples/tools/load_datasets.py
def load_timeseries(conn) -> int:
    df = pd.read_parquet(DATA / "fedbatch_timeseries.parquet")
    buf = io.StringIO()
    df[["ts", "tag", "value", "unit", "quality", "batch_id"]].to_csv(buf, index=False, header=False)
    buf.seek(0)
    with conn.cursor() as cur:
        cur.execute("TRUNCATE ts.sensor_reading")
        with cur.copy("COPY ts.sensor_reading (ts, tag, value, unit, quality, batch_id) "
                      "FROM STDIN WITH (FORMAT csv)") as copy:
            copy.write(buf.read())
    return len(df)

The offline loader deliberately sets an application user before writing, so the audit trail can attribute each result to a responsible actor rather than to an anonymous database connection — the Attributable in ALCOA+ [3]:

# examples/tools/load_datasets.py
def load_offline(conn) -> int:
    df = pd.read_csv(DATA / "offline_assays.csv", parse_dates=["sample_time"])
    n = 0
    with conn.cursor() as cur:
        cur.execute("SELECT set_config('app.user', 'loader', false)")
        for _, r in df.iterrows():
            cur.execute(
                "INSERT INTO lab.sample (sample_id, batch_id, sample_time, sample_point, sample_type) "
                "VALUES (%s,%s,%s,%s,'in_process') ON CONFLICT (sample_id) DO NOTHING",
                (r.sample_id, r.batch_id, r.sample_time.to_pydatetime(), r.sample_point))
            for col, (tid, unit) in OFFLINE_TESTS.items():
                cur.execute(
                    "INSERT INTO lab.result (sample_id, test_id, value, unit, analyst, status) "
                    "VALUES (%s,%s,%s,%s,'auto','verified') ON CONFLICT DO NOTHING",
                    (r.sample_id, tid, float(r[col]), unit))
                n += 1
    return n

When it finishes it prints a one-line receipt. The loader reads the whole campaign CSVs — all six batches — so the totals cover the full dataset, not just the golden batch:

loaded: 322560 sensor readings, 1344 offline results, 66 release results, 30 genealogy edges

That is 322,560 high-rate sensor rows, 1,344 offline results (168 in-process samples × the 8 assays in OFFLINE_TESTS), 66 release results from hplc_results.csv, and 30 genealogy edges from lot_genealogy.csv. The lot-genealogy excerpt below shows the five edges for BATCH-2026-001 alone; the loader writes the equivalent chain for every batch in the campaign.

The loader also writes the lot genealogy: the directed edges that chain a seed train to the bioreactor to the Protein A capture pool to drug substance to drug product. That chain is exactly what 21 CFR 211 expects when it asks you to trace a finished batch back to its component lots:

batch_id,child,child_type,parent,parent_type
BATCH-2026-001,SEED-001,seed_train,WCB-CHO-001,wcb
BATCH-2026-001,BATCH-2026-001,bioreactor,SEED-001,seed_train
BATCH-2026-001,PApool-001,capture_pool,BATCH-2026-001,bioreactor
BATCH-2026-001,DS-001,drug_substance,PApool-001,capture_pool
BATCH-2026-001,DP-001,drug_product,DS-001,drug_substance

Step 3 — turn bare tags into knowledge (make contextualize)

A row in the historian — ('2026-01-12T03:00:05Z', 'BR101.DO.PV', 41.7, '%sat', 0, 'BATCH-2026-001') — is nearly meaningless on its own. Which batch, on which equipment, during which recipe phase? The whole point of the platform is to answer that join. It lives in examples/platform/db/60-views.sql, and it is the move that makes the PAT vision real — building quality on in-process understanding of the run, not on testing it in at the end [6]:

-- examples/platform/db/60-views.sql
-- A reading with its full batch + phase context.
CREATE OR REPLACE VIEW s88.v_batch_sensor AS
SELECT r.ts, r.tag, r.value, r.unit, r.quality, r.batch_id,
       b.product_id, b.recipe_id, b.unit_id,
       bp.phase_id, ph.name AS phase_name
FROM ts.sensor_reading r
JOIN s88.batch b              ON b.batch_id = r.batch_id
LEFT JOIN s88.batch_phase bp  ON bp.batch_id = r.batch_id
     AND r.ts >= bp.start_ts AND (bp.end_ts IS NULL OR r.ts < bp.end_ts)
LEFT JOIN s88.phase ph        ON ph.phase_id = bp.phase_id;

The temporal join on bp.start_ts/bp.end_ts is the clever part: each reading is matched to whatever ISA-88 phase was active at that instant. A second view, s88.v_phase_summary, rolls that up into the per-phase, per-tag statistics that become the "golden batch" reference. make contextualize simply queries the first view for one tag and groups by phase:

select phase_name, count(*) n, round(avg(value)::numeric,1) avg_DO
from s88.v_batch_sensor where batch_id='BATCH-2026-001' and tag='BR101.DO.PV'
group by phase_name order by min(ts);

 phase_name |  n   | avg_do
------------+------+--------
 Inoculate  |  720 |   39.8
 Growth     | 9360 |   38.9
 Production | 8640 |   35.1
 Harvest    | 1440 |   34.4

The four phases are exactly the ISA-88 phases the seed defines — Inoculate, Growth, Production, Harvest — and dissolved oxygen drifts gently downward across them (39.8 → 38.9 → 35.1 → 34.4 %sat) as the culture grows and the cells draw more oxygen, with the controller holding it close to its setpoint the whole time. It is a modest, roughly flat-to-declining trend, not a dramatic crash — which is precisely the point of a well-controlled fed-batch. What matters is that the trend is now legible per phase rather than buried in 322,560 anonymous rows. This is the contextualized record a review-by-exception workflow reads from.

Step 4 — prove the record is tamper-evident (make alcoa)

A batch record is only worth signing if you can trust it has not been quietly altered. Annex 11 requires computerised systems to generate a secure, time-stamped audit trail, and to make that trail reviewable before a disposition decision [2]. Our audit layer, in examples/platform/db/50-alcoa.sql, hash-chains every change to a regulated table so any later edit breaks the chain:

-- examples/platform/db/50-alcoa.sql
-- chain hash = H(prev_hash || payload)
v_hash := encode(digest(
    coalesce(v_prev, '') || TG_TABLE_NAME || TG_OP ||
    coalesce(v_old::text, '') || coalesce(v_new::text, '') ||
    coalesce(v_app, '') || clock_timestamp()::text, 'sha256'), 'hex');

make alcoa runs the verifier and expects zero broken links:

select count(*) as broken_links from audit.verify_chain();

 broken_links
--------------
            0

Here is the honesty the book insists on, written into the schema comment itself: a superuser who disables the trigger can still bypass this — hash chaining makes tampering evident, not impossible. No open-source database gives you Part 11 out of the box. What this layer gives you is detectability: an independent, recomputable check that the contemporaneous record is intact [3]. The validated-system burden — change control, e-signatures with the right meaning, SOPs, the GAMP 5 lifecycle — is the operator's, and it is the hybrid last mile the next chapter scores.

Step 5 — predict release-relevant quality from process data (make soft-sensor)

make soft-sensor trains the Raman→titer Partial Least Squares model in examples/analytics/soft_sensor.py. It learns titer from the in-line Raman spectra and validates on a held-out slice — and it must clear a hard floor or CI fails:

# examples/analytics/soft_sensor.py
if __name__ == "__main__":
    m = train()
    print(f"PLS soft-sensor (titer from Raman): R2={m['r2']} RMSE={m['rmse_g_L']} g/L "
          f"({m['n_components']} comps, {m['n_wavenumbers']} wavenumbers, "
          f"{m['n_train']} train / {m['n_test']} test)")
    assert m["r2"] > 0.85, f"soft-sensor R2 too low ({m['r2']}): dataset not predictive"
    print("ASSERT ok: R2 > 0.85 — the Raman dataset is genuinely predictive of titer.")

PLS soft-sensor (titer from Raman): R2=0.9923 RMSE=0.1498 g/L (6 comps, 701 wavenumbers, 235 train / 101 test)

This is the conceptual heart of real-time release testing (RTRT): evaluating product quality from in-process measurements and process data rather than from end-product testing alone — the model that lets a measured spectrum stand in for a titer assay [4]. We are not claiming a validated soft-sensor here (one Raman model rarely transfers across scales or cell lines without re-training). We are showing that the data path RTRT needs — spectra in, a defensible prediction out, logged and reproducible — runs end to end on the same batch the rest of the record describes.

The whole picture

Every layer the reader built, wired into one run: make data produces the deterministic batch; make load fans it into the TimescaleDB historian and the lab tables; make contextualize joins tags to ISA-88 phases; make alcoa proves the record is tamper-evident; make soft-sensor turns spectra into a release-relevant prediction — ending in one reviewable, FAIR-aligned dataset. Original diagram by the authors, created with AI assistance.

The same flow as a sequence, the way the data actually moves between services:

Step 6 — the reviewable batch record

Stitch the layers together and you have something a quality unit can actually act on. 21 CFR 211.188 says a batch production and control record must reproduce the master record and document, with dates and signatures, that each significant step was accomplished [1]. Our assembled record carries all of it: the recipe and equipment context (ISA-88/95), the contemporaneous in-line trace per phase (the historian + contextualization views), the offline and release results with their specifications, the lot genealogy back to the working cell bank, the tamper-evident audit trail, and the analytics that support disposition. The release results sit in lab.result against lab.test rows that carry the spec limits, so a reviewer sees pass/fail in context:

batch_id        | test            | value  | unit  | spec_low | spec_high | result
----------------+-----------------+--------+-------+----------+-----------+--------
BATCH-2026-001  | SEC_monomer_pct | 98.611 | %     |     95.0 |     100.0 | PASS
BATCH-2026-001  | SEC_HMW_pct     |  1.287 | %     |      0.0 |       3.0 | PASS
BATCH-2026-001  | CEX_main_pct    | 70.686 | %     |     60.0 |      80.0 | PASS
BATCH-2026-001  | HCP_ng_per_mg   | 28.203 | ng/mg |      0.0 |     100.0 | PASS

Because every layer is contextualized and signed, the reviewer does not read 322,560 rows. They review by exception: the system flags only the deviations — a day-7 temperature excursion, an out-of-trend metabolite, an audit link that does not verify — and the human attention goes where it is needed. This electronic-production-record, audit-trail-review model is precisely what GAMP 5 (2nd Edition) describes for compliant computerised systems [7]. And the whole deliverable is a product-realization output inside a pharmaceutical quality system that is meant to stay in a state of control and support release across the lifecycle [5].

Why it matters

For most of the book, each layer stood alone — a collector here, a schema there, a dashboard somewhere else. It is easy to demo a layer; it is hard to make them interconnect such that a single value emitted by a sensor is still attributable, contextualized, and verifiable when it lands in front of a reviewer. The capstone is the proof that they do. It collapses "sensor to submission" from a slogan into a command sequence you can run on a laptop in minutes, and make test re-runs the entire thing on a clean CI runner — the book's adversarial evidence that the build is genuinely implementable, not aspirational.

It also makes the trilogy's argument land. The first book explained the process; the second explained governance; this one builds the thing — and shows that open source can carry a fed-batch CHO + Protein A run from the bioreactor all the way to a reviewable batch record without a single proprietary component in the critical path.

In the real world

A real release is heavier than make test. The contextualized dataset here is genuinely ALCOA+ in its data properties, but a regulator licenses a validated system plus procedures, not a dataset. The hash chain proves tamper-evidence, not Part 11 e-signature compliance; the soft-sensor is a teaching model, not a validated RTRT method; the offline results are simulated, not analyst-witnessed. Real review-by-exception at a site layers an MES electronic batch record, a validated LIMS, and a quality management system on top of exactly this kind of data backbone.

This is where the U.S. NIIMBL (National Institute for Innovation in Manufacturing Biopharmaceuticals, a public-private institute) and its University of Delaware partnership matter. NIIMBL's SABRE facility — a pilot-scale current Good Manufacturing Practice (cGMP) manufacturing facility under construction, which broke ground in April 2024 — is the kind of place where an open data backbone meets the validated, physical reality of making a regulated product. SABRE is a facility, not a data program; the point is that the data architecture this book builds is the substrate such facilities run on, and the honest-hybrid boundary — open source for ~80%, commercial and validated systems for the GxP last mile — is exactly the boundary a real pilot line negotiates. The next chapter scores that boundary tool by tool.

Key terms

• Capstone run — the single end-to-end exercise that drives one batch through every layer of the stack via the make command sequence.
• Lot genealogy — the directed chain of material lots (working cell bank → seed train → bioreactor → capture pool → drug substance → drug product) that lets you trace a finished batch back to its components.
• Contextualization — joining a raw historian reading to its batch, equipment, and active ISA-88 phase, via s88.v_batch_sensor.
• Review by exception — reviewing only the flagged deviations in a contextualized electronic record rather than every raw value.
• Real-time release testing (RTRT) — evaluating in-process and final quality from process data and measured attributes instead of end-product testing alone.
• ALCOA+ — the data-integrity attributes (Attributable, Legible, Contemporaneous, Original, Accurate, plus Complete, Consistent, Enduring, Available) the record is held to.
• FAIR — Findable, Accessible, Interoperable, Reusable: the design goals that make the capstone output a reusable dataset, not a one-off report.
• Deterministic generation — producing the same dataset byte-for-byte from a fixed seed (SIM_SEED=2026), checked against MANIFEST.sha256.

Where this leads

We have proven the open-source stack can carry a real batch end to end — and we have been honest, at each step, about where it stops being enough on its own. The final chapter, The Honest Verdict: Open Source vs Commercial, settles the account: a scored, layer-by-layer comparison of what pure open source gives you, what the GxP last mile demands, and exactly where the hybrid line should fall for a regulated mAb facility.