Fill-Finish, Packaging & Environmental Monitoring

📍 Where we are: Part II, "Capturing the Process." The molecule has been grown, purified, polished and formulated. Now we follow it through its very last industrial steps — into vials, into cartons, into cases — and we watch the air around it. This is where the data stops being about the product and starts being about the units and the space.

The simple version

Imagine the final assembly line at a bottling plant, except the "drink" cost a fortune to make and a single airborne speck can sink an entire batch — because this is a sterile injectable filled aseptically, with no later step that can sterilize a sealed vial, so any contaminant simply ships. Three logbooks run in parallel. One counts vials and weighs each one (fill-finish). One issues every vial a unique, scannable license plate and records which carton and case it went into (serialization). And one is an air-quality monitor that never sleeps, sniffing the cleanroom for dust and microbes (environmental monitoring). This chapter generates all three logbooks with real code as committed CSV goldens — the platform schema is ready for them and a later chapter builds the loader — and then draws the line between the records a regulator will inspect and the facility telemetry that is just for the engineers.

What this chapter covers

• The fill line: per-vial fill volume, in-process-control (IPC) checkweigh, and reject logic, governed by a PackML machine-state model.
• Serialization and aggregation: GS1 SGTIN "license plates" and the vial → carton → case parent/child tree.
• Environmental monitoring (EM): non-viable particle counts and viable CFU against EU GMP Annex 1 Grade A/B/C limits, with a seeded excursion.
• Where the hard GxP boundary falls — and why the same Telegraf-and-VictoriaMetrics tooling you would use for facility dashboards is not allowed to be the system of record (the single authoritative, audited store that is the official version of a regulated record).

The whole chapter runs off one simulator file, examples/sim/bioproc_sim/em_fill.py, which produces three linked datasets plus a PackML log. Everything below comes from that real, deterministic code — manifest-pinned in datasets/MANIFEST.sha256 for byte-for-byte reproducibility.

The fill line: counting, weighing, rejecting

Fill-finish is deceptively simple and unforgiving. A pump doses a target volume into each vial; a checkweigh station weighs it; anything outside tolerance is rejected. The data is high-cardinality (a huge number of distinct identifiers — here one unique serial per vial rather than a handful of repeating tag names) — one row per vial, not one row per tag (the one-row-per-timestamp sensor signals of earlier chapters) — and it is GxP — a Good-Practice regulated record (GxP is the umbrella for the "Good x Practice" regulations — Good Manufacturing, Laboratory and Distribution Practice — that govern records a health authority can inspect) — because the reject decision is a quality decision.

The simulator fills 480 vials at a 6-second cadence, targeting 1.0 mL. From examples/sim/bioproc_sim/em_fill.py:

# examples/sim/bioproc_sim/em_fill.py
def fill_events(batch_id: str = "BATCH-2026-001") -> pd.DataFrame:
    rng = stream_rng("fill", batch_id)
    rows = []
    for i in range(1, N_VIALS + 1):
        ts = FILL_START + pd.Timedelta(seconds=i * 6)
        vol = float(np.clip(rng.normal(TARGET_FILL_ML, 0.020), 0.90, 1.10))
        weight = round(vol * 1.01 + rng.normal(0, 0.004), 4)   # ~1.01 g/mL formulation
        # serial: SGTIN-style (GTIN + serial)
        serial = f"{GTIN}.{i:07d}"
        low, high = 0.95, 1.05
        reject = not (low <= vol <= high)
        rows.append({
            "batch_id": batch_id, "vial_serial": serial, "ts": ts,
            "fill_volume_mL": round(vol, 4), "fill_weight_g": weight,
            "ipc_checkweigh_g": weight, "reject": bool(reject),
            "reject_reason": "low_fill" if vol < low else ("high_fill" if vol > high else None),
        })
    return pd.DataFrame(rows)

Three things are worth pausing on. First, stream_rng("fill", batch_id) gives the fill line its own reproducible random stream derived from SIM_SEED=2026, so the byte-for-byte numbers below are identical on every machine and in CI. Second, the IPC checkweigh is a separate recorded value — in a real line the checkweigher is a different instrument from the filler, and capturing both lets you reconcile dosed volume against measured weight. Third, the reject rule is an explicit, tight band (0.95–1.05 mL) inside the wider physical clip (0.90–1.10 mL): some filled vials are physically possible but commercially unacceptable, and the line rejects them.

The committed golden lives at examples/datasets/fill_events.csv (a 50-row fill_events.sample.csv is committed for CI smoke). The first rows:

batch_id,vial_serial,ts,fill_volume_mL,fill_weight_g,ipc_checkweigh_g,reject,reject_reason
BATCH-2026-001,00361414000017.0000001,2026-01-22 08:00:06+00:00,0.9984,1.0032,1.0032,False,
BATCH-2026-001,00361414000017.0000002,2026-01-22 08:00:12+00:00,1.0116,1.0203,1.0203,False,
BATCH-2026-001,00361414000017.0000003,2026-01-22 08:00:18+00:00,0.9936,0.9923,0.9923,False,

And a rejected vial — vial 152, dosed at 0.9463 mL, below the 0.95 mL floor:

BATCH-2026-001,00361414000017.0000152,2026-01-22 08:15:12+00:00,0.9463,0.9559,0.9559,True,low_fill

That single True is a record an inspector can ask about. Why was it low? Was the reject physically diverted? Was the count reconciled at the end of the run? The data model has to make those questions answerable.

Anatomy of a fill record: what one vial carries

A bioreactor tag is one number sampled over and over; a fill record is the opposite — one row per thing, and that thing is a physical object that either ships or gets destroyed. It pays to dissect a single row of fill_events field by field, because every column is load-bearing and the rejected vial 0000152 is the most instructive specimen in the dataset.

One row of fill_events, dissected: identity and dose at the top, the reject decision highlighted, and the two nested tolerance bands that decided it.

Original diagram by the authors, created with AI assistance.

Read top to bottom, the row tells a complete story. batch_id is the join key — the shared value that links this vial row back to the GMP (Good Manufacturing Practice) batch record (s88.batch, modelled on the ISA-88 batch standard) — so this vial is never an orphan. vial_serial is the unit's identity — the SGTIN we decode in the next section. ts places the fill at 2026-01-22 08:15:12+00, exactly six seconds after vial 0000151, because the simulator advances FILL_START + pd.Timedelta(seconds=i * 6) for vial i. Then come the three numeric measurements that matter: fill_volume_mL = 0.9463 (the dosed volume), fill_weight_g = 0.9559 (the weighed mass, at the ~1.01 g/mL formulation density), and ipc_checkweigh_g = 0.9559 — the same value here because the simulator's checkweigh re-reports the weight, but on a real line it is a separate instrument's reading, captured precisely so dosed volume and measured weight can be reconciled.

The decisive field is reject. For vial 0000152 it is True, and reject_reason is low_fill, because 0.9463 mL fell below the 0.95 mL floor. That is not telemetry — it is a quality decision, and the whole chapter turns on treating it as one. The two tolerance bands in the bottom panel are the logic that produced it: a tight commercial reject band (0.95–1.05 mL) nested inside the wider physical clip (0.90–1.10 mL) that np.clip enforces. A vial can be physically possible yet commercially unacceptable; this one was — it is one of the two vials this run rejects (the other is 0000342, also low_fill), leaving 478 of 480 accepted. Because reject is True, the consequence cascades — as we will see, this serial is barred from the aggregation tree entirely.

PackML: the line has a state, and the state is data

A fill line is not just a stream of vials — it is a machine that is Idle, then Starting, then Execute, sometimes Held, then Completing. That lifecycle is standardized. PackML (the OMAC machine-state model, published by the OPC Foundation as OPC UA for PackML / OPC 30050) defines a finite state machine (a fixed set of named states with only certain transitions allowed between them) and a set of "PackTags" — named values, here Command, Status and Admin tags — that any conformant packaging machine exposes over OPC UA (the open industrial machine-to-software communication standard introduced in Chapter 7) [1]. The Admin PackTags are where production counts and alarm statistics live, which is exactly the reject-and-IPC telemetry this chapter cares about [2]. Because PackML is derived from ISA-88's procedural state model, it slots neatly into the batch-and-equipment model you already built.

The simulator emits the canonical state sequence. From examples/sim/bioproc_sim/em_fill.py:

# examples/sim/bioproc_sim/em_fill.py
PACKML_STATES = ["Idle", "Starting", "Execute", "Holding", "Held",
                 "Unholding", "Execute", "Completing", "Complete", "Resetting", "Idle"]


def packml_log(batch_id: str = "BATCH-2026-001") -> pd.DataFrame:
    rows = []
    t = FILL_START - pd.Timedelta(minutes=10)
    for st in PACKML_STATES:
        rows.append({"batch_id": batch_id, "ts": t, "unit": "FILL-LINE-01",
                     "packml_state": st})
        t += pd.Timedelta(minutes=5)
    return pd.DataFrame(rows)

That packml_state field is really a single PackTag — Status.StateCurrent, the Status family's current-state member — and the simulator captures only it, because the state sequence is what the line-stop story needs. A conformant machine exposes the rest of the set too: Command tags the operator or line controller writes in (Command.UnitMode, Command.CntrlCmd to start/hold/reset, Command.MachSpeed), the rest of Status it reads out (mode, speed, interlocks), and the Admin counters where the production counts and alarm statistics the claim above points at actually live:

Status.StateCurrent             → "Held"   # the one member the simulator emits as packml_state
Admin.ProdProcessedCount[].Count           # running good-vial count
Admin.ProdDefectiveCount[].Count           # reject count — the reject telemetry this chapter cares about
Admin.AlarmHistory[].Message               # the alarm statistics

So the packml_log above is deliberately abridged to the state member; in this chapter those Admin counters are modeled separately — as the fill_events reject rows and the events.equipment_state log — rather than read back as Admin PackTags. On a real line the state member and the Admin counters arrive together off the controller, which is exactly what the next section reads.

The PackML state machine: a line stop, recorded

Notice the run starts ten minutes before the first vial, in Idle → Starting, and passes through a Holding/Held/Unholding excursion mid-run — a line stop, the thing every fill suite dreads. The sequence is not arbitrary: PackML's state model is a finite state machine where Holding is a transitional "wait" state, Held is the stable stopped state an operator clears, and Unholding is the transitional climb back to Execute. Recording all three — not just a single "stopped" flag — is what lets you reconstruct how long the line was down and when it recovered, which is the difference between a defensible batch record and a shrug. At a five-minute cadence the simulated stop spans Holding → Held → Unholding, i.e. fifteen minutes of FILL-LINE-01 not filling, bracketed by timestamps an investigator can quote.

Those state transitions are shaped to land in the platform's events.equipment_state table, which the shared schema defines once for exactly this purpose. From examples/platform/db/30-lab-events.sql:

-- examples/platform/db/30-lab-events.sql
CREATE TABLE events.equipment_state (         -- PackML / serialization (Ch 15)
    state_id   bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    unit_id    text NOT NULL,
    ts         timestamptz NOT NULL,
    state      text NOT NULL,
    batch_id   text REFERENCES s88.batch
);

Reading the line for real: S7 data blocks and python-snap7

On a real line you would not invent these states — you would read them off the controller. The fill machine is almost always a Siemens S7 PLC (programmable logic controller — the ruggedized industrial computer that runs the machine), and python-snap7 (MIT-licensed) is the open-source library you would use to pull state, count and reject data-block tags off an S7 controller [11]. On an S7 those tags live in data blocks (DBs) — named, byte-addressed memory regions — so a reader opens a connection to the PLC's rack and slot, then reads a byte range out of a DB and unpacks it: the PackML state as an integer enum, the running good-count and reject-count as DWORDs. That raw read is exactly where quality is born too: a stale or unreachable DB read is not a zero count, and conflating the two is how a line-stop silently looks like a perfect run.

This repo does not ship a snap7 reader; the closest live example is the Modbus-TCP skid reader in examples/chapters/09-legacy-skids-modbus-s7/modbus_reader.py (using pymodbus), which shows the analogous pattern of reading raw PLC registers and normalizing them into the tag namespace. Here the deterministic simulator stands in for the PLC so the chapter runs on a laptop, but the shape of the data — unit, timestamp, state — is the PackML shape you would persist for real.

Serialization: every vial gets a license plate

Once a vial is filled and accepted, it must become individually traceable. Under the U.S. Drug Supply Chain Security Act (DSCSA), packages carry a standardized numerical identifier that regulators inspect to follow product through the supply chain [4]. The encoding is GS1: a Global Trade Item Number (GTIN, GS1 Application Identifier 01) plus a unique serial number (AI 21) together form a Serialized GTIN (SGTIN), printed as a GS1 DataMatrix on the label [3]. In the simulator the serial is built as {GTIN}.{i:07d} — a deliberately readable stand-in for the encoded SGTIN — which is why every vial_serial above reads like 00361414000017.0000152.

Anatomy of a GS1 SGTIN: decoding the license plate

That vial_serial string is not opaque — it is a structured license plate, and decoding it is the second anatomy worth doing in full. The dot in 00361414000017.0000152 is the simulator's own readable separator; on a real label the two parts are carried under GS1 Application Identifiers, the GTIN under AI (01) and the serial under AI (21).

The SGTIN dissected: the 14-digit GTIN (indicator digit, company prefix and item reference, mod-10 check digit) under AI (01), and the unit serial under AI (21).

Original diagram by the authors, created with AI assistance.

The GTIN-14 splits into an indicator/packaging-level digit, the GS1 Company Prefix and Item Reference (the licensed brand owner plus the specific product), and a trailing mod-10 check digit that lets a scanner self-validate the other thirteen digits. The serial 0000152 is what makes the plate unique — it distinguishes this one vial from the 479 others in the run. Together the GTIN under AI (01) and the serial under AI (21) form the SGTIN, and on the physical label that pair is printed as a GS1 DataMatrix the camera reads at the reject and pack stations. (The GTIN here, 00361414000017, is a fictional example from the simulator — its trailing 7 stands in the check-digit position but is not a computed valid GS1 check digit, which is why we describe the position rather than assert the value.) The punchline is the green block: only accepted vials get a slot in the aggregation tree, so vial 0000152 — rejected — owns a license plate that never aggregates into a carton or a case.

Back along the trilogy spine

The fill_events row and the aggregation tree this chapter builds in Python are the code-and-SQL form of two physical steps from Book 1. The dosed, checkweighed, stoppered vial is made on the line in Formulation and Fill-Finish; the SGTIN license plate and the vial → carton → case parent/child tree are applied in Packaging and Serialization. What is a sterile dosing-and-labeling operation there becomes one row and one genealogy edge here.

The aggregation tree: a queryable genealogy

Serialization on its own is just a list of numbers. The value comes from aggregation: recording which vials went into which carton, and which cartons into which case, so that scanning a case at the loading dock tells you exactly which 120 vials are inside without opening it. That is a parent/child tree, and only accepted vials belong in it. From examples/sim/bioproc_sim/em_fill.py:

# examples/sim/bioproc_sim/em_fill.py
def aggregation_tree(fills, vials_per_carton: int = 24, cartons_per_case: int = 5):
    """Parent/child serialization aggregation: vial -> carton -> case (accepted vials only)."""
    rows = []
    accepted = fills[~fills.reject].reset_index(drop=True)
    for vi, r in accepted.iterrows():
        carton = vi // vials_per_carton + 1
        case = vi // (vials_per_carton * cartons_per_case) + 1
        rows.append({
            "batch_id": r.batch_id, "child": r.vial_serial, "child_level": "vial",
            "parent": f"CARTON-{r.batch_id}-{carton:03d}", "parent_level": "carton",
        })
        rows.append({
            "batch_id": r.batch_id, "child": f"CARTON-{r.batch_id}-{carton:03d}", "child_level": "carton",
            "parent": f"CASE-{r.batch_id}-{case:03d}", "parent_level": "case",
        })
    return pd.DataFrame(rows).drop_duplicates().reset_index(drop=True)

The ~fills.reject filter is doing real regulatory work: a rejected vial must never appear in the aggregation tree, because it never entered saleable inventory. The integer-division arithmetic (vi // 24, vi // 120) is the whole packing geometry — 24 vials to a carton, 5 cartons to a case — and drop_duplicates() collapses the repeated carton→case edge so each parent link is asserted once. The result is a clean, queryable genealogy that, joined to the batch, lets you answer "which case holds vial 0000152?" in one SQL hop. The formal RDF treatment of this parent/child tree — keeping which box something is currently in (contains, a relation that chains so a case automatically holds every vial inside its cartons) strictly separate from where a material came from (its derivedFrom lineage), so that re-packing a vial into a different carton can never be mistaken for a change in its origin — is in Book 4, Formalization: Axioms and Restrictions.

The same tree as triples, a shape, and a competency question

The relational tree above is one face of the artifact; its semantic face is the same edges written as RDF (Resource Description Framework — the subject-predicate-object triple model this series builds its knowledge graph on). Each aggregation row becomes one contains triple — CARTON-… contains VIAL-…, CASE-… contains CARTON-… — and the vial_serial we decoded is not a bare string but a global identity: the SGTIN gives the unit a stable name that survives leaving the plant, which is exactly the IRI (Internationalized Resource Identifier — a global web name, as opposed to a primary key meaningful only inside one database) discipline Book 4 builds in Identifiers and Units. Modeled, a single packed-and-accepted vial reads:

# the aggregation tree as triples — containment, kept off the lineage spine
bp:VIAL-00361414000017-0000017  a bp:SerializedUnit ;
    bp:gtin "00361414000017" ; bp:serial "0000017" ;
    bp:derivedFrom bp:DP-2026-001 .          # lineage: which lot it was filled from
bp:CARTON-BATCH-2026-001-001  bp:contains  bp:VIAL-00361414000017-0000017 .
bp:CASE-BATCH-2026-001-001    bp:contains  bp:CARTON-BATCH-2026-001-001 .

Two things this book has been careful about all along fall out cleanly here. First, contains and derivedFrom stay strictly apart — the vial is contained in a carton (mutable: open the case, it changes) but derived from a drug-product lot (permanent), the very split Conceptualization: Relations and Genealogy draws so a recall scoped by lineage is never confused with a shipping carton. Second, because bp:contains is declared transitive, a reasoner infers CASE-… contains VIAL-… without anyone asserting the long-range edge — the graph analogue of the SQL "one hop to the case" above, but inferred to any depth.

The integrity rule the field-failure section below dreads — every accepted unit gets exactly one parent, asserted once — is itself expressible as a closed-world SHACL shape (Shapes Constraint Language — RDF's validator, which fails on a missing required fact rather than reasoning it away), the same gate-not-guess pattern Book 4's release gate uses: a sh:NodeShape on bp:SerializedUnit with sh:path bp:containedIn ; sh:minCount 1 ; sh:maxCount 1 turns "a vial with no carton link" from a silent gap into a flagged conformance violation. And the case-scan question becomes a one-line SPARQL (the query language for RDF) competency question over the transitive edge — "which serialized vials does this case contain?" — answered by a property path bp:contains+ rather than a hand-written recursive SQL CTE. The point is not that the graph replaces the table; it is that the same aggregation artifact, modeled semantically, makes the missing-link failure a validation error and the contents question a single property-path hop.

When serialization breaks in the field

It is tempting to read all this as solved: a standard exists (GS1), a law mandates it (DSCSA), so package-level traceability must be a routine, working reality. The field evidence is more sobering, and it is worth stating plainly because it is exactly the kind of thing this book refuses to gloss. DSCSA set a 2023 milestone for interoperable, electronic, package-level tracing across the U.S. supply chain — and the FDA, confronting widespread industry unreadiness, used its compliance-policy discretion to grant a one-year stabilization period through November 2024 rather than enforce on schedule, then layered additional staggered exemptions for smaller dispensers beyond that [4]. The standard being unambiguous did not make the data flow. Aggregation in particular — the parent/child tree this section builds in nine lines of Python — is where real programs stumble: a missing or mis-asserted vial→carton link means a case scan no longer resolves to its contents, and the genealogy that was supposed to answer "which case holds this vial?" returns silence. The lesson for a data system is not that serialization is hard to model — it plainly is not — but that the integrity of the model depends entirely on every accepted unit actually getting its link asserted, once, at pack time, with no gaps. That is an operational discipline, not a schema feature, and it is precisely where under-implementation hides.

The last mile and the air around it. The fill line (left) produces PackML states and per-vial records; serialization builds the aggregation tree; cleanroom sensors (top) stream continuously. The dashed boundary is the chapter's whole point: GxP records (vial rejects, EM excursions, serialization) flow into the audited PostgreSQL system of record, while high-cardinality facility observability flows into VictoriaMetrics where it is useful but not a regulated record.

Original diagram by the authors, created with AI assistance.

Environmental monitoring: watching the air

While vials fill, the cleanroom is continuously monitored. EU GMP Annex 1 (the 2022 revision) requires a routine EM program — viable and non-viable particle counts (non-viable = total inert airborne particulate such as dust and fibres, counted by light scattering; viable = living microbes that grow into countable colonies, reported as colony-forming units, CFU), air, surface and personnel monitoring — all governed by a documented Contamination Control Strategy [5]. Non-viable particle telemetry is classified against defined air-cleanliness classes by ISO 14644-1, measured with light-scattering airborne particle counters at thresholds from 0.1 to 5 µm [6]. In a fill suite the critical fill zone is Grade A (the most demanding), surrounded by Grade B, with Grade C/D support areas.

The simulator monitors five locations across an 8-hour shift, sampling hourly, and uses Poisson statistics for particle and microbial counts — the right distribution for rare, independent contamination events. From examples/sim/bioproc_sim/em_fill.py:

# examples/sim/bioproc_sim/em_fill.py
# Annex 1 non-viable particle limits (>=0.5 um, per m3), in operation
GRADE_LIMITS = {"A": 3520, "B": 352000, "C": 3520000, "D": None}

def em_samples(batch_id: str = "BATCH-2026-001") -> pd.DataFrame:
    rng = stream_rng("em", batch_id)
    rows = []
    locations = [("FILL-A-01", "A"), ("FILL-A-02", "A"), ("BKGD-B-01", "B"),
                 ("BKGD-B-02", "B"), ("CORR-C-01", "C")]
    sid = 1
    for hour in range(8):  # an 8-hour shift, hourly samples
        ts = EM_START + pd.Timedelta(hours=hour)
        for loc, grade in locations:
            limit = GRADE_LIMITS[grade]
            base = {"A": 1500, "B": 120000, "C": 1200000}.get(grade, 2000)
            particles = int(rng.poisson(base))
            viable = int(rng.poisson({"A": 0.05, "B": 1.0, "C": 4.0}.get(grade, 6.0)))
            # seed one Grade-A excursion in hour 5
            excursion = grade == "A" and hour == 5 and loc == "FILL-A-01"
            if excursion:
                particles = int(limit * 1.4)
                viable = 2
            rows.append({
                "em_id": f"EM-{batch_id}-{sid:03d}", "batch_id": batch_id, "ts": ts,
                "location": loc, "grade": grade,
                "nonviable_0_5um_per_m3": particles,
                "nonviable_5um_per_m3": int(particles * rng.uniform(0.0, 0.02)),
                "viable_CFU": viable,
                "limit_0_5um_per_m3": limit,
                "excursion": bool(limit is not None and particles > limit),
            })
            sid += 1
    return pd.DataFrame(rows)

The Grade-A limit of 3,520 particles ≥0.5 µm per m³ is the Annex 1 number, not a guess — and for Grade A it is identical at-rest and in-operation, so the GRADE_LIMITS dict needs only the one value. (Grades B and C do differ between states; the dict carries their in-operation limits.) The Grade-A viable expectation is essentially zero (Poisson mean 0.05) — Annex 1 sets a Grade A viable action limit of under 1 CFU for active air, the settle plate (90 mm, 4 h), the contact plate (55 mm) and the 5-finger glove — which is why a Grade-A CFU of 2 is alarming on its own: it is itself a hard breach that would trigger a microbial deviation. And one excursion is deliberately seeded in hour 5 at FILL-A-01: particles jump to int(limit * 1.4) = 4,928, well over the 3,520 limit, and excursion flips to True. Note that the single excursion flag here is driven only by the non-viable particle count — it deliberately ignores the viable limit for simplicity, so a complete EM model would add a parallel viable action-limit check against that under-1-CFU yardstick (the seeded CFU=2 would itself trip it). Note too that the simulated nonviable_5um_per_m3 column is captured but not limit-checked: the 2022 Annex 1 revision dropped the ≥5 µm value from the classification table (cleanroom classification is now particle-count and CCS-based), so there is no ≥5 µm limit to assert here — but the macroparticle count is still a monitored diagnostic worth trending, which is exactly why the simulator keeps the column rather than discarding it.

The committed golden examples/datasets/em_samples.csv shows the calm baseline and then the spike:

em_id,batch_id,ts,location,grade,nonviable_0_5um_per_m3,nonviable_5um_per_m3,viable_CFU,limit_0_5um_per_m3,excursion
EM-BATCH-2026-001-001,BATCH-2026-001,2026-01-22 06:00:00+00:00,FILL-A-01,A,1500,6,0,3520,False
EM-BATCH-2026-001-003,BATCH-2026-001,2026-01-22 06:00:00+00:00,BKGD-B-01,B,120150,1037,0,352000,False
EM-BATCH-2026-001-026,BATCH-2026-001,2026-01-22 11:00:00+00:00,FILL-A-01,A,4928,56,2,3520,True

That last row is a GxP event. An EM excursion triggers an investigation, a deviation record, and a quality decision on the batch. In the platform it is not meant to just sit in a CSV — the row is shaped to land in events.operation_event with event_type = 'excursion', the same table the chromatography phase detector and the bioreactor logic target, so that excursions across the whole process live in one place and join back to the batch. (At this stage of the repo the simulator writes the CSV goldens and the platform defines the schema; the Chapter-12 loader that lands these rows is left as the design the schema anticipates rather than a running flow.)

A single threshold cross is the obvious alarm, but contamination control is statistical, not just pass/fail. Microbiological EM data is trended: alert and action limits behave like SPC control limits, and a slow drift toward the limit matters as much as a single breach [12]. The repo's analytics chapter does the trending; here we capture the raw counts and the per-sample excursion flag that trending consumes.

Why the column shape matters for any model built on it

The per-sample flag is the floor, not the ceiling. The moment anyone wants a model on top of these counts — an early-warning predictor that flags a Grade-A location trending toward its limit before it crosses — the way this chapter shaped the data decides whether that model can be trusted, and two traps from Book 5 land directly on these columns. The first is validation leakage: the eight hourly samples from one location across one shift are not independent draws but an autocorrelated trajectory, so a naive row-wise train/test split would scatter near-twin neighbours across the line and report a flattering score that collapses on the next shift. The honest split groups whole batches (or whole locations) to one side — the GroupKFold and leave-one-batch-out discipline Book 5 makes the default in Data, the Fuel — so the test set is a genuinely unseen room, the only question manufacturing cares about. That is precisely why this chapter keeps batch_id, location and grade as first-class columns and never collapses them: they are the grouping keys an honest cross-validation needs.

The second trap is the difference between process drift and model drift. The seeded hour-5 excursion is process drift — the room genuinely got dirtier, and the right response is a deviation, not a retrain. Model drift is the opposite failure: a counter slowly fouls or a new media lot shifts the particle background, the inputs move into a region the model never saw (its applicability domain — the input range a data-driven model may be trusted in; outside it the model is extrapolating and least reliable), and the prediction silently goes wrong while the room is actually fine. Book 5's MLOps and Lifecycle names these as distinct mathematical kinds — covariate shift in the input counts versus concept drift in the count-to-risk mapping — caught by two different instruments, and insists a model touching a quality decision be locked and only ever changed on purpose, under change control. The quality discipline this chapter already insisted on — a stale or unreachable DB read is not a zero count — is the same signal a drift monitor consumes: a frozen particle-counter feed looks like a perfect room to a naive model and like a covariate-shift alarm to a good one. The same reasoning extends to a hybrid model of the fill line, where physics (a contamination-transport guardrail) covers what the handful of EM batches cannot. None of this modeling lives in this chapter — but the schema choices here are what make it possible to do honestly downstream, and what make every prediction traceable back to the exact dataset hash and batch_id that produced it.

Where it all goes: the GxP boundary, drawn in tooling

Here is the honest, load-bearing distinction of this chapter. EM counts, fill rejects and serialization records are GxP data — records requiring a data-criticality assessment, true copies with a full audit trail, and retention in dynamic (reprocessable) form under MHRA and PIC/S data-integrity expectations [7]. Inspectors apply a risk-based, ALCOA+ lens (the Attributable, Legible, Contemporaneous, Original, Accurate — plus Complete, Consistent, Enduring, Available — data-integrity attributes the trust chapters build out) to these records, which is precisely why the line between a regulated record and a convenience dashboard must be drawn explicitly [8].

So the platform routes fill-finish and EM data two ways:

The collection agent on both paths is the same: Telegraf, a single-binary, plugin-driven metrics agent (300+ plugins, MIT-licensed) ideal for high-cardinality fill-line and facility telemetry [9]. The destinations differ. Differential pressure, relative humidity and temperature trends — thousands of points a minute, every door interlock and HVAC reading — go to VictoriaMetrics (Apache-2.0, pinned at victoriametrics/victoria-metrics:v1.108.1 in the platform compose), whose cardinality explorer and limiter were built for exactly this firehose [10]. That telemetry is enormously useful for keeping the room qualified — but it is not the regulated record. The viable CFU result, the Grade-A excursion, the rejected vial, the aggregation tree: those are written to PostgreSQL, under the same audit triggers and hash chain the trust chapters build, because they are evidence. (In the shipped compose VictoriaMetrics runs under the analytics/ops profile for stack self-monitoring; routing live facility telemetry into it through Telegraf is the architecture this section describes, not a flow the repo wires today — the same honest bracketing as the Chapter-12 loader on the PostgreSQL side.)

Why not just put everything in VictoriaMetrics? Because a time-series observability store is the wrong system of record for GxP: it is optimized for retention windows and downsampling, not for an immutable, attributable, fully audit-trailed history you can reconstruct years later for an inspection. Routing EM excursions through it would quietly demote a regulated record to a dashboard metric. Drawing the boundary in the tooling — Telegraf-to-VictoriaMetrics for observability, Telegraf-to-PostgreSQL for records — is how you keep the honest hybrid honest.

Why it matters

Fill-finish is where a batch worth millions either becomes saleable product or becomes a deviation. The data here is unusually high-stakes for its volume: one excursion row, one reject flag, one missing aggregation link can hold or sink a lot. Modeling it correctly — PackML states for the line, GS1 SGTINs for the units, Annex 1 grades for the air — means the questions an inspector or a quality investigator asks have answers that fall directly out of a query, not out of a frantic spreadsheet reconstruction. And getting the GxP boundary right means you can use cheerful, scalable open-source observability tooling for the firehose of facility data without accidentally turning your system of record into a Grafana panel.

In the real world

A commercial fill line runs at hundreds of vials per minute, with serialization handled by dedicated Level-2/Level-3 software (the line- and site-level layers of the ISA-95 automation hierarchy from the reference architecture — Systech, Optel, SAP ATTP) talking to camera systems and printers, and EM handled by validated particle-counter networks (Lighthouse, TSI, Particle Measuring Systems) feeding a validated EM data manager. Those systems are proprietary and genuinely cannot run on a laptop — so this chapter, like the rest of the book, simulates the data shapes and is explicit that the vendor specifics (camera reject signalling, validated counter calibration, the precise GS1 encoding on the DataMatrix) are where real qualification happens. The honest verdict for OSS here: Telegraf and VictoriaMetrics are excellent and production-grade for facility observability and engineering dashboards, and PostgreSQL is a fully credible GxP system of record once you wrap it in the validated audit-trail, retention and access controls the later chapters build — but no part of this stack is a turnkey, validated EM data manager or serialization repository out of the box. Annex 1, ISO 14644 classification and DSCSA serialization remain the operator's burden to demonstrate; the platform shows you can capture the data correctly and keep it on the right side of the GxP line.

Key terms

• Fill-finish: the final sterile manufacturing steps where bulk drug substance is dosed into vials/syringes, stoppered, capped and inspected.
• IPC (in-process control): a measurement taken during production to control quality in real time — here, the checkweigh on each vial.
• PackML / OMAC (OPC 30050): the standardized packaging-machine state model (Idle/Starting/Execute/Held/…) and PackTags exposed over OPC UA, derived from ISA-88.
• GS1 / GTIN / SGTIN: the global product-identification standard; a GTIN (AI 01) plus a unique serial (AI 21) forms a Serialized GTIN, the unit-level "license plate."
• Aggregation: recording the parent/child containment of serialized items (vial → carton → case) so a scan of the parent reveals its children.
• GTIN check digit (mod-10): the final digit of a GTIN, computed from the others by a GS1 modulo-10 algorithm so a scanner can self-validate the number it read.
• S7 data block (DB): a named, byte-addressed memory region on a Siemens S7 PLC where tags such as PackML state and good/reject counts live, read over the network with a library like python-snap7.
• Environmental monitoring (EM): routine measurement of cleanroom air/surface contamination — non-viable particles and viable colony-forming units (CFU).
• Annex 1 grades (A/B/C/D): EU GMP cleanliness grades; Grade A is the critical fill zone, limited to 3,520 particles ≥0.5 µm per m³ in operation.
• Excursion: a measured value crossing an alert/action limit — a GxP event that triggers investigation.
• GxP: the umbrella for the family of "Good x Practice" regulations — Good Manufacturing Practice (GMP), Good Laboratory Practice (GLP), Good Distribution Practice (GDP) and others — that govern the records a health authority can inspect.
• GxP boundary: the explicit line between regulated records (audited, retained) and non-GxP facility observability (engineering dashboards).
• CFU (colony-forming unit): the count of viable microorganisms recovered from an EM sample.
• High-cardinality: data with very many distinct label values (e.g. one unique serial per vial), which a time-series store like VictoriaMetrics is built to absorb.
• RDF triple / IRI: the subject-predicate-object fact (e.g. CASE-… contains CARTON-…) and the global web name that the aggregation tree maps onto when modeled as a knowledge graph, as opposed to a database-local key.
• SHACL / SPARQL: RDF's validator (which fails on a missing required fact, e.g. a vial with no carton link) and its query language (which walks the transitive contains path to answer "which vials are in this case?").
• Applicability domain: the input range a data-driven model may be trusted in; outside it the model is extrapolating and least reliable — the yardstick a drift monitor watches.
• Model drift vs process drift: a model going stale because its inputs moved (a fouling counter, a new lot) versus the process genuinely changing (a real excursion) — different problems with different fixes (retrain-under-control vs deviation).
• Grouped / leave-one-batch-out cross-validation: holding out whole batches or locations rather than shuffled rows, so an autocorrelated trajectory cannot leak across the train/test line and inflate a model's reported score.
• System of record: the single authoritative, audited store a regulator treats as the official copy of a GxP record — as opposed to a convenience dashboard or analytics copy.
• Audit trail: the tamper-evident log of who changed what data, when and why, that an inspector can reconstruct (a Part 11 / Annex 11 expectation).
• Validated (GxP sense): formally documented evidence that a system does what it is supposed to and protects data integrity — the burden the later trust chapters build out.

Where this leads

We have now captured everything the process emits — bioreactor tags, chromatography decisions, lab results, and the last-mile fill, serialization and environmental data of this chapter. All of it has been landing in TimescaleDB and PostgreSQL almost without comment. It is time to make that store a deliberate choice. The next chapter, The Open-Source Historian: Choosing and Running a Time-Series Store, opens up the historian we have been quietly relying on — how hypertables, continuous aggregates and retention actually work, how TimescaleDB stacks up against alternatives like IoTDB, and which open-source features you can safely build on versus the license traps you must steer around.