Fill-Finish, Packaging & Environmental Monitoring

📍 Where we are: Part II, "Capturing the Process." The molecule has been grown, purified, polished and formulated. Now we follow it through its very last industrial steps — into vials, into cartons, into cases — and we watch the air around it. This is where the data stops being about the product and starts being about the units and the space.

The simple version

Imagine the final assembly line at a bottling plant, except the "drink" cost a fortune to make and a single airborne speck can sink an entire batch. Three logbooks run in parallel. One counts vials and weighs each one (fill-finish). One issues every vial a unique, scannable license plate and records which carton and case it went into (serialization). And one is an air-quality monitor that never sleeps, sniffing the cleanroom for dust and microbes (environmental monitoring). This chapter generates all three logbooks with real code, lands them in the database — and then draws the line between the records a regulator will inspect and the facility telemetry that is just for the engineers.

What this chapter covers

• The fill line: per-vial fill volume, in-process-control (IPC) checkweigh, and reject logic, governed by a PackML machine-state model.
• Serialization and aggregation: GS1 SGTIN "license plates" and the vial → carton → case parent/child tree.
• Environmental monitoring (EM): non-viable particle counts and viable CFU against EU GMP Annex 1 Grade A/B/C limits, with a seeded excursion.
• Where the hard GxP boundary falls — and why the same Telegraf-and-VictoriaMetrics tooling you would use for facility dashboards is not allowed to be the system of record.

The whole chapter runs off one simulator file, examples/sim/bioproc_sim/em_fill.py, which produces three linked datasets plus a PackML log. Everything below comes from that real, tested code.

The fill line: counting, weighing, rejecting

Fill-finish is deceptively simple and unforgiving. A pump doses a target volume into each vial; a checkweigh station weighs it; anything outside tolerance is rejected. The data is high-cardinality — one row per vial, not one row per tag — and it is GxP, because the reject decision is a quality decision.

The simulator fills 480 vials at a 6-second cadence, targeting 1.0 mL. From examples/sim/bioproc_sim/em_fill.py:

# examples/sim/bioproc_sim/em_fill.py
def fill_events(batch_id: str = "BATCH-2026-001") -> pd.DataFrame:
    rng = stream_rng("fill", batch_id)
    rows = []
    for i in range(1, N_VIALS + 1):
        ts = FILL_START + pd.Timedelta(seconds=i * 6)
        vol = float(np.clip(rng.normal(TARGET_FILL_ML, 0.020), 0.90, 1.10))
        weight = round(vol * 1.01 + rng.normal(0, 0.004), 4)   # ~1.01 g/mL formulation
        # serial: SGTIN-style (GTIN + serial)
        serial = f"{GTIN}.{i:07d}"
        low, high = 0.95, 1.05
        reject = not (low <= vol <= high)
        rows.append({
            "batch_id": batch_id, "vial_serial": serial, "ts": ts,
            "fill_volume_mL": round(vol, 4), "fill_weight_g": weight,
            "ipc_checkweigh_g": weight, "reject": bool(reject),
            "reject_reason": "low_fill" if vol < low else ("high_fill" if vol > high else None),
        })
    return pd.DataFrame(rows)

Three things are worth pausing on. First, stream_rng("fill", batch_id) gives the fill line its own reproducible random stream derived from SIM_SEED=2026, so the byte-for-byte numbers below are identical on every machine and in CI. Second, the IPC checkweigh is a separate recorded value — in a real line the checkweigher is a different instrument from the filler, and capturing both lets you reconcile dosed volume against measured weight. Third, the reject rule is an explicit, tight band (0.95–1.05 mL) inside the wider physical clip (0.90–1.10 mL): some filled vials are physically possible but commercially unacceptable, and the line rejects them.

The committed golden lives at examples/datasets/fill_events.csv (a 50-row fill_events.sample.csv is committed for CI smoke). The first rows:

batch_id,vial_serial,ts,fill_volume_mL,fill_weight_g,ipc_checkweigh_g,reject,reject_reason
BATCH-2026-001,00361414000017.0000001,2026-01-22 08:00:06+00:00,0.9984,1.0032,1.0032,False,
BATCH-2026-001,00361414000017.0000002,2026-01-22 08:00:12+00:00,1.0116,1.0203,1.0203,False,
BATCH-2026-001,00361414000017.0000003,2026-01-22 08:00:18+00:00,0.9936,0.9923,0.9923,False,

And a rejected vial — vial 152, dosed at 0.9463 mL, below the 0.95 mL floor:

BATCH-2026-001,00361414000017.0000152,2026-01-22 08:15:12+00:00,0.9463,0.9559,0.9559,True,low_fill

That single True is a record an inspector can ask about. Why was it low? Was the reject physically diverted? Was the count reconciled at the end of the run? The data model has to make those questions answerable.

PackML: the line has a state, and the state is data

A fill line is not just a stream of vials — it is a machine that is Idle, then Starting, then Executing, sometimes Held, then Completing. That lifecycle is standardized. PackML (the OMAC machine-state model, published by the OPC Foundation as OPC UA for PackML / OPC 30050) defines a finite state machine and a set of "PackTags" — Command, Status and Admin tags — that any conformant packaging machine exposes over OPC UA [1]. The Admin PackTags are where production counts and alarm statistics live, which is exactly the reject-and-IPC telemetry this chapter cares about [2]. Because PackML is derived from ISA-88's procedural state model, it slots neatly into the batch-and-equipment model you already built.

The simulator emits the canonical state sequence. From examples/sim/bioproc_sim/em_fill.py:

# examples/sim/bioproc_sim/em_fill.py
PACKML_STATES = ["Idle", "Starting", "Execute", "Holding", "Held",
                 "Unholding", "Execute", "Completing", "Complete", "Resetting", "Idle"]


def packml_log(batch_id: str = "BATCH-2026-001") -> pd.DataFrame:
    rows = []
    t = FILL_START - pd.Timedelta(minutes=10)
    for st in PACKML_STATES:
        rows.append({"batch_id": batch_id, "ts": t, "unit": "FILL-LINE-01",
                     "packml_state": st})
        t += pd.Timedelta(minutes=5)
    return pd.DataFrame(rows)

Notice the run starts ten minutes before the first vial, in Idle → Starting, and passes through a Holding/Held/Unholding excursion mid-run — a line stop, the thing every fill suite dreads. Those state transitions are shaped to land in the platform's events.equipment_state table, which the shared schema defines once for exactly this purpose. From examples/platform/db/30-lab-events.sql:

-- examples/platform/db/30-lab-events.sql
CREATE TABLE events.equipment_state (         -- PackML / serialization (Ch 12)
    state_id   bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    unit_id    text NOT NULL,
    ts         timestamptz NOT NULL,
    state      text NOT NULL,
    batch_id   text REFERENCES s88.batch
);

On a real line you would not invent these states — you would read them off the controller. The fill machine is almost always a Siemens S7 PLC, and python-snap7 (MIT-licensed) is the open-source library you would use to pull state, count and reject data-block tags off an S7 controller [3]. This repo does not ship a snap7 reader; the closest live example is the Modbus-TCP skid reader in examples/chapters/09-legacy-skids-modbus-s7/modbus_reader.py (using pymodbus), which shows the analogous pattern of reading raw PLC registers and normalizing them into the tag namespace. Here the deterministic simulator stands in for the PLC so the chapter runs on a laptop, but the shape of the data — unit, timestamp, state — is the PackML shape you would persist for real.

Serialization: every vial gets a license plate

Once a vial is filled and accepted, it must become individually traceable. Under the U.S. Drug Supply Chain Security Act (DSCSA), packages carry a standardized numerical identifier that regulators inspect to follow product through the supply chain [4]. The encoding is GS1: a Global Trade Item Number (GTIN, GS1 Application Identifier 01) plus a unique serial number (AI 21) together form a Serialized GTIN (SGTIN), printed as a GS1 DataMatrix on the label [5]. In the simulator the serial is built as {GTIN}.{i:07d} — a deliberately readable stand-in for the encoded SGTIN — which is why every vial_serial above reads like 00361414000017.0000152.

Serialization on its own is just a list of numbers. The value comes from aggregation: recording which vials went into which carton, and which cartons into which case, so that scanning a case at the loading dock tells you exactly which 120 vials are inside without opening it. That is a parent/child tree, and only accepted vials belong in it. From examples/sim/bioproc_sim/em_fill.py:

# examples/sim/bioproc_sim/em_fill.py
def aggregation_tree(fills, vials_per_carton: int = 24, cartons_per_case: int = 5):
    """Parent/child serialization aggregation: vial -> carton -> case (accepted vials only)."""
    rows = []
    accepted = fills[~fills.reject].reset_index(drop=True)
    for vi, r in accepted.iterrows():
        carton = vi // vials_per_carton + 1
        case = vi // (vials_per_carton * cartons_per_case) + 1
        rows.append({
            "batch_id": r.batch_id, "child": r.vial_serial, "child_level": "vial",
            "parent": f"CARTON-{r.batch_id}-{carton:03d}", "parent_level": "carton",
        })
        rows.append({
            "batch_id": r.batch_id, "child": f"CARTON-{r.batch_id}-{carton:03d}", "child_level": "carton",
            "parent": f"CASE-{r.batch_id}-{case:03d}", "parent_level": "case",
        })
    return pd.DataFrame(rows).drop_duplicates().reset_index(drop=True)

The ~fills.reject filter is doing real regulatory work: a rejected vial must never appear in the aggregation tree, because it never entered saleable inventory. The integer-division arithmetic (vi // 24, vi // 120) is the whole packing geometry — 24 vials to a carton, 5 cartons to a case — and drop_duplicates() collapses the repeated carton→case edge so each parent link is asserted once. The result is a clean, queryable genealogy that, joined to the batch, lets you answer "which case holds vial 0000152?" in one SQL hop.

The last mile and the air around it. The fill line (left) produces PackML states and per-vial records; serialization builds the aggregation tree; cleanroom sensors (top) stream continuously. The dashed boundary is the chapter's whole point: GxP records (vial rejects, EM excursions, serialization) flow into the audited PostgreSQL system of record, while high-cardinality facility observability flows into VictoriaMetrics where it is useful but not a regulated record.

Original diagram by the authors, created with AI assistance.

Environmental monitoring: watching the air

While vials fill, the cleanroom is continuously monitored. EU GMP Annex 1 (the 2022 revision) requires a routine EM program — viable and non-viable particle counts, air, surface and personnel monitoring — all governed by a documented Contamination Control Strategy [6]. Non-viable particle telemetry is classified against defined air-cleanliness classes by ISO 14644-1, measured with light-scattering airborne particle counters at thresholds from 0.1 to 5 µm [7]. In a fill suite the critical fill zone is Grade A (the most demanding), surrounded by Grade B, with Grade C/D support areas.

The simulator monitors five locations across an 8-hour shift, sampling hourly, and uses Poisson statistics for particle and microbial counts — the right distribution for rare, independent contamination events. From examples/sim/bioproc_sim/em_fill.py:

# examples/sim/bioproc_sim/em_fill.py
# Annex 1 non-viable particle limits (>=0.5 um, per m3), in operation
GRADE_LIMITS = {"A": 3520, "B": 352000, "C": 3520000, "D": None}

def em_samples(batch_id: str = "BATCH-2026-001") -> pd.DataFrame:
    rng = stream_rng("em", batch_id)
    rows = []
    locations = [("FILL-A-01", "A"), ("FILL-A-02", "A"), ("BKGD-B-01", "B"),
                 ("BKGD-B-02", "B"), ("CORR-C-01", "C")]
    sid = 1
    for hour in range(8):  # an 8-hour shift, hourly samples
        ts = EM_START + pd.Timedelta(hours=hour)
        for loc, grade in locations:
            limit = GRADE_LIMITS[grade]
            base = {"A": 1500, "B": 120000, "C": 1200000}.get(grade, 2000)
            particles = int(rng.poisson(base))
            viable = int(rng.poisson({"A": 0.05, "B": 1.0, "C": 4.0}.get(grade, 6.0)))
            # seed one Grade-A excursion in hour 5
            excursion = grade == "A" and hour == 5 and loc == "FILL-A-01"
            if excursion:
                particles = int(limit * 1.4)
                viable = 2
            rows.append({
                "em_id": f"EM-{batch_id}-{sid:03d}", "batch_id": batch_id, "ts": ts,
                "location": loc, "grade": grade,
                "nonviable_0_5um_per_m3": particles,
                "nonviable_5um_per_m3": int(particles * rng.uniform(0.0, 0.02)),
                "viable_CFU": viable,
                "limit_0_5um_per_m3": limit,
                "excursion": bool(limit is not None and particles > limit),
            })
            sid += 1
    return pd.DataFrame(rows)

The Grade-A limit of 3,520 particles ≥0.5 µm per m³ is the Annex 1 number, not a guess — and for Grade A it is identical at-rest and in-operation, so the GRADE_LIMITS dict needs only the one value. (Grades B and C do differ between states; the dict carries their in-operation limits.) The Grade-A viable expectation is essentially zero (Poisson mean 0.05), which is why a Grade-A CFU of 2 is alarming on its own. And one excursion is deliberately seeded in hour 5 at FILL-A-01: particles jump to int(limit * 1.4) = 4,928, well over the 3,520 limit, and excursion flips to True. Note that the simulated nonviable_5um_per_m3 column is captured but not limit-checked: the 2022 Annex 1 revision dropped the ≥5 µm value from the classification table (cleanroom classification is now particle-count and CCS-based), so there is no ≥5 µm limit to assert here.

The committed golden examples/datasets/em_samples.csv shows the calm baseline and then the spike:

em_id,batch_id,ts,location,grade,nonviable_0_5um_per_m3,nonviable_5um_per_m3,viable_CFU,limit_0_5um_per_m3,excursion
EM-BATCH-2026-001-001,BATCH-2026-001,2026-01-22 06:00:00+00:00,FILL-A-01,A,1500,6,0,3520,False
EM-BATCH-2026-001-003,BATCH-2026-001,2026-01-22 06:00:00+00:00,BKGD-B-01,B,120150,1037,0,352000,False
EM-BATCH-2026-001-026,BATCH-2026-001,2026-01-22 11:00:00+00:00,FILL-A-01,A,4928,56,2,3520,True

That last row is a GxP event. An EM excursion triggers an investigation, a deviation record, and a quality decision on the batch. In the platform it is not meant to just sit in a CSV — the row is shaped to land in events.operation_event with event_type = 'excursion', the same table the chromatography phase detector and the bioreactor logic target, so that excursions across the whole process live in one place and join back to the batch. (At this stage of the repo the simulator writes the CSV goldens and the platform defines the schema; the Chapter-12 loader that lands these rows is left as the design the schema anticipates rather than a running flow.)

A single threshold cross is the obvious alarm, but contamination control is statistical, not just pass/fail. Microbiological EM data is trended: alert and action limits behave like SPC control limits, and a slow drift toward the limit matters as much as a single breach [8]. The repo's analytics chapter does the trending; here we capture the raw counts and the per-sample excursion flag that trending consumes.

Where it all goes: the GxP boundary, drawn in tooling

Here is the honest, load-bearing distinction of this chapter. EM counts, fill rejects and serialization records are GxP data — records requiring a data-criticality assessment, true copies with a full audit trail, and retention in dynamic (reprocessable) form under MHRA and PIC/S data-integrity expectations [9]. Inspectors apply a risk-based, ALCOA+ lens to these records, which is precisely why the line between a regulated record and a convenience dashboard must be drawn explicitly [10].

So the platform routes fill-finish and EM data two ways:

The collection agent on both paths is the same: Telegraf, a single-binary, plugin-driven metrics agent (300+ plugins, MIT-licensed) ideal for high-cardinality fill-line and facility telemetry [11]. The destinations differ. Differential pressure, relative humidity and temperature trends — thousands of points a minute, every door interlock and HVAC reading — go to VictoriaMetrics (Apache-2.0, pinned at victoriametrics/victoria-metrics:v1.108.1 in the platform compose), whose cardinality explorer and limiter were built for exactly this firehose [12]. That telemetry is enormously useful for keeping the room qualified — but it is not the regulated record. The viable CFU result, the Grade-A excursion, the rejected vial, the aggregation tree: those are written to PostgreSQL, under the same audit triggers and hash chain the trust chapters build, because they are evidence.

Why not just put everything in VictoriaMetrics? Because a time-series observability store is the wrong system of record for GxP: it is optimized for retention windows and downsampling, not for an immutable, attributable, fully audit-trailed history you can reconstruct years later for an inspection. Routing EM excursions through it would quietly demote a regulated record to a dashboard metric. Drawing the boundary in the tooling — Telegraf-to-VictoriaMetrics for observability, Telegraf-to-PostgreSQL for records — is how you keep the honest hybrid honest.

Why it matters

Fill-finish is where a batch worth millions either becomes saleable product or becomes a deviation. The data here is unusually high-stakes for its volume: one excursion row, one reject flag, one missing aggregation link can hold or sink a lot. Modeling it correctly — PackML states for the line, GS1 SGTINs for the units, Annex 1 grades for the air — means the questions an inspector or a quality investigator asks have answers that fall directly out of a query, not out of a frantic spreadsheet reconstruction. And getting the GxP boundary right means you can use cheerful, scalable open-source observability tooling for the firehose of facility data without accidentally turning your system of record into a Grafana panel.

In the real world

A commercial fill line runs at hundreds of vials per minute, with serialization handled by dedicated Level-2/Level-3 software (Systech, Optel, SAP ATTP) talking to camera systems and printers, and EM handled by validated particle-counter networks (Lighthouse, TSI, Particle Measuring Systems) feeding a validated EM data manager. Those systems are proprietary and genuinely cannot run on a laptop — so this chapter, like the rest of the book, simulates the data shapes and is explicit that the vendor specifics (camera reject signalling, validated counter calibration, the precise GS1 encoding on the DataMatrix) are where real qualification happens. NIIMBL's SABRE facility — the NIIMBL / University of Delaware pilot-scale cGMP (current Good Manufacturing Practice) facility under construction in Delaware, with groundbreaking in April 2024 — is exactly the kind of site where these data flows must be designed in from day one rather than bolted on. The honest verdict for OSS here: Telegraf and VictoriaMetrics are excellent and production-grade for facility observability and engineering dashboards, and PostgreSQL is a fully credible GxP system of record once you wrap it in the validated audit-trail, retention and access controls the later chapters build — but no part of this stack is a turnkey, validated EM data manager or serialization repository out of the box. Annex 1, ISO 14644 classification and DSCSA serialization remain the operator's burden to demonstrate; the platform shows you can capture the data correctly and keep it on the right side of the GxP line.

Key terms

• Fill-finish: the final sterile manufacturing steps where bulk drug substance is dosed into vials/syringes, stoppered, capped and inspected.
• IPC (in-process control): a measurement taken during production to control quality in real time — here, the checkweigh on each vial.
• PackML / OMAC (OPC 30050): the standardized packaging-machine state model (Idle/Starting/Execute/Held/…) and PackTags exposed over OPC UA, derived from ISA-88.
• GS1 / GTIN / SGTIN: the global product-identification standard; a GTIN (AI 01) plus a unique serial (AI 21) forms a Serialized GTIN, the unit-level "license plate."
• Aggregation: recording the parent/child containment of serialized items (vial → carton → case) so a scan of the parent reveals its children.
• Environmental monitoring (EM): routine measurement of cleanroom air/surface contamination — non-viable particles and viable colony-forming units (CFU).
• Annex 1 grades (A/B/C/D): EU GMP cleanliness grades; Grade A is the critical fill zone, limited to 3,520 particles ≥0.5 µm per m³ in operation.
• Excursion: a measured value crossing an alert/action limit — a GxP event that triggers investigation.
• GxP boundary: the explicit line between regulated records (audited, retained) and non-GxP facility observability (engineering dashboards).
• CFU (colony-forming unit): the count of viable microorganisms recovered from an EM sample.

Where this leads

We have now captured everything the process emits — bioreactor tags, chromatography decisions, lab results, and the last-mile fill, serialization and environmental data of this chapter. All of it has been landing in TimescaleDB and PostgreSQL almost without comment. It is time to make that store a deliberate choice. The next chapter, The Open-Source Historian: Choosing and Running a Time-Series Store, opens up the historian we have been quietly relying on — how hypertables, continuous aggregates and retention actually work, how TimescaleDB stacks up against alternatives like IoTDB, and which open-source features you can safely build on versus the license traps you must steer around.