Formulation and Fill-Finish: Computer Vision and the Lyophilizer

📍 Where we are: Part V · Fill-Finish & Release, Learned — Chapter 17. UF/DF handed over DS-001, the bulk drug substance at its target concentration in the formulation buffer. This chapter turns that bulk into the thing a patient actually receives — a filled, inspected, sometimes freeze-dried vial — and meets the single most production-deployed application of machine learning in all of biomanufacturing: deep-learning vision for automated visual inspection.

The drug substance is finished as a molecule. It is not yet a product. Formulation and fill-finish is the operation that takes the bulk DS-001 lot, dilutes or adjusts it to the final formulation, and dispenses it — millilitre by millilitre — into thousands of glass vials or prefilled syringes, which become the drug-product lot DP-001 in our running genealogy. Then, for many biologics, the product is freeze-dried (lyophilized) into a stable cake; every container is checked for fill accuracy, particulates, and closure integrity; and only the units that pass are released — formally cleared by the quality unit to leave the factory. It is the most physically discrete step in the whole process — the continuous broth of upstream has become a countable population of individual objects, each of which can be accepted or rejected on its own.

That discreteness is exactly why this is where machine learning has actually crossed into routine GMP production (Good Manufacturing Practice — the legally binding, regulator-inspected rules for making a medicine to a fixed, validated procedure). Upstream, ML soft-senses (estimates a hard-to-measure value from other signals) a value that a human still acts on; here, in automated visual inspection (AVI), a convolutional network looks at a picture of one vial and makes an accept/reject decision that, after validation, can stand on its own. Amgen has publicly reported releasing roughly 95% of its syringes and vials through automated inspection (vendor/self-reported) [1] — a number that, with all its caveats, is the strongest single data point for production ML in the industry. This chapter builds the two honest, deployable pieces around it: a fill-weight control and reject model on our real fill line, and the framing of the vision problem itself, with a small CNN sketch.

The simple version

Think of the final station on any factory line that makes things people swallow or inject — a person in a hairnet holding each unit up to a bright light, turning it, looking for a fleck, a crack, a low fill, a crooked cap, then putting it in the "good" bin or the "scrap" bin. People are slow, they get tired, they disagree, and they reject far too many good units just to be safe. Automated visual inspection replaces that person with a camera and a trained eye made of software: the camera takes pictures of every vial, and a neural network — the same family of model that recognizes faces in photos — decides accept or reject in a fraction of a second, consistently, all day. The hard part is not the camera. It is teaching it every way a vial can be defective, and then proving to a regulator that it never misses the dangerous ones.

What this chapter covers

• Why fill-finish is the natural home for production ML — discrete units, a binary accept/reject decision, and a defect that is visible.
• Fill-weight and dose-accuracy control — deriving control and action limits from the line's own statistics, the in-process checkweigher, and why a fixed limit beats a learned threshold on a 478-to-2 class imbalance.
• Deep-learning automated visual inspection (AVI) — the task framing, why it needs a CNN and not a tabular model, the convolution-and-pooling structure, the defect library, false-reject reduction, recall as the safety-critical metric, and the production deployments (Amgen, Stevanato, Brevetti CEA, Syntegon, Microsoft Bonsai).
• Lyophilization cycle optimization — soft-sensing product temperature and the primary-drying endpoint, and design-space modeling of the freeze-drying cycle.
• Container-closure integrity (CCIT) and aseptic/isolator environmental monitoring as emerging ML targets.
• A runnable module, examples/platform/ml/fill_control.py, and a CNN sketch in vision_avi.py.
• The anatomy of one AVI accept/reject record — reading a single serialized vial's inspection as an identity card: the imaged ROI, the locked model and its validation library, the per-class confidence and routing threshold, and the genealogy back through DP-001 and DS-001 to release.
• The honest open problem: the validation-versus-learning paradox — how do you keep a model that could keep learning frozen enough to validate, and validated enough to trust?

Why fill-finish is where production ML actually lives

Every prior chapter in this book wrestled with the same handicap: living systems, sparse offline reference, run-to-run variability, and a model that decays fast — the small-data ceiling that keeps pure ML stuck in advisory roles. Fill-finish breaks that pattern in three ways that matter.

First, the unit of decision is discrete and countable. A bioreactor produces one continuous trajectory; a fill line produces 480 individual vials (in our dataset, exactly that — BATCH-2026-001), each an independent object that can be inspected and judged on its own. That turns a hard regression-over-time problem into a clean per-item classification problem (sorting each object into a category — here accept or reject — rather than predicting a continuous number over time, which is regression), the kind machine learning is genuinely good at. It also fixes the sample size: where an upstream model has perhaps a dozen historical batches to learn a CQA (Critical Quality Attribute — a property that must stay in range for the product to be safe and effective) from, an AVI program accumulates one labelled training example per vial, so a single commercial campaign can yield hundreds of thousands of images — the data regime where deep learning actually works.

Second, the decision is binary and consequential in a bounded way. Accept or reject this one vial. A wrong accept is a defect reaching a patient; a wrong reject is a scrapped unit. Both are costly, but neither is the open-ended "did I mis-control a CQA across a 14-day batch" problem that makes upstream ML so fraught. The error is local to one object, the ground truth is checkable after the fact (you can pull the rejected vial and look), and the two error types have a clean asymmetry — a missed defect is a safety event, a false reject is only money — that lets you tune the operating point deliberately rather than chase a single accuracy number.

Third — and this is the deepest reason — the defect is often visible, which means you can collect labelled training data in a way you almost never can upstream. A particulate, a crack, a low meniscus, a missing stopper: these are things a camera can see and a trained human can label. The bottleneck shifts from "we have no ground truth" to "we have to build a comprehensive defect library and validate against it" — a hard problem, but a tractable one, and one the industry has now solved well enough to deploy in commercial GMP. That is why, when you ask where ML has truly reached production in biomanufacturing, the honest answer clusters around monitoring, predictive maintenance, and — above all — vision inspection, not autonomous control of CQAs [2].

Fill-weight and dose accuracy: the case for not using ML

Before the glamorous vision problem, the unglamorous one that runs on every line: did each vial get the right amount of drug? An aseptic filler dispenses a nominal volume — 1.0 mL in our dataset — and an in-process-control (IPC) checkweigher weighs vials (continuously, by 100% net-weight checkweighing, or by sampling) to confirm the delivered dose stays within a dose-accuracy specification, typically about ±5% of target for a liquid fill. The checkweigher reads weight, so the delivered volume is back-calculated as net weight divided by the measured density of the formulation solution (about 1.01 g/mL for this buffered mAb-A (monoclonal-antibody product A) formulation, illustrative) — which is why a 1.0107 g mean weight reads as a 1.0008 mL mean volume rather than the two coinciding at 1.000. This is a control problem, and it is the cleanest possible illustration of a lesson this whole book keeps insisting on: reach for the simplest tool the physics allows, and only escalate when it genuinely helps.

The right tool here is classical statistical process control, not machine learning. From the line's own within-run variation you derive individuals (I) control limits and only then compare against the spec. The math is the same sigma = MR-bar / d2 short-term-sigma estimate from the open-source SPC chapter: for individual measurements you cannot average within a rational subgroup, so you estimate the short-term standard deviation from the average moving range — the mean absolute successive difference, MR-bar — divided by the control-chart constant d2, which for the n equals 2 successive-pair case is 1.128. Three sigma either side of the centre gives the individuals control limits. The reason you use the moving range rather than the raw sample standard deviation is subtle and important: a slow drift inflates the overall spread but barely touches successive differences, so a moving-range sigma stays tight and the chart fires on the drift instead of absorbing it. Control limits come from the process, not the spec — an action limit that trips when the line wanders is what catches a developing fill problem before any vial ever goes out of dose spec.

You then ask a separate question with a capability index, Cpk: given where the process is centred and how tight its spread is, how comfortably does it sit inside the ±5% dose window? Cpk is the distance from the process mean to the nearer spec edge, measured in units of three sigma — formally the minimum of (upper spec minus mean) over three sigma and (mean minus lower spec) over three sigma. A Cpk of 1.33 is the conventional "capable" floor; 1.0 means the nearer spec edge sits exactly three sigma out, so roughly one vial in 740 drifts past it. On our real fill line the within-run moving range gives a centre of 1.0107 g and individuals limits of roughly 0.951 to 1.071 g, while the delivered-volume capability against the ±5% spec is Cpk ≈ 0.84 — below the capable floor, a marginal capability that explains, honestly, why this line does reject the occasional vial: two of the 480, both for low fill. The line is not broken; it is tight against a tight spec, which is exactly the regime where rejects are real and a control chart earns its keep.

Now the tempting move: why not learn the reject boundary with a classifier? You can — and the result is a quiet object lesson. With 478 good vials and 2 rejects, the class imbalance is brutal (about 0.4% positives). Left alone, a classifier would minimise its loss by calling everything good and scoring 99.6% accuracy while catching zero defects — the textbook failure of accuracy on imbalanced data. So you re-weight the loss (class_weight="balanced" up-weights the two rejects by the inverse of their frequency, roughly 240-fold) to force the model to take the minority class seriously. A class-balanced logistic regression on (fill_volume, fill_weight) then does catch both true rejects, but only by drawing its decision boundary so conservatively that it also flags dozens of perfectly good vials as suspect. The learned threshold recovers the fixed limit; it does not beat it — and a learned, data-derived boundary is far harder to validate under GMP than a fixed numeric limit a reviewer can read in one line. This is the false-reject problem in miniature, and it foreshadows exactly the trade-off that makes deep-learning AVI valuable: the win from ML in fill-finish is not catching defects a rule misses, it is catching them without throwing away good product.

The fill-finish station, learned: a marginally-capable fill controlled by statistics not ML, an optional lyophilization cycle with a soft-sensed endpoint, and the production-grade deep-learning vision inspection that accepts or rejects each individual vial — the one place in biomanufacturing where a model's decision routinely stands on its own. Original diagram by the authors, created with AI assistance.

# examples/platform/ml/fill_control.py — fill-weight control + a reject model on
# the REAL fill line (480 vials of DP-001, BATCH-2026-001). The lesson is that the
# fixed checkweigh limit beats the learned threshold; the classifier only recovers it.
TARGET_ML = 1.0
SPEC_LOW_ML, SPEC_HIGH_ML = 0.95, 1.05   # +/- 5% dose-accuracy spec on a 1.0 mL fill
D2 = 1.128                               # moving-range control-chart constant for n=2

def control_limits(weights):
    w = np.asarray(weights, dtype=float)
    mr_bar = np.abs(np.diff(w)).mean()                  # average within-run moving range
    sigma = mr_bar / D2                                 # short-term sigma (drift-robust)
    center = w.mean()
    return {"center_g": round(center, 4), "sigma_g": round(sigma, 5),
            "ucl_g": round(center + 3 * sigma, 4), "lcl_g": round(center - 3 * sigma, 4)}

def capability(volumes):                                 # Cpk vs the +/- 5% dose spec
    v = np.asarray(volumes, dtype=float)
    mu, sd = v.mean(), v.std(ddof=1)
    cpu, cpl = (SPEC_HIGH_ML - mu) / (3 * sd), (mu - SPEC_LOW_ML) / (3 * sd)
    return {"mean_mL": round(mu, 4), "cpk": round(min(cpu, cpl), 3)}

def reject_model(df):                                    # the tempting, inferior ML move
    X = df[["fill_volume_mL", "fill_weight_g"]].to_numpy()
    y = df["reject"].astype(int).to_numpy()
    clf = LogisticRegression(class_weight="balanced", max_iter=1000).fit(X, y)  # ~0.4% positives
    tn, fp, fn, tp = confusion_matrix(y, clf.predict(X), labels=[0, 1]).ravel()
    return {"n": len(y), "n_reject": int(y.sum()),
            "tp": int(tp), "fp": int(fp), "fn": int(fn), "tn": int(tn)}

assert mdl["fn"] == 0, "reject model must not miss a true low-fill vial"
assert mdl["fp"] > 0, "the learned threshold over-rejects vs the fixed limit"

Running it against the committed fill_events.csv prints the whole argument:

Fill line BATCH-2026-001: 480 vials, 2 rejected (reason: low_fill)
Checkweigh control limits (from within-run MR): center=1.0107 g  LCL=0.9507  UCL=1.0708
Dose-accuracy capability vs +/- 5% spec: mean=1.0008 mL  Cpk=0.841
Reject classifier confusion: tp=2 fp=41 fn=0 tn=437
ASSERT ok: a fixed checkweigh limit catches every low-fill vial with no over-rejection; the learned threshold adds false rejects and validation burden without improving detection.

The confusion matrix tallies the four possible outcomes of a yes/no classifier: true negatives (tn, good vials passed), false positives (fp, good vials wrongly rejected), false negatives (fn, defects wrongly passed), and true positives (tp, defects caught). It is the whole argument on one line: the learned threshold catches both true low-fill vials (fn=0) but at the cost of 41 false rejects (fp=41) — over twenty good vials scrapped for every bad one caught. And note these confusion numbers are in-sample: the classifier is graded on the very same 480 vials it learned from (resubstitution), which always flatters a model because it can memorise rather than truly learn. The honest test is a held-out set — data the model never saw in training — that tells you how it generalizes to new vials; but with only two positives there is no way to carve out a leak-free held-out set at all, so you cannot even estimate this boundary's generalization, which is a sharper reason than over-rejection that a learned fill-reject rule is unvalidatable under GMP. A fixed checkweigh limit catches the same two with zero over-rejection and a validation file a reviewer can read in a sentence. The module encodes the lesson as a pair of assertions — fn == 0 (the classifier does catch both) and fp > 0 (but only by over-rejecting) — so the failure mode itself is the test that passes. ML did not lose because it was badly tuned; it lost because the problem did not need it. Remembering which fill-finish problems need learning and which do not is half of doing this well.

Automated visual inspection: the production case for deep learning

Visual inspection is the opposite story. Every parenteral product (an injectable drug, given by needle rather than swallowed) must be inspected for visible particulates and container/closure defects — it is a compendial requirement (USP <790> — a chapter of the U.S. Pharmacopeia, the official drug-quality standards the FDA enforces — is the enforceable General Chapter that mandates 100% visual inspection and sets the acceptance / AQL (Acceptable Quality Level — the sampling-based defect limit) framework for visible particulates, with the informational good-practice guidance of USP <1790> (and Ph. Eur. 2.9.20 in Europe) behind it), historically done by trained humans rotating vials against alternating black-and-white backgrounds under defined lux. Human inspection has two chronic failures: it is inconsistent (inspectors disagree, and the same inspector disagrees with themselves across a shift, with detection probability falling as fatigue rises), and it is over-conservative — to be safe, humans and the older rule-based machines that replaced them reject large fractions of good product. Industry experts have cited rule-based AVI false-rejection rates as high as roughly 20% of good vials (conference-reported) [3], typically because a fixed-threshold machine reads glare off curved glass as a crack, or a transient air bubble as a particle. On a multi-million-vial campaign, a 20% false-reject rate is a staggering, invisible waste of perfectly good medicine.

This is where deep learning genuinely earns its place. The task is image classification: a camera images each vial — often from multiple angles, under controlled lighting, sometimes spinning the vial up and braking it sharply so genuine particulates keep moving against the now-stationary glass features (the same physics that lets a human distinguish a floating fibre from a fixed scratch). A convolutional neural network (CNN) then classifies the resulting image as accept, or as one of several defect types (particulate, crack, low fill level, stopper/closure defect, cosmetic). A CNN is the right tool precisely because this is not a tabular problem: the signal is a spatial pattern of pixels, and convolutional layers are designed to learn local features (an edge, a speck, a meniscus) and combine them hierarchically into a defect/no-defect judgement. You cannot hand a logistic regression a flattened image and expect it to find a small particle against the glass; a flat vector throws away the very thing that defines the defect — where the bright pixels sit relative to each other. You need a model whose architecture matches the structure of the data.

Why a CNN, structurally

A convolutional layer slides the same small learned filter — a 3-by-3 grid of weights, in our sketch — across every position of the image, multiplying and summing under the window at each stop. Because the same filter is reused everywhere (weight sharing), a feature detector that learns "small bright speck against dark glass" fires wherever the speck appears, and the layer needs only a handful of weights per filter regardless of image size. That is the property a flattened dense layer destroys: it would have to learn the speck separately for every pixel location. Stacked convolution-and-pooling blocks then build a hierarchy — early filters catch edges and specks, deeper filters combine those into larger structures (a meniscus line, a crack's branching) — while max-pooling halves the spatial resolution at each step so the deeper layers see a wider field of view for the same filter size. A final classification head maps the learned features to the defect classes. Two more pieces in the sketch matter in practice: batch normalization after each convolution stabilises and speeds training by re-centring each layer's activations, and dropout before the output layer randomly zeros a fraction of features during training to fight overfitting on a finite defect library. The sketch in our example suite makes the shape contract concrete with three convolutional blocks, global average pooling, and a six-class head:

# examples/platform/ml/vision_avi.py — an HONEST CNN sketch for AVI (no real vial
# images ship in the dataset). It fixes the architecture and shape contract; the
# real accept/reject numbers (Amgen ~95% auto-release) are vendor/self-reported.
import torch, torch.nn as nn

IMG_CH, IMG_HW = 1, 128                                  # grayscale ROI cropped to the meniscus
CLASSES = ["accept", "particulate", "crack", "fill_level", "stopper", "cosmetic"]

class VialInspectorCNN(nn.Module):
    def __init__(self, n_classes=len(CLASSES)):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(IMG_CH, 16, 3, padding=1), nn.BatchNorm2d(16), nn.ReLU(),
            nn.MaxPool2d(2),                                          # 128 -> 64
            nn.Conv2d(16, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
            nn.MaxPool2d(2),                                          # 64  -> 32
            nn.Conv2d(32, 64, 3, padding=1), nn.BatchNorm2d(64), nn.ReLU(),
            nn.AdaptiveAvgPool2d(1),                                  # -> 64 x 1 x 1
        )
        self.head = nn.Sequential(nn.Flatten(), nn.Dropout(0.3), nn.Linear(64, n_classes))

    def forward(self, x):                                            # x: (batch, 1, 128, 128)
        return self.head(self.features(x))

torch.manual_seed(2026)                          # reproducible stand-in ROIs and weights
model = VialInspectorCNN().eval()                # LOCKED model: eval() only, no learning in prod
for p in model.parameters():
    p.requires_grad_(False)                      # weights frozen at validation (draft Annex 22)
vials = torch.randn(8, IMG_CH, IMG_HW, IMG_HW)  # stand-in for camera ROIs
with torch.no_grad():
    prob = torch.softmax(model(vials), dim=1)
    decisions = prob.argmax(dim=1)              # class 0 = accept; anything else = a reject reason
print(f"AVI CNN (sketch): {sum(p.numel() for p in model.parameters()):,} params, "
      f"{len(CLASSES)} classes, decisions={decisions.tolist()}")

AVI vial-inspection CNN (sketch) — locked at validation, eval() only
  classes        : ['accept', 'particulate', 'crack', 'fill_level', 'stopper', 'cosmetic']
  parameters     : 23,910
  input  (b,c,h,w): (8, 1, 128, 128)
  output (b,classes): (8, 6)
  decisions (argmax class id): [0, 0, 0, 0, 0, 0, 0, 0]
  NOTE: real accept/reject performance (e.g. Amgen ~95% auto-release) is vendor/self-reported; this sketch only fixes the shape contract.

Our 23,910-parameter sketch is a toy — every decision lands on accept because the weights are random and untrained; the print only proves the tensors flow from an 8-by-1-by-128-by-128 batch of ROIs to an 8-by-6 logit matrix. Production AVI nets are far deeper and trained on proprietary defect libraries running to tens or hundreds of thousands of labelled images. But the structure is the real structure, and two design choices in the sketch are load-bearing in production. The model runs in eval() mode with weights frozen via requires_grad_(False) — a locked model that does not learn in production, the posture every regulator now expects. And the output is not a bare accept/reject but a softmax probability over classes, so the system can compare the accept-probability to a routing threshold and send low-confidence vials to human review rather than guessing — the human-in-the-loop pattern that makes AVI deployable.

Recall is the metric, and false rejects are the cost

The model has two ways to be wrong, and they are not symmetric. A false accept — a defective vial scored as accept — is a safety event: a particulate or a cracked container reaching a patient. A false reject — a good vial scored as defective — is only scrapped medicine. So the metric that governs the safety case is recall (sensitivity): of all the truly defective vials, what fraction did the model catch? A validated AVI system must demonstrate recall at least as good as the manual method it replaces, on every defect class, including the rare and borderline ones. Recall is the acceptance gate a deployed AVI program must clear, and the suite records it as vision_avi.py's intended criterion — the run names its purpose as "defect classifier recall above the gate." But with no labelled vial images in the dataset the sketch only fixes the per-vial six-class shape contract; it cannot measure recall, and it passes the harness by exiting cleanly, not by asserting a held-out number. That gap is the point: a model that quietly misses defects is worse than useless, it is dangerous — which is precisely why the defect library, not the network, is the regulated artifact.

Recall alone is trivial to maximise (reject everything), so the economic case is governed by the false-reject rate, and the whole value of deep-learning AVI is improving both at once: catching more real defects (higher recall) while rejecting far fewer good vials (lower false-reject rate). That two-axis improvement is precisely what separates a learned model from the fixed-threshold machines — and it is the same trade-off the fill_control.py lesson showed in miniature, now flipped to the case where learning wins. You tune the operating point — the accept-probability threshold below which a vial is routed to a human — deliberately, sliding it to buy more recall at the cost of more human review, never the reverse.

The real bottleneck: the defect library and the validation

The neural network is the easy part. The hard, regulated, expensive part of an AVI program is the defect library: a curated, labelled set of images covering every defect type at every severity, including rare and borderline cases, against which the model is trained and — far more importantly — validated. To prove an AVI system is at least as good as the manual method it replaces, you must demonstrate its detection performance on a representative population of known defects and known-good units, the way any compendial inspection method is qualified — with seeded defects (deliberately manufactured cracks, spiked particulates of known size, under-filled vials) and human-adjudicated ground truth on every borderline image. Building and maintaining that library is where the years of work go, and it is also the thing that decays: a new glass supplier, a new stopper, a new particulate type the library never contained, and the validated model is now operating outside the population it was qualified on. Amgen has been explicit that its move to roughly 95%-auto-released vials and syringes took years of effort and direct conversations with the FDA (vendor/self-reported) [1], and the first fully validated retrofit was a syringe line at Juncos, Puerto Rico, not a vial line — a reminder that "the strongest production ML case" is still a multi-year, heavily-scrutinized program, not a plug-in.

The commercial landscape is real and named. Stevanato Group's Vision AI platform markets deep-learning inspection with claimed detection accuracy up to 99.9% and roughly an order-of-magnitude (tenfold) reduction in false rejects (vendor materials) [4]; Brevetti CEA fully absorbed the Danish deep-learning specialist Brevetti AI (formerly Criterion AI, founded 2018) in 2025 to build the same CNN-and-anomaly-detection capability for regulated inspection [5]; Syntegon's AIM (AI-enhanced inspection) reports materially more particle detection with materially fewer false rejects (vendor-reported), and Syntegon was Amgen's equipment partner on the validated retrofit; and Cognex, Antares Vision, Körber, and Microsoft's Bonsai (the reinforcement-learning / machine-teaching platform pitched at industrial control and inspection) play in or adjacent to the same space. The headline numbers are vendor- or conference-reported, not peer-reviewed, and should be read as such — indeed the most defensible Amgen/Syntegon figures published for a critical station are the more modest "roughly 70% higher particle detection and roughly 60% fewer false rejects," not the round 95%. But the direction is consistent and corroborated across vendors and the one named pharma deployment: deep-learning AVI detects more real defects while rejecting far less good product than the rule-based machines it replaces.

Anatomy of one AVI accept/reject record

A deployed AVI station does not emit a bare "reject." Like every artifact in this series, the value is in what travels alongside the decision — the provenance that lets a reviewer, a regulator, or an investigation reconstruct why this vial went to the scrap bin and trust that the model that sent it there was the validated one.

One inspection is a whole record: the imaged region of a single serialized vial, the locked model and the validation library it was qualified against, the per-class confidence and the routing threshold that sends borderline vials to a human, and the links back through DP-001 and DS-001 to the release decision — provenance is what lets an automated reject stand under GMP. Original diagram by the authors, created with AI assistance.

Read the card top to bottom and the chapter's whole argument is laid out as fields. The header pins the unit to a single serialized vial — 00361414000017.0000150, where the leading fourteen digits are the GTIN-14 of the mAb-A drug-product presentation (the product, carried under GS1 AI 01 — the same number on every vial of this product) and the suffix is this vial's serial, the same identity the packaging chapter commissions for track-and-trace; the vial is part_of the DP-001 lot rather than the GTIN naming that lot. That serialized identifier is also the stable join key Book 4 promotes to the vial's IRI, so this record can be joined to every other system that names the same vial. The input rows are the cheap, fast signal: a timestamp, the camera station and view, a reference to the cropped region-of-interest image, and the lighting recipe — because an AVI model is only valid under the imaging conditions it was trained on, and a changed bulb or backdrop silently moves the input distribution out of the validated population. The green core is the decision: a predicted_class (accept, or a named reject reason), the confidence distribution over all six classes, and an accept_confidence compared to a review_threshold that routes uncertain vials to a human rather than letting the model guess. The model-identity block pins the model_id, model_version, and the hash of the validation_library the model was qualified against, plus a flag that the weights were frozen and the model ran in eval() mode — so the record proves which locked model made this call and that it could not have drifted mid-shift. The reconciliation rows tie the vision decision to other evidence on the same serial: the human_verdict for routed vials (the human-in-the-loop adjudication that becomes new ground truth), and the in-process checkweigh_g weight from the fill-control model above, so a low-fill flagged by the camera can be cross-checked against the scale. The violet relationships panel records lineage — validated_by the qualification study, trained_on the defect library, part_of the DP-001 lot deriving from DS-001, and feeds the release decision — the same DP-001 vial that Book 4's instance graph models as a node, with these very derivedFrom edges back through DS-001 to the cell bank (and contains edges up the carton/case/pallet hierarchy — containment, not lineage).

The record is a typed node, not a JSON blob

That identity card is more than a layout: read with Book 4's ontology underneath, every field is a typed statement, and the typing is what makes the record trustworthy enough to feed a model. The serialized vial is a continuant — a thing that persists and bears qualities through time — kept categorically distinct from the inspection event that judged it, which is an occurrent: a happening with a timestamp, a station, and a model that ran once. Fuse the two (a single "reject row" that is half object, half event) and the BFO continuant/occurrent cut collapses, and with it the ability to say this same vial was re-imaged on a second station, or that the camera inspected a hundred thousand other vials. The part_of, derivedFrom, and contains edges are not free-text strings either: each resolves to a named object property in the bp: vocabulary, with derivedFrom aligned up to the OBO Relation Ontology's derives from (RO_0001000) so a recall walk over the vial's lineage means the same thing to any tool that imports the standard term, not just to this line's software. The lineage edge specifically is owl:TransitiveProperty, which is what lets the violet panel claim the vial traces back through DS-001 to the cell bank from only the immediate parent hops the record asserts — the genealogy spine doing the inference, not the AVI station.

This typing pays off twice. First, it is what makes the inspection record a semantically-grounded training feature rather than a fragile column. When the next vintage of the AVI model is trained, a feature is pulled by its ontology IRI (bp:acceptConfidence on a vial that derivedFrom a given DS lot), not by a column name a schema migration can silently rename out from under the learner — the same leak-proof-label-store property the relations chapter argues for the upstream affectsQuality rows. Second, the genealogy is the grouping key for honest validation: two drug-product lots that share a derivedFrom ancestor are not independent examples, so an AVI model re-trained on accumulated inspection records must be scored by a leave-one-batch-out split that holds out a whole derivedFrom-connected lineage at once — the same GroupKFold discipline the rest of this book insists on, here handed to the model for free because the record already records which campaign each vial descends from. A random row-wise split would let near-twin siblings off one cell bank leak across the train/test line and report a validation recall that evaporates on a genuinely new batch.

The same shape gate, pointed at the training set

There is a deeper reuse hiding in the validation_library hash. The release decision the QC chapter makes is, in Book 4, a SHACL shape — a closed-world gate that fails a lot whose required CQA panel is missing, duplicated, or out of range, the question no open-world reasoner can ask because a missing result reads to it as merely "unknown." Point that same shape vocabulary at a candidate training subgraph instead of a lot and it refuses a non-conformant dataset: every inspection record must carry its derivedFrom parent, every confidence its six-class distribution, every vial its lighting recipe and locked model_version, before a single row is handed to a learner. That is the closed-world catch an ML pipeline almost never makes on its own — a silently missing feature trains a model that cheerfully invents what the data never said, exactly the validation-versus-learning failure where a fluent model narrates a false story as smoothly as a true one. SHACL-validating the inputs is the same release-gate rigor, moved one step earlier to the data that makes the model rather than the lot the model judges; it is also what lets the record satisfy the FAIR expectation (findable, accessible, interoperable, reusable) that the analytics stack assumes — an inspection row keyed by a resolvable IRI and typed against a published vocabulary is reusable by a tool that never saw this line.

Why the standards underneath this matter to the model

The inspection record does not live alone; it has to interoperate with the plant systems that already name the same vial, and that interoperability is what keeps the training set honest at scale. The vessel-batch-run separation the ontology draws is the same one ISA-95 (the enterprise-to-plant data model, IEC 62264) carries as separate EquipmentElement, MaterialLot, and production-response objects, and that its B2MML XML serialization keeps in separate elements — so the AVI station's "vial from batch BATCH-2026-001 on this fill line" is not a private string but a MaterialLot reference a semantic-interoperability layer can re-home onto the one bp: IRI for that lot. The live signals the record cross-references — the in-process checkweigh_g, the isolator particle counts — arrive over OPC UA (the open machine-to-machine protocol, OPC Unified Architecture, not the legacy Windows-only OPC DA), whose information model is itself a typed object graph rather than flat tags. Mediating every source through one shared reference model — the data-shadow and governance discipline of Book 2 — is what turns an n-squared tangle of pairwise integrations into a handful of clean alignments, and it is the difference between an inspection dataset whose lineage you can trust and a pile of CSVs whose join keys quietly disagree. A model is only as trustworthy as the data shadow it is trained on, and the data shadow is only as trustworthy as the standards that keep two systems' names for one vial reconcilable.

Lyophilization: soft-sensing the freeze that cannot be rushed

Many biologics are not stable as liquids, so after filling they are lyophilized — frozen solid, then dried under deep vacuum so the ice sublimes directly to vapour (primary drying), after which a final warmer secondary-drying step pulls off the residual bound water, leaving a dry, stable cake that is reconstituted before use. Lyophilization is slow (often days), energy-hungry, and unforgiving: dry too aggressively and the cake collapses or the product is damaged when the product temperature rises above the temperature where the part-dried matrix loses its structure — the collapse temperature in a glassy, non-crystalline (amorphous) formulation, or the lower eutectic-melt temperature in one whose solutes crystallize; cross either and the cake slumps; dry too gently and you waste days of cycle time and freezer capacity. Cycle development and control is therefore a genuine optimization problem with real money and real product risk on both sides — exactly the kind of problem where modeling pays.

The learning targets here are mechanistic-soft-sensing problems — mechanistic meaning grounded in the known physics equations (heat and mass transfer), as opposed to a purely data-learned fit. The quantity you most want to know during primary drying is the product temperature at the sublimation front and the moment primary drying ends (when all the free ice is gone), neither of which is easy to measure in every vial without intruding on the sterile boundary — and the few thermocouple-instrumented vials you can place are unrepresentative, because a probed vial nucleates and dries differently from its neighbours. Classical approaches estimate these from chamber pressure, shelf temperature, and the partial pressure of water vapour, and the field has well-developed first-principles models of coupled heat and mass transfer through the drying cake (sublimation interface temperature as a function of shelf temperature, vial heat-transfer coefficient, and dry-layer resistance). The learning lens adds two things on top of that physics. A soft sensor fuses the indirect signals — several independent, whole-batch ways to watch how much water vapour is still leaving the cake (a Pirani gauge reads high while vapour is present and drops toward the true-pressure capacitance-manometer reading once the ice is gone; comparative / pressure-rise manometric-temperature measurement, MTM, briefly isolates the chamber and watches the pressure rise; tunable-diode-laser water-vapour spectroscopy counts water molecules directly), plus the falloff in sublimation rate — into a continuous estimate of product temperature and a predicted primary-drying endpoint, which arrives the moment those signals all level off, so the cycle advances to secondary drying at the right moment instead of a conservative fixed time that pads every batch with hours of margin. MTM and the Pirani/capacitance-manometer convergence are the established whole-batch endpoint methods the learned soft sensor builds on, not replaces — the classical, well-validated precursor the ML lens extends.

And design-space modeling of the cycle — the same Bayesian-optimization and surrogate-modeling logic from process development — explores the shelf-temperature / chamber-pressure operating space to find a cycle that is fast and keeps product temperature safely below its collapse point, replacing a costly factorial grid of full multi-day cycles with a handful of informative ones. Because the physics is genuinely well understood, the strong play is hybrid: a mechanistic heat-and-mass-transfer backbone with a learned component for the parts that resist clean equations (vial-to-vial heterogeneity, shelf edge effects, cake morphology), the dominant paradigm this book returns to in the digital-twins chapter.

Container-closure integrity and the cleanroom

Two more fill-finish targets round out the picture, both earlier on the maturity curve. Container-closure integrity testing (CCIT) asks whether the seal between the vial and its stopper actually holds — a leak means a compromised sterile barrier, and for a lyophilized product a lost vacuum that ruins the cake. Non-destructive CCIT methods — the deterministic, physicochemical class that USP <1207> now favors over probabilistic microbial-ingress and dye tests for lifecycle integrity: laser-based headspace gas analysis, high-voltage leak detection, vacuum decay, mass extraction — produce continuous signals that a classifier can learn to map to pass/fail, and ML is being layered onto these inline gauges to flag marginal closures the way the checkweigher flags marginal fills. That deterministic, continuous-signal preference is exactly why CCIT is a plausible and current ML target: the methods most suited to learning are the ones the compendium now recommends. The same residual-against-expectation logic from the rest of this book applies: a closure whose headspace oxygen or moisture signature drifts from the population is suspect even when it has not yet tripped a hard limit — the leading indicator that catches a stopper-seating problem before it produces a failed unit.

Aseptic and isolator environmental monitoring is the other frontier. The fill happens inside a Grade-A isolator (the cleanest air classification, reserved for the open sterile operation), continuously monitored for non-viable particles (inert specks) and viable contamination (living microbes) — exactly the em_samples.csv data the QC chapter leans on, with its seeded Grade-A excursion. The learning opportunity is contamination-risk prediction: treating the particle counts, pressure differentials, airflow, and recovery patterns as a multivariate stream and predicting an excursion (a measurement straying outside its acceptable range) before it breaches a limit, rather than reacting after the alarm. This is monitoring, not control — squarely in the human-in-the-loop, advisory category regulators endorse — and it is one of the production-leaning clusters (monitoring and anomaly detection) where ML in pharma genuinely lives today, as opposed to the autonomous-control cluster where it mostly does not [2].

The unsolved part: the validation-versus-learning paradox

The honest open problem here is not a missing algorithm. It is a contradiction at the heart of regulated AI, and AVI is where it bites hardest because AVI is where a model's decision actually stands on its own — there is no human soft-sensor consumer downstream to catch a bad call, the way there is for an upstream titer estimate. The reject goes to the scrap bin, the accept goes toward a patient, and the model's judgement is the gate.

The contradiction is this. The thing that makes machine learning valuable is that it can keep learning — see new defect types, adapt to a new vial geometry, improve as more labelled images accumulate (and an AVI line accumulates them every shift, with every human-adjudicated borderline vial). The thing that makes a regulated process trustworthy is that it does not change without control: a validated system must behave the way it did when it was validated, every time, or the validation is meaningless. A model that keeps learning is, by definition, a moving target that traditional one-time validation was never built for — you cannot certify the behaviour of something whose weights changed since you certified it. Amgen has framed its own AVI journey explicitly around this tension — the validation-versus-learning paradox — and the resolution it and the regulators have converged on is to lock the model at validation (here validation carries its strict GMP meaning — documented evidence proving the system consistently performs as intended, the regulatory bar GMP software must clear before use): freeze the weights, validate the frozen artifact like any other piece of GMP software, and only ever update it through a formal, pre-planned change-control process [1][6]. The vision_avi.py sketch enacts exactly this — eval() plus requires_grad_(False) — not as a coding detail but as the regulatory posture made executable.

The regulatory frame around this is now sharp. The FDA's risk-based credibility framework — its seven-step approach for establishing the credibility of an AI model for a given context of use, explicitly covering the manufacturing phase where AI output affects product quality — scales the evidence you must bring to the consequence of the decision; an AVI model that auto-releases sterile injectables sits at the high-consequence end and carries a correspondingly heavy evidentiary burden [7]. The draft EU/PIC/S GMP Annex 22 (an annex — a topic-specific appendix — to the GMP rules shared by the EU and the international Pharmaceutical Inspection Co-operation Scheme), the first manufacturing-specific AI rule, goes further for critical applications: it permits only static, deterministic models and excludes dynamic, continuously-learning, probabilistic, and generative AI from critical GMP use, demanding a model locked at validation governed by a predetermined change-control plan [8][9]. And the consequences of ignoring the human-review boundary are no longer hypothetical: in April 2026 the FDA issued its first AI-citing cGMP (current GMP) warning letter — an official enforcement notice of regulatory violations — to Purolea, a firm that used AI agents to generate specifications and production records without quality-unit review [10].

So the paradox is managed, not solved. Locking the model buys validatability at the cost of the very adaptability that motivated ML; every improvement now costs a re-validation, and the months between re-validations are months the field's best new idea sits unused. The genuinely open questions are the ones the predetermined-change-control-plan concept only gestures at: how do you pre-specify a model's allowed evolution tightly enough to validate yet loosely enough to be useful; how do you detect that a locked AVI model has silently decayed — a new glass supplier, a new lighting bulb, a new particulate type it was never trained on — before it misses a real defect, given that a locked model by construction never tells you it has fallen behind; and how much of the accept/reject judgement can ever responsibly leave the human entirely. The 95%-auto-release number is real and impressive — and the remaining 5% routed to humans is not a rounding error. It is the standing admission that even the strongest production ML case in biomanufacturing keeps a person in the loop on purpose.

What this chapter adds to the model suite

This chapter contributes two modules to examples/platform/ml/, and they are deliberately a matched pair that makes opposite points:

• fill_control.py — fill-weight control and a reject model on the real fill_events.csv line. It derives individuals control limits from the within-run moving range and a dose-accuracy Cpk from the line's own statistics, then trains a class-balanced logistic classifier on the 478-to-2 imbalance to show that the learned threshold only recovers the fixed checkweigh limit (tp=2, fn=0) while adding 41 false rejects and validation burden. The honest lesson is encoded as the assertions that pass: a fixed limit catches every low-fill vial with no over-rejection; ML does not improve detection here.
• vision_avi.py — a CNN sketch for automated visual inspection. With no vial images in the dataset, it fixes the architecture and the shape contract (a six-class defect head over a 128×128 grayscale ROI, 23,910 parameters), runs one forward pass with frozen weights in eval() mode to embody the locked-model discipline, and labels every real-world performance number (Amgen ~95% auto-release) as vendor/self-reported. Its recorded criterion is recall above threshold — the safety-critical metric a deployed program must clear — but with no labelled images the sketch asserts only the shape contract, not a held-out recall; the gated modules elsewhere assert their acceptance criterion as a real held-out metric where one exists, and as a structural guarantee (here, the per-vial six-class shape contract) where it does not.

Together they encode this chapter's two-sided thesis: ML in fill-finish is sometimes the wrong tool (fill weight) and sometimes the strongest tool in the whole book (vision) — and telling the two apart is the skill.

Both modules run under the suite's open-source reproducibility contract, which is itself part of the honesty. vision_avi.py seeds the stand-in ROIs and weights (torch.manual_seed(2026)) so the printed shapes and parameter count are byte-stable across machines; fill_control.py is deterministic over the committed fill_events.csv. The whole suite pins its dependency versions and runs end to end under run_all.py, so a reviewer reproduces the printed numbers rather than taking them on faith — the same fixed-seed, version-pinned discipline the open-source analytics stack treats as the price of admission for a model anyone is asked to trust. The flip side, stated plainly: no real vial images ship in the dataset, so vision_avi.py is and remains a sketch that fixes a shape contract, not a trained classifier — and the data-split governance that would make a real AVI score admissible (the leave-one-batch-out grouping above, keyed by the genealogy) lives in the ontology's lineage, not in a column of this CSV. The open companion shows the honest skeleton; the regulated defect library and the validated weights are the parts a textbook cannot ship.

Why it matters

Fill-finish is where machine learning stops being a promise and becomes a routine. The discreteness of the unit, the binary decision, and the visibility of the defect together make this the one operation where a learned model's judgement can — after a multi-year, heavily-validated program — stand on its own in commercial GMP. Get the AVI program right and you detect more real defects while scrapping far less good medicine, which is both a safety win (higher recall) and an enormous economic one (a fraction of the old false-reject rate). Get the fill-weight problem right by not over-engineering it, and you keep your validation burden where it belongs. And carry the validation-versus-learning discipline through both — locked models, human-in-the-loop routing, predetermined change control — and you have a template for how ML earns trust everywhere else in the plant. Fill-finish is the proof of concept the rest of biomanufacturing's ML ambitions are measured against.

In the real world

The deepest production deployment is Amgen's: roughly 95% of syringes and vials released through automated visual inspection, the result of a multi-year effort built around the validation-versus-learning paradox and direct FDA engagement, first validated on a syringe line at Juncos, Puerto Rico (vendor/self-reported; the more conservative published critical-station figures are ~70% higher particle detection and ~60% fewer false rejects) [1]. On the equipment side, deep-learning AVI is a real, competitive product market — Stevanato Vision AI, Brevetti CEA (via its Brevetti AI / Criterion AI acquisition), Syntegon AIM, Cognex, Antares Vision, Körber, and Microsoft Bonsai all build in or adjacent to it — with vendor-reported gains of an order-of-magnitude fewer false rejects and substantially higher particle detection than the rule-based machines they replace [4][5]. Lyophilization cycle modeling and soft-sensing of the drying endpoint are mature in cycle development and increasingly in control; CCIT-ML and cleanroom contamination-risk prediction are earlier, in the monitoring cluster where pharma ML is genuinely advancing. What unifies all of it is the regulatory posture now hardening around AI in manufacturing — the FDA credibility framework, draft Annex 22's exclusion of adaptive and generative AI from critical use, and the Purolea warning letter as the enforcement anchor — which together say, unambiguously, that the model may be smart but the medicine's safety still rests on a locked artifact and a human who can be held accountable [7][8][10].

Key terms

• GMP (Good Manufacturing Practice) — the legally binding, regulator-inspected rules a medicine must be made under, to a fixed and validated procedure; "cGMP" is the current GMP in force.
• CQA (Critical Quality Attribute) — a product property (purity, potency, dose) that must stay within range for the product to be safe and effective.
• Soft sensor — software that estimates a hard-to-measure quantity (here product temperature, or the primary-drying endpoint) from other, easier signals.
• Fill-finish — the operation that dispenses bulk drug substance into final containers (vials, syringes), optionally lyophilizes them, inspects them, and releases the passing units as the drug-product lot.
• Automated visual inspection (AVI) — camera-plus-software inspection of every container for particulates and defects; the deep-learning version is the strongest production ML case in biomanufacturing.
• Convolutional neural network (CNN) — the image-classification model family used for AVI, whose convolutional layers learn local-to-global spatial features (a speck, an edge, a meniscus) via weight-shared filters that a tabular model cannot.
• Defect library — the curated, labelled image set of every defect type and severity (with seeded defects and human-adjudicated ground truth), used to train and — critically — validate an AVI model; the real bottleneck of an AVI program.
• Recall (sensitivity) — the fraction of truly defective vials the model catches; the safety-critical metric for AVI and the acceptance gate a deployed program must clear (the vision_avi.py sketch records it as its intended criterion but, lacking labelled images, asserts only the shape contract), because a missed defect is far worse than a false reject.
• False reject (over-rejection) — scrapping a good unit; the chronic failure of human and rule-based inspection that deep-learning AVI most reduces.
• In-process-control (IPC) checkweigher — the gauge that weighs filled vials to confirm dose accuracy against a typically ±5% spec; a control problem best solved with statistics, not ML.
• Capability index (Cpk) — the distance from the process mean to the nearer spec edge in units of three sigma; our fill line sits near Cpk 0.84, below the 1.33 capable floor, which is why it rejects the occasional vial.
• Lyophilization (freeze-drying) — freezing then vacuum-subliming a product into a stable cake; the learning targets are product-temperature soft sensing, primary-drying-endpoint prediction, and cycle design-space optimization.
• Container-closure integrity testing (CCIT) — non-destructive verification that the vial-stopper seal holds; an emerging ML/anomaly-detection target.
• Locked model — a model whose weights are frozen at validation (here, eval() plus no-gradient) and changed only through a predetermined change-control plan; the regulatory resolution of the validation-versus-learning paradox.
• Validation-versus-learning paradox — the contradiction that ML's value is in adapting while GMP's trust requires not changing; managed by locking the model, never fully solved.
• Predetermined change-control plan (PCCP) — a pre-specified, pre-validated envelope for how a model may be updated, so improvements do not require ad hoc re-validation each time.
• Continuant / occurrent — the BFO upper-ontology cut that keeps a persisting thing (the serialized vial) categorically distinct from a happening (the inspection event that judged it); fusing them breaks lineage and re-inspection.
• Semantically-grounded feature — a model input pulled by its ontology IRI (bp:acceptConfidence) and typed object property rather than a fragile column name, so a schema change cannot silently rename it out from under the learner.
• SHACL gate on the training set — pointing the same closed-world release-gate shape at a candidate training subgraph, so a dataset with a silently missing required feature is refused before a model is trained on it, not after.
• Leave-one-batch-out (grouped) validation — scoring a re-trained AVI model by holding out a whole derivedFrom-connected lineage at once, keyed by the genealogy, so near-twin sibling vials cannot leak across the train/test split and inflate recall.
• ISA-95 / B2MML / OPC UA — the plant data-model standard (IEC 62264), its XML serialization, and the open machine-to-machine protocol (OPC Unified Architecture, not legacy OPC DA) that let an inspection record's MaterialLot and live signals reconcile to the one IRI naming each vial.

Where this leads

The vials are filled, dried, inspected, and the good ones counted — but no unit ships until the lot is released. The next chapter, QC and Release: MSPC, Real-Time Release, and Predicting the OOS, turns to the gate every batch must pass: the release assays of the hplc_results.csv panel, multivariate statistical process monitoring against the golden batch, real-time release testing that replaces an end-test with a model, and the hardest learning problem of all — predicting an out-of-specification result before it happens, using the one OOS sibling, BATCH-2026-004, as the example.