Electronic Records & Signatures: Part 11 / Annex 11 with Open Source

📍 Where we are: Part V, "Trust." The previous chapter made our data tamper-evident in code; now we ask the harder question — can an all-open-source stack actually satisfy the rules that say an electronic record may stand in for a signed piece of paper? We build the controls that work, and we write down, honestly, the ones that do not.

The simple version

Think of a paper batch record: every entry is initialled, dated, and never erased — mistakes are struck through with a single line so the old value still shows, and a signature at the bottom means "I, this specific person, reviewed this and I stand behind it." 21 CFR Part 11 and EU Annex 11 are simply the rules for doing all of that on a computer instead of paper: who changed what, when, and why must be recorded automatically, and a signature must be unbreakably welded to the exact record it signs. Open source gets you most of that — automatic audit trails, cryptographic signatures, time-stamps from a trusted clock. The last mile, where the system must prove it is the right human pressing the button, is where you start writing procedures and reaching for commercial parts.

What this chapter covers

This is the chapter where the platform meets the regulators. We are not adding a new sensor or a new dashboard; we are taking the relational backbone we already built — the PostgreSQL relational database whose tables hold the platform's records — and asking whether the records it holds are trustworthy in the legal sense. The roadmap:

• What 21 CFR Part 11 [1] and EU Annex 11 [3] actually require, clause by clause, and how PIC/S PI 041-1 [4] turns those clauses into what an inspector looks for.
• A working audit trail built two ways: pgAudit at the database-session level [7], and the trigger-based, hash-chained audit.change_log table from the companion repo that captures old/new/who/when/why (a trigger is code the database runs automatically on every change; a hash chain links each entry to the one before it so any later edit is detectable — both unpacked below).
• Electronic signatures with eLabFTW and RFC 3161 trusted time-stamps [8][6], plus a reason-for-change signing service backed by Keycloak [10].
• An audit-trail review query — the artifact your quality unit actually runs before batch release.
• A brutally honest gap register: the Part 11 clauses where open source alone falls short, and what closes each gap.

Everything labelled with a file path is real, tested code from examples/ that ran in our continuous-integration pipeline (CI — the automated system that re-runs every test on each change, so a broken control turns the build red). Everything labelled illustrative is a realistic snippet for a service that cannot run on a laptop — shown honestly, never claimed to execute.

What the rules actually say

Part 11 is short and old — it became effective in 1997 [1] — and its genius is that it does not name a single technology. It says an electronic record is acceptable in place of paper if the system that produces it enforces a handful of controls. The ones that matter for us live in three clauses. §11.10(e) demands "secure, computer-generated, time-stamped audit trails" that record operator actions creating, modifying, or deleting records, and crucially that do not obscure previously recorded information. §11.70 requires that electronic signatures be "linked to their respective electronic records to ensure that the signatures cannot be excised, copied, or otherwise transferred" to falsify another record. §11.200 governs the signature itself: it must use at least two distinct identification components (think username + password), and after an initial signing in a session, each subsequent signing must re-execute at least one component — so an unattended, still-logged-in terminal cannot be used to sign in someone else's name.

The 2003 Scope and Application guidance [2] is the document that keeps practitioners sane: FDA narrowed Part 11 to a risk-based posture (focusing the strictest controls where the risk to records is highest) and exercised enforcement discretion on some controls (meaning it chose not to pursue action on them) — including several of the record controls in §11.10 itself: validation, audit trail (§11.10(e)), record copies, and record retention. That discretion did not make those controls optional, though — audit trails and protected retention are still required, now anchored in the predicate rules (the underlying GMP regulations the records exist to satisfy) rather than in Part 11 enforcement itself. What FDA did not relax are the electronic-signature provisions — §§11.50, 11.70, 11.100, 11.200, 11.300 (in plain terms: signature manifestations, record/signature linkage, ID uniqueness, signature components & execution, and ID-code/password controls) — which remain fully enforced under Part 11. That split is the line our gap register is drawn along: predicate-rule audit trail and retention, plus the still-enforced Part 11 signature clauses, are exactly what we must satisfy with open source or admit we cannot.

EU Annex 11 [3] is the European counterpart, and it is in places stricter. Clause 9 requires an audit trail for all GMP-relevant changes and deletions — GMP being Good Manufacturing Practice, the rules governing how medicines are made, so a change is "GMP-relevant" when it touches a record those rules require you to control — with a documented reason; clause 12 demands access controls; clause 14 expects electronic signatures to have the same impact as handwritten ones and to be permanently linked to their record. PIC/S PI 041 [4], the inspectors' data-integrity guide, then adds the operational expectations: audit trails must be reviewed, not merely kept, and the review must happen before the record is relied upon — i.e., before batch release. FDA's own data-integrity Q&A [5] says the same thing in plainer words: audit trails that capture creation and modification of GMP data should be reviewed with the same rigour as the data itself. Keep all of this in mind — the deliverable at the end of this chapter is not "we have an audit trail," it is "we have an audit trail someone reviews."

From left to right: the regulatory requirement, the open-source control that meets it, and the honest gap that remains. The green band (audit trail, attributable change capture, trusted time-stamp) is genuinely achievable in OSS; the amber band (re-authentication at signing, WORM retention, high-availability) is where the validated system, procedure, or commercial tooling carries the load. Original diagram by the authors, created with AI assistance.

The audit trail, two ways

There are two complementary places to capture "who changed what, when, and why," and a serious system uses both.

The first is pgAudit [7], a PostgreSQL extension that logs the actual SQL statements a session executes, into the database server log. It is the closest open-source analogue to a tamper-resistant, system-level transcript: every UPDATE lab.result … is written verbatim, with the database user and a server time-stamp, before the application ever touches it. You enable it as illustrative configuration in postgresql.conf (or via ALTER SYSTEM):

# illustrative configuration — platform/db/pgaudit.conf
shared_preload_libraries = 'pgaudit'
pgaudit.log = 'write, ddl, role'   # capture INSERT/UPDATE/DELETE, schema and grant changes
pgaudit.log_relation = on          # one log entry per affected table
pgaudit.log_parameter = on         # record the bound values, not just the statement text

pgAudit is excellent at one thing — an immutable, append-only statement log — and honest about its limits. Its own documentation states plainly that it cannot reliably audit a superuser — an all-powerful database account — because a superuser can change logging settings mid-session. That single sentence is the first entry in our gap register, and the reason we do not stop at pgAudit.

The second place is the application-meaningful audit trail: a table that records the business change in terms a reviewer understands — old row, new row, the human who did it, and the reason. This is the star of the previous chapter and it is real, tested code. From examples/platform/db/50-alcoa.sql, the change-log table and its hash chain:

-- examples/platform/db/50-alcoa.sql
CREATE TABLE audit.change_log (
    seq        bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    ts         timestamptz NOT NULL DEFAULT clock_timestamp(),
    db_user    text NOT NULL DEFAULT current_user,
    app_user   text,                              -- set via SET app.user = '...'
    table_name text NOT NULL,
    action     text NOT NULL,                     -- INSERT | UPDATE | DELETE
    row_key    text,
    old_row    jsonb,
    new_row    jsonb,
    reason     text,                              -- set via SET app.reason = '...'
    prev_hash  text,
    row_hash   text NOT NULL
);

That schema is, almost line for line, a Part 11 §11.10(e) audit trail expressed in DDL. old_row and new_row mean previously recorded information is never obscured — the strike-through-not-erase rule. app_user is the human (not the database account), satisfying ALCOA+'s "Attributable" — ALCOA+ being the data-integrity principles (Attributable, Legible, Contemporaneous, Original, Accurate, plus Complete/Consistent/Enduring/Available) regulators expect of every record — and Annex 11 clause 12 [3]. reason is Annex 11 clause 9's documented reason for change. And prev_hash/row_hash chain each entry to the one before it, so a deletion or edit anywhere in the history breaks the chain and is detectable.

The trigger is what makes capture automatic — the operator cannot forget to write the audit row, because the database writes it for them on every change, using PostgreSQL triggers and roles [9]. The same file attaches it to the regulated tables and ships a verifier:

-- examples/platform/db/50-alcoa.sql
CREATE TRIGGER audit_result   AFTER INSERT OR UPDATE OR DELETE ON lab.result
    FOR EACH ROW EXECUTE FUNCTION audit.log_change();
CREATE TRIGGER audit_batch    AFTER INSERT OR UPDATE OR DELETE ON s88.batch
    FOR EACH ROW EXECUTE FUNCTION audit.log_change();
CREATE TRIGGER audit_recipe_p AFTER INSERT OR UPDATE OR DELETE ON s88.recipe_parameter
    FOR EACH ROW EXECUTE FUNCTION audit.log_change();

-- Verify the chain is intact: returns rows where the recomputed hash breaks.
CREATE OR REPLACE FUNCTION audit.verify_chain()
RETURNS TABLE(seq bigint, ok boolean) AS $$
    WITH chained AS (
        SELECT c.seq, c.row_hash, c.prev_hash,
               lag(c.row_hash) OVER (ORDER BY c.seq) AS expected_prev
        FROM audit.change_log c
    )
    SELECT seq, (prev_hash IS NOT DISTINCT FROM expected_prev) AS ok
    FROM chained
    WHERE prev_hash IS DISTINCT FROM expected_prev;
$$ LANGUAGE sql;

The application sets the human identity and the reason as session variables before the change, and the trigger picks them up. Our test suite, in examples/tests/test_db.py, proves exactly that round-trip — that an UPDATE records old + new + who + why and keeps the chain intact:

# examples/tests/test_db.py
def test_audit_captures_update(conn):
    # an UPDATE must record old + new + who + why and keep the chain intact
    with conn.cursor() as cur:
        cur.execute("select set_config('app.user','pytest',false), "
                    "set_config('app.reason','test correction',false)")
        cur.execute("update lab.result set value = value where result_id = "
                    "(select result_id from lab.result limit 1)")
        conn.commit()
    last = _scalar(conn, "select action from audit.change_log "
                         "where app_user='pytest' order by seq desc limit 1")
    assert last == "UPDATE"
    assert _scalar(conn, "select count(*) from audit.verify_chain()") == 0

The honest caveat, repeated here because it is load-bearing: a superuser who disables the trigger or rewrites the table can still bypass this. The hash chain makes tampering detectable, not impossible. That is the right design for an OSS stack — but it means the controls preventing privileged misuse (separation of duties, restricted superuser accounts, immutable off-box log shipping) live in your procedures and infrastructure, and an inspector will ask to see them.

Anatomy of an audit-trail row: twelve columns, clause by clause

The single most useful thing you can do with that schema is read one row of it slowly, because every column is a regulatory requirement wearing a DDL disguise. Below is a representative audit.change_log row — the UPDATE that corrects a titer (the concentration of the product antibody in the harvest, here in grams per litre) for the day-13 harvest sample BATCH-2026-001-OFF-028 — dissected field by field, with each column mapped to the exact clause it satisfies.

One row of audit.change_log: twelve columns, each one a Part 11 / Annex 11 control expressed in DDL — who, what, when, why, the old value never erased, and a hash that welds the row to its neighbours. Original diagram by the authors, created with AI assistance.

Walk the columns in three groups. The provenance group — seq, ts, db_user, app_user — answers who and when. seq is an IDENTITY column that fixes append order; ts defaults to clock_timestamp(), the §11.10(e) "computer-generated, time-stamped" requirement made literal; db_user records the database account and app_user records the human, and the distinction is the whole point of ALCOA+ "Attributable" and Annex 11 clause 12 — bioproc is a service account (a non-human login the software itself uses to connect to the database), mlee is a person who can be questioned in a deviation — a documented departure from an approved procedure that triggers a formal investigation. The change group — table_name, action, row_key, old_row, new_row, reason — answers what and why. action is the trigger's TG_OP (INSERT/UPDATE/DELETE); row_key is computed by the trigger as coalesce(batch_id, sample_id) so a reviewer can pull a record's whole history with one filter; old_row and new_row are the full jsonb snapshots that make §11.10(e)'s "do not obscure previously recorded information" physically true — the original 5.87 is struck through, not erased, because it still sits in old_row forever; and reason carries Annex 11 clause 9's documented reason for change, set by the application as a session variable. The integrity group — prev_hash, row_hash — answers has anyone touched this since, and is the subject of the next section.

Where this row comes from — back along the trilogy spine

This single audit.change_log row is the open-source implementation of a story the other two books tell. In Book 1, that titer is born on the floor: QC and release is the physical step where an analyst measures a harvest sample and the quality unit decides whether the batch may ship. In Book 2, that decision becomes a data point with a problem to solve — the anatomy of an audit-trail row frames why an electronic record must carry who, what, when, and the untouched old value. This chapter is where that row finally becomes DDL, a trigger, and a hash chain you can run.

Reading the chain: what verify_chain() does and does not prove

It is tempting to wave the hash chain around as if it were a blockchain. It is not, and the companion repo is scrupulous about saying so — the limitation is written directly into the source. Read the comment that ships above verify_chain() in 50-alcoa.sql:

-- examples/platform/db/50-alcoa.sql
-- Verify the chain is intact: returns rows where a stored prev_hash does not
-- equal the previous row's stored row_hash (a broken/reordered/deleted link).
-- NB: this checks link consistency only; it does NOT recompute row_hash from
-- the payload, so a silent edit to old_row/new_row/app_user is not caught here.

That is an unusually honest piece of code, and it is worth unpacking exactly. verify_chain() compares each row's stored prev_hash against the previous row's stored row_hash using lag(...) OVER (ORDER BY seq). So it catches the things that change the shape of the chain: a deleted row (the link to it dangles), a reordered row, an inserted row whose prev_hash does not match its predecessor. What it does not do is recompute row_hash from the payload and compare. So if an attacker edits old_row in place but leaves both hash columns untouched, verify_chain() still returns zero broken links — the link is intact even though the content lied. Closing that hole is a one-line change (recompute H(prev_hash || payload) per row — where H is the hash function and || means "joined end to end" — and compare to the stored row_hash), and a production deployment should make it; the teaching repo leaves it visible on purpose so the reader understands the difference between link integrity and content integrity.

There is a second, subtler point hiding in the hash formula itself. The trigger computes row_hash over prev_hash || table || op || old_row || new_row || app_user || clock_timestamp() — which means reason and row_key are not inside the hash. The documented reason for change is captured and stored, but it is not cryptographically bound — meaning someone could rewrite reason after the fact and the hash chain would still verify clean, because the reason was never part of what the hash covers. An inspector who cares about Annex 11 clause 9 will want it inside the payload, and that too is a one-line fix. The lesson is the one the whole book keeps teaching: a control is only as strong as the exact bytes it covers, and you only know which bytes those are by reading the code, not by trusting the word "hash."

The capture round-trip: a named test that can never silently rot

A claim like "the database writes the audit row for you, automatically, every time" is worthless unless something proves it on every commit. That something is test_audit_captures_update in examples/tests/test_db.py, and it is the smallest possible end-to-end proof of the whole capture contract: set the human identity and the reason as session variables, make a real UPDATE to lab.result, then assert that an UPDATE row landed in audit.change_log attributed to that user and that the chain still verifies clean.

# examples/tests/test_db.py
def test_audit_captures_update(conn):
    # an UPDATE must record old + new + who + why and keep the chain intact
    with conn.cursor() as cur:
        cur.execute("select set_config('app.user','pytest',false), "
                    "set_config('app.reason','test correction',false)")
        cur.execute("update lab.result set value = value where result_id = "
                    "(select result_id from lab.result limit 1)")
        conn.commit()
    last = _scalar(conn, "select action from audit.change_log "
                         "where app_user='pytest' order by seq desc limit 1")
    assert last == "UPDATE"
    assert _scalar(conn, "select count(*) from audit.verify_chain()") == 0

Two assertions, two distinct guarantees. The first (last == "UPDATE") proves attributable capture: the trigger fired, picked up the app.user session variable, and wrote a row tagged pytest. The second (verify_chain() == 0) proves the new row joined the chain cleanly. A companion test, test_alcoa_chain_intact, asserts the same zero-broken-links property over the seeded history before any test mutation. The value of writing this down as a named, version-controlled test is that it converts a prose promise into inspection evidence: the day someone refactors log_change() and breaks attribution, the build goes red — the audit trail can never silently rot, because a test asserts its behaviour on every run.

Reviewing the audit trail

Capturing the trail is the easy half; PI 041 [4] and FDA [5] both insist on the harder half — review. The companion repo wires the integrity check straight into the command surface. The platform runs in Docker containers (isolated, reproducible boxes started from pinned images — an "image" is the frozen template a container boots from, and "pinned" means locked to one exact version so every run is identical), so the command below uses docker exec … psql to run one SQL statement inside the already-running Postgres container, and make alcoa is simply a shortcut for it. From the repo Makefile:

# examples/Makefile
alcoa: ## verify the ALCOA+ audit hash chain is intact (0 = good)
	docker exec -e PGPASSWORD=bioproc bioprocess-data-stack-postgres-1 psql -U bioproc -d bioproc \
	  -c "select count(*) as broken_links from audit.verify_chain();"

Running make alcoa against the seeded stack returns a single, reviewable number:

 broken_links
--------------
            0
(1 row)

Zero broken links means no entry in the history has been altered since it was written — with the precise caveat from the section above, that "altered" here means the chain shape was changed, not every byte was re-verified.

Reviewing the trail before release: the query your quality unit runs

A reviewer needs more than "the chain is intact" — they need to see the human-meaningful changes for a specific record, and PI 041 [4] is explicit that this seeing happens before the record is relied upon, i.e. before batch release, not in some annual sweep. The trigger in 50-alcoa.sql computes each row's row_key as coalesce(batch_id, sample_id), so a lab.result change — which carries a sample_id but no batch_id (see examples/platform/db/30-lab-events.sql) — is keyed by its sample. A review query for one harvest sample (illustrative SQL over the same real table) is what your quality unit runs before release:

-- illustrative review query over examples/platform/db/50-alcoa.sql
SELECT ts, app_user, table_name, action, reason,
       old_row ->> 'value' AS old_value,
       new_row ->> 'value' AS new_value
FROM   audit.change_log
WHERE  row_key = 'BATCH-2026-001-OFF-028'   -- a lab.result is keyed by its sample_id
ORDER  BY seq;

          ts           | app_user | table_name | action |     reason      | old_value | new_value
-----------------------+----------+------------+--------+-----------------+-----------+-----------
 2026-05-12 09:14:02+00 | aoh      | result     | INSERT |                 |           | 5.87
 2026-05-12 14:32:51+00 | mlee     | result     | UPDATE | transcription   | 5.87      | 5.88
                        |          |            |        | error corrected |           |

A reviewer reading those rows sees the whole story: analyst aoh recorded a titer of 5.87 g/L for sample BATCH-2026-001-OFF-028, analyst mlee later corrected it to 5.88 g/L citing a transcription error, both time-stamped and attributable, and the original value never destroyed. This is the day-13 harvest titer for BATCH-2026-001 — about 5.9 g/L (offline_assays.csv OFF-028 = 5.877). The Protein A capture step (the first purification column; see Capture: Protein A in Book 1) loads that harvest at 5.88 g/L (protein_a_summary.csv), grabs the antibody onto the column, and then releases it in a much smaller volume of acid. Because the same antibody mass leaves the column in far less liquid, its concentration rises roughly four-fold — to a ~22.6 g/L eluate (eluate_titer 22.58). The number goes up because the volume shrinks, not because new product appears. A beginner reading this row is seeing what a real CHO (Chinese-hamster-ovary cell-line) fed-batch mAb (monoclonal-antibody) harvest titer looks like. To roll the sample's history up to its batch, a reviewer joins back through lab.sample (whose batch_id is BATCH-2026-001). That is a Part 11 audit trail doing its job — and it is entirely open source.

Electronic signatures and trusted time

An audit trail says what changed; a signature says I approve this, and I am this specific person. This is where open source gets genuinely good and then hits a wall.

The good part: eLabFTW [8], the open-source electronic lab notebook, can lock an experiment, sign it cryptographically, and stamp it with an RFC 3161 trusted time-stamp [6]. (eLabFTW is sketched as a trust-tier component here — its image pin is recorded but it is not wired into this teaching stack's compose.yaml; only the Postgres audit chain is runnable on a laptop.) RFC 3161 is the standard for asking an independent Time-Stamp Authority (TSA) to issue a signed token proving a particular byte sequence existed at a particular instant — proof of existence that you cannot back-date. eLabFTW's signing flow produces exactly the §11.70 "permanently linked" property: the signature and time-stamp token are bound to the hash of the record's content, so the signature cannot be excised and pasted onto a different record without breaking. You point it at any RFC 3161 TSA in its config (illustrative):

# illustrative configuration — eLabFTW timestamping (config.php-equivalent settings)
ts_authority: custom              # or a managed TSA such as FreeTSA / DigiCert
ts_url: https://freetsa.org/tsr
ts_hash: sha256                   # algorithm for the proof-of-existence token
ts_login: ""                      # credentials if the TSA requires them

That is a real, defensible §11.70 / Annex 11 clause 14 control: tamper-evident, time-anchored, permanently linked. The honest dependency is that the trust now rests on an external TSA you must qualify as a supplier, and on configuration you must validate — eLabFTW out of the box is not a turnkey Part 11 system.

The companion repo carries an illustrative sketch of the integration in examples/ingest/elabftw_ingest.py. The honest point it relies on: eLabFTW is a real service whose image pin is recorded (elabftw/elabimg:5.1.15), but it is sketched, not wired into the shipped compose.yaml, so the script is a teaching sketch of the API pattern, not a tested end-to-end importer. The intended signing behaviour it documents is the design we are dissecting: "the Ed25519ph electronic signature + RFC 3161 trusted timestamp are applied through the eLabFTW UI/API on an entry, which then locks it." Unpacked, that means the signature is an Ed25519ph cryptographic signature over the entry's content, the time-stamp is an RFC 3161 token, and applying them locks the entry so that "later edits create a new, separately signed version" rather than mutating a signed record.

Anatomy of an electronic signature: what "permanently linked" actually welds

Just as one audit row repays a slow reading, so does one signed entry. Here is that signed-and-locked eLabFTW entry dissected field by field, against §11.50 (the signing manifestation), §11.70 (permanent linkage), and Annex 11 clause 14 — with the one honest gap drawn in amber.

An eLabFTW signed-and-locked entry: signer and meaning satisfy §11.50, the welded content-hash/record-hash pair is the §11.70 "permanently linked" property made physical, and the amber callout is the honest dependency — the time-stamp is only as trustworthy as the external TSA that issued it. Original diagram by the authors, created with AI assistance.

The manifestation fields — signer and meaning — are §11.50's requirement that a signing record show who signed and what the signature means (review, approval, authorship); qa_reviewer_02 approving a release review is both, in two columns. The mechanism fields — algorithm (Ed25519ph), locked (true), ts_token (an RFC 3161 token) — are how the signing is made durable: locking converts an editable note into an immutable signed version, so a correction is a new signature on a new version, never a silent overwrite of a signed one. The linkage block is the heart of §11.70: the signature is computed over the content hash, and that signature is welded to the linked_record_hash, so peeling the signature off and pasting it onto a different record breaks the cryptography — exactly the "cannot be excised, copied, or otherwise transferred" property the clause demands. And the amber callout is the field-honest part: ts_token proves the bytes existed at an instant, but the proof inherits the trust of whatever TSA issued it, so the TSA becomes a supplier you must qualify, and the §11.200 step-up re-authentication at the moment of signing is still custom code you own.

The wall is §11.200 — the re-authentication rule. A compliant signing manifestation must capture the meaning of the signature (review, approval, authorship) and, for every signing after the first in a session, re-execute at least one identification component. Keycloak [10], the open-source identity provider, gives us unique user IDs, role-based access control (RBAC — permissions granted by a user's role rather than one by one), and multi-factor authentication, which covers §11.10(d)/(g) and Annex 11 access control cleanly. (Like eLabFTW, Keycloak is sketched as an intended trust-tier component — not a service in this teaching stack's compose.yaml.) What it does not do out of the box is force a step-up re-authentication at the precise moment of signing. You can build it — the repo's signing-service design captures a reason-for-change and forces a Keycloak re-auth before it will sign — but that is custom code (GAMP 5 Category 5 [11] — the software-validation framework's label for bespoke code you wrote yourself, which carries the heaviest validation burden because no vendor has tested it for you — you alone must prove it behaves) that you own and must validate, not a feature you switch on. Here is the intended contract, as illustrative API shape:

# illustrative — proposed examples/services/signing-service contract (not yet in repo)
POST /sign
Authorization: Bearer <fresh Keycloak token from step-up re-auth>
Content-Type: application/json

{ "record": "lab.result:91823", "meaning": "approved", "reason": "release review" }
→ 201 Created
  { "signed_hash": "sha256:9f2c…", "signer": "qa_reviewer_02",
    "ts_token": "rfc3161:MIIE…", "linked_record_hash": "sha256:1b07…" }

The flow is honest open source — Keycloak for identity, the hash chain for linkage, an RFC 3161 token for time — but the enforcement that the token is fresh, and the SOP that says who may hold the qa_reviewer role, are yours to write and defend.

The honest Part 11 gap register

This is the section that earns the book's title. GAMP 5's second edition [11] made critical thinking and a clear-eyed appraisal of supplier evidence the heart of validation; the most professional thing we can do is tabulate exactly where pure OSS stops.

Part 11 / Annex 11 control	OSS status	The honest gap
§11.10(e) / Annex 11 cl.9 — audit trail (old/new/who/when/why)	Met — audit.change_log + triggers + pgAudit	None technically; you must still review it (PI 041)
§11.70 — signature permanently linked to record	Met — hash linkage + RFC 3161 token	Trust depends on an external TSA you must qualify
§11.10(d)/(g) — access control, unique IDs, MFA	Met — Keycloak RBAC + MFA	Role assignment is procedural; segregation of duties is yours
§11.200 — re-authentication at each signing	Partial — needs custom step-up auth	Not a Keycloak default; custom Category-5 code to validate
Superuser/privileged-action auditing	Gap — pgAudit cannot reliably audit superusers	Procedure + restricted accounts + off-box immutable logs
Record retention as WORM (write-once-read-many)	Gap — Postgres is not WORM	Object store with object-lock (SeaweedFS object-lock / commercial)
High availability for the record-of-truth	Gap — single-node Postgres in this stack	TimescaleDB HA is a TSL/commercial feature; needs replication design

Read the table honestly. The green rows are genuinely achievable with open source today, and the companion repo runs them. The amber and red rows are not failures of open source so much as reminders that compliance is a property of a validated system and its procedures, never of a downloaded tool — exactly the framing the whole book opened with.

What the field-failure record actually says

This chapter's controls are not theoretical insurance; they map directly onto the failures regulators write up most often. The agency's own enforcement record is the canonical field-failure source here. A retrospective analysis of FDA warning letters (the agency's official written enforcement notices) issued to pharmaceutical companies from 2010–2020 found that documentation and data-integrity deficiencies were cited as a major deficiency in roughly 20–25% of cGMP warning letters [12] — and the specific failures recur with grim consistency: shared logins that destroy attribution (defeating exactly what app_user exists to capture), audit trails disabled or never turned on (the gap that pgAudit's superuser limitation warns about), and no audit-trail review before release (the half that PI 041 [4] and FDA's data-integrity Q&A [5] insist on, and that the review query above is built to satisfy). Each of those is a clause in our gap register made flesh: a citation is what it looks like when one of these rows is left as a red box and nobody noticed.

The cautionary tale closest to this book's own tooling is open-source maturity itself. SENAITE, the OSS LIMS (Laboratory Information Management System) we treat as a teaching QC (quality-control) system, has exactly one published Part 11 gap analysis, and it dates to 2019, assessing version 1.3.2 [13]. That 2019 review rated electronic-signature controls (§11.50, §11.70, §11.200) as gap/partial — out-of-the-box signing did not meet the full signing-manifestation and signature-to-record-binding requirements — and rated record retention, password policy, and validation as partial-to-gap as well. The image pin recorded for that intended trust-tier component is senaite/senaite:2.6.0 (sketched, not wired into this teaching stack's compose.yaml), so every line of that analysis is a starting point for your own validation, not a current statement of conformance. The point is not that SENAITE is bad; it is excellent at its job. The point is that marketing maturity is not validated maturity — validation, in the regulatory sense, is documented proof that a specific configured system does what you claim, which is a stronger thing than an everyday "we checked it" — the only defensible claim is the one you can show an inspector against the exact version, add-ons, and configuration you deployed.

The same row, as a triple: making the audit trail machine-reviewable

The audit.change_log table is a relational record, but the same row is also a small graph — and saying so is not decoration, because the graph form is what lets a machine ask the review question PI 041 makes a human ask. The book's knowledge-graph chapter loads the plant's relational facts into RDF (the Resource Description Framework, which represents every fact as a subject–predicate–object triple), and a corrected-titer change is exactly such a fact. The mlee-corrects-aoh row above becomes a handful of triples on one PROV-O activity (PROV-O is the W3C provenance ontology — its vocabulary of activity, agent, and entity is the standard way to say "who did what to which record, when," which is precisely an audit trail's job):

# illustrative — the BATCH-2026-001-OFF-028 correction as PROV-O + bp: triples
@prefix prov: <http://www.w3.org/ns/prov#> .
@prefix bp:   <https://example.org/bioproc#> .

bp:change-OFF-028-002 a prov:Activity ;          # the UPDATE itself
    prov:wasAssociatedWith bp:person-mlee ;      # app_user — ALCOA+ Attributable
    prov:used             bp:result-OFF-028-v1 ; # old_row (5.87, struck through, never erased)
    prov:generated        bp:result-OFF-028-v2 ; # new_row (5.88)
    bp:reason "transcription error corrected" ;  # Annex 11 cl.9 documented reason
    prov:atTime "2026-05-12T14:32:51Z"^^xsd:dateTime .  # ts — §11.10(e) time-stamp

Read column-by-triple, the mapping is one-to-one with the clauses the section above walked: app_user is the prov:wasAssociatedWith agent (Attributable), old_row/new_row are the prov:used/prov:generated entities (the strike-through-not-erase of §11.10(e)), and ts is prov:atTime. Because the row is now triples, the review itself becomes a SPARQL competency question — a question the data must be able to answer, the same device Book 4 turns into runnable acceptance tests in Competency questions as queries: "list every change to this sample, who made it and why, oldest first" is one SELECT, and "is any released lot carrying an unsigned correction?" is one ASK that returns the exceptions a reviewer must clear before release. And the completeness rule — a released lot must carry exactly one in-range, signed result for every required test — is not prose a human must remember; it is a SHACL shape (the Shapes Constraint Language, which validates a graph against required structure), exactly the bp:ReleaseShape gate Book 4 builds in The release gate and SHACL, whose sh:property [ sh:path bp:approvedBy ; sh:minCount 1 ] is the graph-native restatement of the §11.70 signature row in our gap register. The closed-world point Book 4 stresses is the same one PI 041 makes: a missing audit-trail review is a failure now, not an open question — and SHACL, unlike a free-text SOP, fails on absence. The honest boundary is the one that book also draws: a SHACL gate proves a record is complete, well-formed, and signed; it cannot prove the value is true — a confidently-mislabeled titer passes the gate, so the graph routes the investigation but data integrity upstream still has to vouch for the number.

Why this is the foundation a learning model stands on

This chapter looks like it has nothing to do with machine learning, and that is exactly backwards: the audit trail, the hash chain, and the four-eyes signature are the substrate every governed model in Book 5 is built on. A model in a GMP plant is not a .pkl file — it is a versioned record, and the MLOps chapter shows that what makes it auditable is precisely the machinery on this page: a model-version record pins its training data by sha256 hash (the same content-binding the chain here gives an audit row), and a retrain is promoted only through a four-eyes gate — a second qualified person signs the change — which is the §11.200 / Annex 11 cl.14 electronic signature this chapter builds, applied to a model instead of a titer. When a deployed soft sensor drifts and is retrained, that event is a change-control record logged into Continued Process Verification (CPV — "Stage 3" of process validation, the ongoing program that keeps a process in a validated state), and the audit trail here is where that record's who/when/why lives.

The connection runs deeper than analogy. The cautionary tale of this page — the Purolea warning letter (April 2026), the first FDA letter to cite AI, where a firm let AI agents generate GMP records without quality-unit review — is the same failure as a shared login or a disabled audit trail in the field-failure record above: a record produced with no attributable human accountable for it. The four-eyes signing service sketched here, the app_user that names the human, and the hash chain that makes an after-the-fact edit detectable are the exact controls whose absence drew that letter. Put plainly: a model is only as trustworthy as the audit trail underneath the data it learned from and the signature on the human who promoted it. Book 5's model-and-validation lifecycle treats these Part 11 / Annex 11 controls and ALCOA+ as the record-integrity layer that turns a .pkl file into a validated object — which is why this chapter, not a modeling one, is where that foundation is poured.

Why it matters

Because a regulator does not inspect your software; they inspect your records and the system that produced them. If a batch of monoclonal antibody is released and a year later a deviation investigation needs to know whether a titer result was edited, by whom, and why, the answer cannot be "the developer assured us the database is fine." It has to be a query a quality reviewer ran, against an audit trail the system wrote automatically, linked to a signature that cannot be peeled off and reattached. The open-source stack we built gets you a remarkable distance toward that — and being precise about the last few feet is what separates a demo from a defensible system.

In the real world

The pattern that actually ships in industry is the hybrid this chapter models. A real CHO (Chinese-hamster-ovary cell-line) + Protein A mAb (monoclonal-antibody) facility runs a validated commercial MES (Manufacturing Execution System — the software that directs and records production on the floor) and historian (a time-series database for process data) for the GxP record-of-truth — GxP being the umbrella for all regulated "good practice" domains, so this is the legally authoritative copy of the record — and increasingly an open-source layer alongside for contextualization, analytics, and engineering — precisely because the OSS layer is faster to evolve and cheaper to own. The discipline is keeping the line between them explicit: the validated system holds the signed record; the OSS layer reads, models, and visualizes. Note carefully: as of mid-2026 no open-source tool — not eLabFTW, not Keycloak, not PostgreSQL — ships as a "Part 11-compliant" product. They ship the mechanisms; you build, validate, and procedurally surround the compliance — as the field-failure record above makes plain, the citations land on the procedures and the review, not on the absence of a hash function.

Key terms

• 21 CFR Part 11 — U.S. FDA regulation setting the criteria under which electronic records and electronic signatures are trustworthy equivalents of paper and handwriting. (The "§" symbol means "section," so "§11.10(e)" is read as "section 11.10, sub-point e" — one of Part 11's numbered sub-sections.)
• GMP / cGMP / GxP — Good Manufacturing Practice, the regulations governing how medicines are made; "c" = current (the cGMP are the U.S. FDA's current GMP rules). GxP is the umbrella for all "Good x Practice" domains (manufacturing, laboratory, clinical, distribution). A change is "GMP-relevant" when it touches a record these rules require you to control.
• ALCOA+ — the data-integrity principles regulators expect of every GxP record: Attributable, Legible, Contemporaneous, Original, Accurate, plus Complete, Consistent, Enduring, and Available. The audit-trail columns in this chapter implement them directly — for example, app_user (the human who made the change) is what makes the "Attributable" principle literal.
• GAMP 5 / Category 5 — GAMP 5 is ISPE's risk-based framework for validating computerised systems; its software categories run from 1 (infrastructure) to 5 (custom/bespoke code you wrote yourself). Category 5 software, like the signing service sketched here, carries the heaviest validation burden precisely because you wrote it.
• Superuser — an all-powerful database account that can change or disable any setting, including the audit logging itself — which is exactly why the controls against privileged misuse must live in procedure and infrastructure, not in the database code alone.
• DDL (Data Definition Language) — the SQL statements (CREATE TABLE, CREATE TRIGGER, …) that define a database's structure; the audit table and its triggers in this chapter are "expressed in DDL." A trigger is code the database runs automatically on every change to a table, and a session variable (set with SET app.user = '…') is a value the application hands the database for the duration of one connection.
• EU Annex 11 — the European GMP guideline for computerised systems; the EU counterpart to Part 11, in places stricter (e.g., the documented reason for every change).
• PIC/S PI 041 — the inspectors' data-integrity guidance that makes audit-trail review (not just retention) an explicit expectation.
• Audit trail — a secure, time-stamped, computer-generated record of who created, modified, or deleted data, and why, without obscuring the prior values.
• Deviation — a documented departure from an approved procedure that triggers a formal investigation; it is the situation in which an audit trail is consulted to learn who changed a value, when, and why.
• pgAudit — a PostgreSQL extension that logs executed SQL statements for session/object-level auditing.
• Hash chain — linking each audit entry to a cryptographic hash of the previous one so any later alteration is detectable.
• RFC 3161 / TSA — the trusted-timestamp protocol and the Time-Stamp Authority that issues signed proof-of-existence tokens.
• Re-authentication (§11.200) — the requirement that each signing after the first in a session re-execute at least one identity component.
• WORM — write-once-read-many storage that physically prevents alteration of a committed record.
• Object store / object-lock — storage that holds files as "objects"; object-lock physically blocks overwrite or deletion of an object for a retention period (the WORM property). SeaweedFS is an open-source object store that offers it.
• TimescaleDB HA / replication — high availability (HA) keeps a second live copy of the database (a process called replication) so a node failure does not lose the record-of-truth. TimescaleDB's built-in HA is a TSL (Timescale License — source-available, not OSI-approved open-source) / commercial feature, so the single-node stack in this book does not have it.
• Strike-through-not-erase — the paper-batch-record discipline, carried into the database by storing both old_row and new_row, that a corrected value is preserved alongside its correction rather than overwritten (§11.10(e)).
• Ed25519ph — the pre-hash variant of the Ed25519 elliptic-curve signature scheme used by eLabFTW to sign an entry over the hash of its content, so the signature is bound to that exact content.
• PROV-O — the W3C provenance ontology, whose activity / agent / entity vocabulary expresses "who did what to which record, when" — the same who/what/when an audit-trail row records, written as RDF triples a machine can query.
• SHACL / competency question — SHACL (the Shapes Constraint Language) validates a graph against required structure, so "every released lot carries one in-range, signed result per test" becomes an enforced gate rather than an SOP a human must remember; a competency question is a question the data must be able to answer (a SPARQL SELECT/ASK), which is what an audit-trail review becomes once the row is triples.
• Model-version record — the Book 5 governed equivalent of a signed batch record: a model pinned to its training data by sha256, validated against written acceptance criteria, and promoted through a four-eyes signature — the §11.200 / Annex 11 cl.14 control of this chapter applied to a model instead of a result.
• Continued Process Verification (CPV) — "Stage 3" of process validation, the ongoing program keeping a process in a validated state; a retrained model's revalidation is a change-control event logged here, with its who/when/why living in the audit trail this chapter builds.

Where this leads

We now have records the regulators would recognise — automatically audited, attributably changed, cryptographically signed — and an honest map of where open source needs procedure or commercial parts to finish the job. But a defensible system is more than its controls: someone has to prove the whole stack was installed, configured, and behaves as intended, with no vendor quality system to lean on. That is validation. The next chapter, Validating an Open-Source Stack: GAMP 5 & CSA, turns these same artifacts into IQ/OQ/PQ evidence — running the test suite you have already seen as inspection-ready proof that the platform does what we say it does.