Electronic Records & Signatures: Part 11 / Annex 11 with Open Source

📍 Where we are: Part V, "Trust." The previous chapter made our data tamper-evident in code; now we ask the harder question — can an all-open-source stack actually satisfy the rules that say an electronic record may stand in for a signed piece of paper? We build the controls that work, and we write down, honestly, the ones that do not.

The simple version

Think of a paper batch record: every entry is initialled, dated, and never erased — mistakes are struck through with a single line so the old value still shows, and a signature at the bottom means "I, this specific person, reviewed this and I stand behind it." 21 CFR Part 11 and EU Annex 11 are simply the rules for doing all of that on a computer instead of paper: who changed what, when, and why must be recorded automatically, and a signature must be unbreakably welded to the exact record it signs. Open source gets you most of that — automatic audit trails, cryptographic signatures, time-stamps from a trusted clock. The last mile, where the system must prove it is the right human pressing the button, is where you start writing procedures and reaching for commercial parts.

What this chapter covers

This is the chapter where the platform meets the regulators. We are not adding a new sensor or a new dashboard; we are taking the relational backbone we already built and asking whether the records it holds are trustworthy in the legal sense. The roadmap:

• What 21 CFR Part 11 [1] and EU Annex 11 [2] actually require, clause by clause, and how PIC/S PI 041-1 [3] turns those clauses into what an inspector looks for.
• A working audit trail built two ways: pgAudit at the database-session level [4], and the trigger-based, hash-chained audit.change_log table from the companion repo that captures old/new/who/when/why.
• Electronic signatures with eLabFTW and RFC 3161 trusted time-stamps [5][6], plus a reason-for-change signing service backed by Keycloak [7].
• An audit-trail review query — the artifact your quality unit actually runs before batch release.
• A brutally honest gap register: the Part 11 clauses where open source alone falls short, and what closes each gap.

Everything labelled with a file path is real, tested code from examples/ that ran in our continuous-integration pipeline. Everything labelled illustrative is a realistic snippet for a service that cannot run on a laptop — shown honestly, never claimed to execute.

What the rules actually say

Part 11 is short and old — it became effective in 1997 [1] — and its genius is that it does not name a single technology. It says an electronic record is acceptable in place of paper if the system that produces it enforces a handful of controls. The ones that matter for us live in three clauses. §11.10(e) demands "secure, computer-generated, time-stamped audit trails" that record operator actions creating, modifying, or deleting records, and crucially that do not obscure previously recorded information. §11.70 requires that electronic signatures be "linked to their respective electronic records to ensure that the signatures cannot be excised, copied, or otherwise transferred" to falsify another record. §11.200 governs the signature itself: it must use at least two distinct identification components (think username + password), and after an initial signing in a session, each subsequent signing must re-execute at least one component.

The 2003 Scope and Application guidance [8] is the document that keeps practitioners sane: FDA narrowed Part 11 to a risk-based posture, exercising enforcement discretion on some controls, but it was explicit that it still expects compliance with the audit-trail, record-retention, and electronic-signature clauses (§§11.10, 11.30, 11.50, 11.70, 11.100, 11.200, 11.300). That sentence is the line our gap register is drawn along: the clauses FDA still enforces are exactly the ones we must satisfy with open source or admit we cannot.

EU Annex 11 [2] is the European counterpart, and it is in places stricter. Clause 9 requires an audit trail for all GMP-relevant changes and deletions, with a documented reason; clause 12 demands access controls; clause 14 expects electronic signatures to have the same impact as handwritten ones and to be permanently linked to their record. PIC/S PI 041 [3], the inspectors' data-integrity guide, then adds the operational expectations: audit trails must be reviewed, not merely kept, and the review must happen before the record is relied upon — i.e., before batch release. FDA's own data-integrity Q&A [9] says the same thing in plainer words: audit trails that capture creation and modification of GMP data should be reviewed with the same rigour as the data itself. Keep all of this in mind — the deliverable at the end of this chapter is not "we have an audit trail," it is "we have an audit trail someone reviews."

From left to right: the regulatory requirement, the open-source control that meets it, and the honest gap that remains. The green band (audit trail, attributable change capture, trusted time-stamp) is genuinely achievable in OSS; the amber band (re-authentication at signing, WORM retention, high-availability) is where the validated system, procedure, or commercial tooling carries the load. Original diagram by the authors, created with AI assistance.

The audit trail, two ways

There are two complementary places to capture "who changed what, when, and why," and a serious system uses both.

The first is pgAudit [4], a PostgreSQL extension that logs the actual SQL statements a session executes, into the database server log. It is the closest open-source analogue to a tamper-resistant, system-level transcript: every UPDATE lab.result … is written verbatim, with the database user and a server time-stamp, before the application ever touches it. You enable it as illustrative configuration in postgresql.conf (or via ALTER SYSTEM):

# illustrative configuration — platform/db/pgaudit.conf
shared_preload_libraries = 'pgaudit'
pgaudit.log = 'write, ddl, role'   # capture INSERT/UPDATE/DELETE, schema and grant changes
pgaudit.log_relation = on          # one log entry per affected table
pgaudit.log_parameter = on         # record the bound values, not just the statement text

pgAudit is excellent at one thing — an immutable, append-only statement log — and honest about its limits. Its own documentation states plainly that it cannot reliably audit a superuser, because a superuser can change logging settings mid-session. That single sentence is the first entry in our gap register, and the reason we do not stop at pgAudit.

The second place is the application-meaningful audit trail: a table that records the business change in terms a reviewer understands — old row, new row, the human who did it, and the reason. This is the star of the previous chapter and it is real, tested code. From examples/platform/db/50-alcoa.sql, the change-log table and its hash chain:

-- examples/platform/db/50-alcoa.sql
CREATE TABLE audit.change_log (
    seq        bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    ts         timestamptz NOT NULL DEFAULT clock_timestamp(),
    db_user    text NOT NULL DEFAULT current_user,
    app_user   text,                              -- set via SET app.user = '...'
    table_name text NOT NULL,
    action     text NOT NULL,                     -- INSERT | UPDATE | DELETE
    row_key    text,
    old_row    jsonb,
    new_row    jsonb,
    reason     text,                              -- set via SET app.reason = '...'
    prev_hash  text,
    row_hash   text NOT NULL
);

That schema is, almost line for line, a Part 11 §11.10(e) audit trail expressed in DDL. old_row and new_row mean previously recorded information is never obscured — the strike-through-not-erase rule. app_user is the human (not the database account), satisfying ALCOA+'s "Attributable" and Annex 11 clause 12 [2]. reason is Annex 11 clause 9's documented reason for change. And prev_hash/row_hash chain each entry to the one before it, so a deletion or edit anywhere in the history breaks the chain and is detectable.

The trigger is what makes capture automatic — the operator cannot forget to write the audit row, because the database writes it for them on every change, using PostgreSQL triggers and roles [10]. The same file attaches it to the regulated tables and ships a verifier:

-- examples/platform/db/50-alcoa.sql
CREATE TRIGGER audit_result   AFTER INSERT OR UPDATE OR DELETE ON lab.result
    FOR EACH ROW EXECUTE FUNCTION audit.log_change();
CREATE TRIGGER audit_batch    AFTER INSERT OR UPDATE OR DELETE ON s88.batch
    FOR EACH ROW EXECUTE FUNCTION audit.log_change();
CREATE TRIGGER audit_recipe_p AFTER INSERT OR UPDATE OR DELETE ON s88.recipe_parameter
    FOR EACH ROW EXECUTE FUNCTION audit.log_change();

-- Verify the chain is intact: returns rows where the recomputed hash breaks.
CREATE OR REPLACE FUNCTION audit.verify_chain()
RETURNS TABLE(seq bigint, ok boolean) AS $$
    WITH chained AS (
        SELECT c.seq, c.row_hash, c.prev_hash,
               lag(c.row_hash) OVER (ORDER BY c.seq) AS expected_prev
        FROM audit.change_log c
    )
    SELECT seq, (prev_hash IS NOT DISTINCT FROM expected_prev) AS ok
    FROM chained
    WHERE prev_hash IS DISTINCT FROM expected_prev;
$$ LANGUAGE sql;

The application sets the human identity and the reason as session variables before the change, and the trigger picks them up. Our test suite, in examples/tests/test_db.py, proves exactly that round-trip — that an UPDATE records old + new + who + why and keeps the chain intact:

# examples/tests/test_db.py
def test_audit_captures_update(conn):
    # an UPDATE must record old + new + who + why and keep the chain intact
    with conn.cursor() as cur:
        cur.execute("select set_config('app.user','pytest',false), "
                    "set_config('app.reason','test correction',false)")
        cur.execute("update lab.result set value = value where result_id = "
                    "(select result_id from lab.result limit 1)")
        conn.commit()
    last = _scalar(conn, "select action from audit.change_log "
                         "where app_user='pytest' order by seq desc limit 1")
    assert last == "UPDATE"
    assert _scalar(conn, "select count(*) from audit.verify_chain()") == 0

The honest caveat, repeated here because it is load-bearing: a superuser who disables the trigger or rewrites the table can still bypass this. The hash chain makes tampering detectable, not impossible. That is the right design for an OSS stack — but it means the controls preventing privileged misuse (separation of duties, restricted superuser accounts, immutable off-box log shipping) live in your procedures and infrastructure, and an inspector will ask to see them.

Reviewing the audit trail

Capturing the trail is the easy half; PI 041 [3] and FDA [9] both insist on the harder half — review. The companion repo wires the integrity check straight into the command surface. From the repo Makefile:

# examples/Makefile
alcoa: ## verify the ALCOA+ audit hash chain is intact (0 = good)
	docker exec -e PGPASSWORD=bioproc sensor-to-submission-postgres-1 psql -U bioproc -d bioproc \
	  -c "select count(*) as broken_links from audit.verify_chain();"

Running make alcoa against the seeded stack returns a single, reviewable number:

 broken_links
--------------
            0
(1 row)

Zero broken links means no entry in the history has been altered since it was written. But a reviewer needs more than "the chain is intact" — they need to see the human-meaningful changes for a specific record. The trigger in 50-alcoa.sql computes each row's row_key as coalesce(batch_id, sample_id), so a lab.result change — which carries a sample_id but no batch_id (see examples/platform/db/30-lab-events.sql) — is keyed by its sample. A review query for one harvest sample (illustrative SQL over the same real table) is what your quality unit runs before release:

-- illustrative review query over examples/platform/db/50-alcoa.sql
SELECT ts, app_user, table_name, action, reason,
       old_row ->> 'value' AS old_value,
       new_row ->> 'value' AS new_value
FROM   audit.change_log
WHERE  row_key = 'BATCH-2026-001-OFF-007'   -- a lab.result is keyed by its sample_id
ORDER  BY seq;

          ts           | app_user | table_name | action |     reason      | old_value | new_value
-----------------------+----------+------------+--------+-----------------+-----------+-----------
 2026-05-12 09:14:02+00 | aoh      | result     | INSERT |                 |           | 4.81
 2026-05-12 14:32:51+00 | mlee     | result     | UPDATE | transcription   | 4.81      | 4.18
                        |          |            |        | error corrected |           |

A reviewer reading those rows sees the whole story: analyst aoh recorded a titer of 4.81 g/L for sample BATCH-2026-001-OFF-007, analyst mlee later corrected it to 4.18 g/L citing a transcription error, both time-stamped and attributable, and the original value never destroyed. To roll the sample's history up to its batch, a reviewer joins back through lab.sample (whose batch_id is BATCH-2026-001). That is a Part 11 audit trail doing its job — and it is entirely open source.

Electronic signatures and trusted time

An audit trail says what changed; a signature says I approve this, and I am this specific person. This is where open source gets genuinely good and then hits a wall.

The good part: eLabFTW [6], the open-source electronic lab notebook in our lab profile, can lock an experiment, sign it cryptographically, and stamp it with an RFC 3161 trusted time-stamp [5]. RFC 3161 is the standard for asking an independent Time-Stamp Authority (TSA) to issue a signed token proving a particular byte sequence existed at a particular instant — proof of existence that you cannot back-date. eLabFTW's signing flow produces exactly the §11.70 "permanently linked" property: the signature and time-stamp token are bound to the hash of the record's content, so the signature cannot be excised and pasted onto a different record without breaking. You point it at any RFC 3161 TSA in its config (illustrative):

# illustrative configuration — eLabFTW timestamping (config.php-equivalent settings)
ts_authority: custom              # or a managed TSA such as FreeTSA / DigiCert
ts_url: https://freetsa.org/tsr
ts_hash: sha256                   # algorithm for the proof-of-existence token
ts_login: ""                      # credentials if the TSA requires them

That is a real, defensible §11.70 / Annex 11 clause 14 control: tamper-evident, time-anchored, permanently linked. The honest dependency is that the trust now rests on an external TSA you must qualify as a supplier, and on configuration you must validate — eLabFTW out of the box is not a turnkey Part 11 system.

The wall is §11.200 — the re-authentication rule. A compliant signing manifestation must capture the meaning of the signature (review, approval, authorship) and, for every signing after the first in a session, re-execute at least one identification component. Keycloak [7], the open-source identity provider in our trust profile, gives us unique user IDs, role-based access control, and multi-factor authentication, which covers §11.10(d)/(g) and Annex 11 access control cleanly. What it does not do out of the box is force a step-up re-authentication at the precise moment of signing. You can build it — the repo's signing-service design captures a reason-for-change and forces a Keycloak re-auth before it will sign — but that is custom code (GAMP 5 Category 5 [11]) that you own and must validate, not a feature you switch on. Here is the intended contract, as illustrative API shape:

# illustrative — proposed examples/services/signing-service contract (not yet in repo)
POST /sign
Authorization: Bearer <fresh Keycloak token from step-up re-auth>
Content-Type: application/json

{ "record": "lab.result:91823", "meaning": "approved", "reason": "release review" }
→ 201 Created
  { "signed_hash": "sha256:9f2c…", "signer": "qa_reviewer_02",
    "ts_token": "rfc3161:MIIE…", "linked_record_hash": "sha256:1b07…" }

The flow is honest open source — Keycloak for identity, the hash chain for linkage, an RFC 3161 token for time — but the enforcement that the token is fresh, and the SOP that says who may hold the qa_reviewer role, are yours to write and defend.

The honest Part 11 gap register

This is the section that earns the book's title. GAMP 5's second edition [11] made critical thinking and a clear-eyed appraisal of supplier evidence the heart of validation; the most professional thing we can do is tabulate exactly where pure OSS stops.

Part 11 / Annex 11 control	OSS status	The honest gap
§11.10(e) / Annex 11 cl.9 — audit trail (old/new/who/when/why)	Met — audit.change_log + triggers + pgAudit	None technically; you must still review it (PI 041)
§11.70 — signature permanently linked to record	Met — hash linkage + RFC 3161 token	Trust depends on an external TSA you must qualify
§11.10(d)/(g) — access control, unique IDs, MFA	Met — Keycloak RBAC + MFA	Role assignment is procedural; segregation of duties is yours
§11.200 — re-authentication at each signing	Partial — needs custom step-up auth	Not a Keycloak default; custom Category-5 code to validate
Superuser/privileged-action auditing	Gap — pgAudit cannot reliably audit superusers	Procedure + restricted accounts + off-box immutable logs
Record retention as WORM (write-once-read-many)	Gap — Postgres is not WORM	Object store with object-lock (SeaweedFS object-lock / commercial)
High availability for the record-of-truth	Gap — single-node Postgres in this stack	TimescaleDB HA is a TSL/commercial feature; needs replication design

Read the table honestly. The green rows are genuinely achievable with open source today, and the companion repo runs them. The amber and red rows are not failures of open source so much as reminders that compliance is a property of a validated system and its procedures, never of a downloaded tool — exactly the framing the whole book opened with.

Why it matters

Because a regulator does not inspect your software; they inspect your records and the system that produced them. If a batch of monoclonal antibody is released and a year later a deviation investigation needs to know whether a titer result was edited, by whom, and why, the answer cannot be "the developer assured us the database is fine." It has to be a query a quality reviewer ran, against an audit trail the system wrote automatically, linked to a signature that cannot be peeled off and reattached. The open-source stack we built gets you a remarkable distance toward that — and being precise about the last few feet is what separates a demo from a defensible system.

In the real world

The pattern that actually ships in industry is the hybrid this chapter models. A real CHO + Protein A mAb facility runs a validated commercial MES and historian for the GxP record-of-truth, and increasingly an open-source layer alongside for contextualization, analytics, and engineering — precisely because the OSS layer is faster to evolve and cheaper to own. The discipline is keeping the line between them explicit: the validated system holds the signed record; the OSS layer reads, models, and visualizes. NIIMBL — the U.S. public-private Institute for Innovation in Biopharmaceutical Manufacturing — funds work on exactly these data-integration and digital-maturity problems, and its SABRE facility (a pilot-scale current Good Manufacturing Practice, or cGMP, facility at the University of Delaware that broke ground in April 2024) is the kind of site where these controls have to be real, not theoretical. Note carefully: SABRE is a facility under construction, not a data program, and as of mid-2026 no open-source tool — not eLabFTW, not Keycloak, not PostgreSQL — ships as a "Part 11-compliant" product. They ship the mechanisms; you build, validate, and procedurally surround the compliance. SENAITE, the OSS LIMS this book treats as a teaching system, is the cautionary tale: its only published Part 11 gap analysis dates to 2019 and lists real gaps in e-signatures, retention, and password controls. Marketing maturity is not validated maturity.

Key terms

• 21 CFR Part 11 — U.S. FDA regulation setting the criteria under which electronic records and electronic signatures are trustworthy equivalents of paper and handwriting.
• EU Annex 11 — the European GMP guideline for computerised systems; the EU counterpart to Part 11, in places stricter (e.g., the documented reason for every change).
• PIC/S PI 041 — the inspectors' data-integrity guidance that makes audit-trail review (not just retention) an explicit expectation.
• Audit trail — a secure, time-stamped, computer-generated record of who created, modified, or deleted data, and why, without obscuring the prior values.
• pgAudit — a PostgreSQL extension that logs executed SQL statements for session/object-level auditing.
• Hash chain — linking each audit entry to a cryptographic hash of the previous one so any later alteration is detectable.
• RFC 3161 / TSA — the trusted-timestamp protocol and the Time-Stamp Authority that issues signed proof-of-existence tokens.
• Re-authentication (§11.200) — the requirement that each signing after the first in a session re-execute at least one identity component.
• WORM — write-once-read-many storage that physically prevents alteration of a committed record.

Where this leads

We now have records the regulators would recognise — automatically audited, attributably changed, cryptographically signed — and an honest map of where open source needs procedure or commercial parts to finish the job. But a defensible system is more than its controls: someone has to prove the whole stack was installed, configured, and behaves as intended, with no vendor quality system to lean on. That is validation. The next chapter, Validating an Open-Source Stack: GAMP 5 & CSA, turns these same artifacts into IQ/OQ/PQ evidence — running the test suite you have already seen as inspection-ready proof that the platform does what we say it does.