How this system works

In plain language — what happens to your documents, where you come in, and where things can go wrong.

The one-paragraph version

Your project 9076 documents — lab notebooks, spreadsheets, test reports, spectra, handwritten pages — are scanned into text, and every statement about an experiment is collected onto one page per experiment (a "record"). The old ELN reports and the handwritten notes are the source of truth; the summary spreadsheets are each researcher's condensed view of the same experiments and are used to cross-check, not to overrule. Nothing is ever typed in from memory: every value on a record points back to the exact page or spreadsheet row it came from. Where two documents disagree, the record shows both values and asks a scientist to decide. Everything runs on this machine only; nothing leaves it.

The pipeline, step by step

The Pipeline tab has four steps — Discover → Extract → Link → Browse — and every one of them can be re-run at any time without losing human work. A "Next:" banner always points at the one thing waiting for you.

1Discover. A crawler scans the document drop and builds a categorized table of every file: what it is, when it changed, which experiments its name and contents mention, and whether it has been read in yet. It also makes suggestions — "these two spreadsheets have the same name but different contents", "this notebook covers experiments 48–60 and must be split", "this report only names a sample code and will need linking". When new files arrive, re-running the crawler shows exactly what appeared or changed. Clicking any file opens it in the Extract step — the original document on the left, whatever was read from it on the right. (Scanned PDFs are turned into text by OCR; Word and Excel files are converted directly. Originals are never modified — everything works on copies.)

2Extract — and check as you go. Each source has its own extraction route, by role and trust:

• Old ELN reports and handwritten notes (source of truth) are read by a local AI model (Qwen, on this machine), one document at a time. It is instructed to copy values verbatim, never convert units, and leave a field blank rather than guess; it locates facts with a bounded search tool rather than reading whole files. Handwriting-derived numbers are never trusted above "medium" confidence.
• The experiment table (cross-check) is each researcher's human-readable summary — its columns are not reliable enough for fixed code (the A.V. column can hold acid value, amine value or prose; temperature/time cells are step-by-step ramps; extra columns appear per batch; some rows are blend recipes in plain language). So a small program slices out one batch's rows and Qwen interprets just that block — one batch per run, resumable, skipping batches already done.
• Application & analytical testing data — the pass/fail test reports and the IR/NMR spectra headers — are read for verdicts and sample identifiers so they can be linked to the right batch. The APP DATA summary sheet keeps a fixed no-AI parser (its layout is regular, and it is the only application-result source for most batches), but it is marked as a derived summary: where an individual test report exists, the report wins.
• Not extracted at all: the near-duplicate copy of the experiment table (its 18 differing cells are on record), the 3-row ELN summary export, the review presentation, and the patents/vendor sheets — reference and context only.

Each run produces fragments — "this document says the following about experiment 48". Then the checking happens right there: the original document on the left, the extracted values on the right, the cited row or page highlighted. Press Enter (or click OK) to confirm each value, or fix anything wrong, tagging the cause. Fields the extractor was unsure about are flagged amber. Every confirmation and fix is recorded, survives every re-run, and never touches the extractor's raw output underneath.

3Link. The same experiment goes by many names — "Lab Batch 48", "LBN 48", "Exp 48", sample "P 76 B.NO 48". Fixed naming rules map these to one identity automatically. Whatever they can't place is shown to you as a short list: for each unmatched document you either pick the experiment it belongs to (with suggestions based on the numbers it mentions) or ignore it if it isn't experiment data — both reversible, both recorded. Merging then combines all fragments about one experiment into its record in source-of-truth order — ELN report first, then test reports and spectra, then handwritten notes, then summary tables — so the primary source fills each field and a summary can only confirm it, fill a gap, or raise a conflict. Where two sources disagree, the record keeps both values clearly marked as a conflict — the software never quietly picks one.

4Browse and review. The Experiments tab is the result: one page per experiment, every value pointing back to its source. The review queue lists what still needs a scientist's judgement, most important first: conflicts between sources, unresolved success/failure verdicts, and spot-checks of "confident" values — checking those is how we measure accuracy rather than assume it.

Corrections make the system better. Every fix you make — on a fragment or a record — is permanent (stored separately, field frozen, no re-run can overwrite it) and tagged with its cause (bad OCR, AI misread the layout, wrong column mapping, wrong name-match). Those tags tell us which part of the pipeline to repair, and every repair is re-tested against all previously corrected examples so old mistakes cannot quietly return.

Fragments — the unit everything is built from

A fragment is one source's testimony about one experiment: "this document says the following about batch 48". One ELN report produces one fragment; one batch block of the experiment table produces one fragment; the APP DATA sheet produces one per sample. A fragment never mixes sources and never mixes experiments, which is what makes everything auditable: it records which file it came from, the exact sheet row or page (locator), which extractor read it and when, the experiment names it saw verbatim (target_aliases), a per-field confidence, and the extracted values themselves. Fragments are immutable testimony — your corrections are stored beside them, never inside them — and merging replays all fragments in source-of-truth order to build each experiment's record. Re-running any extractor just re-writes its fragments under the same stable ids; nothing human-made is lost.

The experiment record, field by field

This is the schema every fragment feeds into (machine-checked against schema/record.schema.json on every merge) — one record per experiment:

record_id, aliases[]       exp-9076-048 + every name seen for it ("Lab Batch 48", "LBN 48", "#48")
record_type              synthesis | blend (e.g. "21A = LBN 21 + 3% AMP" — no reaction, straight to
                         testing; carries preparation + derived_from parent batch) | rework ("24CR")
date · scientist · aim   experiment header facts
eln_form                 the LAB JOURNAL v2.0 form fields: equipment, reaction scheme,
                         safety precautions, journal version, any rare labels
materials[]              Input Raw Material table, one row per material — name, source, CAS no,
                         RM no, quantity, purity %, RM %, mol. wt., moles, molar ratio, role
                         (role intermediate = a "Step-N product" charged into another step)
process_steps[]          per step (1/2/3 — a DAG, not a fixed order; 1 and 2 can run in
                         parallel into 3): its materials[], its procedure[] as ordered events
                         (seq, time h, temperature, atmosphere — ramps read positionally:
                         "40/0.15, 55/0.5" = 40 °C for 0.15 h then 55 °C for 0.5 h),
                         conditions, results[] (A.V. / amine / I.V. … as measurements), remarks
conditions               the optimized/roll-up reaction conditions (temperature, time, atmosphere)
critical_process_parameters[]  the ELN's CPP table — source, time, temp, observation
critical_parameters[]    target-check measurements: parameter, reference value, measured value
observations             experiment-level notes (incl. "Expt Table remarks: …")
samples[]                what the batch produced — each with analytical[] (IR / NMR:
                         technique, date, solvent, file) and application_tests[] (pigment
                         system, result, vs same-day BYK 9076 standard, verdict ok / not ok,
                         failure reason)
conclusion → outcome     the verbatim conclusion; status derived only from explicit wording,
                         with its citation
documents[]              every file linked to this experiment and *why* it linked (alias
                         match or human link)
provenance · validation  per-field: which file + row/page it came from, extractor + version,
                         confidence; which fields a human validated (frozen against re-runs)

Any single value may instead hold a {conflict, by_source} pair when two sources disagree — both values, each naming its source, waiting for a scientist. Measurements keep the original cell text in verbatim, so an interpreted reading (was that A.V. an acid value or an amine value?) can always be re-read by a person.

Known gaps in the current data

• Only failed application tests are on file. All nine ink-test reports in the data drop are "not ok" — reports for successful tests weren't included. Until they are, "no failure on record" does not mean success.
• Handwritten pages are the weakest source. Experiments ~48–60 exist partly as handwriting; numbers read from them are always flagged and never trusted above "medium" confidence until a person confirms them.
• Some experiments will be thin. Only 14 of the ~58 batches have an ELN report in this drop — the rest rest on the summary table alone until FOIL shares the remaining scans. A thin record shows what is missing rather than papering over it.
• Vendor and patent documents are reference only. Target values from the BYK-9076 datasheet appear on records clearly labelled as spec targets — they are never mixed into FOIL's measured data.

Where it can fail — and the safety net for each

• OCR misreads a character (a 3 becomes an 8, especially in handwriting). Net: low-confidence flagging, review queue, and the original page shown beside every value.
• The AI misunderstands a layout (reads a two-column page in the wrong order). Net: it must copy verbatim into a fixed format that is machine-checked; malformed output is rejected and retried, and the audit sample catches what slips through.
• A document is linked to the wrong experiment (naming rules mismatch an alias). Net: every link states which name it matched on; ambiguous ones are parked as stubs instead of guessed.
• Two sources genuinely disagree. Net: that's not a failure the system hides — it's surfaced as a conflict for a scientist to settle.
• Duplicate files disagree (two copies of the experiment table exist, differing in 18 cells — a catalyst renamed AIBN vs FOIL-1). Net: only the original copy is extracted; the full cell-by-cell difference is on record, and which name is authoritative stays an open question for FOIL rather than a silent choice.
• A re-run changes something unexpectedly. Net: every run produces a difference report ("2 records added, 4 fields changed") — silent changes are impossible, and human-validated fields cannot change at all.

What this system is not

It does not decide anything. It retrieves, structures, and cites; scientists validate and conclude. Search on this site is plain keyword matching — the AI is used only during extraction, under the rules above, and its every contribution is labelled with the model, prompt version, and source it came from.