Extract blood test data from PDF documents that have been OCR'd

The objective is to build a structured blood test database that allows pathology results to be viewed, edited, filtered, and exported to Excel via a web-based HTML interface. The system stores results in a clean, standardised format so trends can be analysed accurately over time.

Using AI-assisted OCR, I have built a local Python extraction pipeline that converts PDF pathology reports into machine-readable text and inserts structured data into a SQLite database. The majority of blood tests extract correctly, including canonical test name, result value, unit, and reference range.

However, I have reached a specific technical issue with three markers:

• CRP (C-reactive protein)
• ESR
• GLU (Glucose)

The OCR output clearly contains the correct lines, and debug logs confirm they are processed. Yet no rows are inserted for these markers.

The failure appears to occur between canonical matching, numeric extraction, or validation logic.

Current System Architecture

The system runs locally and consists of:

• extraction_core_2.py (main engine)
• Supporting modules for OCR preprocessing, lab dictionary building, regex matching, and validation
• SQLite backend
• Schema-driven canonical lab dictionary
• Controlled fuzzy fallback logic
• HTML viewer for results display and Excel export

Pipeline flow:

Convert PDF to image (pdf2image)

Preprocess

Run Tesseract OCR

Clean and normalise text

Match against canonical lab dictionary

Extract:

canonical test name

numeric result

unit

reference range

Validate

Insert into SQLite

The engine is deterministic and rule-based.

The Specific Problem

Example OCR line:

CRP H 5.2 mg/L 0-5

OCR text is correct.
NUMBER_PATTERN matches.
The canonical dictionary contains the test.

Yet:

Inserted 0 rows from 0126251OrderReport_23B00006604_CRP.pdf

Likely failure points include:

• Canonical containment match failing due to normalisation
• Flag tokens (“H”, “L”) interfering with numeric capture
• Numeric extraction anchored incorrectly
• Validation rejecting due to strict range formatting
• Unit pattern mismatch (e.g. mmol/L)
• Dictionary indexing issue
• Match overridden by another lab name
• Guard conditions too strict

If validation fails, the row is rejected silently.

All other panels extract correctly. The issue appears isolated.

What Is Required

This is not a rebuild.

We do not want:

• Re-architecture
• Experimental AI guessing logic
• Large-scale changes
• Expanded fuzzy matching

We need:

1. Precise Diagnosis

Identify exactly where CRP, ESR, and GLU are failing insertion and which rule is causing rejection.

2. Minimal Safe Fix

Implement a targeted correction that:

• Adjusts canonical matching if required
• Anchors numeric extraction correctly
• Allows flag tokens without blocking capture
• Relaxes only necessary validation checks
• Preserves deterministic behaviour

3. Zero Regression

• No impact to currently working panels
• No performance degradation
• No uncontrolled fuzzy expansion

4. Modular Implementation

If appropriate:

• Implement as small isolated module
or
• Cleanly adjust matching block

The existing architecture should remain intact.

Constraints

The system is designed to be:

• Deterministic
• Schema-driven
• Reproducible
• Forensic-grade

We cannot introduce probabilistic or unpredictable behaviour.

Longer-Term Goal

After stabilising extraction:

• Migrate to web deployment
• Enable structured uploads
• Add trend analysis
• Later incorporate AI-assisted interpretation

Immediate priority:

Stabilise deterministic extraction for CRP, ESR, and GLU without breaking the existing engine.

Materials Provided

Uploaded:

• Full extraction_core_2.py (text format)
• Screenshot of HTML viewer
• Sample PDF files
• Export showing required output

Additional materials available on request:

• Sample OCR blocks
• Canonical dictionary entries
• Regex patterns
• Validation logic
• Database schema
• Debug logs

This is a focused debugging and refinement request. I have spent many hours attempting to isolate the issue and now require an experienced developer to identify the blocking condition and implement a practical fix.

I have been advised this should take 1–2 hours for a senior developer.

Looking for a swift turnaround.