How Diffing Works
Pipeline: extract → (ocr) → align → diff → enrich → report, implemented in core/src/pipeline.rs. Fully local, fully deterministic — no wall-clock reads or randomness in decision paths.
Extraction
Each format extracts into a normalized DocumentModel (ordered blocks, format tag, page count, warnings):
| Format | Engine | Notes |
|---|---|---|
PDFium (pinned chromium/8165, via pdfium-render) | per-char boxes rebuilt into paragraphs; >30% textless pages ⇒ “scanned PDF suspected” + OCR eligibility | |
| DOCX | raw OOXML (zip + roxmltree) | own parser; tracked changes: w:ins as accepted-inserted, w:del/w:delText as accepted-removed |
| XLSX | calamine | cached values + formula strings; row cap with truncation warning |
| PPTX | zip + XML | <a:t> runs in slide order |
| TXT/MD | direct (64 MB cap) | heading/paragraph structure |
Zip safety: ZipBudget reserves declared sizes, caps entries while reading, catches lying headers; roxmltree’s billion-laughs protection stays on. Text is Unicode NFC-normalized and whitespace-collapsed. The two documents must share the same format.
Alignment (3 passes)
- Anchors (deterministic, no embeddings): clause headings (
Điều N,Article N,Mục N,Phụ lục,1.2.3) matched by unique clause key; unique headings / long identical text (≥ 40 chars); long money strings (≥ 7 digits, unique). Hard constraints — pass 2 never realigns them. - Semantic Needleman–Wunsch: cosine matrix over e5-small block embeddings (mean pooling,
query:/passage:prefixes); monotonic NW with gap penalty −0.2 (headings ×1.5); pairs below similarity 0.55 rejected; exact-float traceback with fixed preference order; chunked at 2000 blocks/side in lockstep windows — determinism requirements. - Move repair: unmatched blocks get a ±3 window search around the predicted position; matches need cosine ≥ 0.85 with (sim, dist, index) tie-breaks. Result: moved clauses become moved pairs, not noise.
Every pair records its source (anchor/semantic/move_repair) and cosine similarity — the alignment half of the audit trail. CI gates quality on a gold corpus: precision ≥ 0.85, recall ≥ 0.80.
Word-level diff
similar (Myers + Patience) produces inline equal/insert/delete spans inside aligned pairs; added/removed changes carry full block text; stats summarize the change list.
Heuristic intelligence
Deterministic regexes + fixed lexicons over VN+EN text:
- Entities: money (
triệu×1e6,tỷ×1e9,nghìn/ngàn/k×1e3; bare1.500.000without a currency marker is rejected), dates (ngày 15 tháng 3 năm 2026→ day numbers), durations → day-equivalents, percent, references, clause names. - Deltas: money relative
(new−old)/|old|(guarded against ÷0) + absolute; date day differences; duration relative on day-equivalents (30 → 15 days = −50%); percent relative + points. - Rules:
MONEY_CHANGED,DATE_CHANGED,DURATION_CHANGED,PERCENT_CHANGED,REFERENCE_CHANGED,CLAUSE_RENAMED,ENTITY_ADDED,ENTITY_REMOVED,KEYWORD_RISK— each with an i18n label key covered by EN/VI parity tests. - Severity:
score = min(100, max(rule points) + keyword uplift); rule points money 35, date/duration/percent 30, reference 20, clause-renamed 15, entity-side 15/10; ≥ 3 lexicon categories add +5. Bands: 0 info, 1–29 low, 30–59 medium, 60–79 high, ≥ 80 critical. Formatting-only changes score 0.
Determinism: lexicon hits sorted before scoring, anchor candidates sorted before pairing, pairing keys quantized to i64, move-repair tie-breaks fixed, timestamps injected from the shell so the core stays wall-clock-free — with a test asserting byte-identical JSON across runs. That is what makes the audit trail’s “why this severity” honest.
Report
DiffResult (JSON) feeds the UI, SQLite history, and the HTML/Markdown exporters (hostile-content escaping with golden tests, zero JS in reports).
What the AI layer does and does not do
| Does | Does not |
|---|---|
| Explain one selected change (EN/VI), streamed | Re-score, reorder or alter any change |
| Receive only that change’s context, budgeted | See your other documents |
| Run loopback-only, fully local | Touch the network |
Not legal advice. Severity comes from rules with published point values; the LLM’s text is generated by a local Apache-2.0 model and DiffLens never depends on it for any decision. The app is fully functional with no model installed.