Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

How Diffing Works

Pipeline: extract → (ocr) → align → diff → enrich → report, implemented in core/src/pipeline.rs. Fully local, fully deterministic — no wall-clock reads or randomness in decision paths.

Extraction

Each format extracts into a normalized DocumentModel (ordered blocks, format tag, page count, warnings):

FormatEngineNotes
PDFPDFium (pinned chromium/8165, via pdfium-render)per-char boxes rebuilt into paragraphs; >30% textless pages ⇒ “scanned PDF suspected” + OCR eligibility
DOCXraw OOXML (zip + roxmltree)own parser; tracked changes: w:ins as accepted-inserted, w:del/w:delText as accepted-removed
XLSXcalaminecached values + formula strings; row cap with truncation warning
PPTXzip + XML<a:t> runs in slide order
TXT/MDdirect (64 MB cap)heading/paragraph structure

Zip safety: ZipBudget reserves declared sizes, caps entries while reading, catches lying headers; roxmltree’s billion-laughs protection stays on. Text is Unicode NFC-normalized and whitespace-collapsed. The two documents must share the same format.

Alignment (3 passes)

  1. Anchors (deterministic, no embeddings): clause headings (Điều N, Article N, Mục N, Phụ lục, 1.2.3) matched by unique clause key; unique headings / long identical text (≥ 40 chars); long money strings (≥ 7 digits, unique). Hard constraints — pass 2 never realigns them.
  2. Semantic Needleman–Wunsch: cosine matrix over e5-small block embeddings (mean pooling, query:/passage: prefixes); monotonic NW with gap penalty −0.2 (headings ×1.5); pairs below similarity 0.55 rejected; exact-float traceback with fixed preference order; chunked at 2000 blocks/side in lockstep windows — determinism requirements.
  3. Move repair: unmatched blocks get a ±3 window search around the predicted position; matches need cosine ≥ 0.85 with (sim, dist, index) tie-breaks. Result: moved clauses become moved pairs, not noise.

Every pair records its source (anchor/semantic/move_repair) and cosine similarity — the alignment half of the audit trail. CI gates quality on a gold corpus: precision ≥ 0.85, recall ≥ 0.80.

Word-level diff

similar (Myers + Patience) produces inline equal/insert/delete spans inside aligned pairs; added/removed changes carry full block text; stats summarize the change list.

Heuristic intelligence

Deterministic regexes + fixed lexicons over VN+EN text:

  • Entities: money (triệu ×1e6, tỷ ×1e9, nghìn/ngàn/k ×1e3; bare 1.500.000 without a currency marker is rejected), dates (ngày 15 tháng 3 năm 2026 → day numbers), durations → day-equivalents, percent, references, clause names.
  • Deltas: money relative (new−old)/|old| (guarded against ÷0) + absolute; date day differences; duration relative on day-equivalents (30 → 15 days = −50%); percent relative + points.
  • Rules: MONEY_CHANGED, DATE_CHANGED, DURATION_CHANGED, PERCENT_CHANGED, REFERENCE_CHANGED, CLAUSE_RENAMED, ENTITY_ADDED, ENTITY_REMOVED, KEYWORD_RISK — each with an i18n label key covered by EN/VI parity tests.
  • Severity: score = min(100, max(rule points) + keyword uplift); rule points money 35, date/duration/percent 30, reference 20, clause-renamed 15, entity-side 15/10; ≥ 3 lexicon categories add +5. Bands: 0 info, 1–29 low, 30–59 medium, 60–79 high, ≥ 80 critical. Formatting-only changes score 0.

Determinism: lexicon hits sorted before scoring, anchor candidates sorted before pairing, pairing keys quantized to i64, move-repair tie-breaks fixed, timestamps injected from the shell so the core stays wall-clock-free — with a test asserting byte-identical JSON across runs. That is what makes the audit trail’s “why this severity” honest.

Report

DiffResult (JSON) feeds the UI, SQLite history, and the HTML/Markdown exporters (hostile-content escaping with golden tests, zero JS in reports).

What the AI layer does and does not do

DoesDoes not
Explain one selected change (EN/VI), streamedRe-score, reorder or alter any change
Receive only that change’s context, budgetedSee your other documents
Run loopback-only, fully localTouch the network

Not legal advice. Severity comes from rules with published point values; the LLM’s text is generated by a local Apache-2.0 model and DiffLens never depends on it for any decision. The app is fully functional with no model installed.