Vedabase Original Edition

Complete Diff Summary — All Books

Note: 4 posthumous compilations (A Second Chance, Life Comes from Life, Light of the Bhagavata, Path of Perfection) have no original edition to compare.

Full Book Replacements (8 Books)

Surgical Patching

Zero-Diff Volumes (16 Confirmed Identical)

Scan Files Used (68 PDFs)

Architecture Pipeline

┌─────────────────────────────────────────────────────────────────────────────┐
│                           VEDABASE CORRECTION PIPELINE                      │
└─────────────────────────────────────────────────────────────────────────────┘

    ┌──────────────┐     ┌──────────────┐     ┌──────────────┐
    │  SCAN PDFs   │     │   VEDABASE   │     │   OUTPUT     │
    │  (68 files)  │     │  (current)   │     │  (corrected) │
    └──────┬───────┘     └──────┬───────┘     └──────▲───────┘
           │                    │                    │
           ▼                    ▼                    │
    ┌──────────────────────────────────────┐        │
    │          PyMuPDF TEXT EXTRACTION     │        │
    │  • Unicode IAST preservation         │        │
    │  • Page-by-page processing           │        │
    │  • Header/footer removal             │        │
    └──────────────────┬───────────────────┘        │
                       │                            │
                       ▼                            │
    ┌──────────────────────────────────────┐        │
    │        NORMALIZATION LAYER           │        │
    │  • Smart quote → ASCII               │        │
    │  • Hyphenated line-break repair      │        │
    │  • Diacritic-aware matching          │        │
    └──────────────────┬───────────────────┘        │
                       │                            │
                       ▼                            │
    ┌──────────────────────────────────────┐        │
    │      PARAGRAPH ALIGNMENT (Jaccard)   │        │
    │  • Trigram similarity scoring        │        │
    │  • Best-match paragraph linking      │        │
    │  • Orphan detection                  │        │
    └──────────────────┬───────────────────┘        │
                       │                            │
                       ▼                            │
    ┌──────────────────────────────────────┐        │
    │       DIFF GENERATION (difflib)      │        │
    │  • SequenceMatcher comparison        │        │
    │  • Line-level diff extraction        │        │
    └──────────────────┬───────────────────┘        │
                       │                            │
                       ▼                            │
    ┌──────────────────────────────────────┐        │
    │         5-LAYER NOISE FILTER         │────────┤
    │  • OCR character confusion           │        │
    │  • Diacritic normalization           │        │
    │  • Punctuation variants              │        │
    │  • Whitespace artifacts              │        │
    │  • Alignment false positives         │        │
    └──────────────────┬───────────────────┘        │
                       │                            │
                       ▼                            │
    ┌──────────────────────────────────────┐        │
    │         MANUAL VERIFICATION          │        │
    │  • Scan-by-scan confirmation         │        │
    │  • Semantic change flagging          │────────┘
    │  • Apply corrections                 │
    └──────────────────────────────────────┘

Types of Changes Detected

5-Layer Noise Filter (Detail)

Accuracy & Error Analysis

Tools & Libraries

Process Pipeline

Scripts & Code

compare.py — Main comparison pipeline (2,500+ lines)

def normalize_for_comparison(text: str) -> str:
    # Fix hyphenated line breaks
    text = re.sub(r'(\w)[\-\u00ad]\s*\n\s*(\w)', r'\1\2', text)
    # Strip diacritics
    text = strip_diacritics(text)
    # Remove page headers/footers from scans
    text = re.sub(r'\d+\s+Bhagavad-g\w*\s+As\s+It\s+Is', '', text)
    # Normalize quotes/dashes
    text = text.replace('\u201c', '"').replace('\u201d', '"')
    text = text.replace('\u2018', "'").replace('\u2019', "'")
    return text.strip()

is_noise() — 5-layer OCR noise filter

def is_noise(orig: str, veda: str) -> bool:
    """Filter OCR errors, diacritics, transliteration variants.
    Returns True if difference is noise, False if real edit."""
    o = strip_diacritics(orig.lower().strip())
    v = strip_diacritics(veda.lower().strip())

    # After diacritics normalization, same?
    if o == v: return True

    # OCR zero/O confusion
    if o.replace('0','o') == v.replace('0','o'): return True

    # Only alpha chars — same?
    o_alpha = re.sub(r'[^a-z]', '', o)
    v_alpha = re.sub(r'[^a-z]', '', v)
    if o_alpha == v_alpha: return True

    # Alignment false positive check
    if len(o_alpha) > 25 and len(v_alpha) > 25:
        ratio = difflib.SequenceMatcher(None, o_alpha, v_alpha).ratio()
        if ratio < 0.25: return True

    return False

strip_diacritics.py — IAST diacritic handling

DIACRITIC_MAP = str.maketrans({
    'ā': 'a', 'Ā': 'A', 'ī': 'i', 'Ī': 'I',
    'ū': 'u', 'Ū': 'U', 'ṛ': 'r', 'Ṛ': 'R',
    'ṁ': 'm', 'ṃ': 'm', 'ṅ': 'n', 'ñ': 'n',
    'ṇ': 'n', 'ś': 's', 'ṣ': 's', 'ḥ': 'h',
    'ṭ': 't', 'ḍ': 'd',
})

def strip_diacritics(text):
    text = text.translate(DIACRITIC_MAP)
    # NFD normalization for remaining combining chars
    normalized = unicodedata.normalize('NFD', text)
    return ''.join(c for c in normalized
                   if unicodedata.category(c) != 'Mn')