Check Quran quotations in a converted Arabic book

A classical Arabic book can quote the Quran hundreds of times: in the author's argument, in the commentary, in the footnotes of the editor. Once the book has been scanned and converted to Word, each of those quotations is a place where a dropped letter, a missing word or a misread hamza changes the text of a verse. Proofreading them by eye is slow, and the eye tends to read the verse it already knows rather than the letters on the page.

This guide is for the muhaqqiq, editor, publisher or translator who has a converted book and must be sure the quotations are right before editing, typesetting or translating. It compares every quotation in the file with a verified Quran text using a short Python script, then tells you which differences to fix and which to leave. It assumes the conversion is already done (see digitizing Arabic books and the general QA checklist before delivery); this is one deeper check from that list.

Why quotations need their own check

Three things make Quranic quotations different from the rest of the book:

• The reader expects zero tolerance. A typo in the author's prose is a typo. A wrong letter in a verse is a misquotation of scripture, and it is the first thing a reviewer will notice.
• They are usually fully vowelled. Printed editions often add complete tashkeel to verses while the prose around them is bare. Diacritics are small marks that sit close together, which makes them hard for any recognition step to read and hard for a proofreader to see.
• They can be checked mechanically. Unlike the author's prose, there is a reference text to compare against. That is what makes a script worth writing.

Step 1: get a verified reference text

The Tanzil Project publishes the Quran text as plain files. Its download page offers several text types and output formats, and its documentation describes them: "Simple" is the modern (imla'i) spelling, "Simple Clean" is that text "without any diacritics or symbols. Suitable for easy search", and the Uthmani texts follow the older script, described as "similar to Medina Mushaf".

1. Open the Tanzil download page.
2. Choose Simple (Clean) if the book quotes verses in ordinary modern spelling, which most printed scholarly books do. Choose Uthmani if the book's quotations were set from a Mushaf font (you will see spellings like «الصلوٰة» and «ٱلرَّحْمَـٰنِ»).
3. Choose Text (with aya numbers) as the output format. Open the file and look at its first line: the script below expects each line as sura|aya|text, and stops with a message if the file is laid out differently.
4. Read the terms of use on that page. They allow verbatim copies but not changes to the text, and ask that Tanzil be credited with a link. Don't "fix" the reference file; if you think it is wrong, check a printed Mushaf.

Step 2: export the book as UTF-8 text

The script reads plain text, not .docx. In Word, use File > Save As, set Save as type to Plain Text, and when the File Conversion dialog appears choose Other encoding and then Unicode (UTF-8). Microsoft's page on text encoding describes this dialog. Keep the .docx as your working file; the .txt is only for checking.

In plain text, each Word paragraph becomes one line. The script reports line numbers from this file, so search for the quotation's words in Word to find it.

Step 3: make sure quotations are marked

The script finds quotations by the ornate brackets ﴿ ﴾ that most Arabic editions put around verses. In Unicode these are U+FD3E and U+FD3F (ORNATE LEFT and RIGHT PARENTHESIS); the script accepts either as the opening or closing mark, because books and fonts are not consistent about which is which.

Two problems to fix before you run it:

• Brackets turned into something else. If a conversion read ﴿ ﴾ as ordinary parentheses or curly braces, the script won't see those quotations. Search the Word file for «قال تعالى» and «قوله تعالى» and look at what follows.
• A missing bracket. If one side of the pair was lost, the script reports the line as unbalanced and skips it, so check that quotation by hand.

Step 4: run the comparison

Save this as check_quran_quotes.py and run it with Python 3: python3 check_quran_quotes.py book.txt quran-simple-clean.txt. It uses only the standard library.

What it does, in order:

1. Removes tashkeel, Quranic annotation marks and tatweel from both texts, turns alef wasla (ٱ) into a plain alef, and drops digits, verse-end signs (U+06DD) and punctuation. That lets a fully vowelled quotation with verse numbers inside it match the reference.
2. Looks for each quotation as an exact run of words in the reference, across verse boundaries, and prints OK with the verse or range where it starts and ends.
3. If there is no exact match, it relaxes hamza seats, alef forms, ya/alef maqsura and ta marbuta to find the nearest verses, then prints CHECK with each word that differs. If it still finds no close verse, it prints NOT FOUND.

import re, sys, difflib

BRACKETS = "﴾﴿"  # the ornate parentheses ﴾ ﴿
MARKS = re.compile("[ؐ-ًؚ-ٰٟۖ-ۭ࣓-ࣿـ]")
NOISE = re.compile("[0-9٠-٩۰-۹۝()\\[\\]{}«»,.:;!?،؛؟۔-]")
LOOSE = str.maketrans({"أ": "ا", "إ": "ا", "آ": "ا",
                       "ٱ": "ا", "ى": "ي", "ة": "ه",
                       "ؤ": "و", "ئ": "ي"})

def strict(s):
    s = MARKS.sub("", s).replace("ٱ", "ا")
    return " ".join(NOISE.sub(" ", s).split())

def loose(s):
    return strict(s).translate(LOOSE).replace("ء", "")

def load_reference(path):
    verses = []
    for n, line in enumerate(open(path, encoding="utf-8-sig"), 1):
        line = line.strip()
        if not line or line.startswith("#"):
            continue
        parts = line.split("|")
        if len(parts) != 3:
            sys.exit(f"{path} line {n}: expected sura|aya|text; download 'Text (with aya numbers)'")
        verses.append((f"{parts[0]}:{parts[1]}", strict(parts[2])))
    return verses

def locate(words, verses, norm):
    """Return the run of consecutive verses whose words best match `words`."""
    target = set(words)
    scores = [len(target & set(norm(v).split())) for _, v in verses]
    best = None
    for i in sorted(range(len(verses)), key=lambda i: -scores[i])[:5]:
        for a in range(max(0, i - 3), i + 1):
            for b in range(i + 1, min(len(verses), i + 4) + 1):
                cand = " ".join(norm(v) for _, v in verses[a:b]).split()
                sm = difflib.SequenceMatcher(None, words, cand)
                r = sum(m.size for m in sm.get_matching_blocks()) / len(words)
                if not best or (r, a - b) > (best[0], best[1] - best[2]):
                    best = (r, a, b)
    return best

def main(book_path, ref_path):
    text = open(book_path, encoding="utf-8-sig").read()
    verses = load_reference(ref_path)
    joined = " " + " ".join(v for _, v in verses) + " "
    starts, pos = [], 1
    for ref, v in verses:
        starts.append(pos); pos += len(v) + 1

    for n, line in enumerate(text.splitlines(), 1):
        if sum(line.count(b) for b in BRACKETS) % 2:
            print(f"line {n}: unbalanced ornate bracket, check the quotation by hand")

    quotes = re.findall(f"[{BRACKETS}]([^{BRACKETS}\n]+)[{BRACKETS}]", text)
    problems = 0
    for k, raw in enumerate(quotes, 1):
        q = strict(raw)
        if not q:
            continue
        hit = joined.find(" " + q + " ")
        if hit < 0:
            hit = joined.find(q)
        if hit >= 0:
            i = max(j for j, s in enumerate(starts) if s <= hit + 1)
            e = max(j for j, s in enumerate(starts) if s <= hit + len(q))
            span = verses[i][0] + ("" if e == i else "-" + verses[e][0].split(":")[1])
            print(f"[{k}] OK {span}  {q}")
            continue
        problems += 1
        best = locate(loose(q).split(), verses, loose)
        if not best or best[0] < 0.5:
            print(f"[{k}] NOT FOUND  {q}")
            continue
        r, a, b = best
        refs = verses[a][0] + ("" if b - a == 1 else "-" + verses[b - 1][0].split(":")[1])
        ref_words = " ".join(v for _, v in verses[a:b]).split()
        kind = "spelling (hamza/alef/ya/ta marbuta) only" if loose(q) in loose(" ".join(ref_words)) else "words differ"
        print(f"[{k}] CHECK {refs} ({kind})")
        ops = difflib.SequenceMatcher(None, q.split(), ref_words).get_opcodes()
        while ops and ops[0][0] == "insert":
            ops.pop(0)  # reference words before the quotation starts
        while ops and ops[-1][0] == "insert":
            ops.pop()   # reference words after the quotation ends
        for op, i1, i2, j1, j2 in ops:
            if op != "equal":
                book = " ".join(q.split()[i1:i2]) or "(missing)"
                ref = " ".join(ref_words[j1:j2]) or "(not in reference)"
                print(f"      book: {book}  |  reference: {ref}")
    print(f"{len(quotes)} quotations, {problems} to check")

if __name__ == "__main__":
    main(sys.argv[1], sys.argv[2])

Illustrative example

To test the script in this session, we built a small book extract (book.txt, below) and a reference file of four short surahs (Al-Fatiha, Al-Asr, Al-Kawthar and Al-Ikhlas) typed by hand in the same sura|aya|text layout. That reference is a stand-in for the test only. It is not the Tanzil file, and for real work you should download the full text as in Step 1. The extract contains errors we put there on purpose: a dropped alef, a missing word, quotations without hamza, a hadith wrongly set in ornate brackets, an unclosed bracket, and a quotation in a mixed spelling.

قال المصنف رحمه الله: والتوحيد أصل الدين، قال تعالى: ﴿قُلْ هُوَ اللَّهُ أَحَدٌ (١) اللَّهُ الصَّمَدُ﴾ [الإخلاص: ١-٢].
وقال سبحانه: ﴿إِلَّا الَّذِينَ آمَنُوا وَعَمِلُوا الصَّالِحَاتِ وَتَوَاصَو بِالْحَقِّ﴾ [العصر: ٣].
وقال: ﴿إِنَّ الْإِنْسَانَ خُسْرٍ﴾ [العصر: ٢].
وفي الفاتحة: ﴿اياك نعبد واياك نستعين﴾ [الفاتحة: ٥]، و﴿ٱلرَّحْمَـٰنِ ٱلرَّحِيمِ﴾.
وحديث ﴿إنما الأعمال بالنيات﴾ ليس من القرآن.
وقال تعالى: ﴿إِنَّا أَعْطَيْنَاكَ الْكَوْثَرَ [الكوثر: ١].
ومنه: ﴿وَتَوَاصَوْا بِالْحَقِّ وَتَوَاصَوْا بِالصَّبْرِ﴾ وقوله ﴿فصل لربك﴾ ﴿وَالْعَصْرِ (١) إِنَّ الْإِنْسَانَ لَفِي خُسْرٍ﴾ و﴿عملوا الصلحات وتواصوا بالحق﴾

Running python3 check_quran_quotes.py book.txt reference-test.txt printed:

line 6: unbalanced ornate bracket, check the quotation by hand
[1] OK 112:1-2  قل هو الله أحد الله الصمد
[2] CHECK 103:3 (words differ)
      book: وتواصو  |  reference: وتواصوا
[3] CHECK 103:2 (words differ)
      book: (missing)  |  reference: لفي
[4] CHECK 1:5 (spelling (hamza/alef/ya/ta marbuta) only)
      book: اياك  |  reference: إياك
      book: واياك  |  reference: وإياك
[5] OK 1:1  الرحمن الرحيم
[6] NOT FOUND  إنما الأعمال بالنيات
[7] OK 103:3  وتواصوا بالحق وتواصوا بالصبر
[8] OK 108:2  فصل لربك
[9] OK 103:1-2  والعصر إن الإنسان لفي خسر
[10] CHECK 103:3 (words differ)
      book: عملوا الصلحات  |  reference: إلا الذين آمنوا وعملوا الصالحات
10 quotations, 5 to check

How to read it:

• [1], [5], [7], [8], [9]: OK. The vowelled quotation in [1] has a verse number inside it and still matches 112:1–2. [5] was typed in Uthmani style (ٱلرَّحْمَـٰنِ) and still matches after marks are removed. It is reported as 1:1 because the phrase occurs there first (it is also all of 1:3); the script reports the first place a phrase occurs, so compare short phrases with the printed reference after the quotation.
• [2] and [3]: real errors. A dropped alef in «وتواصوا» and a missing «لفي». These are what the check is for. Fix them against the scan, not only against the reference: if the printed page also has the error, it belongs in your apparatus as the author's or printer's error, not silently corrected.
• [4]: spelling only. «اياك» without hamza. Many older printings write hamza inconsistently. Whether you add it is an editorial decision for your house style, not a conversion error.
• [6]: not found. A hadith inside ornate brackets. Either the conversion turned other brackets into ﴿ ﴾ or the book itself uses them loosely; either way it is worth a look.
• [10]: check, with a wide span. The quotation starts in the middle of 103:3 and its first word differs (عملوا for وعملوا, الصلحات for الصالحات), so the comparison lists the earlier words of the verse too. Read such a line against the full verse; here the cause is a quotation set in a partly Uthmani spelling and a missing «و».
• Line 6: unbalanced bracket. The Al-Kawthar quotation lost its closing bracket, so it wasn't compared at all.

Checklist for every quotation the script flags

• Open the scan at that page and read the quotation there, letter by letter.
• If the scan matches the reference, correct the Word file.
• If the scan itself differs from the reference, keep the printed text and note it, following your edition's rules for the apparatus.
• If the difference is only hamza, alef or ya/alef maqsura spelling, apply your house style consistently across the book rather than case by case.
• Compare the surah name and verse number printed after the quotation with the location the script reports.
• Re-run the script after your corrections until only deliberate differences remain, and keep the final output with the project files.

The diacritics are still your job

The script removes every diacritic before comparing, which is why a vowelled quotation can match an unvowelled reference. It also means a wrong fatha, a missing shadda or a misplaced tanween passes as OK. For a book whose quotations are fully vowelled, add a second pass:

1. Download the Simple text (with diacritics) from Tanzil as well.
2. For each quotation the script marked OK, open the verse it reports in that file and compare the vowelling by eye, or paste both into Word and use Review > Compare.
3. Remember that the Unicode Standard encodes Quranic annotation marks "as separate combining marks". A quotation can look identical on screen and still contain different characters, so a visual check catches wrong vowels but not every encoding difference.

Errors to watch for

• Reading the reference instead of the page. When a line is flagged, look at the scan first. The goal is a faithful edition of this book, which is not always the same as the standard text.
• Quotations from another reading (qira'a). If the author quotes a different reading from the one in your reference file, the script will flag it. That is a difference, not an error.
• Paraphrase in brackets. Some authors put a paraphrase or a partial verse with their own connecting words inside ﴿ ﴾. Expect CHECK lines for these and leave them.
• Unmarked quotations. Verses quoted without brackets, or embedded in the author's sentence, are invisible to the script. Search for introductory phrases (قال تعالى، لقوله، كقوله) and check those by hand.
• Changing the reference file. Tanzil's terms don't allow it, and an edited reference defeats the check.

Limitations

• The script has been run only on the small test files described above, not on a full book or on any ScribeTools export. Treat its output as a list of places to look, not a verdict.
• It finds quotations only inside ornate brackets on a single line (one Word paragraph).
• It ignores diacritics entirely, so vowelling errors need the second pass described above.
• The relaxed matching treats ه and ة, ي and ى, and the hamza forms as equal to find the verse, then shows you the difference. Very short quotations (one or two words) can match several verses; the script shows the first.
• The script doesn't read the printed surah name and verse number, so you compare those yourself.
• No reference file settles questions of reading or edition. For a critical edition, the printed Mushaf your edition follows has the final word.

Whatever tool converted the book, its output should be reviewed before anyone relies on it, and scripture is where that review matters most. If you are converting the book in the first place, Arabic OCR and Arabic PDF to Word explain what the conversion gives you to start from.