BlogBooks & scholarship8 min read
Turn a converted Arabic book into clean plain text for corpus work
Keep one faithful UTF-8 master of the converted book with printed page markers, strip running heads and page numbers, move footnotes out of the main text, then scan for presentation forms, tatweel, Persian letters and invisible marks. Normalize only in copies made for analysis, never in the master.
A Word file is the right deliverable when someone will read, edit or typeset a book. It is the wrong one when the next step is a concordancer, a word-frequency script, a text-reuse search or an AI tool that answers questions about the book. Those tools want plain text, and they treat everything in the file as the author's words: the running head printed on every page, the page numbers, the editor's footnotes, a stray tatweel or an invisible direction mark.
This guide is for a researcher, muhaqqiq or digital-humanities team that has a scanned Arabic edition converted to text and wants a file it can analyse and still cite from. It sits after conversion. If you are still choosing how to convert the book, start with digitizing Arabic books.
Decide what the text file is for
Make two kinds of file, and never confuse them:
- The master. UTF-8 plain text that matches the printed edition as closely as you can: same letters, same hamza and tashkeel as printed, printed page markers, notes kept but clearly separated. You correct OCR errors here and nowhere else.
- Analysis copies. Derived from the master by a script you keep, with whatever normalization your tool needs: tashkeel removed, alef forms merged, notes dropped. You can regenerate these at any time, so you never edit them by hand.
Normalization destroys information. Once أ, إ and آ are all ا, you cannot get the printed spelling back. Keeping a master is what lets you quote the edition exactly in a paper while your search index treats those letters as one.
Step 1: get UTF-8 text out
If your conversion tool exports plain text directly, use that. If you are starting from a reviewed Word file, Word can save it as text: File > Save As, choose Plain Text as the type, then in the File Conversion dialog pick Other encoding and Unicode (UTF-8). Microsoft's documentation notes two things worth knowing here. Any encoding other than Unicode may lose characters, and Word marks in red what it cannot save. Text in the Symbol font and field codes are removed when you save as encoded text, so check how any automatic table of contents or cross-references came out before you trust the file.
If Word offers Allow character substitution, leave it unticked. Its job is to swap characters for "the closest equivalent", which is the opposite of what a master file needs.
Some editors write a byte order mark (BOM) at the start of a UTF-8 file. The Python
documentation says that in UTF-8 "the use of the BOM is discouraged", and its utf-8-sig
codec reads files with or without one. Pick one convention for the corpus and note it.
Step 2: keep printed page markers
You will cite the analysis results by page of the printed edition, so the master needs page
markers. Use one format for the whole corpus. If you already follow our
page-number guide, markers such as [ج٢ ص٤٢]
at the start of each printed page work. If your texts are going into the OpenITI ecosystem,
its mARkdown scheme uses PageV02P042: a two-digit volume and a three-digit page, padded with
zeros. Note the difference in placement. The OpenITI documentation says these tags are
"inserted at the end of the corresponding page", while a bracket marker as described above
marks where a page begins. Mixing the two conventions in one file shifts every citation by a
page.
Step 3: strip the page furniture
Running heads (often the book or chapter title), printed page numbers and catchwords are the largest source of noise. A running head printed on every page of a long book adds the same words hundreds of times to every count. Delete them from the master, but only after checking a handful against the scan, because some editions put a real section title in the same position.
Then repair paragraphs that the page break split. When a sentence runs from one page to the next, the converted text can have a hard line break, sometimes a blank line, where the page ended. Join the two halves and keep the page marker inline at the exact point where the new page begins. A tool that splits text into paragraphs or sentences will otherwise treat each half as separate.
Step 4: separate the notes
The muhaqqiq's footnotes are a different text by a different author, often centuries later. Left inline, they inflate word counts, show up as text reuse between unrelated books and put modern vocabulary into a classical corpus. Choose one approach for the corpus:
- move each page's notes to a companion file, keyed by page marker and note number; or
- keep them in the master after a clear separator line per page, so a script can drop them from the analysis copy.
Editorial matter at book level (title pages, the editor's introduction, indexes) deserves the
same treatment. OpenITI tags such sections ### |EDITOR| so that analysis can skip them, and a
similar separator in your own scheme does the same job.
Step 5: check the characters
Text that looks correct on screen can still hold characters that break counting and matching. These are the usual suspects in OCR'd Arabic:
| Problem | Characters | Why it matters |
|---|---|---|
| Presentation forms | U+FB50–U+FDFF, U+FE70–U+FEFF | A shaped glyph such as ﻣﻦ is not the word من to a search or a counter |
| Tatweel | U+0640 | الـــماء and الماء count as different words |
| Persian or Urdu letters in Arabic text | ک U+06A9, ی U+06CC | Look like ك and ي, but don't match them |
| Invisible and directional marks | U+200B–U+200F, U+202A–U+202E, U+2066–U+2069, U+FEFF | Split words silently |
| Extended Arabic-Indic digits | ۰–۹ U+06F0–U+06F9 | Don't match ٠–٩ in dates, verse numbers or page markers |
Fix these in the master only when the printed page agrees, for example a Farsi yeh that should be an Arabic yeh. Tatweel is sometimes printed deliberately, so check the scan before deleting.
A short check script
The script below runs on Python 3 with no extra packages. It reports the characters above
with line numbers, flags page markers that skip or repeat within a volume, and lists short
lines that recur at the top or bottom of three or more pages, which are probably running heads
or page numbers you have not removed yet. It understands [ص ٤٢], [ج٢ ص٤٢] and
PageV02P042 markers. It reports and changes nothing.
import re, sys
from collections import defaultdict
path = sys.argv[1]
raw = open(path, 'rb').read()
if raw.startswith(b'\xef\xbb\xbf'):
print('note: file starts with a UTF-8 BOM')
text = raw.decode('utf-8-sig')
lines = text.splitlines()
MARKER = re.compile(r'\[(?:ج\s*([0-9٠-٩]+)\s*)?ص\s*([0-9٠-٩]+)\]|PageV(\d\d)P(\d{3})')
CHECKS = [
('presentation form', re.compile('[ﭐ-﷿ﹰ-]')),
('tatweel', re.compile('ـ')),
('keheh or Farsi yeh', re.compile('[کی]')),
('invisible or directional mark', re.compile('[---]')),
('Extended Arabic-Indic digit', re.compile('[۰-۹]')),
('replacement character', re.compile('�')),
]
pages, edges, last = [[]], defaultdict(set), None
for i, line in enumerate(lines, 1):
for name, rx in CHECKS:
if rx.search(line):
print(f'line {i}: {name}: {line.strip()[:50]}')
m = MARKER.search(line)
if m:
vol = int(m.group(1) or m.group(3) or 1)
page = (vol, int(m.group(2) or m.group(4)))
if last and page[0] == last[0] and page[1] != last[1] + 1:
print(f'line {i}: page sequence jumps from {last} to {page}')
last = page
pages.append([])
elif line.strip():
pages[-1].append(line.strip())
for n, page in enumerate(pages):
for line in page[:2] + page[-2:]:
key = re.sub(r'[0-9٠-٩]+', '#', line)
if len(key) <= 60:
edges[key].add(n)
for key, where in edges.items():
if len(where) >= 3:
print(f'possible running head or page number on {len(where)} pages: {key}')
print(f'{len(pages) - 1} page markers, {len(lines)} lines checked')
Save it as check_text.py and run python3 check_text.py book.txt. Python's int() accepts
Arabic-Indic digits directly, so ص ٤٢ and ص 42 both read as page 42. In the report, #
stands for any number, so a page number line shows up as a bare #.
Illustrative example
This is an invented test file, not a real book. We wrote a three-page sample in this session with a running head ("كتاب الطهارة") and a page number on every page, a marker that jumps from page 43 to page 45, a tatweel, the ligature ﷺ, a shaped ﻣﻦ, a right-to-left mark in the middle of a sentence and the Persian letters ی and ک. Running the script with Python 3.11 printed:
note: file starts with a UTF-8 BOM
line 8: presentation form: الـــماء ﷺ
line 8: tatweel: الـــماء ﷺ
line 10: page sequence jumps from (1, 43) to (1, 45)
line 12: presentation form: ومن الفقهاء من فرّق بين القليل والكثير وهو قول ﻣﻦ
line 12: invisible or directional mark: ومن الفقهاء من فرّق بين القليل والكثير وهو قول ﻣﻦ
line 13: keheh or Farsi yeh: ی ک
possible running head or page number on 3 pages: كتاب الطهارة
possible running head or page number on 3 pages: #
3 page markers, 14 lines checked
Every planted problem was reported. The ﷺ line shows why normalization belongs in the analysis copy. Unicode normalization form NFKC (described in UAX #15) does turn ﻣﻦ back into من, but in the same session it also expanded the single character ﷺ into the full phrase صلى الله عليه وسلم, three words and 18 characters. That may be what a frequency count wants. It is not what a faithful master should contain if the edition printed the ligature.
Checklist before you hand the text to a tool
- The master is UTF-8, BOM convention recorded, opened once in a plain text editor to confirm the Arabic is intact
- One page-marker format for the whole corpus, at page start or page end, not both
- The script reports no page jumps, or each jump is explained (plates, missing pages)
- Running heads, printed page numbers and catchwords are gone, checked against the scan
- Paragraphs split by page breaks are rejoined, with the marker inline
- Footnotes and editorial matter are separated from the author's text
- No presentation forms, stray directional marks or replacement characters remain
- Persian letters, tatweel and digits match the printed page
- Analysis copies come from a saved script, and the script and its settings are recorded with the corpus
- A sample of pages has been read against the scan by someone who reads Arabic
Where ScribeTools fits
ScribeTools converts scanned Arabic books into editable Word or plain text, so you can start from a text file instead of retyping. The output still needs review against the scan before anyone cites it, and the structural work here (page markers, notes, running heads) is a decision only the editor can make. If you also deliver a Word file from the same conversion, the QA checklist before delivery covers the Word side, and why Find can't locate Arabic words explains the letter variants that matter for search.
Limitations
- The script finds candidates. It does not know whether a repeated line is a running head or a real repeated phrase, whether a tatweel was printed, or whether a page jump is a missing page or a plate. A person decides each case.
- It only recognizes the three marker formats shown. A different convention needs a different pattern.
- It does not check OCR errors in ordinary letters: a wrong dot or a dropped word in otherwise clean Arabic passes every check here. Only reading against the scan catches those.
- We tested it on one small invented file in this session, not on a full book or on any particular tool's output.
- Which normalization an analysis copy needs depends on the tool and the research question. This guide does not recommend one scheme for every project.
Sources
- OpenITI mARkdown documentation: page numbers, headers and paratext tags
- Microsoft Support: Choose text encoding when you open and save files
- Python documentation: codecs, Encodings and Unicode (utf-8-sig and the BOM)
- Unicode Standard Annex #15: Unicode Normalization Forms
- Unicode Standard: Arabic Presentation Forms-A code chart (U+FB50–U+FDFF)