Correctors

class manuscript.correctors.CharLM(weights=None, vocab=None, lexicon=None, device=None, force_download=False, mask_threshold=0.05, apply_threshold=0.95, max_edits=2, min_word_len=4, max_len=32, correction_levels=('word',), **kwargs)[source]

Bases: BaseCorrector

Character-level language model corrector using ONNX Runtime.

CharLM uses a Transformer-based masked language model to correct OCR substitution errors. It estimates each character’s probability from its masked context; recognition_confidence from OCR is not used.

Parameters:

  • weights (str or Path, optional) –

    Path or identifier for ONNX model weights. Supports:

    • Local file path: "path/to/model.onnx"

    • HTTP/HTTPS URL: "https://example.com/model.onnx"

    • GitHub release: "github://owner/repo/tag/file.onnx"

    • Google Drive: "gdrive:FILE_ID"

    • Preset name: "prereform_charlm_g1" or "modern_charlm_g1" (from the model registry)

    • None: auto-downloads default preset (prereform_charlm_g1)

  • vocab (str or Path, optional) – Path to vocabulary JSON file. Required for local weights; registry presets supply their own vocabulary artifact.

  • lexicon (str, Path, or set, optional) –

    Word list for dictionary-based validation. Supports:

    • Local file path: "path/to/words.txt"

    • Model registry key: "prereform_charlm_g1" or "modern_charlm_g1" (loads its lexicon artifact)

    • Python set: {"word1", "word2", ...}

    • None: auto-downloads default lexicon for model preset (prereform_words for prereform_charlm_g1, modern_words for modern_charlm_g1)

  • device ({"cuda", "coreml", "cpu"}, optional) – Compute device. If None, selects CUDA when ONNX Runtime reports GPU support, otherwise CPU. Select CoreML explicitly with device="coreml".

  • mask_threshold (float, optional) – Confidence threshold below which characters are considered for correction. Default is 0.05.

  • apply_threshold (float, optional) – Minimum model confidence required to apply a correction. Default is 0.95.

  • max_edits (int, optional) – Maximum number of edits per word. Default is 2.

  • min_word_len (int, optional) – Minimum word length to attempt correction. Default is 4.

  • max_len (int, optional) – Model context length. Longer words are preserved. Default is 32.

  • correction_levels (sequence of str, optional) – TextRegion levels to correct: word, line or block. Default is (“word”,). Text is split into words while preserving case and separators.

  • **kwargs – Additional configuration options.

  • force_download (bool)

Examples

>>> from manuscript.correctors import CharLM
>>> corrector = CharLM()
>>> corrected_page = corrector.predict(page)
predict(page, image=None)[source]

Apply character-level correction to a Page.

Parameters:

  • page (Page) – Input Page object with recognized text.

  • image (np.ndarray, optional) – Optional source image associated with page. Reserved for integrations that may pass the original image to correctors. CharLM does not use this argument.

Returns:

Page copy with corrected text at the selected levels.

Return type:

Page

static train(words_path=None, text_path=None, pairs_path=None, charset_path=None, *, exp_dir='exp_charlm', max_words=1500000, max_pairs_edits=3, max_len=32, emb_size=192, n_layers=8, n_heads=6, ffn_size=1024, dropout=0.1, batch_size=256, accumulation_steps=2, use_amp=True, compile_model=False, epochs=50, lr=0.001, weight_decay=0.01, grad_clip=1.0, min_len=3, mask_prob=0.3, span_min=1, span_max=3, spans_min=1, spans_max=2, pairs_ratio=0.8, eval_ratio=0.01, seed=42, checkpoint=None, **extra_config)[source]

Train CharLM character-level language model.

Parameters:

  • words_path (str or Path, optional) – Path to words file (one word per line).

  • text_path (str or Path, optional) – Path to text file for n-gram dataset.

  • pairs_path (str or Path, optional) – Path to CSV file with incorrect/correct pairs.

  • charset_path (str or Path) – Path to charset file (allowed characters).

  • exp_dir (str, optional) – Experiment directory. Default is “exp_charlm”.

  • max_words (int, optional) – Maximum words to use from words file. Default is 1_500_000.

  • max_pairs_edits (int, optional) – Maximum number of character edits in pairs to include. Default is 3.

  • max_len (int, optional) – Maximum sequence length. Default is 32.

  • emb_size (int, optional) – Embedding size. Default is 192.

  • n_layers (int, optional) – Number of transformer layers. Default is 8.

  • n_heads (int, optional) – Number of attention heads. Default is 6.

  • ffn_size (int, optional) – Feed-forward network size. Default is 1024.

  • dropout (float, optional) – Dropout rate. Default is 0.1.

  • batch_size (int, optional) – Batch size. Default is 256.

  • accumulation_steps (int, optional) – Gradient accumulation steps. Default is 2.

  • use_amp (bool, optional) – Use automatic mixed precision (AMP). Default is True.

  • compile_model (bool, optional) – Use torch.compile for faster training. Default is False.

  • epochs (int, optional) – Number of epochs. Default is 50.

  • lr (float, optional) – Learning rate. Default is 1e-3.

  • weight_decay (float, optional) – Weight decay. Default is 0.01.

  • grad_clip (float, optional) – Gradient clipping. Default is 1.0.

  • min_len (int, optional) – Minimum word length. Default is 3.

  • mask_prob (float, optional) – Probability of using span masking. Default is 0.3.

  • span_min (int, optional) – Minimum span length for masking. Default is 1.

  • span_max (int, optional) – Maximum span length for masking. Default is 3.

  • spans_min (int, optional) – Minimum number of spans. Default is 1.

  • spans_max (int, optional) – Maximum number of spans. Default is 2.

  • pairs_ratio (float, optional) – Ratio of OCR pairs in mixed dataset (0.8 = 80% pairs, 20% ngrams). Default is 0.8.

  • eval_ratio (float, optional) – Evaluation set ratio. Default is 0.01.

  • seed (int, optional) – Random seed. Default is 42.

  • checkpoint (str, optional) – Path to checkpoint to resume from.

  • **extra_config – Additional config options.

Returns:

Path to the final checkpoint. After training, CharLM also attempts to export the final checkpoint to ONNX in exp_dir.

Return type:

str

static export(weights_path, vocab_path, output_path, max_len=32, emb_size=192, n_layers=8, n_heads=6, ffn_size=1024, opset_version=14, simplify=True)[source]

Export CharLM PyTorch model to ONNX format.

Parameters:

  • weights_path (str or Path) – Path to PyTorch checkpoint (.pt file).

  • vocab_path (str or Path) – Path to vocabulary JSON file.

  • output_path (str or Path) – Path to save ONNX model.

  • max_len (int, optional) – Maximum sequence length. Default is 32.

  • emb_size (int, optional) – Embedding size. Default is 192.

  • n_layers (int, optional) – Number of transformer layers. Default is 8.

  • n_heads (int, optional) – Number of attention heads. Default is 6.

  • ffn_size (int, optional) – Feed-forward network size. Default is 1024.

  • opset_version (int, optional) – ONNX opset version. Default is 14.

  • simplify (bool, optional) – Apply ONNX simplification. Default is True.

Return type:

None