Instructions to use srivassid/drishti with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use srivassid/drishti with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("google/gemma-4-12B-it") model = PeftModel.from_pretrained(base_model, "srivassid/drishti") - Notebooks
- Google Colab
- Kaggle
Drishti β Gemma 4 12B Indic OCR & caption LoRA (v1)
A LoRA adapter (r=16) for google/gemma-4-12B-it, trained with QLoRA
(NF4) on about 36k Indic page-OCR and image-caption samples. It is an adapter, not a full model: load it on top
of the base model.
Headline result (IndicVisionBench, 10 languages): page-OCR character error rate falls from 0.991 to 0.379, but English-to-Indic translation gets worse (chrF 57.1 to 47.4). See Results and Limitations before using it.
Usage
import torch
from transformers import AutoModelForImageTextToText, AutoProcessor, BitsAndBytesConfig
from peft import PeftModel
from PIL import Image
base = "google/gemma-4-12B-it"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True,
bnb_4bit_compute_dtype=torch.bfloat16,
llm_int8_skip_modules=["vision_tower", "multi_modal_projector", "embed_vision",
"embed_audio", "vision_embedder", "lm_head"])
model = AutoModelForImageTextToText.from_pretrained(base, quantization_config=bnb, dtype=torch.bfloat16,
attn_implementation="sdpa", device_map={"": 0})
model = PeftModel.from_pretrained(model, "srivassid/drishti").eval()
processor = AutoProcessor.from_pretrained(base)
msgs = [{"role": "user", "content": [{"type": "image", "image": Image.open("page.jpg").convert("RGB")},
{"type": "text", "text": "Extract the text from this image exactly as written."}]}]
enc = processor.apply_chat_template(msgs, tokenize=True, return_dict=True, return_tensors="pt",
add_generation_prompt=True).to("cuda")
enc["pixel_values"] = enc["pixel_values"].to(torch.bfloat16)
out = model.generate(**enc, max_new_tokens=768, do_sample=False)
print(processor.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))
Use the same prompt wording as training (see code/prompts.py); the model was trained on four OCR phrasings and four
caption phrasings, English only.
Training data
36,000 samples drawn at random (seed 42) from a 72,605-sample mix; 244 were dropped for exceeding 2,048 tokens, so
35,756 were trained on. The exact 36,000 are listed in
srivassid/indic-vlm-ocr-caption-manifest-v1
(text only; no images).
| Source | Task | Samples | Languages |
|---|---|---|---|
| Internet Archive (Digital Library of India scans, OCR text) | page OCR | 15,879 | 22 |
| Wikisource page scans + transcriptions | page OCR | 14,025 | 12 |
| Vaani (spoken descriptions of images, transcribed) | caption | 5,355 | 19 |
| WIT (Wikipedia image captions) | caption | 741 | Hindi only |
Per-language caps were applied when the mix was built (IA 1,500, Wikisource 2,500, Vaani 700 per language), so large languages are not over-represented. This is a small slice of the data available (the Internet Archive set alone has about 710k samples before capping); the caps and the 36k subset were chosen to fit a ~15-hour training budget, not for quality.
Training procedure
| Method | QLoRA: NF4 4-bit base, double quantization, bf16 compute; vision/audio embedders and lm_head not quantized |
| LoRA | r=16, alpha=32, dropout 0.05, on q/k/v/o/gate/up/down projections of the language model only |
| Optimizer | paged AdamW 8-bit, lr 1e-4, cosine schedule, 100 warmup steps, grad clip 1.0 |
| Batch | global 16 (2 GPUs x 2 samples x 4 accumulation), 2,235 steps = 1 epoch |
| Sequence | max 2,048 tokens; images resized to <=1,344 px, 280 image tokens |
| Loss | answer tokens only (prompt and image tokens masked); computed in 256-token chunks so the 262k-vocab logits are never fully materialised |
| Hardware / time | 2 x RTX 3080 (20 GB), ~14.8 h, ~0.67 samples/s |
Training loss (mean of first / last 10 logged points): 3.40 to 2.31. Validation loss (200 held-out mix samples):
2.438, 2.277, 2.185, 2.146, 2.145 at 0.22, 0.45, 0.67, 0.90, 1.0 epochs. Curve in results/training_curve.json.
Code and config are in code/.
Results
Evaluated on IndicVisionBench (Krutrim AI Labs), which is not part of the training data. Greedy decoding, same prompts for both models, same sampled items.
| Task | Metric | Base | This adapter |
|---|---|---|---|
| OCR (200 pages, 20 per language) | CER, lower is better | 0.991 | 0.379 |
| Translation, English to Indic (106 items) | chrF, higher is better | 57.1 | 47.4 |
Per language (base / adapter):
| Language | OCR CER | Translation chrF |
|---|---|---|
| Bengali | 0.91 / 0.20 | 67.1 / 60.8 |
| Gujarati | 0.93 / 0.32 | 55.8 / 45.1 |
| Hindi | 0.91 / 0.35 | 65.0 / 59.2 |
| Kannada | 0.95 / 0.44 | 54.6 / 45.2 |
| Malayalam | 1.10 / 0.60 | 56.4 / 42.4 |
| Marathi | 0.89 / 0.33 | 58.6 / 44.7 |
| Odia | 1.12 / 0.57 | 32.4 / 24.3 |
| Punjabi | 1.14 / 0.26 | 45.8 / 46.0 |
| Tamil | 0.95 / 0.22 | 65.8 / 46.3 |
| Telugu | 1.00 / 0.49 | 61.2 / 57.8 |
Aggregates only are published (results/indicvisionbench_aggregate.json); the benchmark's pages are not redistributed.
Scoring script: code/eval_bench.py.
Limitations
- The OCR gain is overstated by the baseline. With the training-style prompt, the base model usually does not transcribe at all: it writes an English description of the page (and often names the wrong script). So 0.991 reflects instruction-following, not only reading ability. A stricter prompt for the base model was not tested.
- Translation regressed in 9 of 10 languages (Punjabi flat), by about 10 chrF on average. The training mix contains no translation or general instruction data. The cause was not investigated (for example, whether the adapter answers a different task). Do not use this adapter for translation or general chat.
- Small evaluation: 200 OCR pages and 106 translation items; differences of a few points are noise. OCR CER is computed on the first 600 characters of each page. VQA parts of the benchmark were not run.
- Noisy training labels: the Internet Archive OCR text (44% of the training samples) contains OCR errors, so the model can learn to reproduce them. Some languages have little data in the subset (Urdu Wikisource 343, Kashmiri and Santali Vaani under 20).
- Hindi is the only WIT language. The adapter was not evaluated for safety, bias, or non-OCR behaviour.
- This is a first run with an arbitrary 36k subset and one set of hyperparameters, with no ablations.
Licence
The adapter is released under Apache-2.0, matching the base model. The training data comes from sources with their own terms (Internet Archive / Digital Library of India scans, Wikisource, Vaani, WIT); see the dataset card. Check those before commercial use.
- Downloads last month
- 19