Image-Text-to-Text
PEFT
Safetensors
gemma4
lora
qlora
ocr
indic
vision-language

Drishti β€” Gemma 4 12B Indic OCR & caption LoRA (v1)

A LoRA adapter (r=16) for google/gemma-4-12B-it, trained with QLoRA (NF4) on about 36k Indic page-OCR and image-caption samples. It is an adapter, not a full model: load it on top of the base model.

Headline result (IndicVisionBench, 10 languages): page-OCR character error rate falls from 0.991 to 0.379, but English-to-Indic translation gets worse (chrF 57.1 to 47.4). See Results and Limitations before using it.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor, BitsAndBytesConfig
from peft import PeftModel
from PIL import Image

base = "google/gemma-4-12B-it"
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True,
                         bnb_4bit_compute_dtype=torch.bfloat16,
                         llm_int8_skip_modules=["vision_tower", "multi_modal_projector", "embed_vision",
                                                "embed_audio", "vision_embedder", "lm_head"])
model = AutoModelForImageTextToText.from_pretrained(base, quantization_config=bnb, dtype=torch.bfloat16,
                                                    attn_implementation="sdpa", device_map={"": 0})
model = PeftModel.from_pretrained(model, "srivassid/drishti").eval()
processor = AutoProcessor.from_pretrained(base)

msgs = [{"role": "user", "content": [{"type": "image", "image": Image.open("page.jpg").convert("RGB")},
                                     {"type": "text", "text": "Extract the text from this image exactly as written."}]}]
enc = processor.apply_chat_template(msgs, tokenize=True, return_dict=True, return_tensors="pt",
                                    add_generation_prompt=True).to("cuda")
enc["pixel_values"] = enc["pixel_values"].to(torch.bfloat16)
out = model.generate(**enc, max_new_tokens=768, do_sample=False)
print(processor.decode(out[0, enc["input_ids"].shape[1]:], skip_special_tokens=True))

Use the same prompt wording as training (see code/prompts.py); the model was trained on four OCR phrasings and four caption phrasings, English only.

Training data

36,000 samples drawn at random (seed 42) from a 72,605-sample mix; 244 were dropped for exceeding 2,048 tokens, so 35,756 were trained on. The exact 36,000 are listed in srivassid/indic-vlm-ocr-caption-manifest-v1 (text only; no images).

Source Task Samples Languages
Internet Archive (Digital Library of India scans, OCR text) page OCR 15,879 22
Wikisource page scans + transcriptions page OCR 14,025 12
Vaani (spoken descriptions of images, transcribed) caption 5,355 19
WIT (Wikipedia image captions) caption 741 Hindi only

Per-language caps were applied when the mix was built (IA 1,500, Wikisource 2,500, Vaani 700 per language), so large languages are not over-represented. This is a small slice of the data available (the Internet Archive set alone has about 710k samples before capping); the caps and the 36k subset were chosen to fit a ~15-hour training budget, not for quality.

Training procedure

Method QLoRA: NF4 4-bit base, double quantization, bf16 compute; vision/audio embedders and lm_head not quantized
LoRA r=16, alpha=32, dropout 0.05, on q/k/v/o/gate/up/down projections of the language model only
Optimizer paged AdamW 8-bit, lr 1e-4, cosine schedule, 100 warmup steps, grad clip 1.0
Batch global 16 (2 GPUs x 2 samples x 4 accumulation), 2,235 steps = 1 epoch
Sequence max 2,048 tokens; images resized to <=1,344 px, 280 image tokens
Loss answer tokens only (prompt and image tokens masked); computed in 256-token chunks so the 262k-vocab logits are never fully materialised
Hardware / time 2 x RTX 3080 (20 GB), ~14.8 h, ~0.67 samples/s

Training loss (mean of first / last 10 logged points): 3.40 to 2.31. Validation loss (200 held-out mix samples): 2.438, 2.277, 2.185, 2.146, 2.145 at 0.22, 0.45, 0.67, 0.90, 1.0 epochs. Curve in results/training_curve.json. Code and config are in code/.

Results

Evaluated on IndicVisionBench (Krutrim AI Labs), which is not part of the training data. Greedy decoding, same prompts for both models, same sampled items.

Task Metric Base This adapter
OCR (200 pages, 20 per language) CER, lower is better 0.991 0.379
Translation, English to Indic (106 items) chrF, higher is better 57.1 47.4

Per language (base / adapter):

Language OCR CER Translation chrF
Bengali 0.91 / 0.20 67.1 / 60.8
Gujarati 0.93 / 0.32 55.8 / 45.1
Hindi 0.91 / 0.35 65.0 / 59.2
Kannada 0.95 / 0.44 54.6 / 45.2
Malayalam 1.10 / 0.60 56.4 / 42.4
Marathi 0.89 / 0.33 58.6 / 44.7
Odia 1.12 / 0.57 32.4 / 24.3
Punjabi 1.14 / 0.26 45.8 / 46.0
Tamil 0.95 / 0.22 65.8 / 46.3
Telugu 1.00 / 0.49 61.2 / 57.8

Aggregates only are published (results/indicvisionbench_aggregate.json); the benchmark's pages are not redistributed. Scoring script: code/eval_bench.py.

Limitations

  • The OCR gain is overstated by the baseline. With the training-style prompt, the base model usually does not transcribe at all: it writes an English description of the page (and often names the wrong script). So 0.991 reflects instruction-following, not only reading ability. A stricter prompt for the base model was not tested.
  • Translation regressed in 9 of 10 languages (Punjabi flat), by about 10 chrF on average. The training mix contains no translation or general instruction data. The cause was not investigated (for example, whether the adapter answers a different task). Do not use this adapter for translation or general chat.
  • Small evaluation: 200 OCR pages and 106 translation items; differences of a few points are noise. OCR CER is computed on the first 600 characters of each page. VQA parts of the benchmark were not run.
  • Noisy training labels: the Internet Archive OCR text (44% of the training samples) contains OCR errors, so the model can learn to reproduce them. Some languages have little data in the subset (Urdu Wikisource 343, Kashmiri and Santali Vaani under 20).
  • Hindi is the only WIT language. The adapter was not evaluated for safety, bias, or non-OCR behaviour.
  • This is a first run with an arbitrary 36k subset and one set of hyperparameters, with no ablations.

Licence

The adapter is released under Apache-2.0, matching the base model. The training data comes from sources with their own terms (Internet Archive / Digital Library of India scans, Wikisource, Vaani, WIT); see the dataset card. Check those before commercial use.

Downloads last month
19
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for srivassid/drishti

Adapter
(171)
this model