Fine-Tuning Qwen2.5-VL with LoRA

MAI 656 ā€” Natural Language Processing | Canadian University Dubai | Spring 2026

Stage 3: Arabic Handwriting Recognition ā€” Transcription Fine-Tuning Pipeline

Model: Qwen2.5-VL-7B Method: 4-bit LoRA Task: Arabic HTR Hardware: NVIDIA A10G (24 GB) Training: ~30 min

End-to-End Pipeline Flow

flowchart TD
    subgraph INPUT["šŸ“ Input Data"]
        A1["data/train/train.jsonl\n1,120 samples"]
        A2["data/eval/eval.jsonl\n280 samples"]
    end

    subgraph PREP["šŸ”§ Setup & Configuration"]
        B1["GPU Check\nnvidia-smi\nā‰„24 GB VRAM"]
        B2["Install Dependencies\ntransformers Ā· peft Ā· bitsandbytes\naccelerate Ā· qwen-vl-utils"]
        B3["Configuration Cell\nModel Ā· LoRA Ā· Training\nImage Ā· Quantization"]
    end

    subgraph CONFIG["āš™ļø Configuration Parameters"]
        C1["BASE_MODEL\nQwen/Qwen2.5-VL-7B-Instruct"]
        C2["LoRA\nr=32 Ā· Ī±=64 Ā· dropout=0.05\ntargets: q/k/v/o Ā· gate/up/down"]
        C3["Training\nepochs=3 Ā· bs=1 Ā· grad_accum=2\nlr=2e-5 Ā· cosine Ā· warmup=0.1"]
        C4["Quantization\n4-bit NF4 Ā· double quant\nbfloat16 compute"]
        C5["Image Processing\nmin_pixels=3,136\nmax_pixels=100,352\nDPI=100"]
        C6["MAX_SEQ_LENGTH\n8192"]
    end

    subgraph MODEL["šŸ§  Model Loading"]
        D1["Snapshot Download\nVerify HF Disk Cache"]
        D2["BitsAndBytesConfig\n4-bit Quantization"]
        D3["Qwen2_5_VLForConditionalGeneration\nfrom_pretrained\ndevice_map='auto' Ā· sdpa"]
        D4["AutoProcessor\nmin/max_pixels applied"]
    end

    subgraph LORA["šŸŽÆ LoRA Adaptation"]
        E1["prepare_model_for_kbit_training\nGradient hook setup"]
        E2["LoraConfig\nr=32 Ā· Ī±=64 Ā· dropout=0.05"]
        E3["get_peft_model\nInject LoRA adapters"]
        E4["95.18M trainable\n/ 8.39B total (1.13%)"]
    end

    subgraph DATA["šŸ“Š Dataset Pipeline"]
        F1["ArabicHTRDataset\nCustom torch Dataset"]
        F2["extract_images()\nbase64 ā†’ PIL Image"]
        F3["apply_chat_template\nTokenize chat format"]
        F4["processor(text, images)\ninput_ids Ā· attention_mask\npixel_values Ā· image_grid_thw"]
        F5["labels = input_ids.clone()\n-100 for padding (collator)"]
        F6["DataCollatorForSeq2Seq\nDynamic batch padding"]
    end

    subgraph TRAIN["šŸ‹ļø Training Loop"]
        G1["TrainingArguments\nbf16 Ā· grad checkpoint\neval/save per epoch"]
        G2["Trainer\nHuggingFace Trainer"]
        G3["Forward Pass\nVision Tower ā†’ Projector\nā†’ Language Model"]
        G4["Loss Computation\nCross-entropy"]
        G5["Backward + Optimizer\nAdamW Ā· gradient accumulation"]
        G6["Eval per Epoch\nValidation Loss"]
    end

    subgraph OUTPUT["šŸ’¾ Output"]
        H1["model.save_pretrained\nLoRA adapter weights"]
        H2["processor.save_pretrained\nTokenizer + image processor"]
        H3["models/lora_adapter/run-2/\n~380 MB adapter"]
    end

    subgraph NEXT["āž”ļø Next"]
        I1["06_evaluation.ipynb\nCER Ā· WER Ā· Exact Match"]
    end

    A1 --> F1
    A2 --> F1
    B1 --> B2 --> B3
    B3 --> C1
    B3 --> C2
    B3 --> C3
    B3 --> C4
    B3 --> C5
    B3 --> C6
    C1 --> D1 --> D2 --> D3 --> E1
    C2 --> E2
    C4 --> D2
    C5 --> D4
    C6 --> F4
    D3 --> E1 --> E2 --> E3 --> E4
    E4 --> G2
    D4 --> F4
    A1 --> F1 --> F2 --> F3 --> F4 --> F5 --> F6 --> G2
    C3 --> G1 --> G2
    G2 --> G3 --> G4 --> G5 --> G6
    G6 --> G3
    G5 --> H1 --> H2 --> H3 --> I1

Component Architecture

šŸ“„ Input Data Layer

  • OpenAI-style chat messages per sample
  • Image as base64 data:image/png;base64,...
  • Target: JSON string {"transcription": "..."}
  • Reference transcriptions generated by gpt-5.4-mini
  • Train: 1,120 samples Ā· Eval: 280 samples

šŸ”© Dependencies

  • transformers ā‰„ 4.52.0
  • peft == 0.14.0
  • accelerate ā‰„ 1.3.0
  • bitsandbytes ā‰„ 0.46.1
  • torchvision
  • qwen-vl-utils

šŸ§  Model Architecture

  • Qwen2.5-VL-7B-Instruct ā€” Vision-Language Model
  • Vision Tower ā†’ Projector ā†’ LLM backbone (7B)
  • 4-bit NF4 quantization reduces VRAM ~4Ɨ
  • Double quantization for further compression
  • bfloat16 compute dtype for stability
  • SDPA attention implementation

šŸŽÆ LoRA Adaptation

ParameterValue
Rank (r)32
Alpha (Ī±)64
Dropout0.05
Task TypeCAUSAL_LM
Trainable Params95,178,752 (1.13%)
Total Params8,387,345,408

q_proj k_proj v_proj o_proj gate_proj up_proj down_proj

šŸ“Š Dataset Pipeline

  • ArabicHTRDataset: custom PyTorch Dataset
  • extract_images(): base64 decode ā†’ PIL Image (RGB)
  • apply_chat_template(): chat ā†’ tokenizable string
  • processor(text, images): full tokenization + image features
  • DataCollatorForSeq2Seq: dynamic padding per batch
  • Padding positions set to -100 (loss ignored)

šŸ‹ļø Training Configuration

ParameterValue
Epochs3
Batch Size1 (per device)
Gradient Accumulation2
Effective Batch2
Learning Rate2e-5
SchedulerCosine
Warmup10%
Precisionbf16
Gradient CheckpointingEnabled
Total Steps1,680
Duration~30 min

Training Results

šŸ“ˆ Loss Per Epoch

EpochTraining LossValidation Loss
11.62981.6350
21.62081.6248
31.45501.6387

Training loss decreases steadily. Validation loss follows a U-shape ā€” slight overfitting begins at epoch 3 (train ā†“ 1.455 vs val ā†‘ 1.639).

šŸ’¾ Output Artifacts

  • Saving path: models/lora_adapter/run-2/
  • Contents: adapter_config.json, adapter_model.safetensors
  • Processor: tokenizer + image preprocessor config
  • Adapter size: ~380 MB (2.3G du reported)
  • Versioning: Each run writes to its own run-N/

Key Techniques & Why They Matter

šŸ”‹ 4-bit Quantization (NF4)

Loads the 16 GB (fp16) model in ~4 GB VRAM, enabling fine-tuning on a 24 GB GPU. NF4 is a non-uniform 4-bit type optimized for normally-distributed weights. Double quantization compresses the quantization constants themselves for marginal extra savings.

šŸŽÆ Parameter-Efficient Fine-Tuning (LoRA)

Only trains low-rank adapter matrices (1.13% of params). Freezes the base model, so the full 8.3B backbone is never updated during backprop. Rank 32 is the max that fits 24 GB ā€” higher ranks OOM.

šŸ–¼ļø Vision-Language Processing

min_pixels / max_pixels control the vision tower's token budget. Below min ā†’ image is upsampled; above max ā†’ downsampled. DPI=100 during transform and running must be identical. Mismatch ā†’ distorted or unreadable input.

šŸ“ Sequence Length = 8192

Arabic page transcriptions are long. Below 8192 many samples are truncated to near-zero tokens ā†’ degenerate training with 0 useful steps. This was a critical requirement: too-small MAX_SEQ_LENGTH silently produced a broken adapter.

šŸ—œļø Gradient Checkpointing

Trades ~20% longer training time for ~60% VRAM reduction on activations. Essential here ā€” 7B model + vision tokens + LoRA would OOM without it.

āš” base64 Image Decoding

The notebook's extract_images() walks all messages, decodes data: URLs to PIL Images, and feeds them to the processor. Without this step, the vision tower receives empty slots and the adapter learns no visual features.

Critical Constraints

āš ļø The notebook will silently fail if any of these are wrong:

  • MAX_SEQ_LENGTH < 8192 ā†’ samples truncated to near-zero ā†’ 0 training steps
  • min_pixels / max_pixels mismatch between transform and training ā†’ unreadable images
  • DPI mismatch (transform vs training) ā†’ distorted input resolution
  • Forgetting images= in the processor call ā†’ vision tower gets empty slots ā†’ adapter learns no visual features
  • LoRA rank >32 on 24 GB ā†’ OOM (rank 16 is the safe fallback)
  • BATCH_SIZE > 1 ā†’ instant OOM; use gradient accumulation instead

Troubleshooting Reference

ProblemCauseFix
CUDA OOMLoRA rank too high or images too largeReduce LORA_R to 16, reduce MAX_PIXELS
Training: 0 stepsMAX_SEQ_LENGTH too lowSet to 8192 or higher
Killed / OOMBatch size too largeKeep BATCH_SIZE=1, increase GRADIENT_ACCUMULATION
Blurry imagesDPI too lowUse DPI=100 in transform step

Next Step

LoRA adapter saved to models/lora_adapter/run-2/. Continue to:

ā†’ Notebook 6: 06_evaluation.ipynb ā€” Evaluate CER, WER, and exact match against gpt-5.4-mini reference transcriptions.

MAI 656 ā€” Canadian University Dubai Ā· Spring 2026