TrVLM: A Turkish Vision-Language Model for Captioning and Visual Question Answering
9th International Symposium on Multidisciplinary Studies and Innovative Technologies, ISMSIT 2025, Ankara, Türkiye, 14 - 16 Kasım 2025, (Tam Metin Bildiri)
- Yayın Türü: Bildiri / Tam Metin Bildiri
- Doi Numarası: 10.1109/ismsit67332.2025.11268077
- Basıldığı Şehir: Ankara
- Basıldığı Ülke: Türkiye
- Anahtar Kelimeler: LLaMA, LoRA Fine-Tuning, Projection Layer, SIGLIP, Turkish Vision-Language Model
- Ankara Üniversitesi Adresli: Evet
Özet
We address the scarcity of Turkish multimodal resources by introducing TrVLM, a Turkish vision-language model for image captioning and visual question answering. The model couples a frozen SigLIP image encoder with a learnable projection into a Turkish LLaMA decoder, and is trained with a two-stage recipe: feature-alignment pretraining on Turkish caption pairs followed by multitask fine-tuning, augmented with a targeted LoRA adaptation for short captions. On Turkish benchmarks, TrVLM delivers strong results. For example, it achieves BERTScore-F1 0.67 for short captioning and 0.65 for VQA, alongside consistent gains across ROUGE, BLEU, and METEOR. A GPT-4o judge study further supports instruction following and semantic fidelity. Our contributions are: a compact SigLIP and Turkish-LLaMA architecture tailored for low-resource Turkish multimodal AI, curated task-balanced Turkish datasets (short/long captioning and VQA), and an efficient training and inference strategy (LoRA with prompt-conditioned checkpointing) that balances specialization and generalization.