PocketVLM · Technical paper
P / VLM
Machine Learning · Model Compression
Technical project paper

PocketVLM.
Smaller vision-language models for local inference

An experimental comparison of quantization, LoRA, and distillation to find a model that fits on a laptop without losing most of its visual question-answering accuracy.

Fine-tuned reference7.51 GBQwen2.5-VL-3B · bf16
→
Compression study4 pathsPTQ, GGUF, adapters, distillation
→
Selected local model1.54 GBDistilled 2B · Q3_K_M + Q8 vision
AuthorDev Sharma
ImplementationQwen2-VL · GGUF · llama.cpp
ProjectSource code ↗ · YouTube demo ↗

Abstract

PocketVLM studies how far a fine-tuned vision-language model can be compressed before its visual question-answering accuracy breaks down. Starting from Qwen2.5-VL-3B-Instruct, the project compares GPTQ, AWQ, GGUF quantization, LoRA adaptation, and distillation into a Qwen2-VL-2B student. Across a controlled VQAv2 evaluation split, the selected student reduces total model size from 7.51 GB to 1.54 GB while retaining 97.6% of the fine-tuned baseline accuracy. It reaches 41.6 decode tokens per second and 2.69 seconds per image question on a MacBook Air M3. The main finding is that a smaller, moderately quantized model can offer a better local deployment tradeoff than an aggressively quantized larger model.

Vision-language modelsQuantizationKnowledge distillationVQAv2llama.cppApple Silicon

The question

Vision-language models can answer questions about images, but storing and running them on a laptop is another problem. Their weights, vision encoders, and projection layers all consume memory. Reducing weight precision makes the model smaller, but at some point the answers become unreliable.

PocketVLM tests that boundary. It starts with a fine-tuned 3-billion-parameter model, applies several compression methods, and compares accuracy, total model size, and inference speed. It also asks whether a smaller distilled model is a better choice than pushing the original model to very low bit widths.

Research question. What is the smallest practical vision-language model that preserves at least 95% of the fine-tuned baseline’s VQA accuracy and runs locally on a 16 GB Apple Silicon laptop?

Experimental approach

The starting point is Qwen2.5-VL-3B-Instruct. PocketVLM fine-tunes its language decoder with LoRA on VQAv2 while keeping the vision tower frozen. The merged, bf16 fine-tuned checkpoint serves as the reference for every compression result.

End-to-end workflow
Prepare VQAv2Split by image ID; write common images and questions for each backend
Fine-tune 3BLoRA on decoder, frozen vision tower; merge into bf16 baseline
CompressGPTQ, AWQ, GGUF, adapters, and student distillation
Check accuracyScore held-out questions with the same prompt and image limits
Benchmark devicesNVIDIA A40 for training and evaluation; MacBook Air M3 for local runs
Choose a modelBalance size, accuracy retention, end-to-end latency, and decode speed
Every branch is compared with the same fine-tuned baseline, rather than with an unmodified base checkpoint.

Dataset and controls

The experiment uses 20,000 VQAv2 training questions, 512 calibration examples, and 3,000 held-out evaluation questions. The evaluation set is separated by image ID, so different questions about the same image do not leak into training or calibration. All backends use the same short-answer prompt and a cap of 512 visual tokens.

This is a controlled comparison within the project’s held-out VQAv2 split. It is not a score on the official VQAv2 test-dev leaderboard.

Compression methods

PocketVLM compares several ways to reduce storage and memory. They do not behave the same at low precision, which is why bit width alone is not enough to select a model.

GPTQ

Post-training weight quantization that uses calibration information to reduce reconstruction error. The sweep tests 4-bit, 3-bit, and 2-bit weights.

AWQ

Activation-aware post-training quantization. It protects weights judged important using activation statistics and tests the same three precisions.

GGUF with llama.cpp

Converts the model and vision projection into formats the Mac can run. The sweep includes Q8_0, Q6_K, Q4_K_M, Q3_K_M, Q2_K, and IQ2_XS.

LoRA and QLoRA

Compares merging an adapter before quantization with loading an adapter over a quantized base. NF4-backed QLoRA tests adaptation during low-bit training.

Knowledge distillation

Trains a Qwen2-VL-2B student from the fine-tuned 3B teacher, combining teacher-logit matching with supervised answer loss.

Vision encoder precision

Measures the language model and multimodal projector together. GGUF variants use F16 or Q8_0 projection weights, rather than excluding vision components from the size total.

Distillation objective

The student learns from teacher probabilities and ground-truth answers. The project uses a temperature of 2, with a weighted combination of KL divergence and cross-entropy:

L = 0.7 × T² × KL(student, teacher) + 0.3 × CE(student, answer)
T = 2

This tests whether using a smaller architecture at moderate precision can beat a larger model forced into extreme quantization.

Measured results

The full-precision fine-tuned 3B model scores 0.8137 VQA accuracy on 3,000 held-out questions, compared with 0.8056 before LoRA fine-tuning. The recommended distilled 2B model scores 0.7945 while reducing the deployable footprint from 7.51 GB to 1.54 GB.

4.9×Smaller model package
97.6%Baseline VQA retained
2.69 sPer image question on M3
Selected variants
Model / formatSize (GB)VQA accuracyRetentionM3 s/questionDecode tok/s
3B fine-tuned bf167.510.8137100.0%Not reportedNot reported
3B GPTQ W43.390.8162100.3%Not reportedNot reported
3B AWQ W43.400.806899.2%Not reportedNot reported
3B GGUF Q4_K_M + Q8 vision2.780.809899.5%3.1618.5
3B GGUF Q3_K_M2.930.805699.0%3.1322.4
3B GGUF Q2_K + Q8 vision2.120.786996.7%3.4039.9
3B GPTQ W22.740.07449.1%Not reportedNot reported
3B AWQ W22.750.00050.1%Not reportedNot reported
2B distilled GGUF Q3_K_M + Q8 vision1.540.794597.6%2.6941.6
Accuracy and model sizes are reported by the project. Latency and decode speed are shown only for variants benchmarked on the Mac; blank comparisons must not be read as zero.

Retention is computed relative to the fine-tuned bf16 score, not the original base model. The figures compare complete loadable model packages, including the vision components required for inference.

Total model package size
3B bf16 baseline
7.51 GB
3B GGUF Q4
2.78 GB
3B GGUF Q2
2.12 GB
2B distilled Q3
1.54 GB
Package sizes include model and required multimodal weights. Smaller does not necessarily mean more accurate or faster.

What the comparisons show

Two bits can work, but the method matters

The 2-bit GPTQ and AWQ variants lose most of their VQA accuracy: 0.0744 and 0.0005. GGUF Q2_K keeps 0.7869, or 96.7% of the fine-tuned baseline. The practical limit depends on the format and quantization procedure, not just a nominal bit count.

Lower weight precision does not fix every latency bottleneck

The Mac benchmarks put the 3B GGUF models in roughly the same end-to-end range despite sizable differences in stored weights. Image encoding and prompt processing still take time. Decode tokens per second improves with some lower-bit variants, but that does not translate directly into the same improvement per image question.

The smaller student is the better local tradeoff

The 1.54 GB distilled 2B model is smaller than the 3B Q2_K model and scores higher: 0.7945 versus 0.7869. It is also faster in the reported Mac runs: 2.69 versus 3.40 seconds per question. Distillation contributed a modest gain on its own; much of the benefit comes from choosing a model with fewer parameters.

Four-bit methods retain useful accuracy

GPTQ W4 scores 0.8162, slightly above the 0.8137 reference on this evaluation split. AWQ W4 scores 0.8068 and the 3B GGUF Q4_K_M variant with Q8 vision scores 0.8098. Small changes around the baseline should not be treated as broad improvements without testing across more datasets and runs.

The best local model was not the original 3B pushed to the fewest bits. It was a smaller student compressed to a practical format.

Evaluation and implementation

The pipeline is built around repeatable scripts. Data preparation writes common inputs, fine-tuning produces the reference checkpoint, quantization generates comparable variants, and evaluation records accuracy before local deployment tests are run.

Implementation stages
StageImplementationWhat it measures or produces
Data00_prepare_data.pyImage-disjoint train, calibration, and evaluation splits
Training02_train_lora.py, 03_merge_lora.pyFine-tuned 3B baseline
Quantization04_quantize_llmc.py, 06_export_gguf.pyGPTQ / AWQ and GGUF variants
Distillation05_distill.py2B teacher-supervised student
Evaluation01_eval_hf.py, 07_eval_llamacpp.pyVQA scores, latency, and peak memory
Benchmark08_bench_llamacpp.sh, 11_bench_vllm.pyDecode throughput and GPU kernel benchmarks
Reporting and demo09_report.py, 10_demo.pyComparison tables, charts, and local browser demo

Two measurement environments

Training and the main accuracy sweep were run on a single 48 GB NVIDIA A40. Local inference was measured on a MacBook Air M3 with 16 GB memory, using llama.cpp with Metal. This matters because CUDA quantization kernels and on-device GGUF execution have different performance characteristics.

The headline 41.6 tokens/s is a decode throughput measurement from llama-bench, while 2.69 seconds is an end-to-end image-question latency. The two metrics answer different questions and should not be compared as if they were interchangeable.

Cross-backend checks

The project reports agreement between Hugging Face and llama.cpp on the full-precision model, along with matching behavior for merged and runtime-loaded LoRA adapters in smoke tests. Its Mac cross-device check reports 0.7976 VQA accuracy versus 0.7945 on the A40 for the recommended configuration. These checks reduce the risk that conversion or prompt differences are mistaken for model-quality changes.

Running PocketVLM locally

The selected configuration uses a GGUF language model with a Q8_0 vision component. A local llama-server provides inference, and the demo presents an image upload, a question field, an answer, and the measured response time. After setup, image questions can be processed on the laptop without a remote inference endpoint.

Local inference path
Image + questionEntered in the local web interface
Vision encoderConverts image content into multimodal features
Qwen2-VL-2B GGUFProcesses the prompt and generates an answer through llama.cpp / Metal
The demo runs locally, but reproducing the reported results still requires the correct checkpoint, matching conversion artifacts, and compatible llama.cpp setup.

There is a practical distinction between weights on disk, memory needed at runtime, and time per answer. PocketVLM measures all three rather than using file size as a proxy for the full user experience.

Limits and next steps

The evaluation is strong as a controlled within-project comparison, but it has a defined scope. The accuracy numbers come from a held-out partition of VQAv2, not multiple public benchmarks. Other tasks such as document reading, OCR, chart interpretation, and multi-image reasoning may react differently to compression.

Broader accuracy tests

Evaluate OCR, document QA, charts, and harder visual reasoning before assuming the same retention outside VQAv2.

Statistical variation

Repeat evaluation with additional seeds or splits. Differences of a few tenths of a point may be smaller than run-to-run variation.

Hardware coverage

Measure more Apple Silicon chips and memory configurations. Results from one M3 Air do not establish performance on every laptop.

Low-bit robustness

Test long prompts, rare answers, and image-heavy requests where quantization errors may be more pronounced.

The repository notes that full quantization-aware training and ONNX export are outside the current scope. It also distinguishes real 4-bit GPU measurements from methods without a corresponding optimized GPU or Metal kernel. Those boundaries matter when interpreting speed comparisons.

Conclusion

PocketVLM shows why model compression should be evaluated as a system tradeoff. GPTQ, AWQ, GGUF, adapter strategies, and distillation can produce very different outcomes even when the reported precision looks similar.

The recommended distilled 2B model reduces the deployed package from 7.51 GB to 1.54 GB, keeps 97.6% of the fine-tuned reference accuracy, and answers an image question in 2.69 seconds in the reported MacBook Air M3 test. The result supports a practical conclusion: for local vision-language inference, a smaller model with moderate quantization can be a better choice than forcing a larger model into its lowest possible bit width.

References & source material

[1] Sharma, D. PocketVLM, source repository, implementation, benchmarks, and README. github.com/b4batunde/PocketVLM.

[2] PocketVLM video demonstration, model compression and local inference walkthrough. youtu.be/mBcHVPjh1uE.