The question
Vision-language models can answer questions about images, but storing and running them on a laptop is another problem. Their weights, vision encoders, and projection layers all consume memory. Reducing weight precision makes the model smaller, but at some point the answers become unreliable.
PocketVLM tests that boundary. It starts with a fine-tuned 3-billion-parameter model, applies several compression methods, and compares accuracy, total model size, and inference speed. It also asks whether a smaller distilled model is a better choice than pushing the original model to very low bit widths.
Research question. What is the smallest practical vision-language model that preserves at least 95% of the fine-tuned baseline’s VQA accuracy and runs locally on a 16 GB Apple Silicon laptop?
Experimental approach
The starting point is Qwen2.5-VL-3B-Instruct. PocketVLM fine-tunes its language decoder with LoRA on VQAv2 while keeping the vision tower frozen. The merged, bf16 fine-tuned checkpoint serves as the reference for every compression result.
Dataset and controls
The experiment uses 20,000 VQAv2 training questions, 512 calibration examples, and 3,000 held-out evaluation questions. The evaluation set is separated by image ID, so different questions about the same image do not leak into training or calibration. All backends use the same short-answer prompt and a cap of 512 visual tokens.
This is a controlled comparison within the project’s held-out VQAv2 split. It is not a score on the official VQAv2 test-dev leaderboard.
Compression methods
PocketVLM compares several ways to reduce storage and memory. They do not behave the same at low precision, which is why bit width alone is not enough to select a model.
Post-training weight quantization that uses calibration information to reduce reconstruction error. The sweep tests 4-bit, 3-bit, and 2-bit weights.
Activation-aware post-training quantization. It protects weights judged important using activation statistics and tests the same three precisions.
Converts the model and vision projection into formats the Mac can run. The sweep includes Q8_0, Q6_K, Q4_K_M, Q3_K_M, Q2_K, and IQ2_XS.
Compares merging an adapter before quantization with loading an adapter over a quantized base. NF4-backed QLoRA tests adaptation during low-bit training.
Trains a Qwen2-VL-2B student from the fine-tuned 3B teacher, combining teacher-logit matching with supervised answer loss.
Measures the language model and multimodal projector together. GGUF variants use F16 or Q8_0 projection weights, rather than excluding vision components from the size total.
Distillation objective
The student learns from teacher probabilities and ground-truth answers. The project uses a temperature of 2, with a weighted combination of KL divergence and cross-entropy:
T = 2
This tests whether using a smaller architecture at moderate precision can beat a larger model forced into extreme quantization.
Measured results
The full-precision fine-tuned 3B model scores 0.8137 VQA accuracy on 3,000 held-out questions, compared with 0.8056 before LoRA fine-tuning. The recommended distilled 2B model scores 0.7945 while reducing the deployable footprint from 7.51 GB to 1.54 GB.
| Model / format | Size (GB) | VQA accuracy | Retention | M3 s/question | Decode tok/s |
|---|---|---|---|---|---|
| 3B fine-tuned bf16 | 7.51 | 0.8137 | 100.0% | Not reported | Not reported |
| 3B GPTQ W4 | 3.39 | 0.8162 | 100.3% | Not reported | Not reported |
| 3B AWQ W4 | 3.40 | 0.8068 | 99.2% | Not reported | Not reported |
| 3B GGUF Q4_K_M + Q8 vision | 2.78 | 0.8098 | 99.5% | 3.16 | 18.5 |
| 3B GGUF Q3_K_M | 2.93 | 0.8056 | 99.0% | 3.13 | 22.4 |
| 3B GGUF Q2_K + Q8 vision | 2.12 | 0.7869 | 96.7% | 3.40 | 39.9 |
| 3B GPTQ W2 | 2.74 | 0.0744 | 9.1% | Not reported | Not reported |
| 3B AWQ W2 | 2.75 | 0.0005 | 0.1% | Not reported | Not reported |
| 2B distilled GGUF Q3_K_M + Q8 vision | 1.54 | 0.7945 | 97.6% | 2.69 | 41.6 |
Retention is computed relative to the fine-tuned bf16 score, not the original base model. The figures compare complete loadable model packages, including the vision components required for inference.
What the comparisons show
Two bits can work, but the method matters
The 2-bit GPTQ and AWQ variants lose most of their VQA accuracy: 0.0744 and 0.0005. GGUF Q2_K keeps 0.7869, or 96.7% of the fine-tuned baseline. The practical limit depends on the format and quantization procedure, not just a nominal bit count.
Lower weight precision does not fix every latency bottleneck
The Mac benchmarks put the 3B GGUF models in roughly the same end-to-end range despite sizable differences in stored weights. Image encoding and prompt processing still take time. Decode tokens per second improves with some lower-bit variants, but that does not translate directly into the same improvement per image question.
The smaller student is the better local tradeoff
The 1.54 GB distilled 2B model is smaller than the 3B Q2_K model and scores higher: 0.7945 versus 0.7869. It is also faster in the reported Mac runs: 2.69 versus 3.40 seconds per question. Distillation contributed a modest gain on its own; much of the benefit comes from choosing a model with fewer parameters.
Four-bit methods retain useful accuracy
GPTQ W4 scores 0.8162, slightly above the 0.8137 reference on this evaluation split. AWQ W4 scores 0.8068 and the 3B GGUF Q4_K_M variant with Q8 vision scores 0.8098. Small changes around the baseline should not be treated as broad improvements without testing across more datasets and runs.
Evaluation and implementation
The pipeline is built around repeatable scripts. Data preparation writes common inputs, fine-tuning produces the reference checkpoint, quantization generates comparable variants, and evaluation records accuracy before local deployment tests are run.
| Stage | Implementation | What it measures or produces |
|---|---|---|
| Data | 00_prepare_data.py | Image-disjoint train, calibration, and evaluation splits |
| Training | 02_train_lora.py, 03_merge_lora.py | Fine-tuned 3B baseline |
| Quantization | 04_quantize_llmc.py, 06_export_gguf.py | GPTQ / AWQ and GGUF variants |
| Distillation | 05_distill.py | 2B teacher-supervised student |
| Evaluation | 01_eval_hf.py, 07_eval_llamacpp.py | VQA scores, latency, and peak memory |
| Benchmark | 08_bench_llamacpp.sh, 11_bench_vllm.py | Decode throughput and GPU kernel benchmarks |
| Reporting and demo | 09_report.py, 10_demo.py | Comparison tables, charts, and local browser demo |
Two measurement environments
Training and the main accuracy sweep were run on a single 48 GB NVIDIA A40. Local inference was measured on a MacBook Air M3 with 16 GB memory, using llama.cpp with Metal. This matters because CUDA quantization kernels and on-device GGUF execution have different performance characteristics.
The headline 41.6 tokens/s is a decode throughput measurement from llama-bench, while 2.69 seconds is an end-to-end image-question latency. The two metrics answer different questions and should not be compared as if they were interchangeable.
Cross-backend checks
The project reports agreement between Hugging Face and llama.cpp on the full-precision model, along with matching behavior for merged and runtime-loaded LoRA adapters in smoke tests. Its Mac cross-device check reports 0.7976 VQA accuracy versus 0.7945 on the A40 for the recommended configuration. These checks reduce the risk that conversion or prompt differences are mistaken for model-quality changes.
Running PocketVLM locally
The selected configuration uses a GGUF language model with a Q8_0 vision component. A local llama-server provides inference, and the demo presents an image upload, a question field, an answer, and the measured response time. After setup, image questions can be processed on the laptop without a remote inference endpoint.
There is a practical distinction between weights on disk, memory needed at runtime, and time per answer. PocketVLM measures all three rather than using file size as a proxy for the full user experience.
Limits and next steps
The evaluation is strong as a controlled within-project comparison, but it has a defined scope. The accuracy numbers come from a held-out partition of VQAv2, not multiple public benchmarks. Other tasks such as document reading, OCR, chart interpretation, and multi-image reasoning may react differently to compression.
Evaluate OCR, document QA, charts, and harder visual reasoning before assuming the same retention outside VQAv2.
Repeat evaluation with additional seeds or splits. Differences of a few tenths of a point may be smaller than run-to-run variation.
Measure more Apple Silicon chips and memory configurations. Results from one M3 Air do not establish performance on every laptop.
Test long prompts, rare answers, and image-heavy requests where quantization errors may be more pronounced.
The repository notes that full quantization-aware training and ONNX export are outside the current scope. It also distinguishes real 4-bit GPU measurements from methods without a corresponding optimized GPU or Metal kernel. Those boundaries matter when interpreting speed comparisons.
Conclusion
PocketVLM shows why model compression should be evaluated as a system tradeoff. GPTQ, AWQ, GGUF, adapter strategies, and distillation can produce very different outcomes even when the reported precision looks similar.
The recommended distilled 2B model reduces the deployed package from 7.51 GB to 1.54 GB, keeps 97.6% of the fine-tuned reference accuracy, and answers an image question in 2.69 seconds in the reported MacBook Air M3 test. The result supports a practical conclusion: for local vision-language inference, a smaller model with moderate quantization can be a better choice than forcing a larger model into its lowest possible bit width.
References & source material
[1] Sharma, D. PocketVLM, source repository, implementation, benchmarks, and README. github.com/b4batunde/PocketVLM.
[2] PocketVLM video demonstration, model compression and local inference walkthrough. youtu.be/mBcHVPjh1uE.