vlm

Replicating the LLaVA series of vision-language models.

sh scripts/setup.sh

The script creates a conda environment named vlm_env and automatically detects your platform to install PyTorch with the appropriate backend:

• Mac M-series (Apple Silicon): PyTorch 2.9.0 with MPS support
• Linux aarch64: PyTorch 2.9.0 with CUDA 12.8+ support
• CUDA systems (NVIDIA GPUs): PyTorch 2.9.0 with CUDA 12.8+ support
• Other platforms: PyTorch 2.9.0 CPU version

Make sure you have conda or miniconda installed first.

After setup, activate the environment:

conda activate vlm_env

Verify installation:

conda run -n vlm_env python scripts/verify_pytorch.py
conda run -n vlm_env python scripts/verify_llava.py
conda run -n vlm_env python scripts/verify_instruct_dataset.py

Or if you've activated the environment:

conda activate vlm_env
python scripts/verify_pytorch.py
python scripts/verify_llava.py
python scripts/verify_instruct_dataset.py

Download LLaVA-Pretrain dataset (~100GB):

conda run -n vlm_env python scripts/prepare_dataset.py

Or with the environment activated:

conda activate vlm_env
python scripts/prepare_dataset.py

# Training Phase 1 (with conda run)
conda run -n vlm_env python src/vlm/train/phase1_run.py --data_path ~/dataset/llava-pretrain/blip_laion_cc_sbu_558k.json --image_folder ~/dataset/llava-pretrain

# Training Phase 1 (with activated environment)
conda activate vlm_env
python src/vlm/train/phase1_run.py --data_path ~/dataset/llava-pretrain/blip_laion_cc_sbu_558k.json --image_folder ~/dataset/llava-pretrain

# Inference
conda run -n vlm_env python src/vlm/inference/inference.py --checkpoint ~/models/llava/checkpoint_phase1.pt --image_path <path> --text "Describe this image"

• Python version error: The conda environment will automatically install Python 3.11
• conda not found: Install conda or miniconda from https://docs.conda.io/en/latest/miniconda.html
• PyTorch issues: Try recreating the environment: conda env remove -n vlm_env && sh scripts/setup.sh
• Environment activation issues: Run conda init and restart your terminal

• Python 3.11+, PyTorch 2.9, conda package manager
• MPS support (Apple Silicon), CUDA support (NVIDIA GPUs)