GGUF (llama.cpp)
The standard for running models on CPU or Apple Silicon. It allows mixed-precision and offloading layers to the GPU, making it the most flexible format for local experimentation.
AWQ and GPTQ
Formats optimized for pure GPU inference. AWQ (Activation-aware Weight Quantization) generally offers better accuracy retention than GPTQ at 4-bit precision and is highly recommended for production GPU deployments (e.g., via vLLM).
Internal Resources
- Inference Cost Calculator
- Latency Estimator
- Tabular vs Deep Learning
- Local LLMs vs Managed APIs
- Time Series Baselines
- Tabular Model Selector
- Vision Architecture Selector
- VRAM Calculator
- Choosing Embeddings
- CNN vs ViT in 2024
- RLHF vs DPO
- RAG Chunking Strategies
- RAG Chunk Size Calculator
- Token Ratio Estimator
- Synthetic Data Generation
- Quantization Methods Explained
- Quantization Memory Savings
- Fine-Tuning vs LoRA
- LoRA Rank Calculator
- Multimodal Model Landscape