Embeddings

Five open source vector databases are worth a shortlist in 2026. Qdrant is Rust-based and wins on single-node latency and filtered ANN. Milvus 2.5 is the billion-scale pick with disk and GPU indexes. Weaviate bundles hybrid search and generative modules. Chroma is the simplest Python option for prototypes and agent memory. pgvector 0.8 is the smart bet when Postgres already runs your data. LanceDB earns a mention for multimodal, read-heavy work on S3. The right pick depends on where your data sits, how big the index gets, and whether you want strict p95 latency or built-in RAG glue.

RAG and long context windows are not competing replacements. They are different tools built for different problems. If you are trying to choose between them, the short answer is: it depends on the size and nature of your data, your latency and cost constraints, and how much infrastructure complexity you are willing to maintain. The longer answer involves understanding what each approach actually does, where each one breaks down, and what teams running production LLM systems are doing in 2026 - which is usually some combination of both.

You can build a multi-modal RAG pipeline that searches text, diagrams, and screenshots at once. The trick is to mix CLIP-based image embeddings with text embeddings in one shared vector space. Store them in a ChromaDB or Qdrant collection. Route queries through a retrieval layer that returns both passages and images. Feed it all to an LLM. With OpenCLIP ViT-G/14 for images plus a self-hosted Llama 4 Scout as the LLM, the whole pipeline runs offline on an RTX 5070 or better.

Gemma 4 shipped on April 2, 2026 with four model variants under the Apache 2.0 license. The 31B dense model ranks third on the Arena AI text leaderboard with a score of 1452. The 26B MoE model scores 1441 while firing only 3.8B of its 26B total parameters per forward pass. So what design choices make this possible? Three of them break from the standard transformer recipe: Per-Layer Embeddings (PLE), Shared KV Cache, and Dual RoPE. Each one shifts the math for inference cost, memory use, and fine-tuning. The rest of this post covers those three, plus the Mixture-of-Experts layer and the multimodal encoders.

To build a private Retrieval-Augmented Generation (RAG) system, pair a local vector database like Qdrant with an embedding model like BGE-M3 . Add a local LLM through Ollama , and you can index hundreds of documents and ask questions about them. Your data stays on your machine.

Why RAG? The Problem With Pure LLM Memory

Large language models sound smart, but they are poor knowledge stores. They learn from old training data and know nothing about files you created later or keep private. Ask about your own data, and the model will often guess. Even strong open weight models like Llama 4.0 can invent plausible but wrong answers about content they never saw. For a deeper breakdown of why LLM hallucinations happen and how to measure them, the issue goes beyond missing context.