Five open source vector databases are worth a shortlist in 2026. Qdrant is Rust-based and wins on single-node latency and filtered ANN. Milvus 2.5 is the billion-scale pick with disk and GPU indexes. Weaviate bundles hybrid search and generative modules. Chroma is the simplest Python option for prototypes and agent memory. pgvector 0.8 is the smart bet when Postgres already runs your data. LanceDB earns a mention for multimodal, read-heavy work on S3. The right pick depends on where your data sits, how big the index gets, and whether you want strict p95 latency or built-in RAG glue.

I’ll show you how to build a real URL shortener in under 200 lines of Python. We’re going to use FastAPI for the web layer, SQLite for storage, and base62 encoding for short codes. I’ll walk you through a redirect endpoint, a click counter, and rate limiting with SlowAPI . In my experience, this simple stack handles millions of links on one server.

Architecture and Tech Stack Choices

Before I write any code, I want to walk you through why I picked this stack. Picking the wrong stack for a small project either over-engineers it or under-builds it. I’ve seen systems fall over at a few hundred users, and I want to help you avoid that.

SQLite can now run at the edge. It works inside Cloudflare Workers via D1, on Fly.io via LiteFS replicated volumes, and in any V8 isolate through embedded WASM builds. This gives you sub-millisecond read queries. You get them by placing your database close to your users on a global CDN. A few tools made this practical: LiteFS for transparent SQLite replication, Cloudflare D1 as a managed edge service, Turso for libSQL with server mode and replication, and Litestream for streaming the WAL to S3. SQLite ships as a single file with zero dependencies. So you get a relational database that deploys with your app binary, needs no connection pooling, and handles thousands of reads per second per node.

Redis Streams give you a light, self-hosted option versus Apache Kafka for event-driven data pipelines. You get append-only log semantics, consumer groups with ack tracking, and sub-millisecond latency on a single Redis 7.4+ instance. Producers XADD events to a stream. Consumer groups read with XREADGROUP in Python via redis-py . Manual XACK calls plus a pending entry list (PEL) give you at-least-once processing.

What follows covers stream basics, consumer groups with failure recovery, a full producer and consumer pipeline with a dead-letter queue, and the ops practices to keep Redis Streams healthy in production.

SQLite is the right default database for most apps. With WAL mode on, it gives you unlimited concurrent readers and one writer. That writer can sustain thousands of transactions per second on modern NVMe drives. SQLite also handles files up to 281 TB and needs zero config, zero extra processes, and zero network hops. Start with SQLite. Move to PostgreSQL only when you hit a real, measured limit, not a guess.

Testcontainers spins up real databases and services as Docker containers inside your test suite. Tests run against production-grade PostgreSQL, Redis, or Kafka instead of flaky mocks. The testcontainers-python v4.14.2 library works with pytest . It automates the container life cycle. You get isolated, reproducible integration tests that catch bugs unit tests miss.

Below: setup with pytest, testing services beyond databases, performance patterns, and CI/CD configuration.

Why Mocks and In-Memory Databases Are Not Enough

Mocking db.execute() only checks if your code calls the function. It does not check if the SQL is valid. It also misses schema errors and type mismatches. You might have passing tests while your queries fail in production.