The Cloud Was Too Expensive
When I started building Cluster, the first problem was not search. It was storage.
A search engine has an appetite. It crawls raw HTML, extracts metadata, builds indexes, saves screenshots and PDFs, and then asks for more. Eventually you are talking about petabytes of data.
The normal startup answer is to put all of this in the cloud. I did the math. The cloud was too expensive, so I put a server rack in my basement.
Servers in the Basement
The logic was pretty simple: if I wanted to store and process this much data on a startup budget, I could either learn how to run the hardware myself or give up on the idea.
So I sourced rack-mount servers, wired the network, and built the storage layer from scratch.
There is something satisfying about hearing your search engine physically humming in the next room. This was the first rack up and running:
Owning the hardware means owning every layer of the stack. Nobody rate-limits my IOPS. Nobody sends me a surprise bill because the crawler got busy at 3 AM.
Of course, it also means I am the one replacing a failed drive on Sunday morning. Fair enough.
Why ScyllaDB?
The crawler produces a ridiculous number of writes, while search queries need low-latency reads at the same time. This is not the workload where I wanted to spend my days persuading a traditional relational database to behave like a distributed key-value store.
I used ScyllaDB instead. It is a C++ rewrite of Cassandra built for high-throughput, low-latency workloads on modern hardware. It can saturate NVMe drives and the network without adding JVM overhead and garbage collection pauses to the list of things I need to worry about.
For the first version of Cluster's data layer, it was a good fit. The crawler wrote page content and extracted metadata into ScyllaDB, partitioned by domain. The query path pulled ranked results from the index with single-digit millisecond latency under concurrent load.
The Rest Goes in Object Storage
Not everything belongs in a database. Raw HTML, screenshots, PDFs, and cached assets are just blobs, and a search engine accumulates an awful lot of them.
For those, I set up MicroCeph across the same physical servers. It provides S3-compatible object storage that scales in the obvious way: add drives, add nodes, and let the cluster rebalance.
Ceph can be an operational beast. MicroCeph cuts down enough of that complexity to make it reasonable for a small team. I configured erasure coding so the data could survive failures without consuming three copies' worth of disk space.
That gave Cluster its first complete storage architecture: ScyllaDB for fast access to structured data, MicroCeph for cheap bulk storage, and a pile of machines I could walk downstairs and touch.
What I Learned
Running your own infrastructure has a wonderful way of turning abstract trade-offs into concrete problems.
A replication factor of two sounds reasonable until two drives fail in the same week. A pile of CPU sounds impressive until the network between racks becomes the bottleneck. Monitoring, alerting, and backups sound like the boring parts until they are the only reason the system is still alive.
This was only the first version, and the architecture has changed since then. But I understand every later version better because I started with those servers humming in my basement.