Classifying documents into 77 categories without training a model
A document management company receives dozens of emails with attached documents every day: meeting minutes, maintenance invoices, insurance policies, vendor quotes, official notices. They all come in through the same place and none of them is labeled. When someone needs the minutes of the extraordinary meeting from two years ago, the manual search begins.
The platform was already transcribing that client's calls and analyzing their conversations. Attachments came in, their text was extracted, and there they stayed. Classifying them was the missing step, and the underlying decision that solved it goes against the grain: no model was trained.
The categorizer is the work of the product team at Irontec, where I worked as a software architect and platform architect. I am telling this story because of the design, which strikes me as one of the most sensible I have seen for this kind of problem.
The obvious option, and why not
The natural thing would be to train a classifier. The taxonomy has 77 subcategories spread across 15 families —minutes, contracts, finance, taxation, insurance, maintenance, HR…—, so we are talking about a multiclass problem with many classes and heavy imbalance.
That is where it breaks. Training requires a labeled corpus with enough examples of each of the 77, and the long tail never has them: for «supply contracts» or «health and safety» you get four documents a year. And every time the company wants to split a category in two, it is back to retraining and relabeling. The cost is not in the initial training; it is in keeping it alive.
Prototypes instead of weights
The module moves the problem elsewhere. Instead of learning a boundary between classes, it places each category as a point in embedding space and measures distances. A new document is turned into a vector, and the category whose point is closest wins.
Each subcategory has two anchors, not one: the centroid of its example documents and the embedding of its written description. The first captures what real documents look like; the second, what the definition says. They weigh the same, fifty-fifty. When a category has only four examples, the description holds the anchor; when the description is vague, the documents rule. It is a risk split between two sources that fail for different reasons.
The hub problem
Here is the detail that separates this from a run-of-the-mill similarity search. In high-dimensional spaces, vectors appear that come up as neighbors of almost everything: hubs. A generic category —«communications, other»— ends up sitting in the middle of the cloud and wins comparisons it should not win, not because it fits better, but because it is close to everything. If you classify by cosine and call it a day, the entire long tail ends up in the miscellaneous drawer.
The correction is called CSLS and consists of subtracting that popularity from both sides:
sims_c = protos.centroids @ doc_vec # coseno contra los 77 centroides
sims_d = protos.description_embeddings @ doc_vec # y contra las 77 descripciones
# cuánto de "cerca de todo" está este documento (media de sus 10 mejores)
rd_c = np.sort(sims_c)[-k:].mean()
rd_d = np.sort(sims_d)[-k:].mean()
# rc_* viene precalculado: cuánto de "cerca de todo" está cada categoría
csls_c = 2.0 * sims_c - rd_c - protos.rc_centroid
csls_d = 2.0 * sims_d - rd_d - protos.rc_description
mix = 0.5 * csls_c + 0.5 * csls_dFrom the raw similarity you subtract how popular the document is and how popular the category is. What is left is specific affinity. Each category's popularity vector is computed once, when the artifacts are built, and travels with them in a JSON file: at classification time the only computation is a matrix product.
Without the hubness correction, classifying by similarity in 1024 dimensions systematically favors generic categories. It is the silent failure of half the RAG setups out there.
The LLM comes in last, and only if needed
With the scores sorted, you have to decide whether the result is reliable. The system looks at the raw cosine similarity of the top candidate —not the CSLS score, nor the confidence— and compares it against 0.88.
The distinction matters. CSLS reorders well but warps the scale, so its number means nothing in absolute terms. And confidence is a softmax over the five finalists: it measures how much the first stands out over the other four, not whether the first is good. Five bad, similar candidates give low confidence; five bad candidates where one stands out give high confidence and an equally bad answer. To decide «I have got this one» you need an absolute measure, and the only one available is cosine.
Above the threshold, the document is resolved without calling anyone. Below it, the top five go to a language model along with their descriptions, the document text, and the email subject as context, and it picks one. If it picks a different one, it is substituted and marked as decided by the model.
What it takes to hold up in production
The rest of the module is small decisions that only occur to someone who has already been burned.
- Long documents. Meeting minutes do not fit in a single embedding call. They are split into chunks of 6,000 tokens with 200 of overlap and their vectors are averaged, but with decreasing weight: the first chunk weighs more than the second, and so on. The header of the minutes says much more about what the document is than its last page of signatures.
- Insufficient text. Below 50 extracted characters, the attachment is skipped without error and the rest keep being classified. A scanned PDF with no extractable text does not break the whole email.
- Integrity. Each attachment is checked against the SHA-256 stored when it was received. If it does not match, the email's categorization fails instead of classifying content that is no longer what arrived.
- Models by alias. Neither the embedding model nor the reranker names a provider: they request an alias from the gateway —one for embeddings, one for reranking— and it decides what is behind it. It is the same pattern I described in the LLMOps post, and here it shows: swapping the reranking model does not touch the module.
The human closes the loop
Every classification is born marked as unreviewed. In the conversation's analysis tab you can see what was assigned to each attachment, and anyone with access can correct it by choosing among the five candidates or searching the entire taxonomy. When correcting, the parent category is derived automatically —you cannot leave a subcategory hanging from a family it does not belong to— and, this is the important part, the original prediction is preserved.
That detail is what turns review into more than a one-off fix. With the prediction and the correction stored together, a manager can export the set filtering by dates, category, or only human-reviewed items, and out comes a JSON with the document text, what the system predicted, and what the human said. Every correction is a free label for the next version of the prototypes.
When I would copy this design
Whenever there are many classes, few examples per class, and a taxonomy that is still moving. Under those three conditions, training is committing to a snapshot of the problem that will go stale; prototypes regenerate in minutes, and a category's written definition is worth almost as much as its examples.
What I like most is where the cost lands. The expensive part —asking a language model— is the exception, not the usual path: only those who need it pay, and the threshold that decides is a visible number that can be discussed, measured, and moved. It is the opposite of putting an LLM at the center of the system and discovering the bill at the end of the month.
More real-world cases
Konect: audio transcription and analysis
LLMOps on your own cluster with LiteLLM, Langfuse, and ArgoCD
Huygens: a SurrealDB graph as memory for agents
From standard to graph to project
Setting up a homelab from scratch: backups and restore - Homelab (06/06)
