LLMOps on your own cluster with LiteLLM, Langfuse, and ArgoCD

The question I started with was a reasonable one: if I already have my infrastructure declared in Git and ArgoCD reconciling the cluster, what's the next rung on the ladder? The answer you find everywhere is a reference architecture with six or seven pieces —Crossplane, Argo Workflows, MLflow, KServe, Kubeflow— and the feeling that you're missing all of them.

I crossed them off one by one and ended up where I already was. But with one difference that does matter: now I know why. This post is that journey —what I ruled out and on what grounds—, what runs on the cluster today, and the problem I still haven't solved.

What runs today

Three pieces, and each one solves a problem that genuinely hurts when you put agents in production.

Provider keys don't live in the repository. There's a single source of secrets, and the External Secrets operator materializes them where they're needed: today that's thirteen distinct secrets in the production namespace, each one declared in Git as a reference, never as a value. Rotating a provider key means changing it in one place.

In front of the models there's a gateway, LiteLLM, and that's the decision you appreciate most over time. Applications don't talk to Anthropic or OpenAI: they talk to a stable alias. Switching the model behind that alias doesn't touch a single line of product code. And since all traffic goes through it, you can set a budget per key: each agent or each team has its own, with its cap, and when it's gone it's gone.

And on top of that, self-hosted Langfuse, with its own Application in ArgoCD and its secrets kept separate. Every call ends up there with its cost, its tokens, its latency, and its session. It's been running in production for months, and so has the gateway. Both are declared in Git like anything else in the cluster, so the model gateway and the spend observability deploy with the same mechanism as the rest of the platform. That's all the "Ops" there is in LLMOps.

77 / 51days in production · tracing and gateway

13secrets, none in the repository

1alias per model, stable

0models hosted by me

To avoid depending on production for trying things out, the same setup lives in a reproducible lab: gateway, tracing, storage, and an API that consumes them, all in a dev container. It has one detail I'm happy with: a fake model that calls no provider and needs no key, but carries declared prices. It follows exactly the same cost-calculation path as a real model, so you can validate that the whole chain works —that spend is calculated, returned, and shows up in the traces— without spending a cent.

# Modelo sintético: no llama a ningún proveedor y no necesita API key.
# Lleva pricing declarado para que el gasto sea visible end-to-end sin gastar dinero.
- model_name: mock
  litellm_params:
    model: openai/mock
    api_key: not-needed
    mock_response: "El stack funciona: se ha calculado el coste y se ha enviado a las trazas."
    input_cost_per_token: 0.000003
    output_cost_per_token: 0.000015

The rejects, one by one

This is the useful part, because ruling things out well is harder than adding them.

Crossplane. It's for provisioning cloud infrastructure from cluster manifests. My infrastructure is local and already switched on: the machines exist, Kubernetes runs on top, and the dependencies run as pods inside. It would make sense if I wanted a commit to spin up virtual machines on the fly in the hypervisor, or if I managed databases outside the cluster. That's not the case, and adding it would mean adding a layer that provisions nothing.
Argo Workflows. It's a cluster-native workflow engine, and it shines when you need to process large volumes in parallel or chain steps with dependencies. I don't have data pipelines: I have requests. What I do have —building images, running tests, publishing releases— is already handled by the repository's continuous integration, which is its natural home.
MLflow. It's a model registry and an experiment log. I don't train any. A model registry without training is an empty cupboard with an inventory.
KServe and Seldon. They serve a model by declaring it, without hand-writing the API. The thing is, I have no weights to serve: I consume third-party models through their APIs. And to have it you need to install a certificate manager and a networking or serverless layer underneath, which is a lot of machinery to host nothing.
Kubeflow. The whole suite, which internally uses several of the ones above. If I don't need the individual pieces, I don't need the package that bundles them either.

It wasn't MLOps, it was LLMOps

By the fourth rejection the pattern becomes clear: all those tools assume the model is yours. That you train it, version it, store it, and serve it. My problem is the opposite. The models belong to others and sit behind a paid API; what I have to govern is the traffic toward them.

The reference architecture wasn't a plan: it was a menu. The work wasn't in installing, it was in crossing things off.

Takeaway from the review — from MLOps to LLMOps

And once the problem is called by its name, the priorities rearrange themselves. What becomes critical is where the keys live, who can spend and how much, what happens when a provider returns an error and you have to route to another one, and exactly how much each response costs. None of that is covered by a model registry. All of it is covered by three pieces that fit in the same Git as the rest of the platform.

What the traces don't see

This is where the setup falls short, and I'd rather say so than pretend it's solved.

When an answer goes wrong in a system that retrieves documents before responding, there are two possible culprits. Either the search returned the wrong fragments and the model had nothing to answer with, or the search got it right and the model ignored it. Traces show the second case well: there's the prompt, the response, and what it cost. About the first they say almost nothing.

I looked at tools specialized in that and ruled them out for the general case —they overlap almost entirely with what I already have— except in precisely this scenario. And this scenario is mine: the portal's search is hybrid, mixing keyword matching with semantic similarity, and I've tuned the balance between the two by hand, looking at results and deciding they seemed better.

"Seemed better" is not a metric. Today I have no automatic way of knowing whether touching those weights, or changing a prompt, improves or worsens the answers. What's needed is not one more tool: it's a set of reference questions with their good answers, an evaluation that runs in the pipeline when someone touches the search, and a threshold below which the change doesn't go in. That's still pending, and it's the next thing I'll build.

The other path: serving your own models

That leaves the other half of the question: stopping paying per token and serving open models on your own hardware. Fair warning that this section is research, not experience: I haven't built it, and the cluster has no GPU. It's the map I made for myself before deciding whether to buy hardware, and the map already says a lot.

First comes the operating system. My nodes run Talos, which is immutable and has no package manager: drivers are baked into the image as extensions. For a modern card you have to choose the open kernel modules, and for anything older than Turing the proprietary ones; and the container toolkit has to be from the same branch as the driver, or they won't talk to each other. Then you have to load the modules explicitly in the machine configuration, a step that's easy to forget.

# Extensiones de la imagen (misma rama en driver y toolkit)
#   siderolabs/nvidia-open-gpu-kernel-modules-production
#   siderolabs/nvidia-container-toolkit-production

# Y los módulos, cargados a mano en el machine config:
machine:
  kernel:
    modules:
      - name: nvidia
      - name: nvidia_uvm
      - name: nvidia_drm
      - name: nvidia_modeset

There are also two frictions that don't show up in the tutorials. Talos doesn't install on a partition: it claims the whole disk and creates its own table, so a machine that already has a hypervisor on it forces you to choose between dedicating a disk to it or wiping one. And if, instead of installing it on bare metal, you want to pass the card through to a virtual machine, on consumer motherboards the GPU usually shares an isolation group with other devices, which forces a patch that buys separation at the cost of stability.

Above that there's another fork. You can declare the inference service in a minimal way, relying on the networking you already have, or install the serverless layer that shuts the pod down when nobody is asking and frees the card. The first is lighter and leaves the GPU permanently reserved; the second frees it, in exchange for one more networking piece to maintain.

And then there's the constraint that really decides whether this is worth it, and it has little to do with Kubernetes. A pod that requests a GPU reserves the whole thing: while one model is loaded, any other waits for the first to shut down. Bringing one up isn't instant, because the weights have to be loaded into the card's memory. And those weights take up space: an eight-billion-parameter model in fp16 fills a sixteen-gigabyte card on its own, leaving no room for the conversation's context; quantized to four bits it drops to a quarter —about four gigabytes— and can breathe. If what you want is to serve many variants at once with a single card, the answer isn't running many models: it's one base model with small adapters that are hot-swapped.

With all of that on the table, the question stops being technical. Hosting your own models makes sense when volume is high and steady, when the data can't leave, or when you want independence from a provider. With irregular traffic, a pay-per-use gateway wins almost always — and that's exactly my situation today.

What I'm taking away

That a reference architecture is not a shopping list. Each of the tools I crossed off is excellent at solving the problem it was made for, and none of them solved one of mine. The useful question in front of any of them isn't whether it's on the list, but which of my problems it removes today — and if I can't name it, I don't need it yet.

It's the same conversation I have when someone asks me to "set up MLOps". Almost always, what they need isn't training or hosting anything: it's knowing what they're spending, being able to switch models without touching the product, and not having keys lying around. That's three pieces and a repository, not a suite. The hard part isn't installing them: it's resisting the urge to install the other six.