Asynchronous. Inference. Primitives.
I had started this essay to introduce what I was building but I had to go back and write two more to surface the intent. I wanted to document the journey covering the pivots in intent and product on my way to building Asynchronous Inference Primitives.
My first attempt at turning my abstract ideas into code was pnpl. Push Now and Pop Later (you see naming is hard even outside of Computer Science). The combination of requiring CPU inference and my hardware being a 2017 MacBook Pro with Intel chip led me to the legendary Georgi Gerganov who single handedly shipped llama.cpp. I will write a separate appreciation essay for the llama.cpp community but back to my side quest of making AI work locally for me and my laptop.
The existing universe was seeded with chat completion APIs and we built on top of them. I was interested in what becomes possible if there is another universe alongside it, seeded with asynchronous inference primitives.
I was building to fill this gap.
But.
Unlike the last time the small model ecosystem was not small anymore. The combination of Hugging Face and the strong llama.cpp community meant small models were doing real work. Multimodal inputs, a ton of options for a variety of tasks and thriving open source had finally set me and my local hardware up for success.
As they say nothing works like work.
But here is my bigger takeaway.
What you put in the center determines what gets built around.
When the model provider, the agent runtime or the session is at the center, the problems you solve and eventually what gets shipped as features all start gravitating to that center.
Coding, a personal pain point, AI itself, enterprise, small models. Whatever is at the center of your universe is what you build for. The tools and software you use start conforming to that center.
There are no gaps to fill, only centers to find and build around.
We can have multiple centers. We build around some. We subscribe to others. Small galaxies of tools form around each. Here is the center I spent close to 3 summers and you all about 3 essays to find.
It was the work I wanted done.
Once the work is at the center, the intelligence doesn't have to be.
I don't have to go somewhere, submit something and wait. Maintain a session. Lose it and recover it. The work generates as I go. I submit it when I want. Intelligence can get to it when it can. I retrieve the results whenever I can.
I don't control which model wins. How small they get. How capable they become. What runs on my laptop next year. I just have to ask what intelligence is generally available to me today.
Bring in a model when I want to. Swap the model. Run the work again with another one. Maybe submit jobs first and bring in a model later to drain them.
The work should not interrupt my life. And it should survive interruptions to the lifetime of the agent or the model doing it. I wanted this durability closest to the model.
I did not want to build another work queue service, add a database, run another server or introduce a new framework to get there. The stack was already on my machine. Files. Directories. Processes. It was always local. I just had to stop looking at the cloud.
The complexity can come from what we build on top. The primitives underneath can stay simple.
All the letting go and eventually recognizing the right center led me to find
nrvna.
A set of Unix-like asynchronous inference primitives to make work and results durable.
You can submit work to a workspace. A daemon can bring in a model and process it. Once the results are ready you can retrieve them at your convenience. All are files. Inputs and outputs. All saved in that workspace.
wrk submits work
nrvnad runs the model
flw retrieves work outputs