Kimi3 just murdered Claude Sonnet

Qwen 3.8 just made OpenAI’s GPT 5 look like a child’s plaything

Deepseek v4 just obliterated Google’s Gemini 3.6 on ChudBench XXL, outperforming the smartest person you know on the hardest thing ever

AI Lab CEOs HATE this one set of weights. You won’t believe where they came from

These models are the new kids on the block, and they’re open weights. This means that, and we’ll get into this in more detail in a bit, the researchers that built them openly publish their secret mathematical sauce on the web for anyone with a big enough computer to use. Contrast this with models like Claude(s) and ChatGPT, where their weights are carefully guarded behind lock and key.

But despite all the progress and hype for open weights models, almost all of us are still using Claude and ChatGPT. This is no accident. The frontier labs have spent an unprecedented amount of capital to do a few key things:

1. Create amazing AI models that write code, build financial models, and draft legal memos better and faster than you.

2. Create great, user friendly applications like Claude Code and ChatGPT desktop, and pair them with highly optimized user acquisition funnels such that you slide down it into their ecosystem as if covered in venture-backed baby oil.

And, and this is the one I take issue with,

3. position themselves as the only viable options for serious work. The Coke and Pepsi; the Marlboro and Camel; the McDonalds and Burger King of AI. Would you bring RC cola to your niece’s quinceanera? Would you light her up a Parliament? Absolutely not. Me neither, my niece is going to be a Marlboro girl.

The truly superior model and harness quality of the big labs was a reality for a brief moment around 2024, which made their claims about being the only serious options generally true unless you were a turbonerd (we are turbonerds, which is why you pay us).

But over the past 2 years, the rate of advances in open source models and harnesses have outpaced the frontier labs, and that old RC cola setup is starting to taste pretty damn good.

In this series, we’ll cover what the open source AI community has been up to, why you might want to leverage their work, and how to get started if you’re serious about churning off the big two.

The components of an AI stack

When you ask ChatGPT what to make for dinner or why your life didn’t go how you wanted it to go, you’re not just using a model. You are using two things:

1. A model

2. A harness

These terms are worth defining in detail. Recently, my colleague at Technically, Paul Iusztin, wrote a great article on Harness engineering that lays out the components of an AI stack beautifully, so I’m going to steal heavily from him here.

A model can only talk. A model answers a prompt with the words it predicts are most likely to come next in an appropriate response based on its weights. Weights is an astoundingly unsexy term for the most important component of an AI model. Weights tell the model how to mathematically predict the best responses to given inputs. Anthropic and OpenAI serve closed weights models, keeping their math behind a wall, but allowing you to talk with the model through their products. Open weights models from companies like Qwen and Moonshot publish their weights on the internet for anyone to host and use.

A harness is everything around the model and how you use it; and increasingly where much of the real value in AI comes from. Harnesses include pretty much everything you need to use a model that’s not the model itself:

• The literal application you use to interact with the model’s API

• What allows the model to run code that it writes, using sandboxes

• What makes it feel like the model “remembers” things about you between sessions, by using agent memory

• What allows it to search the web for answers or draft word documents via tools

• …and more, depending on your setup

The harness is what makes AI useful for real day to day work, as opposed to the raw model which is really just the brains of the operation. This combination of a model and harness is known as an agent.

Claude and ChatGPT are examples of closed source chat harnesses, and Cursor, Codex, and Claude Code are closed source coding harnesses. Specialized harnesses are increasingly popular as well, such as Harvey for legal matters and Figma AI for design.

On the open source side, OpenCode and Pi lead the coding harnesses. LibreChat, AnythingLLM, and Jan are open-source chat harnesses, giving you a self-hosted ChatGPT-style interface that can plug into any model. For general-purpose personal assistance, OpenClaw does the same thing outside a chat window, connecting any model to messaging apps like Telegram and WhatsApp with persistent memory and a skills system to let it run your life. OpenClaw is also a harness! It’s a piece of software that’s primary job is to talk to a model and make its output useful for your task.

For a deeper dive into harnesses, check out Paul’s post here.

Alright, so we’ve got the components down. With these definitions in mind, let’s discuss what’s been going on in the open weights/source world.

What’s going on in the open weights AI world?

Models

In the beginning there was GPT from OpenAI. Anthropic quickly followed with Claude and Google with Gemini, and for a while, that was about it. Then a number of Chinese AI labs came out with good LLMs, and unlike the above players, published their weights on the internet for anyone to use.

The strategic decision to open source is complicated, kind of out of scope for this piece, and not made by one company or government unilaterally. One plausible high level explanation is that Chinese labs realized that to compete for global influence in the AI market (the market largely propping up the global economy), they couldn’t just make another chat app. The world already had a few good ones. Instead, they needed to target a different group of consumers - developers. Developers love open source technology, and choose open source options to build atop of. And so they decided to open source their models and capture influence and business through the hosting and harness layers. But first they had to make them good.

And make them good, they did.

One way of measuring “good” is benchmarks, standardized exams for model performance (that are not without their pitfalls). If a model is good, it gets into Stanford (gets released, marketed, and adopted); else it’s SUNY with the rest of us.

A year ago the best open-weights model scored roughly 22 on the Artificial Analysis Intelligence Index. The best closed model scored 35. That’s a pretty meaningful gap.

Currently though, the top open models are 54-57, while GPT-5.5 and Claude Opus sat at 50 to 61. So the gap is barely there. And this narrowing of the gap is true across several important benchmarks, not just AAII.

Another way of measuring “good” is adoption, which is harder to game than a benchmark. Alibaba’s Qwen model passed a billion cumulative downloads on Hugging Face (a model registry) by March 2026, reaching that milestone faster than any open-source model family in history. It generated 153.6 million downloads in February alone, more than the next eight competitors combined. The ATOM project, which tracks open-source model dynamics, found that from late 2023 to March 2026, 70% of developers making new models from existing models, known as derivative models, were based on Qwen, while Llama’s share (remember them?) fell from about 40% to around 10% over the same stretch.

A third way of measuring “good,” and probably the most important one for our context, is what happens after the download. OpenRouter’s token-routing data shows Chinese open-weight models crossing from a negligible share to a majority share of all tokens processed on the platform between late 2024 and mid-2026. Nerds can run all the tests they want that say their model is good, but the market tells the real story. Real companies are voting with their tokens, sending live requests through these models because the cost and quality trade favors them.

Read more

Vibe coding is amazing. But do you actually understand what you’ve built, so you can maintain and improve it over time?

This 6-part Technically series, Software Engineering for Vibe Coders, teaches you everything you need to know about how the apps you’re vibe coding work under the hood, and (crucially) how they might break as your user base scales.

Thanks to our friends at Railway, who help you peacefully deploy those vibe coded apps, for sponsoring this series. Spin up anything your vibe coded app might need at railway.com/new.

The TL;DR

Every app that you vibe code into existence has two major parts: the frontend and the backend. The AI builds both parts for you, but they’re very, very different:

• The frontend is everything the user sees and interacts with: text, buttons, forms, shapes…all the visual and interactive stuff.

• The backend is the factory behind the scenes that makes the app work: APIs, databases, and business logic.

• Web frontends are only written in HTML, CSS, and JavaScript, while backends can be in practically any programming language that runs on a computer (like Rust).

• The magic is how frontends and backends talk to each other, sending data and logic back and forth so you can do things like log in and actually use the app.

If you want to be a better vibe coder, understanding this distinction is absolutely critical. So let’s dive in.

The restaurant analogy for frontends and backends

Technically readers by now know that I’m a “big food guy.”1 Ergo, the best analogy I’ve thought of for frontends and backends is a restaurant. Say you’re out on a hot date and you stroll into your favorite upscale Manhattan eatery. As you chow down on your Michelin Star Crab Rangoon, you might notice that your experience is interfacing with a few things:

• A nice table with comfortable chairs

• High quality food and drink brought to your table

• Servers catering to your every whim

Fantastic. Behind the scenes though, there’s tons of logistical work that goes into being able to provide you with said experience. In particular, there’s an entire kitchen staff that worries about:

• What should be on the menu (Rangoon)

• How to cook everything consistently and perfectly (deep fry the Rangoon)

• Where to source quality ingredients from (good crabs for the Rangoon)

• How to prepare and store those ingredients

• What to do in situations where ingredients are missing or go bad (buy frozen Rangoon)

As a patron, you don’t need to think or deal with any of this – you interface with the “frontend” of the restaurant, and the “backend” takes care of making sure the food that arrives on your table makes you want to leave a 5 star review (unless you’re Pete Wells, in which case 4 stars will suffice).

It’s not a perfect comparison, but this is pretty much how I think of application frontends and backends: except instead of food, it’s data.

The frontend’s job is to interface with the user. Whenever any sort of data (or now, AI responses) needs to get involved in that user experience, it is time to call upon the backend.

The backend’s job is to take care of all of the data and logistics behind the scenes. It consists of the database, API endpoints for interacting with that database, and increasingly today, calls to AI models (and HarNeSSeS) plus other such highly interesting and obscure tasks.

The backend, of course, can only do what you (or Claude) programs it to do. There’s a discrete menu (no pun intended) of things that the backend can do for the frontend – these are API endpoints – like sending a user’s profile information, creating a new user account, or accepting credit card information.

Getting into the import / export business

The best way to better understand these concepts is to walk through an actual vibe coded example.

Meet Vandelay Industries, importer and exporter of fine potato chips, the finest in all the land (where from specifically, we cannot reveal). Art Vandelay – proprietor – has decided to digitize the family business, which he runs out of a one-room windowless office (Penske material).

Thanks to the advent of vibe coding tools, he prompted his way to a website. But he has no idea what’s happening inside it, so we will need to explain it to him.

This is the website homepage. Everything that you see here – text, dividers, a navigation bar – is some combination of HTML, CSS, and JavaScript…our frontend. In fact, most marketing websites can be created as what we in the biz call “static.” Static means that they are completely frontend with no backend (data) component at all.

But Art has suppliers and vendors who need to access their accounts and move shipments around, so unfortunately a simply static website won’t do. Here, for example, is a page of his app (yes, app now, not website) where he can review applications submitted from new potential suppliers. Hm…where are these applications coming from…could it be…data???

Yes, data. As Art’s suppliers submit their applications, their responses get stored somewhere like a database. And on our app’s Supplier application page, we get our first glimpse of the backend. The data in the table here – an application from a seemingly fraudulent supplier named Kostanza Logistics LLC – is backend data. More specifically, when the page loaded, it sent a request via API to the backend to pull all open applications, which the backend happily sent back.

This same logic holds for any pages in Art’s app that have a data component. One way to think about it is: if any information on the page is liable to change from day to day, it’s probably backend related.

The confusing thing here is that when you prompt something like Claude Code of v0, it just goes ahead and builds the whole app. It doesn’t always (but sometimes does) explain to you which pieces are which. It might suggest you use Supabase (backend), and React (frontend), and perhaps Railway (infrastructure, backend) and maybe Tailwind (frontend). If you want to begin to understand what any of these tools do, it all starts with understanding frontend / backend.

More practically, every time something breaks, ask “is this a frontend or a backend problem?”

For example, Art is frustrated because clicking the “Become a Vandelay” button is supposed to open the form for a supplier application, but instead does absolutely nothing. Seems simple enough, but there are actually several things that could be going wrong here:

• The button is broken under the hood. The backend logic that pulls the latest version of the form isn’t working and needs fixing.

• The button isn’t clickable because of a CSS property that tells the browser to disallow the user from clicking. This is a frontend problem!

If you can do this, you are already ahead of most vibe coders!

Frontend and backend concepts, 201

If you’re curious to go deeper (evidently you have time on your hands) there are a few more things worth knowing about frontends / backends for vibe coding.

What and where the frontend actually is

There are all different types of frontends depending on where they’re built.

Many (most?) apps that people access today are web apps, meaning that you use them in a web browser (Chrome, Safari, etc.). For these apps, you only have 3 languages you can use, because only 3 languages can run in the browser: HTML, CSS, and JavaScript.

On top of these languages, Claude Code will likely use some frameworks to make things easier to generate. The most popular frontend framework for building UIs is React. React helps turn your frontend into reusable components and also handles complicated stuff like data flow and when to update the page.

You’ve probably seen a React component in the wild before. It’s just a reusable chunk of code that under the hood is another chunk of code.

When you want to use a component in your code, you write it like a tag:

And the component itself is just… a function with a name that returns some UI.

In addition to React, your AI model will likely choose to use a web framework. This is a piece of software that’s somewhere between a frontend and a backend, doing little bits of both. One of the most popular web frameworks (for React) is Next.js, which helps with things like defining URLs, pre-loading data so your pages are really fast, and making it easy to add authentication.

Outside of web apps, you have native apps – ones that run on phones as iOS or Android apps, or tablets and desktops, or even special-purpose hardware in factories, doctors offices, and other fun places. Though the languages and frameworks for these frontends are slightly different, the concepts are the same. For example, on mobile there is React Native, often used through Expo.

What and where the backend actually is

Backends, for the most part, do not run in a browser or on your device. They instead run in the cloud, on one or many giant servers specially built for exactly this purpose.

The most important component of most backends is the database. It’s the backbone of the application you’re building because it’s where the data goes. Recall that so much of what you’re seeing on a webpage is actually just data: your username and password (hopefully not), your profile information, your emails in Gmail, your playlists in Spotify…all data. All stored in a database somewhere.

Databases are one of the areas where you tend to see the most competition among vendors leaking into vibe coding. If you’ve ever seen Claude propose using Supabase, or Neon, or Planetscale, or (unlikely) RDS, you are choosing a database provider. There are hundreds, maybe even thousands, all optimized for different types of data and different types of customers. The advent of vibe coding has been good to the database industry; Supabase co-founder Paul Copplestone has tweeted about how a large percentage of their growth is actually coming from agents and vibe coders.

You might think the frontend just talks directly to the database when it needs data, but it does not. Instead, you have APIs that act as a translation layer for a few reasons:

• Protecting your backend from strangers: the data in your database is proprietary, you don’t want just anyone to be able to query it.

• Data cleaning, formatting, and organization: databases store data for many reasons, and your frontend is only one of many consumers of that data.

• Centralizing query logic for re-use: several pages in your app might reuse the same data. Instead of rewriting that query in every single place, you can just use a single API endpoint each time.

Not all APIs exist just to protect your database. You might build APIs to kick off workflows, or even use APIs from 3rd party providers like Stripe.

APIs also have a language they speak, and it is (for the most part) called JSON. When your code asks an API for something, this is the format you’ll usually get the data back in. JSON looks like this:

And btw if you weren’t sure:

The structure is fairly self explanatory, but it’s worth noting that this is not how the data is actually stored. Part of the hard work of API building and design is exactly this: formatting, structuring, and aggregating. You’ll see the word JSON every day for the rest of your life as a builder. Now you know: it’s just data.

If I had written this post a few years ago, I would have stopped here. But there’s a newly popular component of backends, and that’s AI models. If you’re building a product that has AI as part of it, there’s an entire new discipline of AI Engineering™ that encompasses picking the right model, prompting it correctly, doing evals, context management…there’s a lot. All of this conceivably is backend work.

The code that defines your APIs, when they get used, this AI nonsense, and other stuff like that also needs to run on a server somewhere. You may have also seen vibe coding tools suggesting places to do this. If you use something like Replit or Bolt, they provide their own hosting service for a fee (often including database). If you use something like Claude Code or Codex, they might suggest a hosting provider like Vercel, Netlify, or even AWS.

Bringing all of this back to the Vandelay Industries app; when Art prompted Claude, the model went and built out a whole backend too. There’s a database for storing supplier and shipment data, plus a set of APIs the frontend uses to access that database.

Art is smart, so he decided to host all of this on Railway, the all-in-one intelligent cloud provider. It gives him one place to run his database, API server, frontend, scheduled jobs, and anything else Claude might cook up for him in the future.

The messy middle between modern frontends and backends

Everything I just wrote is…mostly true. Unfortunately for those of us who explain things for a living, the lines between frontend and backend are starting to blur, and one particular culprit is Next.js.

In Next.js (and other web frameworks), you write your frontend and backend pretty much in the same place; it takes care of figuring out which is which and what runs where. In fact, your backend often lives in the same folder as your frontend. If you drop a file at app/api/applications/route.ts you’ve created a backend endpoint at /api/applications. Same project, same language! This used to be two separate codebases maintained by two different teams who occasionally yelled at each other.

Another area where the lines blur is server side rendering. When an app or website needs to send you a page that has data in it – in other words, a page that has some backend content – there are two places that can actually create that page.

• You can do it on the server side, pulling the requisite data and generating and formatting the whole page before sending it off to the client.

• Or you can do it on the client side, sending just the data from the backend and then actually melding it with the webpage on the client (in the browser, phone, etc.).

In practice, the difference is in where the backend logic happens: all on the server, or mostly in the browser calling APIs on the server.

Traditionally, server side rendering was the dominant form of building web applications – it usually leads to much faster web pages. But as Javascript has gotten better and web apps have gotten more complicated, client side rendering is getting more popular, to the point where it’s now pretty much the default for most tech startups building their stuff from scratch. React is the poster child of that movement.

So what’s really frontend, and what’s really backend? It’s complicated. We can safely say that HTML, CSS, and JavaScript in your browser is definitely frontend. We can also safely say that databases, APIs, and AI models running in the cloud are definitely backend. Everything else…well, the devil is in the details.

We’re going to start breaking some of what’s happening down in the signature Technically style.

Today, we’ve got 3 news stories to break down:

• Intrigue in AI model licensing

• A new AI lab to watch

• A newly accessible way to ship better code

Enjoy! Feedback welcome on this format.

Business model watch: open weight models as a service

A couple weeks back, the AI lab Moonshot published the weights for Kimi K3. It made waves for being fairly close in performance to frontier models from the big labs, but also because it carried with it a very…unusual new license.

The model has 2.8 T parameters, a 1M token context window, and is pretty smart according to the independent model analysis group Artificial Analysis (although they should start grading on a curve, 60 doesn’t inspire confidence).

Within a few days you could rent servers running it from Modal, Baseten, Fireworks, DigitalOcean, RunPod, Together AI, Databricks and probably others. You could not rent it from Blockbuster, because Blockbuster is, famously, out of business.

All of the providers charged the same rate for it, which is not-coincidentally the rate Moonshot charges through their API: $3 per million input tokens, $15 per million tokens out. This is drastically cheaper than the latest and greatest from Anthropic ($10/in, $50/out) and their ilk, while only being slightly less good.

Prior to this, open weight models were priced as a commodity. The inference provider, rather than the lab training the model, set the price. So they were pretty cheap to run, b/c competition (see Kimi 2.7 Code, Moonshot’s last model generation, above for comparison).

Why is this?

License innovation. Kimi K3 shipped not under the Apache 2.0 license (or any other flavor of open source license), but rather the innovatively named Kimi K3 License.

I’m not a lawyer, but I did watch Law & Order a few times, so let’s try and unpack this:

The Kimi K3 license stipulates that if you:

• Run a “Model as a Service” business (like those inference providers above)

• Clear $20mm in revenue over 12 months (unclear whether that’s total, or just with a Kimi K3 service, lawyers help me out)

• Are not a “certified partner”

Then you need to negotiate a (presumably commercial) license with Moonshot.

This is basically the freemium model, but for the relationship between frontier AI labs + inference providers. To be clear, you as a user (or developer of AI products) must pay either way (to buy your own chips, or more likely rent them from a cloud provider).

But why bother with all this complexity? If you’re Moonshot, why not just be like OpenAI and Anthropic and rent out your model directly to users / developers as an API?

Because the inference providers are much better than a lab like Moonshot at, well, inference.

When Moonshot released Kimi K3, the overwhelming demand for that piping-hot 60 intelligence score left them unable to keep it online:

So, the splendor of comparative advantage unfolded.

Moonshot worked with the inference providers to package up their model as a service, to deliver it reliably. And that’s actually great in other ways too.

The inference providers are competing on speed (in terms of tokens/second, tps), which they achieve by optimizing every tiny little piece of the system (eeking out literally tiny edges in performance, it’s impressive engineering work). This is what they do well, about 4x faster than Moonshot in Modal’s case (with Fireworks and others not far behind).

Company to watch: Liquid AI

Kimi K3 (as mentioned above) needs about 1.4TB of memory and some of NVIDIA’s newest GPU chips to run.

Liquid AI is an AI lab going the other way, training models that run on your device (on the mythical edge) on a plain old CPU. I try to not run on (or near) edges, but we can talk about that another time.

Last week they released LFM2.5-2.6B. It’s a paltry 2.6B parameters (Anthropic’s Fable is rumored to be in the trillions), and ergo fits on 2.5 GB of disk, which my laptop and phone hypothetically should have if I cleared videos properly out of my iMessage.

It’s also fast - Liquid AI reports 220 tokens/second (tps) on an M5 Max. Like actually fast - Fable is 40 tps according to OpenRouter, although Opus 5 Fast’s 115 tps is a better comp.

Why?

Like we’ve covered before, GPUs are very good at doing thousands of small calculations simultaneously. That advantage is large when you’re training a model.

But during inference, if a model is small enough, you can eke out sufficient (maybe better) performance from a CPU as well.

So a laptop (or a phone, or toaster) with enough RAM (again, not mine, but hypothetically someone’s) can run a model like LFM2.5.

I can’t speak to how great of a model LFM2.5’s is, and tbh it doesn’t really matter. You are almost certainly not going to use it, I am not going to use it, but, BUT! You will very likely use a similar model, running on your device, quite soon, and it’ll unlock new AI use cases + business models.

One interesting sidenote: LFM2.5’s license allows you to run the model as part of your commercial product, so long as your company makes less than $10mm.

>$10mm in revenue, you need to negotiate a license with them. Seems pretty fair, and we need a way for smaller AI labs to be commercially successful (or else we’ll have less innovation).

New tool alert: Stacked pull requests on GitHub

Last week GitHub released stacked pull requests, and if you use a coding agent it’s worth checking out.

To understand why, let’s go back to the early days of coding agents - when Cursor was just a few months old, and the brief but monumental “Cursor for X” startup wave hadn’t yet washed ashore.

It was September 2023, and a startup called Graphite (later acquired by Cursor later acquired by SpaceXAI) launched stacked git branches and pull requests for their GitHub integration to little fanfare.

They were early to a problem that coding agents created:

Coding agents can write a lot of code quickly.

You + your team can not review that code as quickly.

A coding agent’s code is like those crab legs you ambitiously scooped up at the Wynn buffet in Vegas. Fun at the start, tedious AF at the finish.

There are 3 solutions to this:

• Don’t review the code and just ship it to production anyway (YOLO mode).

• Have AI review the code. Lots of startups (including later Graphite) and now every coding agent do this.

• Split the code into smaller chunks that are easier to review.

#3 is a useful approach in any area of work, but the problem comes when those small chunks of work depend on each other.

You have to then keep track of the order in which those changes need to be shipped, and that becomes difficult on a team.

Stacked pull requests codify the order in which changes need to be deployed.

So you (more likely your coding agent) can lock in small wins, then when you’re ready to actually deploy your big batch of work, GitHub (or any git provider that supports stacked PRs) will merge those dependencies in the right order.

If we’re going to enlist coding agents to work independently for long periods of time, these are the types of DevOps adaptations we need to keep them on track.

–

This is all just our take though. Wdyt?

And thanks to our faithful Technically correspondents, especially Sam Schneider, for their notes.

-David, Justin and the Technically team

They use your computer to do things like order a smashburger, or book a flight to Cancun (YOLO). You might even have one running, right now, on a new Mac Mini.

The term ‘computer use agent’ gets thrown around for three different things. The first two are components of the real thing:

1. A vision-language model (VLM) looks at your screen and decides what to do (clicks + keystrokes).

2. A browser agent operates websites.

3. A computer use agent controls the entire operating system.

The OSWorld benchmark confirms that computer use is the hard part, as anyone who’s tried to get a computer use agent to open a file in Excel can attest.

Let’s get into why that is, and where we’ve made progress.

What does “computer use agent” mean?

A computer use agent is an LLM-powered agent that operates a full computer.

It does so through the same interfaces a human uses: the filesystem, terminal, browser, and applications like Minesweeper when it’s bored.

Computer use agents click buttons, fill in forms, create files, and work across multiple Git repositories at once.

Most importantly, computer use agents retain a memory of historical sessions + context, and all the software installed to pick up where they left off.

The interface is the same, but the way agents work on the computer diverges significantly from humans. We think we can multi-task, but computer use agents actually can, by spinning up multiple sub-agents to work on smaller tasks in parallel.

Most people would say that computer use agents were introduced by Anthropic in October 2024 as really a model capability vs a full agent, and that it didn’t work very well at the outset.

Just a couple short years later, there are now over a dozen options:

OpenAI gave Codex the ability to operate your Mac or Windows desktop, Claude Cowork is one of the most praised products from Anthropic, Google folded Project Mariner into Gemini Agent, Factory offers persistent Droid Computers, Perplexity launched its Computer, and open-source agents like OpenClaw and Hermes have gone viral.

How an AI agent uses a computer

Agents “seeing” the screen and moving your cursor carry a certain sense of magic, but it’s not all the same magic. There are a few technical approaches:

1. Screenshot-based vision + click coordinates

This was the early approach of Anthropic’s Computer Use (2024) and OpenAI’s CUA powering Operator (2025).

The agent takes a screenshot, feeds it to a vision-language model (VLM), which then decides to “click at pixel (432, 187)” or “type ‘hello’”. It executes the action, takes a new screenshot, and repeats the loop. The agent controls the mouse, keyboard, and applications.

This approach is expensive, and the screenshot → model inference → action workflow has high latency (although companies like Moondream are working on that). It’s also fragile, and errors compound when interacting with UI elements, low contrast, or dense layouts.

Moving or scaling UI elements, or changing from light to dark mode, can cause the agent to fall over.

2. Accessibility tree / DOM parsing

The challenges above are why we rely on structured approaches too. Every operating system and every browser exposes a structured tree of UI elements (buttons, text fields, labels) with their positions, roles, and states.

An agent can read this tree as text, which is orders of magnitude cheaper and faster than reading it as images.

What does this actually mean? When you visit a website, your browser builds two internal data structures:

First, the DOM (Document Object Model), a tree of every HTML element on the page.

Second, on top of the DOM, the browser builds an accessibility tree, a simplified version that strips away decorative elements and keeps only what you can interact with: buttons, links, form fields, headings. Each element gets a role (”button”, “link”), a name (”Submit”, “Search”), and a state (”disabled”, “checked”).

One common accessibility-tree tool is Microsoft’s Playwright MCP, which is also useful for coding agents to review their work on an app’s UI.

But even structured input has real limits. In the original WebVoyager benchmark, a text-only agent reading just the accessibility tree scored 40.1% versus 59.1% for the same agent with vision.

Dynamic, JavaScript-heavy sites like Booking.com and Google Flights were among the hardest categories, because the DOM gets messy.

3. Hybrid (the 2026 consensus)

Hybrid can mean two different things:

1. It means defaulting to the cheap, structured signal (DOM or the accessibility tree), and falling back to screenshots (VLM) only when the structure runs out.

2. Routing each task to whichever method suits the computer use problem at hand.

At Factory (where I work) our computer-use agents do the second. They route across four backends depending on the task:

• Accessibility-tree snapshots drive native desktop apps, letting the agent operate Finder or Slack by element rather than by pixel.

• The Chrome DevTools Protocol drives web and Electron apps,reading the page’s structure directly and screenshotting only when it has to.

• A virtual pseudoterminal (PTY) reads terminals as pure text, consuming the character stream and typing keystrokes back. This is how the agent runs git or installs a package.

• A terminal emulator captures actual pixels, for what the text stream can’t express: for apps with a full-screen terminal UI (TUI), that might contain elements like progress bars + colored outputs.

Only one of the four (the terminal emulator) works primarily from pixels.

The rest read text or structure, because structured signal (if it exists) is cheaper and steadier.

Computer use agents stack

Every product that gets called a “computer use agent” is some combination of three layers:

• The model (the LLM or vision-language model) that decides what to do next

• The harness (the agent software wrapped around the model). This one contains loops, tools, or sub-agent orchestration. IMO harnesses and models are converging a little.

• The computer, the environment the agent actually works in. Before, some people used containers, but the consensus nowadays is using microVMs.

Most arguments about what “counts” as computer use come from mixing up these layers.

Anthropic’s original computer use was a model capability. Some agents are harnesses you install on your own computer. Others sell the harness and the computer together, with automatic routing between models.

What a computer use agent needs

Computer environment (local or cloud)

The agent needs its own work environment. It can have its own literal computer (your Mac Mini), or a secure cloud environment like an E2B sandbox.

Persistence and long-running sessions

True computer use means that the agent has a persistent relationship with a machine - it doesn’t break up, fly to Cancun, and then get back together later.

An agent working across a real engineering workflow (opening a pull request, waiting on CI, responding to a code review, pushing a follow-up commit) needs to resume from where it stopped.

Memory and state

An LLM remembers nothing between sessions, so that state has to live somewhere outside the prompt.

You might hear “state” mentioned a lot online, and it might mean a few things:

• Files. Memory and instructions stored as Markdown files within folders. In OpenClaw, HEARTBEAT.md is the task list, or Hermes saves finished workflows as reusable SKILL.md files.

• The environment. Git repos, package dependencies, and test results just stay put, so the agent resumes where it left off.

• External memory stores. Vector databases or services like Mem0, Letta, or Zep hold what the agent learned and can retrieve it later.

Parallelism

An agent can spin up sub-agents to run tasks in parallel.

For example, in Factory, worker agents work in sequence, but each of them can spawn specialized sub-agents during the workflow.

Action space

A human reaches for the mouse, the keyboard, the terminal, or a browser tab depending on what the task needs.

An agent needs the same flexibility expressed as a defined set of primitives: click, type, scroll, run a shell command, read or write a file, call an API.

The action space is a design decision with real tradeoffs: too narrow and the agent can’t complete tasks that fall outside it, too broad and it becomes harder to audit what the agent actually did and why.

Most production systems restrict irreversible actions (file deletion, form submission, outbound messages) behind explicit confirmation steps.

Tool integration

Whenever an external service has an API, the agent should call it rather than navigate a UI to get the same result. This is where MCP (Model Context Protocol) becomes important.

It’s a standardized protocol that lets agents discover and call tools across calendars, codebases, databases, and communication apps without custom wiring for every combination.

As more services expose APIs and more MCP servers get built, the portion of work that requires the agent to actually “see” a screen keeps shrinking.

Verification and error recovery

A human notices when something went wrong: a form didn’t submit, a build failed, a page didn’t load. An agent needs to check this explicitly at each step, and validate that the output actually works, not just looks good in the code.

Why are computer use agents going mainstream now?

Not long ago, AI models couldn’t reason well enough to drive a computer use agent.

Sandbox environments made for agents were just being built. There was no standard way for agents to connect to tools. And context windows were so small your agent forgot what it was doing halfway through a three-step task.

All of that changed, more or less simultaneously. On OSWorld, agents went from roughly 12% at the benchmark’s launch in 2024 to the low-to-mid 80s by mid-2026. The estimated human baseline is ~72%, which seems high if the task is to use Microsoft Teams.

Comparison of computer use products

All computer use agent products look similar on the surface, but work very differently under the hood. Here’s a quick summary of the field, current as of publication but moving quickly:

Manus

Manus originally ran each task in a fresh cloud sandbox, but it now also offers Cloud Computer (launched April 30, 2026), a persistent Ubuntu VM for 24/7 bots, databases, and scheduled automations. The standard Sandbox still recycles after 7 days (free) or 21 days (paid).

OpenClaw

OpenClaw is the project that made “an agent living on your machine” a mainstream idea. It’s an agent that lives on a machine you provide (a VPS, a Mac Mini, your laptop) rather than a full OS-automation engine, though it can reach desktop tools and services via MCP servers.

Hermes Agent

Architecturally, Hermes is in the same family as OpenClaw: a self-hosted, persistent agent that lives on your server or PC, reachable over Telegram, Discord, Slack, or WhatsApp. Its defining feature is the self-improvement loop: persistent cross-session memory plus a skills system where the agent writes down how it solved a task and reuses that skill next time.

Factory Droid Computers

(Disclosure: I work at Factory.)

Factory runs persistent cloud machines that pause when idle and resume with full filesystem and memory snapshots. You can Bring Your Own Machine or use virtual computers.

Factory’s observation layer is hybrid. Its Droid Control tooling picks the right backend for the target: Playwright-backed automation with Chrome DevTools Protocol support for web apps and Electron apps, accessibility-tree snapshots with ref-based element selection for native desktop apps, virtual-terminal automation for driving real TUI apps, and video recording of what the agent did.

OpenAI Codex computer use

In 2026, Codex got its own virtual cursor and can see, click, and type across all the apps on your Mac. Windows support followed in May 2026. It’s a vision-driven approach — screenshots plus simulated input — with one important twist: if a plugin or integration exists for an app, Codex prefers the structured path and falls back to pixels only when it has to.

Claude Cowork

Claude Cowork now runs remote persistent sessions by default (beta, rolling out in 2026). The agent loop and code execution run on Anthropic’s servers, and sessions persist across desktop, web, and mobile. For local desktop control, Claude Cowork can also use Computer Use via the Claude Desktop app.

Perplexity Computer

Perplexity Computer is fundamentally different from all of the above: in its cloud form it does not control your screen at all, running entirely in Perplexity’s cloud, in isolated VMs, orchestrating 20+ AI models as sub-agents. Think of it as a task orchestration layer.

In April 2026 Perplexity shipped Personal Computer, a Mac app that runs Perplexity Computer locally on a Mac or Mac mini. Personal Computer can control local files, native Mac apps, the Comet browser, and Perplexity’s cloud sandbox. It is available to all Mac users as of May 2026.

Devin

Each Devin session runs in its own isolated Linux VM with a full desktop environment, browser, terminal, and filesystem. Since 2.2, Devin can launch and test desktop applications, take screenshots to visually verify its work, and send back screen recordings for human review. In 2026 Devin got its own Windows VMs and a desktop command center for managing fleets of agents.

Zo Computer

Zo has the most literal implementation of this article’s definition — the agent and you share one persistent machine with a real filesystem — aimed at people who want personal software running on infrastructure they control rather than an enterprise SDLC. Every Zo user gets a persistent Linux server with real file storage, hosting, scheduled automations, and an always-on agent you can text via iMessage, SMS, Telegram, or email.

Infrastructure, not agents

Worth separating out: providers like E2B (Firecracker microVMs), OpenComputer (persistent full VMs with hibernate/wake and checkpoints), and Orgo (headless cloud VMs for agents) provide the machines that agents run on, but not the agent itself.

What is still a challenge for computer use agents

The PC was redefined once before, from terminal to GUI. It’s being redefined again, from a computer you use to a computer that uses itself (sometimes with a human).

I still see the three approaches to computer use (screenshot-based, accessibility tree, and hybrid) as an interim solution stitched onto human software, not as an agent-first solution.

On the agent side, there are other specific challenges.

CAPTCHAs were literally designed to block non-human users, but now we might need to distinguish it from CAPTCHAs for agents.

Authentication is still a mess: agents need scoped tokens with limited permissions, not your username and password.

More challenges like this could be summarized as “the web wasn’t built for agents, and it shows.”

On the human side, we need to decide what role agents play in organizations. Are they autonomous AI employees, or just pieces of software? Should they be given high autonomy, or more frequent humans in the loop? How specialized vs general should they be?

Either way, we are heading to an exciting future, where agents and humans collaborate together.

In part 1 I explained why there are so many different programming languages: because languages come with tradeoffs, and everyone wants to choose their tradeoffs differently. We discussed how memory and how it’s managed is one of if not the most important of those tradeoffs, and how when it comes to memory management you pretty much need to choose between your program being fast, or your program being safe.

The speed vs. safety thing was always around, but it was ultimately manageable. Slow safe programs were fast enough for most users, and fast unsafe languages could broadly keep up with the speed of attackers trying to exploit any memory-related vulnerabilities that snuck into production.

But the game hath changed. Today, increasingly lag-intolerant consumers rely on massive multiplayer games with millions of concurrent players, trading platforms where a millisecond of latency costs real money, and critical infrastructure that nation-states are actively and effectively hacking. Which begged the question (again): can we have a programming language that is both fast to write and safe from a memory perspective?

Rust is a programming language that says maybe you don’t have to choose.

It was created by a Mozilla engineer named Graydon Hoare, started as a personal project in 2006, got picked up by Mozilla as an official project in 2009, and hit its stable 1.0 release in 2015. It was built from scratch to be as fast as C and C++ while making an entire category of bugs impossible to write due to its unique approach to the memory problem.

To understand how Rust pulls this thing off, we need to go a little deeper on memory (again). What does it look like to write in a programming languages that do or don’t require you to manually work with memory?

First, the simpler one. In higher level languages like Python there is something called the garbage collector – a glamorous position if there ever was one. The garbage collector is a sort of “engine” that runs periodically in Python programs to help handle your memory for you. It scans your memory story and cleans it up. And much like a garbage truck if you’ve ever been stuck behind one while 10 minutes late for something important, it’s slow as hell because garbage collection is a computationally intensive task. Again, tradeoffs.

In a low level language like C, you must manually interact with the memory system, allocating data for each piece in your program. Before you can store a user’s birthday to display it to them in an app, you need to allocate a slot for it, and when you are done with the birthday, you need to release the memory. By allocating a slot, I mean literally telling the computer: hey, I need you to store this variable (an allocation), and it will require 8 bytes and data, and you should do it here in particular. And then Mr. Computer, when we are done with this variable, you should delete that data (a release). Like I said, computers both dumb and smart.

This manual process skirts around the garbage truck – which means it’s much faster while running – but it’s incredibly time consuming and error prone to write. If you miss an allocation or release anywhere in your code, it’s likely your program will crash or be vulnerable to hackers.

Like I said, for what seems like forever these were your two choices. Easy breezy but very slow garbage collection, or rigorous and fast but highly labor intensive (and vulnerable) manual memory allocation. Rust does a third thing.

What makes Rust, Rust

The entire point of Rust is to enable developers to write manually allocated code that’s safe. It’s all about avoiding the vulnerabilities and bugs that tend to happen when you have to manually allocate memory. Both ways, western man.

So Rust’s primary thing, the thing it will not stop talking about at parties, is that it approaches memory management differently. It aims to be fast, so runtime memory management (the garbage truck) like in Python is a no-go. But those damn lower level languages are so unsafe. I saw three of them go into the bathroom together!

Rust’s Raison d’être is the ownership system, which approaches our heralded tradeoff with a twist:

• You in fact do have to manage memory on your own. No garbage collection.

• BUT, instead of manual memory allocation being the Wild West, Rust gives you a helpful partner of sorts to do that with you.

Specifically, Rust elevates the role of the compiler from a simple translation machine to the world’s strongest and smartest bouncer. As you may recall, early computer scientists invented the compiler as the translation layer between human writable higher level languages and the 1s and 0s that the computer can actually run. The Rust compiler, however, does more than just translate.

Where older manually allocated memory languages like C allow whatever bugs you write to end up in the compiled program ready to leak data or get exploited, the Rust compiler has an extremely strict set of rules that code must adhere to before being compiled. You physically cannot ship the bug because the compiler won’t let you build a program that isn’t responsible in regards to memory.

“I don’t see how you can hate from outside of the club. You can’t even get in.”

- Chris Brown on his 2011 hit Look at me Now, and, the Rust compiler to improperly used memory

So you get the speed of C, because there’s no garbage collector running in the background, but you don’t get the myriad memory bugs that have plagued C and C++ codebases for decades. The amazing Rust compiler catches crashes from accessing memory you already freed, security holes from forgetting to clean something up, all of it, before a single user ever touches the code.

This fabulous compiler is not a silver bullet. The issue is that Rust, with all its rules and requirements, is hard to write. Despite this however, developers love Rust. It’s been the most loved programming language in Stack Overflow’s annual developer survey every year since 2016, and every year it seems like more and more of the world’s software is written, or rewritten in Rust.

The receipts: Rust in action

The fast, safe, and beloved language is finally getting its flowers in the form of massive grassroots community growth, adoption by hardcore and influential dev teams, and endorsement by standards organizations and governments.

For years, “Rewrite It in Rust” was a joke. Any time someone on the internet mentioned a performance problem, a security bug, or really any problem at all, a Rust evangelist would show up in the comments and suggest, with complete sincerity, that the solution was to rewrite it in Rust. It became a meme, then a term of light mockery for a certain type of engineer who had discovered a new favorite language and would not shut up about it. The Rust community even has a mascot, Ferris the Crab, and a habit of spamming the 🦀 emoji into threads where it is not needed or wanted.

But as the years scrolled by, the Rust-hater became the Rust-curious after Mozilla rewrote a major service in Rust and wrote a blog about it in 2017. The Rust-curious turned to Rust-fan after Dropbox called it a “force multiplier” for their core file syncing service, and little by little Rust has started eating the world, turning skeptics into believers, one codebase at a time.

• Discord rewrote the service that tracks who’s online and in which servers, switching from Go to Rust, and saw latency improvements so dramatic the periodic spikes that plagued the Go version disappeared entirely.

• Cloudflare rewrote a core piece of their network infrastructure and got 25% better performance at half the compute cost. Cloudflare sits in front of about 20% of all web traffic.

• Microsoft has been rewriting parts of Windows in Rust for years. The reason is that roughly 70% of the security vulnerabilities Microsoft has patched since 2006 trace back to memory safety issues.

• The Linux Kernel added Rust as a supported language in 2022 and researchers estimate it could eliminate around two-thirds of the security vulnerabilities in the components written in it.

• Hugging Face rewrote their tokenizer library in Rust, and it now processes a full gigabyte of text in under 20 seconds on a standard CPU, dramatically faster than the Python-based tokenizers it replaced.

But don’t take my word for it, take Uncle Sam’s. The federal government published a report in 2024 recommending that the software industry move away from C and C++ toward memory-safe languages, naming Rust specifically. The NSA and CISA have issued similar guidance. When the feds start weighing in on what programming language you should use, the meme phase is probably over.

The trolls who mocked rust and spammed the meme were right that you probably shouldn’t undertake a full rust rewrite to change the color of a button, but wrong about most of the rest. Rust didn’t come with a flashy million-dollar marketing campaign; it came with pudding… in which there was proof that if your users want snappy, secure software (and they do) Rust can help make that happen.

Lets try this again

I think now we’re ready for that definition of Rust.

Rust is a statically-typed, compiled programming language that enforces memory safety through compile-time ownership semantics without requiring a garbage collector.

If you ask me, Rust adoption will continue to rise for its killer performance, tight memory safety, and growing presence in systems-level infrastructure, but also due to the rise of LLM-generated code. The fact that Rust is hard to write was a significant barrier to entry when actual humans were coding. But LLMs don’t feel that friction the same way; they can bang out thousands of lines of Rust no problem, which effectively lowers the cost of choosing it. And there seems to be a calcifying effect around language choices, with LLMs knowing the most about the languages that already have the most code written in them. Rust has a large, high-quality open source footprint, which means models are well-trained on it.

There will likely be downstream consequences when it comes to maintaining the inevitable deluge of rust-slop given how steep the human rust-learning curve is, but in any case, we see no signs of slowing Rust’s massive growth in popularity.

An AI model on its own can only talk.

A harness is the software wrapped around it that lets it actually do things: touch your files, run your tools, and remember what happened.

• An AI model by itself only answers a prompt. A harness is what lets it act: use tools, remember, and take steps until a job is done.

• Picture harnessing a horse: the horse is the raw power, the harness is what lets you put it to work without it bolting.

• The AI coding agents you’ve heard of (Claude Code, Codex, Cursor) are all harnesses.

• The frontier model is becoming a commodity (everyone rents the same few). Your harness configuration is what you want to own.

The next decade of software won’t be “an app with AI added on.”

It’ll be harnesses, and the build-vs-buy call is yours to make. So read this!

For the past 6 months I’ve done a bet with myself: do everything through agentic tools, change a line of code or text only when I absolutely have to.

At the beginning this made me incredibly slow, but as I progressed and turned into a power user, I started to see the value in harnessing AI models myself. Currently I am interacting with my computer 90% via Claude Code and I can see how harnesses have the potential to revolutionize how we think about software and computers in general.

Now let’s see why a raw model can’t do the job on its own, what a harness actually is, the everyday parts every harness is built from, and what all of it means for your company.

Why a Raw Model Isn’t Enough

On its own, an AI model does exactly one thing: you give it one input, it hands you one output, and then it stops. No memory of the last step, no ability to take the next.

That’s not the magic you feel in Claude Code or Codex. Those tools take dozens of steps on their own: reading your files, running things, catching their own mistakes, pulling what they need from memory or the internet, looping until the job is actually done. A raw model can’t do any of that by itself.

That repeat-until-done cycle (read what’s there → decide the next step → do it → check the result → go again) has a name. It’s the agent loop, and it’s the engine that turns a one-shot answer into finished work.

And the best ones don’t just act. They check their own work. Boris Cherny, who created Claude Code, says the most important thing for getting great results is giving the AI a way to verify its own work: a feedback loop that, in his words, can 2–3× the quality of the final result. That self-correction is a harness feature, not a model one.

This takes us from “answers a question” to “does the job”:

If the model is the easy part and what’s built around it is the hard part, the obvious question is: what is a harness?

So What’s a Harness?

Picture a wild horse. It’s raw power it’s nearly useless until you harness it: reins to steer it, blinders to keep it focused on the road, a saddle so it can carry weight.

A model is the horse. The harness is the software wrapped around it. It adds the hands (tools), the memory, the reins (guardrails), and the loop that lets it take many steps. That combination (not the horse alone) is what does real work.

An agent = a model + a harness. The agent is the working thing you actually use: a model with a harness wrapped around it. The model is the raw intelligence.

The harness is what puts it to work. LangChain (a popular toolkit for building these systems) frames it bluntly: the harness is everything that isn’t the model itself, and that “everything else” is where almost all of the engineering, and the magic, actually lives.

There’s a name for the work of building one, too. Mitchell Hashimoto (a veteran engineer, co-founder of HashiCorp) calls it harness engineering: “anytime you find an agent makes a mistake, you take the time to engineer a solution such that the agent never makes that mistake again.”

So a harness is what that work leaves behind. Lessons about how the AI fails gets built into the wrapper so it stops failing that way (and instead finds new ways to fail :).

What’s Inside a Harness

At the center of the harness, we have the loop, the engine that keeps calling the model until the job is done. Plus a kit of 7 parts that allows us to steer the model’s behavior:

Tools. Without them, the AI model can only provide “talk”. In other words, it can only output text, images or audio. With them, it can interact with the outer world, such as reading files from your computer, executing code, searching the web or editing a document.

Memory. The AI forgets everything between sessions, so the harness gives it files to write to and read back: its long-term memory and to-do list.

The files are what let it remember anything at all and pick up where the last session left off. The simplest memory implementations are based on files, but depending on the complexity of your use case, you may need a more sophisticated solution based on vector or graph databases.

Context. The context window is what the LLM sees at a time, in each iteration of its loop. The problem is that it’s limited to 128K, 256K, up to 1M tokens. As the context window grows, the model starts to get confused and produce incorrect results, a problem known as “context rot.” The role of the harness is to keep the context window sanitized by adding/removing components at each iteration. Ideally, you want to keep the context with exactly the right information that gets the job done.

A sandbox. It’s a safe room where the AI can make a mess (run things, break things) without the risk of touching anything real in your business, such as deleting files, accessing sensitive data or even running malicious code on the computers from your private network.

The goal is to give the agent exactly as much access as it needs to get the job done. Otherwise, during its agentic loop, it can always decide to go rogue just to achieve its goals. It’s very common to start looking around for API keys to access data from your database or other services.

Guardrails. A big part of guardrails is the permission layer, which sets the rules for what the AI may do on its own: what runs freely, what needs your okay first, and what’s automatically denied. These need to be carefully balanced to avoid asking the human to accept each command (which adds a ton of friction, forcing the user to babysit the agent) while still asking for user input at essential points, such as deleting a file or changing one that’s not versioned.

Orchestration. For complex jobs, 1 AI acts as a project lead, delegates to a team of specialist AIs, each handling a piece and then pulls the results together. A caution from experience: don’t reach for the multi-agent systems too early. I once split a job across 5 or 6 specialist AIs, and a single, well-set-up one beat the whole team.

Interfaces. The same harness is reachable from your phone, laptop, browser, Slack, or WhatsApp.

The AI Tools You Already Know Are All Harnesses

Every popular AI tool you are using is a harness. Claude Code, Cursor, Codex: the same kind of model inside, a different harness wrapped around it.

Here’s what usually happens: when a product feels better than a plain chatbot (it remembers your project, fixes its own mistakes, actually finishes the task), that feeling is the harness doing its job. Two products can call the exact same model and feel worlds apart, purely because their harnesses are worlds apart.

And this isn’t just a vibe. In one public test done by LangChain, they changed only the harness around a coding agent (same model throughout) and it climbed from roughly 30th place into the top 5 on the Terminal Benchmark used to evaluate agents.

Before we get to what this means for your company, let’s clear up 3 words that get thrown around as if they’re the same thing.

So harness engineering is the last step: the whole AI application around the model. Which raises the real question: should your team buy or build its own harness?

What This Means for Teams

Claude Code is a harness someone else built for general coding. It’s superb at that. But it will never know your finance rules, your data, your guardrails or your customers. The moment your AI needs to do your job, a generic harness hits a wall.

Still, a harness you build becomes its own product, with its own bugs, issues and maintenance overhead. Owning the layer can give you an advantage, but it’s an ongoing investment, not a one-and-done build. The real question isn’t whether you can (Claude Code can build you a harness), it’s whether you should.

That’s why most people I know, including myself, are customizing harnesses (via skills and MCP servers) intended for coding, such as Claude Code, for their own use cases, or coupling them with Notion or Obsidian to transform them into their personal assistants.

But using coding harnesses for everything has one major drawback: they are optimized for coding, and the people building them will keep optimizing them for coding. This might lead them to behave more poorly with each version update.

Another huge drawback is that it’s almost impossible to build your own product on top of them. They are amazing as internal tools, but once you want to stitch a UI on top of them or distribute them to your users, you hit another wall.

That’s why we started seeing harnesses specialized for different domains, such as:

• Creative: Descript (video), Canva Magic Studio & Adobe Firefly (visuals), Figma AI (design)

• Legal: Harvey (drafting contracts), CoCounsel (reviewing contracts)

• Medical & Healthcare: Abridge & Nabla (generating clinical notes from doctor-patient conversations)

• Financial: I personally worked at ZTRON as a harness for financial advisors

• Productivity: Notion AI

But there is a middle ground between building and buying. As harnesses are being standardized, major frameworks such as Pydantic AI’s Harness, Pi or LangChain’s Deep Agents have recently entered the game.

They’re good open source starting points that provide the agentic loop, the 7 key parts and the option to easily add your own UI/UX and distribution.

You don’t need to build a harness tomorrow, but it’s important to know that you can build a custom harness when you’re ready, and it’s a piece of your stack that’s worth owning.

Harness in Conversation

“We’re still building out the harness around the model.”

We have the AI. Now we’re building the software that lets it actually do the job.

“That’s a model problem, not a harness problem.”

The AI itself is getting it wrong, versus we just haven’t given it the right tools or memory yet. (Usually it’s the second one.)

“We don’t want to be locked into someone else’s harness.”

We want to own the layer that makes our AI ours, instead of renting it from a vendor.

“We should switch to an open-source harness.”

Move off a closed tool like Claude Code to an open-source alternative you can see inside and bend to your needs, such as OpenCode or Pi.

Further Reading

• If this isn’t enough harness engineering talk for you, I’m publishing a series on harness eng soon on my Substack, DecodingAI.

• Letting an AI check its own work is the single biggest lever for getting good results out of it, straight from the engineer who built Claude Code.

• If you want the full parts-list of a harness, plus the test where swapping only the wrapper took a coding agent from 30th place into the top 5, it’s all in the anatomy of an agent harness.

• More on the “notebook” that lets an agent remember across sessions, in Anthropic’s guide to effective harnesses for long-running agents.

• Learn how OpenAI’s Codex actually works.

• More on keeping the AI’s context window clean, and avoiding “context rot,” in Anthropic’s guide to effective context engineering.

Turbopuffer is a search engine built on top of object storage (like AWS S3).

• AI applications need to quickly search through lots of unstructured data (text + embeddings). Cursor searches through your codebase, Notion AI searches through your documents.

• In the early days of AI applications, most vector search options (like vector databases) stored data on RAM to ensure fast query times. This turned out to be quite expensive.

• Turbopuffer offers a search engine on data stored in object storage (like AWS S3), and only pulls data into RAM when it’s needed. They claim this makes it 10x cheaper while still being fast.

• Competitors like Pinecone and Elastic also followed suit with “serverless” vector search tooling.

Search is one of the most competitive spaces in AI infrastructure. Turbopuffer reportedly crossed $100M in ARR recently, so their early bet on object storage is paying off.

Their story is one of an obsessive focus on knowing what can be done with the tools cloud providers like AWS + GCP have on offer. It’s a tale as old as 2006.

Separation of storage and compute

In 2006, AWS launched a simple product without fanfare. It let developers store + retrieve unlimited numbers of files, and it promised not to lose track of them:

All software does is basically manipulate files, and S3 (Simple Storage Service) was a bottomless filing cabinet.

Renting it was much cheaper than buying your own filing cabinet of merely moderate size.

AWS launched S3’s soulmate, EC2 (Elastic Compute Cloud) in a similarly low-energy way in August of 2006. It let you rent as many servers as you needed, whenever you needed them.

This pairing (the foundation of the Cloud) made developers rather thirsty. By separating compute from storage as separate cloud services, you could reimagine stodgy, ubiquitous infrastructure like databases.

The creators of Snowflake were two of those developers, and the analytics warehouse was a ripe first target. As Snowflake co-founder Benoit Dageville put it:

I always said that as older software guys we were at the mercy of hardware and its resources. The cloud was a game changer for analytic workloads that has lots of peaks and troughs in usage. It really needed that elasticity the cloud offered.

Him and his co-founder Thierry Cruanes left Oracle and started Snowflake in 2012, which today has an ~$80B market cap. Larry Ellison had no comment.

These two AWS launches from 2006 unlocked an untold number of novel data infrastructure companies to be built.

• Analytics + machine learning: Databricks, Snowflake’s nemesis for data engineering workloads

• Transactional databases: Neon, the transactional database (acquired by Databricks)

• Streaming: WarpStream, the real-time event streaming service (acquired by their competitor Kafka)

• Search: Turbopuffer

All have succeeded by rebuilding foundational data tooling on top of object storage like S3.

Enter vector search

AI agents make analytics workloads look quaint by comparison. Just the vector search piece is a perfect storm of how to run up a cloud bill:

1. Embeddings are big. If a chunk of text is 1KB, an embedding vector representing the meaning of that text could be 6KB.

2. Agents search more than humans. Way more. If you watch a coding agent work, it runs tons of searches. I usually only search “Oprah net worth” once a week.

3. Agents produce more vectors. More slop code + documents means the search index grows faster than it would if it was just people making stuff.

Search engines like Algolia or Elastic at this time were designed for a person searching in a search box on a website. They needed instant feedback, so data was stored in RAM. High cost was a tradeoff for low latency.

For agents using vector search, the tradeoff needed to be different. High cost was untenable, so the solution would be to trade a little latency.

Read more

Woah woah woah. Let’s back up.

What is a programming language?

Computers have the notable distinction of being at once one of the most intelligent inventions in human history and also being super dumb. They can technically do almost anything, but only if you tell them exactly what to do in exactly their language – the language of 1s and 0s.

See, a computer is a bundle of physical circuits that can be open (represented as 0) or closed (represented as 1), and all the data a computer uses and all the logic it performs is articulated to it in the language of 1s and 0s called binary; the OG programming language. It’s just ones and zeros all the way day.

Writing complicated logic with only two numbers at your disposal is obviously hard. Accordingly, telling computers what to do was a massive pain for many many years, so much so that it was generally confined to the domain of nerds at universities like Grace Hopper and Ada Lovelace. They’d write out the literal 1s and 0s the hardware understood, sometimes by physically flipping switches or feeding in stacks of cards with holes punched in them. This was, as you might expect, difficult, error prone, and a pretty far cry from the work you see modern software engineers doing at the specialty coffee shop.

As these early computer scientists wrote more code, they realized a lot of their code did the same things, like adding two numbers or storing a name, and they concluded they needed an easier way to write these common instructions. They landed on the idea of a “higher level” programming language that read like English, which would then be run through a translator called the compiler that turned it into 1s and 0s for the machine. Easy for humans to write, easy for computers to run, everybody wins! Thus the first mainstream programming language was born: FORTRAN.

Why are there so many programming languages?

We don’t talk about programming languages much here at Technically because, well, for the purpose of understanding the world of tech at a high level, we don’t really think it matters. Most languages are more alike than they are different, and they all do basically the same things give or take a few things. So, why even have more than one language? You just said computers only speak 1s and 0s right and so we made FORTRAN to solve that. Why do developers today write in over a hundred of them?

The answer is in tradeoffs. Since a language is a layer on top of 1s and 0s, each one makes slightly different design decisions on its journey to being English-like. The authors of FORTRAN made a thousand little judgment calls on your behalf. What gets to be easy and what stays hard? What happens automatically and what do you have to spell out by hand?

Some of FORTRAN’s decisions people liked, others they didn’t. So they wrote a different language, and then another, and then another, and each of these human-friendly programming languages prioritizes something different at the expense of something else. There is no best language, there is only the spectrum on which a host of tradeoffs sit.

For the purposes of understanding the wonderful world of Rust, we’re going to zoom in on one specific tradeoff: memory.

A trip down memory lane

When a program (some code) runs, it needs somewhere to keep the stuff it’s actively working with like:

• The user’s name

• The current score of your little game

• The contents of the text box you’re typing in right now

This is what memory is for. You can think of it as the program’s desk; not long-term storage like a filing cabinet (that’s your hard drive), but the open workspace where things sit while you’re actually using and working with them.

The desk needs to be managed: periodically cleaned and organized. After all, it’s small, and the program never stops working. In practice this means setting new data down, grabbing data back when it’s needed, and clearing off the old stuff to make room. This is the harrowed discipline of memory management.

There are lots of different ways to manage memory. Some approaches prioritize speed, and others deliver safety.

• By speed, I mean exactly what it sounds like. How fast the program runs. How quickly the page loads, how little the game stutters, how fast the trade goes through.

• By safety, I mean how well the language stops you from hurting yourself. A safe language makes it hard to write the kind of bug that crashes the program, leaks private data, or hands an attacker a way in. An unsafe language trusts you to never screw up, which, reader, you will.

By way of example, many higher level languages, like Python, manage memory for you automatically. That makes them safe and forgiving, but it costs in terms of performance because the program is spending energy doing memory housekeeping chores in the background while it runs. I.e. Python is considered slow.

Many lower-level languages, like C, are wicked fast because they don’t do any automatic memory management, but that means they’re hard to write (you have to manage the memory yourself) and prone to failure.

There are other tradeoffs inherent in different programming languages too, which won’t be the focus of this series, but are worth mentioning:

• Static vs. dynamically typed. Does the language require you to declare what kind of data every variable holds (a number, a word, true/false) before it runs? Static typing catches bugs early but is more rigid to write. Dynamic typing is faster and looser to write, but errors show up at runtime in production, where they hurt.

• Compiled vs. interpreted. Does the translation from code to binary happen upfront, or on the fly? Compiled languages do it all before the program runs, producing something fast and optimized. Interpreted languages translate as they go, which makes them more flexible and easier to experiment with, but slower when running.

• Ecosystem. A language comes with a community, a library of pre-built tools, and a job market. Python dominates data science not because it’s the fastest language (it isn’t) but because the tooling built around it is so mature that nothing else comes close.

In any case, the memory tradeoff often ends up being the big one - the granddaddy of them all, the primary question a developer must ask themselves before embarking on a new engineering expedition: Do you want it fast or do you want it safe? Which way western man?

You might be wondering why memory in particular is so important, such that (as I’ll demonstrate) someone built an entire new language around making working with it better. In part two, we’ll go deeper on why the memory problem is more consequential than ever in the age of high speed trading, AI, and online gaming, and what one cracked engineer at Mozilla decided to do about it.

Read more

Though for most of my readers (who for some reason I assume were born in the 90’s) this word may evoke a quaint yet hauntingly-long-ago visage of their suburban childhoods, it means something else entirely to a software developer.

The TL;DR

Sandboxes are isolated computing environments where you can run code that you don’t entirely trust. Think of them as solitary confinement, but for a program.

• Much of software today requires you to run untrusted code – code written by a Large Language Model or even a human you don’t know.

• Running untrusted code on your system is risky: it can breach your security, delete your data, and generally just mess with things.

• A sandbox puts iron walls around this untrusted code so it can’t mess with things, while still allowing it to run and (hopefully) accomplish its task.

• Sandboxes are absolutely exploding in popularity because of several different LLM-related use cases.

Sandboxes have been around for as long as I’ve been in tech. For years, products like Retool (my former employer) have needed ways to let their users run arbitrary code…without being subject to the risks of their users running arbitrary code.

But with LLMs taking the center stage, they’ve gotten a second wind for the ages. There are now dozens (dozens!) of sandboxes for LLM startups vying for the business of small companies and enterprises alike who need a place to run random code. And I will explain them to you.

Coding agents, RL training, and other forms of untrusted code

To understand why sandboxes are having such a moment, we must first detour to understand this concept of “untrusted code” a little more deeply.

Simply put, untrusted code is code that you, yourself, have not examined closely line by line. Because you haven’t examined it closely line by line, you don’t know what it does. And because you don’t know what it does, running the code in the same places that you have your other code running is very risky. Who knows what it’s going to do?

Imagine you’re a prison warden (if I have any readers who are prison wardens, let me know in the comments). You’ve got your hundreds of inmates who are, for the most part, playing pretty nicely together. Then you get a call that a new prisoner is on his way; you don’t know anything about him, just that he’s rumored to be a high-profile gangbanger. Is he well behaved? Could he incite a riot? Will he try to escape? Who knows.

With that uncertainty, you’re certainly not going to put him in Cell Block A with everyone else. Instead, you’ll cordon him off in solitary confinement where whatever he does won’t impact the orderly conduct of the rest of your subordinates.

In software engineering this is referred to as isolation. The prison is your infrastructure – the server your existing code runs on, the database it’s backed by, the APIs powering it. Everything for the most part is working well, and you don’t want to risk a new inmate fucking around with things.

So who are the gangbangers in our analogy? There are a few:

• Internal coding agents – developers using LLMs to generate code for new features, fixing bugs, etc. This code is untrusted, so it needs a sandbox to test it before merging into the rest of the codebase.

• Reinforcement Learning (RL) – when teams train their own coding agents, they need environments where the agents can play around and learn how to get better. Since the code they’re generating and running is (very) untrusted, they need a sandbox.

• (old school) Users – sometimes you want your users to be able to write code and run it on your systems.

Sandboxes give developers an isolated environment to run and test these outputs – these gangbangers if you will – where they physically cannot negatively impact other parts of your stack. If they mess up, if they turn malicious, if they start a riot, the scope of damage can never extend beyond the sandbox itself.

By the way, this gangbanging is not theoretical. Code can absolutely get nasty and mess with your stuff. You’ve probably seen examples on X of coding agents “accidentally” deleting someone’s entire database. And for RL, logs of coding agents in sandboxes often show them attempting destructive behavior because they’re basically try anything (that’s how RL works).

Read more

Within minutes, the agent was crawling the repository and shipping pull requests across multiple pages at once. Then it hit a wall.

For the parts of our site that lived in GitHub as code, the agent ran wild:

18 pages updated in a single PR for a handful of AI tokens. But the pages that mattered most (like our blog), were locked behind a Content Management System (CMS) the agent couldn’t touch.

Moments like this led us to decide to kick off a migration. Four months later, ~2500 pages of Sentry‘s marketing site live as Markdown and code in a Git repo, and we’re 100% out of our CMS. Here is what we learned.

Why we did it

Before the Claude Code session, we were actually expanding our CMS footprint. We wanted scalability for non-technical team members (me included at the time), guardrails to keep things on-brand, and a foundation for localization. Watching an agent do a month of SEO work in an afternoon and then stop dead at the CMS boundary changed the calculation.

Velocity was the real reason, not cost. We weren’t trying to escape a Contentful bill (though Cursor’s CMS had cost them nearly $57,000 in one year, which is non-trivial). We were trying to escape the situation where an agent could update half of our site instantly and the other half not at all. That asymmetry was getting more painful by the week as we built more workflows around Claude Code.

There’s been a small wave of teams writing about similar moves: Lee Robinson migrated cursor.com from a headless CMS back to Markdown in three days, spending $260 in AI tokens. Prefect rebuilt their entire site from scratch. Anita Kirkovska at Vellum took a different path, staying inside Sanity CMS but giving each marketer their own AI assistant with agent.md files that codify the team’s writing rules.

But it’s still early enough that best practices are still being written. What was clear, was that we wanted to work on the website w/ coding agents, and having pages gated behind a CMS was a blocker.

How we actually did it

Eli, our Sr. Growth Software Engineer wrote up the full technical playbook here, which is the right reference if you want the deep dive on Astro, Vite, the Vercel Blob cache we built for our marketing automation forms, and the schema-to-frontmatter mapping work.

The highlights:

• ~2500 pages migrated by a team of 2.5 developers in a two-month window. Most of the actual code was written by Claude Code; the engineers spent most of their time planning, reviewing, and directing the agent.

• One directory at a time. Docs and marketing pages migrated first (easiest to template), then blog content, then the most complex interactive pages. We kept the CMS on a free tier for legacy content during the transition, which gave us a soft cutover instead of a hard one.

• We changed frameworks at the same time, from Gatsby to Astro. Build times went from 14 minutes to under 6 in the migration itself, and after we shipped a Marketo form cache (the forms had been hitting an external API on every build), we got down to under 4. That works out to about 15.8 build hours saved daily across our ~95 builds/day.

• We consolidated ~200 bespoke pages into 3 templates during the migration. This made the codebase dramatically simpler, which makes it easier to use by coding agents. The agent doesn’t have to learn the quirks of 200 one-off pages; it learns three templates.

• Claude Code wrote most of the migration scripts that pulled content from Contentful’s API into our new Markdown files. We also experimented with Claude running Playwright visual regression tests against the old Gatsby site to catch unintended shifts.

• Broken staging builds are down ~95% and our Web Vitals score went from 89 to 97. Most of the broken-build reduction came from cutting external API dependencies out of the build process.

What it unlocked

Things that used to require a Jira ticket, a project request, or a Looker dashboard now happen in a Claude Code session.

In the last four months, our growth team has shipped a lot more than previously possible:

• 147 pages edited across 2 PRs to fix link issues surfaced by our Screaming Frog crawler.

• 50 legacy blog redirects cleaned up in about 5 minutes. This used to be a backlog ticket that nobody got to.

• 116 internal-link improvement PRs opened by a hub-and-spoke tool I built that prioritizes pages by SEO value. It runs as a Claude Code skill and pulls authority data from Ahrefs via MCP.

• 10 pages updated or built same-day for product launches and A/B test outcomes. Content that used to sit stale for months because nobody had cycles to update it.

• 11 active A/B tests running concurrently, including a new solutions page I prototyped in an afternoon. Each of these would have been a cross-functional project request before.

Two of the things we built deserve a longer look because they’re the artifacts most likely to be useful to other teams.

Skills as templates

We have Skills (AI instruction files) for each page type: landing pages, product pages, solutions, blog posts, cookbook recipes, workshop templates. Each one encodes the structure, the SEO requirements, the brand patterns, and the component library for that page type.

When anyone on the team asks Claude to create a new landing page, it doesn’t start from a blank file. The Skill interviews them for the required fields (slug, display name, SEO metadata, hero copy), produces a structured Markdown or JSON file that matches the page-type schema, and opens a PR for review.

This is the same consistency our CMS templates used to give us, except the agent actually follows the rules (humans are less observant than agents to pedantic instructions).

The developer cookbook

A few weeks ago we shipped the Sentry Cookbook, a collection of recipes that solve specific developer problems with copyable code.

It contains recipes like:

• “Debug undefined properties in React Native with Sentry Logs”

• “Automate weekly performance digests with Claude Code + Sentry MCP”

• “Monitor your MCP server with Sentry”

We built it because of how we were watching our traditional blog get consumed: organic search traffic to long-form posts has been falling for two years, but LLMs and AI Overviews keep citing structured, executable content.

The cookbook format collapses the distinction between content for SEO and content for users. Both audiences (developers + LLMs) need the same shaped answer: structured, short, and code-first.

We’re early on with this approach, but early returns are looking good, and the structure feels right.

What’s harder than we expected

A few honest things that didn’t go smoothly:

• Design fidelity took longer than the data migration. Anita made the same point in her Vellum writeup, and we hit it too. Pulling content from Contentful into Markdown was a scripted job. Getting the new Astro pages to render the same as the old Gatsby pages, pixel for pixel, was the slow part. Claude Code is good at producing functional code; matching a specific existing design 1:1 is still hard.

• Human Error. Someone on our team added an email validation rule to a form during the migration that they thought was innocuous. It tanked a key conversion metric for a few days before we caught it.

• Build infrastructure was an under-appreciated source of failure. The Marketo cache we built late in the migration to fix recurring build breaks turned out to be one of the highest-leverage changes of the whole project, not because it sped up builds (though it did), but because it eliminated a class of failure we’d been quietly working around for years. Worth doing before you “need” to.

• The team learning curve is real but smaller than expected. I couldn’t write code four months ago. Most of our marketing team couldn’t either. A handful of us are opening PRs now, not because anyone learned to be an engineer, but because the environment forced enough fluency to keep up. If your team is going to do this, plan for the first month being slower than feels comfortable.

What we’re working on next

Next on our team we’re tackling autonomous workflows: agents that can run on a schedule.

We’ve shipped one already, a workflow that routes specific kinds of Linear tickets directly to Claude. A ticket gets filed, an agent picks it up, a PR shows up for review.

So far it works best for small, well-scoped fixes (broken links, typo corrections, blurb updates), but we’re still figuring out which tasks are agent-ready and which need a human in the loop on every step.

The other thing we’re investing in is AI code review. The biggest risk with letting agents touch production is that quality control falls on whoever’s reviewing PRs, and reviewers miss things in a batch of 50 changes.

We’re still just scratching the surface. The migration itself is done, the velocity has been surprising, and the quick examples above are already more than we shipped in the year before this.

But every week we’re finding another workflow that could be moved off of SaaS and onto our codebase, another report that could be a Claude Code skill, another recurring task that could become a scheduled loop. Over time, I assume we’ll shift from spending time building this tooling to just using the tools we’ve built.

Resources:

• Eli’s technical playbook on the Sentry migration (Astro, Vite, Markdown, the Marketo cache)

• Lee Robinson’s migration writeup for cursor.com

• Prefect’s rebuild post

• Anita Kirkovska’s in-CMS approach using Sanity + agent.md

• The Sentry Cookbook

• The hub-and-spoke SEO tool referenced above

Read more

Next.js is a web framework for React. It helps developers (and now agents) create web applications in React quickly and easily, as long as you follow a few rules.

• The magic we take for granted – type in a URL, see a website – relies on a million complicated things going on in the background

• Web frameworks help developers focus less on the logisti…

Read more

Intro to package management

The vast, vast majority of software, be it consumer apps like Snapchat and Strava, or B2B platforms like Salesforce and Workday, use open source software as building blocks. This is to say, they use code written and made public by other people to save themselves time when developing their own applications.

A classic example of a use case for leveraging open source code is if your app has anything to do with time, such as displaying the date and time to a user. Time is surprisingly complex: what time zone are users in, are they in daylight savings, do you want to display the day of the week as well? You could code up all this logic yourself, but it’s much easier to use code someone else already wrote to do this.

The world of package managers emerges when you ask: how do you actually use someone else’s code?

One naive way might be copy-pasting code from someone else’s app or website into your own codebase, line for line. This would work, but is very inefficient. What happens if the author makes changes? What if there’s a security vulnerability discovered down the road?

For these reasons and more, the most common way that developers of open source software distribute their code is by bundling it into packages and publishing them for the world to use via a package manager. You can think of a package as a nice wrapper around the code that someone has authored, boxing up your little (or a lot) bit of code nicely so you can ship it to others without things getting messed up.

🚨 Confusion Alert 🚨

The terms package, library, module, and framework are used mostly interchangeably in the industry, for simplicity, I’ll stick to package here.

Package managers are programs that deal with the logistics of these “packages” of code: in particular, fetching and installing these open-source software packages. They are used almost anywhere people (or robots) write, test, or run code. The key elements of a package management system are the package manager and the package registry. Let’s quickly review how they work.

If I’m a developer starting a new project and I want to use someone else’s code for something (like handling dates), one of the first things I’m going to do is ensure I have a package manager installed on my system. The package manager makes it extremely easy to install the packages that I need. Each language maintains one or more official package manager, such as pip for Python, and npm for Javascript.

For the example where I’m coding up an app where I need to display time to my user, if I was building in Python I’d use pip by opening up my terminal and typing pip install pendulum (pendulum is the package name). The package manager does a few things:

First, the package manager checks my system to see if pendulum is installed already. If it is, fantastic, it does nothing. If it is not, it goes out and downloads the package. But where does it go to get the package? It goes to the registry.

The package registry is a public, trusted repository of open source software packages on the internet. Each language maintains one or more official registries, such as PyPI (Python Package Index) for Python, not to be confused with pip, and the npm registry for JavaScript.

You can think of the registry as the warehouse where all of the boxed software goes to be shipped out. The authors of the software are (ideally) constantly uploading new and improved versions of their packages so users like you can benefit from the great wheels of progress.

This general process of ensuring I have the packages I need is referred to as dependency resolution – a dependency just means a package that your software needs in order to run – and it’s the core job of the package manager. And it’s a full-time job: as I code up more features, my dependencies will change, and the package manager needs to make sure what I need is installed at all times, and what I no longer need is purged from the system.

Over time, many package managers grew beyond just fetching code. See, going from code to real running software requires a few other steps. For example, after code is written and external packages are installed, something called a compiler or bundler needs to translate that code into a language that’s readable by the machines. After that, a different program actually needs to run the code, called the runtime. At least that’s how it was for a long time — different tools for different stages of the pipeline.

But newer tools like Bun in the JavaScript world started collapsing this entire toolchain into one fast program that handles everything. This bundling is important context for understanding what AI labs actually bought, which we’ll get to shortly.

The thing I want you to know is that for the entirety of my time building software, and for decades before that, the package manager was a thing you used manually, slowly, and pretty infrequently. Starting or pulling a new project, updating a package, or running some tests locally. It was not software that anyone loved or waxed poetic about, it was just a thing you had to do.

This is because coding — making your software do what you want it to do by typing out instructions — takes a long time when humans do it. For human engineers, package management only has to be as fast as humans can decide what packages they need and can use them in their development cycle. Until recently, this was fine, because humans were the only entities that could write code.

And then came the agents

LLMs, on the other hand, can generate code really quickly. AI-powered coding agents can now take a loose set of requirements for a feature or whole app, plan the implementation, and then code it up across a massively parallel fleet of agents. And though the speed at which they can generate code is fast, they still need to run that code: each member of this fleet needs to resolve package dependencies, install packages in isolated environments, run tests and code, and clean up after themselves after they’re done. Just like humans do.

Even AI features geared toward non-developers increasingly require the ability to quickly write and run code. For example, when I ask Claude to produce a Word document, it installs an open-source JavaScript package called docx-js, writes code that uses that package to produce a document, and runs that code all within an isolated environment in the cloud.

So, using package managers went from a slow thing humans did once in a while to a thing that needs to happen tens or hundreds of times a minute when it’s agents writing the code. Where code runs went from a permanent thing I’d think about once per project, to something more ephemeral that needs to be rapidly spun up and torn down many times an hour.

What this means is that package managers are now critical infrastructure for AI products. They’re in the “hot path”, as we say in the biz. Every time Claude Code produces a file, or Codex writes and tests a function, package managers and runtimes are doing work on every single invocation. If they’re slow, the product feels slow. If they break, the product breaks.

Anthropic and OpenAI both have whole suites of products that involve agentic coding, from terminal-based offerings like Claude Code and Codex to their desktop apps and more. I believe that the folks at these labs realized early on that a big bottleneck in the quality of their products was going to be the sluggish package manager ecosystem. So they bought the best ones out there. And the tools they bought were the ones that had already grown beyond just package management into all-in-one toolchains that handle package management, building, and code execution.

Read more

It is not the first time that such a paradox has shown up in the deployment of a new technology. In 1987, Robert Solow remarked that you can see the computer age everywhere except in the productivity statistics. Erik Brynjolfsson later coined the term “productivity paradox” in 1993 to describe this very phenomenon. The massive investment in information technology through the 1970s and 1980s produced no measurable uptick in productivity growth. It took nearly a decade of organizational restructuring before the gains showed up in the late 1990s. We may be in a similar lag period with AI.

One answer to why this delay happens may lie in how early technology develops through the under-studied activity of kit-making.

A kit is a set of components that are meant to be tinkered with and have no single “correct” method of usage. Steam engines were famously kits. In 1763, James Watt was asked to repair a scale-model Newcomen steam engine at Glasgow University. The job led him to see how wastefully the Newcomen design used steam, and a year and a half later he hit on the idea of a separate condenser — building his first prototype using a brass surgical syringe as the cylinder. When he partnered with Matthew Boulton to commercialize the design, they didn’t produce finished steam engines. Instead, they sold engineering kits with extensive instructions that required on-site assembly. Boulton & Watt made a killing and transformed their age. Someone even named a startup incubator after them.

This rough template has foreshadowed technological revolution ever since. Whether in radio, automobiles, aircraft, electronics, or personal computers, communities of talented kit-building amateurs have disproportionately influenced early innovation. Michael Schrage, a research fellow at MIT’s Sloan School, puts this well: kitonomic innovation doesn’t follow the money, the money follows the kits. The proliferation of cheap kits signals a market sector ripe for revolution more reliably than the presence of expensive cutting-edge products. On the influence of kit-making on the information age, he writes:

So while there may be no “Steve Jobs of Kits” yet, there is surely no Steve Jobs without kits. There’s no Bill Gates or Akio Morita without kits either. Their market-transforming entrepreneurial leaps all emerged from kit-enabled cottage industries. The two Steves — Jobs and Wozniak — literally built Apple from kits. Gates and Paul Allen started Microsoft as a software systems supplier for DIY computer kit builders. Morita and Masaru Ibuka launched Sony with kits to turn AM radios into shortwave receivers. From the prewar “cat’s-whisker” playfulness of crystal radio kits to postwar floods of surplus electronics, kits became a medium, mechanism, and marketplace for next-generation invention.

Even within kits there is an important difference between amateurs tinkering to make money directly from the technology itself, and users repurposing kits to interpret a technology in a way that fits their own context. The first kind of kit-builder is someone like Wozniak at Homebrew, building computers because the computer itself is the product. The second kind is a farmer in 1915 Iowa jacking up the rear wheel of his Model T to run a corn sheller. He doesn’t care about the car as a product. He cares about shelling corn.

Users Interpret New Technology

There is an excellent paper by Ronald Kline and Trevor Pinch arguing that this second kind of creativity, what they call the “interpretative flexibility” of rural users, is what eventually led to the production of tractors and specialized trucks. Their framework comes from the Social Construction of Technology (SCOT) tradition, which holds that different social groups assign different meanings to the same artifact. What counts as a “working” technology is not settled by engineering alone. It is settled by what communities of users actually do with it.

The paper documents how the period from roughly 1903 to 1950 was dominated by rural users employing automobiles in ways that manufacturers never intended and sometimes actively discouraged. Farm men saw the car not merely as transport but as a general source of power. As early as 1903, a Kansas farmer advised readers of the Rural New Yorker to block up the hind axle and run a belt from the wheel to a corn sheller, grinder, saw, pump, or any other machine the engine was capable of running. Cars powered washing machines, cream separators, water pumps, hay balers, wood saws, cider presses, and corn grinders. From the paper:

“A rancher even used a Cadillac to shear his sheep. A Maine farm man put a car to so many uses in 1915 that tax assessors did not know whether to classify it as a pleasure vehicle or a piece of agricultural machinery. Farm men also used their cars as snowmobiles, tractors, and agricultural transport vehicles.”

Then kit manufacturers took advantage of this. Although firms brought out kits to convert the car into a stationary power source as early as 1912, advertisements for these kits did not appear in large numbers until 1917, during wartime shortages of farm labor and horses. Some companies simply sold a pulley to be attached to a jacked-up wheel.

Eventually, most kit manufacturers realized that jacking up one wheel put undue strain on the differential gear, since one wheel would spin while the other sat stationary on the ground. Most kits were therefore designed to overcome the differential problem, taking power from the crankshaft or rear axle instead. The Lawrence Auto Power Company in St. Paul sold a $35 kit consisting of a tie-rod, two pulleys, and a metal stand that could operate a feed grinder, corn sheller, silo filler, wood saw, and cream separator.

More elaborate kits allowed the car to act as an agricultural tractor. Food shortages during the war led the federal government to encourage farmers to “plow to the fences,” which gave added incentive to buy tractors or conversion kits. Kline and Pinch found three prewar instances where farm men yoked the automobile to the plow. The conversion kits that came out in a flurry in 1917 typically consisted of tractor-like drive wheels, a heavy axle, reduction gears, a larger radiator, and forced-feed lubrication, selling for $97.50 to $350. They counted twenty-two companies manufacturing these kits.

Then came what Kline and Pinch call “closure.” By the late 1910s and 1920s, manufacturers like Ford had observed these “creative” uses by people of the farm and ergo started producing dedicated tractors and trucks as specialized products. Ford told its dealers in 1916 that it did not want them converting Ford cars into trucks and other makeshifts not sanctioned by the company. Ford released its first one tonne truck in 1916, hoping to put an end to truck conversion kits. The American Tractor Association, a powerful trade group, even requested that the War Industries Board prohibit entirely the manufacture of kits that converted automobiles into tractors. Even though commercial tractors were available in 1918, widespread use of kits continued well into the great depression, until the New Deal when families had an influx of capital to invest in specialized farm equipment.

Post-closure, the social interpretation of the car as a vehicle for transporting people stabilized. The interpretative flexibility that Kline and Pinch described completely disappeared by the early 1950s. Farm people had stopped using their autos for grinding grain, plowing fields, or carrying produce to town. Instead, they bought tractors and pickup trucks in large numbers. These new artifacts were ones that manufacturers had developed partly in response to the new interpretations of the car that rural users had pioneered decades earlier. As Kline and Pinch put it, “The users, so easily overlooked in writing the story of technology, had made their mark.”

The interesting bit about this for our argument is that this general-purpose use of cars, where they were modified and repurposed as farm equipment, is not something that shows up in economic statistics from the time. Kline and Pinch note that most of the kit companies seem to have led a relatively short life. The Pullford Company of Quincy, Illinois, was an exception, bringing out a $135 kit in 1917 and advertising it continuously until at least 1940. But most of these businesses were ephemeral. The products they made were local solutions to local problems, sold in small quantities to a dispersed population of tinkerers. Government surveys of the era counted how many farms had automobiles, tractors, trucks, and stationary gasoline engines, but they did not count how many farms were using a jacked-up Ford to run a cream separator.

Living in Experiential Kit Time

The kit stage in the development of a technology is characterized by rapid experiments from amateurs and tinkerers. Most of these experiments turn out to be just interesting experiments or remain relevant for a short enough time that nobody bothers to record them. Healthy innovation cultures need the human capital that this tinkering produces. But that human capital is invisible to economic stats.

Just like how people throw away or don’t look at vibe coded projects, they don’t attempt to remember or productize the general-purpose uses they engineer during the kit era. The focus is on the act itself, on developing tacit skills through doing. There is a local development of skills in a small group, but it is rarely documented because there is no time to waste on documentation. And besides, chances are that the project is not worth documenting to begin with.

The subjective experience of kit making is similar to a religious procession or participation in Burning Man, where you have to be there and do the thing to understand what the hype is about. The memory it generates lives in the public and embodied layers, in islands of early adopters and makers willing to put up with the jankiness of this stage of a technology’s development.

Kits and the Firm

In his 1937 paper “The Nature of the Firm,” Ronald Coase asked a simple question: why do firms exist at all? If markets are efficient, why don’t individuals just contract with each other for every task?

His answer was that firms exist to reduce transaction costs. The costs of discovering prices, negotiating contracts, and coordinating production are often so high that it becomes cheaper to organize activity within a single entity. The boundaries of the firm are drawn where the cost of organizing one more transaction internally equals the cost of doing it through the open market. In simpler terms, if you had to bid on an hourly paid engineer every time you have some work to do, your organization will not get much done.

But Coase’s framing leaves out the other half of the ledger. Paul Lawrence and Jay Lorsch, in their 1967 book Organization and Environment, showed something that will sound intuitive to anyone reading this with a desk job: that coordinating activity inside a firm isn’t free either. As organizations take on more diverse work, their teams become differentiated — they develop their own time horizons, their own goals, their own ways of thinking about problems. The engineering team working on production ready software will move at a different speed and have a slightly different culture than the R&D team working on a new AI product.

Holding those worlds together requires integration: liaison roles, cross-functional teams, and formal integrator positions whose entire job is translating between teams that no longer share a common frame. The more differentiated you become, the harder and more expensive integration gets. Integration is expensive, but the ability to integrate different functions and teams is also a defensible moat for a company. You may know from your own working experience that not all companies are successful at integration. Successful companies solve for integration over and over again at different scales.

One way firms have historically reduced both kinds of cost (external transaction cost and internal integration cost) is by creating an accumulated memory layer of the experiments and skills from the kit phase. From the Kline and Pinch paper, you could think of the tractor as a product of the skills and knowledge that accumulated through the general-purpose use of cars by farmers in the 1910s and 1920s, combined with the evolution of the underlying engine technology. Ford didn’t invent the idea of using an automobile engine to pull a plow. Farmers did. Ford’s contribution was to take that scattered, tacit knowledge and embed it in an organization capable of producing a dedicated gasoline run machine at scale.

This integration of information and skill was possible because the environment had become less uncertain - prior to the Second World War, most families preferred to use automobiles over tractors so as to avoid additional expenditure. Then the New Deal provided families with capital to invest in farm specific equipment, which created a larger, more stable market for tractors.

Similarly, the first Apple computer was an accumulation of kit-stage experiments at Homebrew computing clubs, combined with the R&D capabilities of Xerox labs. Firms turn accumulated memory of the kit stage of a technology into flows of capital through products. That process is itself an integration feat — pulling differentiated knowledge from workshops, labs, and factory floors into a single coherent artifact. This is undoubtedly a capability that only exists for the organization and not the individual.

To summarize, a modern firm accumulates memory in the form of organizational design, documentation, and specialization, then turns that into flows of capital and productivity through products. The really good firms keep doing this over and over again. Each product cycle is a chance to absorb what was learned in the previous kit phase and formalize it into the integration machinery — the roles, routines, and shared vocabularies that let differentiated subunits cooperate on the next product.

The Firm and LLMs

The firm’s traditional advantage was twofold: it could internalize transaction costs the open market couldn’t handle cheaply, and it could build integration machinery to hold differentiated expertise together under one roof. LLMs erode both advantages, but not symmetrically.

On the market side, LLMs are rapidly making it cheaper for an individual to find specialists, evaluate contractors, synthesize scattered knowledge, and write enforceable specifications — these classic transaction costs are collapsing fast.

On the internal side, the picture is stranger. LLMs can automate some integration work, but they also accelerate the pace and volume of output inside each subunit, deepening the differentiation that integration has to span. The interpretive burden of reconciling AI-assisted outputs across teams lands on individuals rather than disappearing. At least for now, internal integration costs in most firms are going up, not down. Several surveys of AI use in the workplace have shown that people are producing more work than previously and dealing with task expansion, scope creep, and ‘AI brain fry’.

This brings us back to the question we started with. If AI is so impactful, why isn’t it showing up in the productivity stats? The Solow paradox answer is that firms haven’t reorganized yet. The computer took nearly a decade to show up in productivity numbers because the organizational work — flattening hierarchies, redrawing workflows, retraining workers, rebuilding integration machinery around the new technology — took nearly a decade to do. We are in the same lag now but with a different shape. Inside firms, integration costs are rising because differentiation between individuals and teams is accelerating faster than integration mechanisms can adapt.

The intensification of work that people feel with AI use points to this increasing integration cost. Outside firms, transaction costs are falling faster than people have habits for exploiting them — workers are not yet used to reaching past the org chart and engaging with markets constantly, and firms are not yet structured to let them. Caught in the middle, the output of AI is real but illegible to the statistics.

What comes out the other side of this lag is probably a bifurcation. Firms that can afford the integration machinery to harness AI across many differentiated teams and individuals will get very large, because the returns to scale on that machinery are enormous once it’s built — the hyperscalers are already showing what this looks like. Anthropic for example is not just a research lab, it’s also involved in consulting, economics and media.

Firms that cannot afford that machinery will get very small, because the falling transaction costs in the market make the solo operator with an LLM stack a viable competitor to the mid-sized integrated firm. A single person with the right tools is, in a real sense, running a differentiated organization of one, paying none of the integration costs that would have made that organization impossible a decade ago. The mid-market — big enough to need serious integration but not big enough to amortize AI-era integration infrastructure — gets squeezed from both sides.

The productivity gains from this transition will without a doubt eventually show up in the statistics. But they will show up as the rise of a few enormous firms and a long tail of very small ones, with the integrated middle hollowing out. By the time the numbers confirm it, we will already have moved on to arguing that quantum computing, or whatever comes next, is failing to show up in the stats — forgetting that the kit stage is always invisible, and the reorganization always takes longer than anyone expects.

As a technical person, but by no means a frontend engineer, I like to vibe code as much as I can - from growth experiments, to POCs, internal tooling, new designs, the list goes on.

It’s a balance, and I’m always trying to figure out what I should try to build myself, vs looping in the engineering team. We have a culture of being as independent as possible, without creating burden on other teams.

The most common use case I encounter is shipping a new landing page. I can open Claude Code (or any coding agent), describe the goal of the page, and voila, I have an MVP. However, the vibe coded result is just that: an MVP.

The dilemma then arises: should I ship it myself, or should I loop in engineering?

I’ve found that answering that requires considering two factors: consequences and ownership.

You build it, you own it

Even if a coding agent wrote every line of code, I still own the resulting product. I own both the successes and the failures.

So I need to consider:

What happens when what I shipped breaks? A formatting bug on an internal dashboard is annoying, but a 404 on a live page serving millions of users is an incident.

You could call this the blast radius. Is an issue just a nuisance to me + my direct team, or does it extend to our users + customers?

Who owns making the fix? Whoever is paged at 2am needs to have all the context to fix the underlying issue. For most production web infrastructure, this person is likely not me, the marketer.

Ship internal tooling yourself

Some of the most useful tools I’ve built with coding agents don’t need an engineer, because they never touch production infrastructure. Internal tools for data analysis or customer research don’t need an immediate fix if they break.

For example, each of the tools below took less than 4 hours to create on my own, and could even be public on the web by being deployed to Railway. They save me + our customer engineers tons of manual research time.

While not perfect, internal tools help me both understand behavior and solve problems in a much more automated way than I would otherwise.

This type of tooling is internal, not a core part of the product, used weekly by a few people, and if it breaks, the blast radius is mostly just my own time.

A few examples of internal tools I’ve vibe coded:

Template verification tool

Before admitting a company into Railway’s partner program, we ensure they have a template that meets our best practices guidelines. Verifying this used to involve a support engineer launching the template and QAing it.

Now, I have an app to verify whether templates meet the standards by just entering the URL. This enables me to be self-sufficient and makes verification fairly instant.

Awareness campaign metrics dashboard

In 2026, Railway is prioritizing broad-market awareness. This means sponsorships + brand campaigns. These are notoriously hard to track, particularly because they depend on metrics that we can’t directly collect, like YouTube views. I built a dashboard where I can enter links of YouTube videos and OSS tools we sponsor, and with the Youtube API and public `npm` data, map their engagement metrics and release dates over our growth metrics to view correlation.

Content quality checker app

Community is everything to us - which is why we’ve invested in our affiliate program to reward users who share about Railway.

As an exploration, I looked into what it would be like to automate monetary rewards (we call them bounties) for user-generated content.

This app takes a link or plain text and audits the content against a set of criteria to check for authenticity and accuracy. Note: this isn’t live, and purely a POC.

Claude Code has completely changed my workstream - by investing just a few hours into vibe coding a tool that’ll help me do my core work, I’m able to make more informed decisions while burdening adjacent teams less with new ideas.

When you need engineering

Going back to vibe coding that landing page.

What if it’s not just a landing page, but a completely new section of the site, at a new subdomain, with new structure. That makes it a bit more complex.

I’d need to consider:

Traffic and scale. What happens to the site if it reaches 1M daily users? At Railway, our website infrastructure is maintained by our engineering team - so we have load handling, caching, CDN configuration, and graceful degradation already in place. These aren’t things as a marketer I’d want to setup or own.

Security. If the page collects user data or touches payments, this is particularly important. Even simple landing pages can be subject to attacks. There’s no way I would vibe code anything related to our Stripe integration, for instance.

Defensive thinking. I caught a gnarly bug once where all the buttons on one marketing page broke, because we switched from capital case to sentence case, but button paths were hardcoded to capital case. An engineer might’ve caught this with defensive programming, but the coding agent did not.

Monitoring. Anything user-facing should have tests and checks in place so if there’s an issue, I know about it before our users do.

All of these combine to form the underlying engineering system.

Wherever possible, I much prefer to ship within existing systems that my engineering team has already set up, vs adding new systems myself. This allows me to iterate quickly, while leaving ownership to the engineers.

A few examples where I could implement improvements myself on top of existing engineering systems:

• Implementing a new landing page for partners when launching our partner program in September 2025. It used existing design components, existing form integrations, and existing web infrastructure. The only truly new piece was the URL and copy.

• Testing a new banner within the product experience that promotes Railway’s referral program. The banner design existed, and the GraphQL query for new users was already exposed to the frontend to show it to only a subset of users.

Railway’s partners page, mostly vibe-coded by me

The key insight here: while I am no frontend or design engineer by any means, I could self-serve growth experiments both on the website and within the product that are visual and copy-based, without introducing new infrastructure complexity.

How I choose when to vibe code

There’s a time and a place for non-engineers making code changes (in my case to web properties) with agents. For many of us, the time is now, so let’s pick the place.

Start with tasks you own end to end. If you own the outcome and have all the context for any fixes required, coding agents enable us to ship things we don’t have the skillset or time to do by hand. Research, internal tools, and analysis all fall into this bucket.

For frontend (web) work, separate ownership. Engineering maintains the systems, non-engineers build on top of those systems, with review. The key question is not “can I build this page”, it’s “does the system this page would live on already exist”.

Engineering-heavy work, like actual product implementation, should be owned by engineers. Non-engineers might implement MVPs to illustrate strategy and vision, but ultimately the user-facing stakes require engineering rigor in a production system.

The biggest shift coding agents have facilitated is where the handoff happens.

Non-engineers (in marketing, growth, dev rel, product, etc) have far more leverage today than we did a year ago. But for our use of coding agents to be sustainable, we have to work within the right boundaries, so we don’t get our friends in engineering paged at 2am.

Hey friends, quick note for you:

There are a few sure things in life. Death, taxes, and website rebrands. We cannot help you with the first two, but…

ICYMI, we recently redesigned the technically.dev site. It should be a lot easier to read and look through now. We also brought back a crowd favorite: learning tracks. We’ve added new tracks like How AI Models Actually Work + From Spreadsheets to Databases.

And to celebrate, we’ve created a new All-access subscription there that gets you access to all learning tracks (and the entire Technically archives).

If you’re a paid Substack subscriber, you will get automatic access to this All-access subscription — just make sure to sign up on technically.dev with the same email as your Substack account.

If you’re curious to test the waters, you can start by creating a free account that lets you unlock a couple free posts a month. Enjoy!

Amidst the frenzy of AI, a new category of infrastructure vendor has emerged: The inference provider. This white-hot newish(ish) vertical is producing some pretty big headlines for a category that, as we’ll see, is often overshadowed by big AI labs like OpenAI and Anthropic. TogetherAI is in talks to raise at $7.5B, Fireworks raised at $4B last year, and Modal is in talks to raise at $2.5B – all of these are doubling and doubling every few months it would seem. NVIDIA splashed $20 billion for Groq’s inference chip technology in December, its largest acquisition to date.

So yes, inference is white hot. But what do these companies actually do? And why are they worth so much money?

Let’s dig in, shall we?

What’s inference?

AI didn’t take long to become totally ubiquitous. It answers your questions, summarizes your meeting notes, codes up your apps, and, if you’re a bad friend, writes your birthday cards. All of it, every AI product you touch, boils down to two things: training and inference.

Training is how a model learns. You take a massive amount of text and use it to produce a mathematical equation that captures patterns in language: what words tend to follow what, how ideas relate to each other, what a reasonable response to a question looks like. Models like GPT and Claude were trained on enormous amounts of text from the internet, and what emerged are systems that can produce generally fluent, useful language.

Inference is when the model actually does its job. Every time you send a message to ChatGPT and it responds, that’s inference. Every time Claude summarizes a document or an AI coding assistant autocompletes a function, that’s inference too. It’s the model taking what it learned during training and applying it to your input, in real time.

Training models has historically been the sexy part of AI, done by people in glasses, wearing turtlenecks, and holding PhDs. They are paid many dollars. Serving models, on the other hand is plumbing, but as it turns out, as every company races to add AI to their app, the plumbers are doing pretty well for themselves. Dedicated inference providers have emerged as a major force in the AI application stack, offering a convenient managed inference service for developers building AI-powered features.

In its purest and most managed form, the OpenAI and Anthropic (we’ll call them frontier labs) APIs are AI inference providers. They have both trained the models and set them up for you to use easily. You make an API call: (“write my wedding vows”) and get back an inference: (“Webster’s dictionary defines love as…”). So what are we talking about here? Why would you use something special?

Why use an inference provider?

As someone using AI or building into your app, you have two major choices. You can use a closed-source model like ChatGPT or Claude. These are proprietary models – nobody knows how they were trained – and the only way to use them is through the people who trained them, OpenAI and Anthropic. These labs (as well as other closed-source providers like Runway, Midjourney, and Google) are very good at training models, and generally their models will be the best on the planet, but also cost the most.

Then there are open source models. These are built by companies or the community, and released for all to use. The weights of these models (the key to the fancy prediction equation) and sometimes even the training code are open to the public. You can download and run them yourself or even tweak them if you’d like. The most popular ones are Llama from Meta, Qwen from Alibaba, and DeepSeek.

With open-source models, you have a choice in how you actually use them. You can self-host by spinning up your own infrastructure, loading the model weights, and managing everything yourself. Or you can instead use an inference provider: a company that hosts open-source models for you and lets you access them through an API, just like you would with a closed-source lab.

There are a few reasons a developer might use an inference provider over just calling the inference endpoints of the frontier labs directly, or running OSS models themselves.

Cost. The newest Frontier models are expensive. If your use case doesn’t actually need a state of the art model like GPT-5.4 or Claude Opus, you can run an open-weights model like LLaMA on an inference provider for a fraction of the price. For high-volume, straightforward tasks like summarization, classification, or extraction, you probably don’t need the best model on the market, and the savings add up fast.

Speed. The big frontier labs generally optimize for model capability over latency. If you’re building something where response time matters (autocomplete, real-time agents), inference providers running on optimized hardware can get you significantly faster performance.

Resilience at scale. If you’re big enough that OpenAI having an outage means your product has an outage, you have a problem. Inference providers let you route across multiple models and backends through a single API, so you’re not in hell it every time one provider has a bad day.

Native integration with your cloud. The public cloud inference provider offerings like AWS Bedrock or Google VertexAI have an advantage in the enterprise because their models and endpoints are colocated with your other infrastructure, which can matter to security and data-residency-conscious buyers. They also tend to offer the most straightforward paths to fine-tuning and customization, which is when you take a base model and train it further on your own data so it performs better for your specific use case.

All this, at least for me, actually reverses the question - why would you call the frontier labs’ APIs when the inference providers are cheaper and faster? There are at least a few reasons:

• You need the latest and greatest models (remember, the inference providers are often hosting the open weights models, which are older)

• Cost and/or latency are not a concern for you (congrats, happy for you)

• You’re a noob and didn’t know about inference providers (me, until recently)

The inference provider spectrum

Like many software categories, there is a spectrum from:

• Easy to use with fewer customizations and control for a lot of money...

• …to pure infra providers that you configure to do what you want, with full control (and responsibility over) the infrastructure for less money.

Let’s take a look at that spectrum.

Most Managed: First-Party Model APIs

I already touched on this, but the AI labs themselves provide inference endpoints and, as such, are technically inference providers. You call their API, and you get a response without managing infra. It’s the simplest way to add AI to a product, and where most teams start. You get access to the newest and best models, but you also pay per call (among other things), so it’s usually the most expensive.

• OpenAI: The company that started the current wave; their API format has become the de facto standard that most other providers copy.

• Anthropic: Makes Claude; strong enterprise adoption, and known for handling very long documents.

• Google (Gemini): Lots of models; deeply integrated into Google Cloud and Workspace suites.

Technically, the platform of the model provider isn’t the only way to access them.

Cloud Hyperscalers: Enterprise AI Platforms

Read more

This is a story from Giorgio Liapakis of wibci. If you have a story to tell, drop us a note at editors@technically.dev.

In January, I gave an AI agent $1,500, full control of a Meta Ads account, then walked…

Read more

Maybe. This increasingly popular deployment model goes by a few names (self-managed, customer-managed, customer-cloud-deployed), but we’ll call it BYOC (Bring Your Own Cloud), and if you’re building B2B software, it’s worth understanding what makes some BYOC offerings successful, and some a huge dumpster fire.

Trickiness aside, the BYOC ask from customers is becoming commonplace. As data regulations get stricter and security teams get more power over vendor decisions, more enterprise buyers are walking into sales calls with a version of the same question: “Can this run in our cloud?”. Those vendors that can confidently say yes are making buku bucks.

The SaaS deal

SaaS works because of a simple trade: as the customer you give up control, and in return, you get convenience. Instead of managing your own servers, you pay the vendor to do it for you. No servers to manage, no upgrades to coordinate, no infra team needed. The vendor handles all of that, and you get a website to log into. For most companies, this is great. You should probably stop reading here if your customers are all happy with this arrangement.

But “most companies” is not “all companies,” and the ones where this deal falls apart tend to have very, very, large budgets.

Mo money mo problems

For buyers in financial services, healthcare, defense, and large tech (among others), multi-tenant SaaS creates problems that aren’t really solvable with a better sales pitch.

🚨 Confusion Alert

Multi-tenant just means that multiple customers are using software that resides on a single physical (or virtual) server. Your data is in the same shared database as everyone else’s. If you use Gmail, that’s multi-tenant. So is X, Sheets, Claude, you name it.

Here are some of those problems:

• Data residency and sovereignty: some companies can’t have their data leave their own servers, period.

• Security posture inheritance: in SaaS, your customer’s security is only as good as yours…and they might not want to take a bet that you know what you’re doing.

• Performance and egress costs: SaaS requires moving a lot of data around the internet, which is expensive AF, in some cases, even more expensive than the software itself.

• Vendor lock-in: if your SaaS runs on one specific cloud provider, your customer is now locked into that cloud, even if they already use a different one.

If you’re a cool, lean, startup moving fast and breaking things, these may sound like lame problems to you. But these are Fortune 500 problems. Get ya money up.

The spectrum

For all of these reasons and more, companies want to keep things running in their own clouds, your software included. But what that means in practice is very squishy. There’s a whole spectrum of deployment models that blend vendor-managed and customer-managed infrastructure. Understanding where you sit on it is probably a good first step.

Fully managed multi-tenant SaaS. You run everything. Customer gets a login. Everyone’s data sits in the same general pool of infrastructure. This is the default, and it works great for most buyers.

Single-tenant SaaS. You still run it, but each customer gets their own isolated infrastructure like databases and API servers, in the cloud. Better for compliance and sometimes performance, but you’re still holding the data.

Hybrid Architecture. This one is a bit wonky, but worth mentioning. Basically, sensitive data lives in the customer’s account (called the data plane in this architecture), and non-sensitive data and functionality lives on centralized infrastructure hosted by the vendor (control plane). This can help alleviate some of the pains of fully self-hosted setups, such as a lack of observability for debugging, because the data plane can communicate error logs and such back up to the control plane.

Fully self-hosted (pure BYOC). This is the most extreme, but also most lucrative version of BYOC, and the one we’ll generally focus on here. The customer runs the entire software stack in their own environment. These days, that usually means you ship a containerized (fancy word for packaged) version of your software that runs on Kubernetes, which helps with installs and upgrades. But you lose visibility into what’s actually happening and debugging without cluster access is painful. Works if your customers have strong platform teams. Doesn’t if they don’t.

The economics of BYOC

If you’re selling a SaaS product at $50K/year to a mid-market customer, the BYOC version of that same product can go for $150-250K/year. Sometimes more. The functionality is the same. What the customer is paying for is the right to run it inside their own environment, on their own infrastructure, under their own security and compliance posture. So the obvious question is: why would they pay more for less?

Read more

Imagine my surprise, as I became a writer to write actual content, not to become a prompt engineer. The idea that I’d turn over my job to a bot (who, it just so happens, I’m afraid is coming …

Read more