Models keep getting cheaper and more available, and now they slot into pretty much any project you’ve already got lying around.

In the corporate world, it means the evaluations for the traditional build-vs-buy process will change. On one side, you still have those thinking the software companies will deliver the same services and products. On the other side, some argue this means you could just build it all yourself from the ground up, no SaaS or software purchases needed. Just build it and host it.

For years that gap was the whole pitch for managed platforms: don’t self-host, let us carry the operational grind, you focus on the product. Fair trade, mostly. Self-hosting open source has always been free the way a stray dog is free. No purchase price, plenty of upkeep. Someone still has to patch it, watch it, and debug the config at midnight when it quietly stops working.

Open source just got easier.

To me, it seems like something in between, a hybrid, will be the new normal; pre-built frameworks which have all the core components you need, but with an AI helping out with the ops and customization. Turns out that player has been here for years already - open source.

Even though open source is old news, the introduction of open weight language models and smaller edge llms is a new paradigm. Maybe running an agent locally is not so far fetched anymore.

That’s what agents actually change: not the software, the babysitting around it. I run a home lab for fun, nothing critical riding on it, and I’ve felt the shift. Used to be me at 23:00, staring at a log file trying to figure out why a container won’t come back up. Now I describe the problem and let an agent go dig through it. Same interesting work, a lot less of the tedious kind, the kind that was never why I wanted a home lab in the first place.

Scale that up to a company and the math changes too. Enterprises didn’t avoid self-hosted open source because the software was worse. They avoided it because someone had to babysit it, and a vendor’s managed offering made that cost someone else’s problem. Shrink the babysitting bill and build-vs-buy tilts back toward build, or at least toward self-host.

Data sovereignty sharpens this. The industries and regions that already care where their data sits were paying the biggest premium for managed convenience. Take away the operational tax on self-hosting, and keeping things in-house stops being the expensive, principled choice. It’s just the obvious one.

This also applies to individuals who would like to take more control of their privacy and digital footprint, and host their data at home rather than on someone else’s computer. I would not be surprised to see home-labbing also get more popular, even among non-geeky individuals.

There’s a risk angle too. Once you can reasonably hand some share of your codebase to an agent, you have to actually decide what’s critical and what’s just glue and scaffolding nobody wanted to spend senior engineering time on anyway. That split forces teams to be deliberate about where their human attention actually goes.

None of this makes big tech’s platforms useless, scale and compliance guarantees are still real. But a lot of what people were paying for was never the software, it was not having to deal with the toil around it. Agents chip away at exactly that toil.

Maybe AI didn’t so much fix open source’s problem as remove the excuse for avoiding it. So big tech maybe didn’t kill itself, but big tech’s own pitch is getting a lot more contested, by the same AI it helped build.

Making a data platform functional was never a technical problem. It’s an organizational one, and it gets harder as your org grows, not as your data does.

A data platform for 50 users vs. 500 users are totally different.

In a five-person team, everyone knows the handful of datasets that exist and who built them. At fifty people, nobody holds that map anymore. Teams stick to their own corner, don’t know what already exists next door, and quietly rebuild it: same numbers, slightly different logic, different name. Not because anyone wanted three versions of “active customers.” Because finding the existing one was harder than writing a new fitting query.

“Functional” moves too. For one team, functional means: ask Sarah, she built it. Across ten teams, that stops working the moment Sarah changes roles, and nobody agrees on who inherited what she built.

Reorgs make this visible fast. Boundaries move overnight; the data doesn’t move with them. A dataset’s owning team gets merged or split, and the dataset just sits there, kept alive by whoever still has write access, while the new domains around the new chart get all the attention. Nobody decided to abandon it. It just stopped being anyone’s problem the day the boxes moved. Whoever officially inherits it later usually doesn’t want it: owning code you didn’t write and can’t explain isn’t a promotion, (traditionally) it’s been debt with your name on the pager (more on this in a coming post).

Distributing ownership to the teams closest to the data only works if those teams can actually hold onto it, through growth and through reorgs. Most companies aren’t there yet, which is how “we’re doing mesh” turns into the mess from part 1.

The version that survives contact with an org chart: build data domains, not just datasets. A domain is a bounded, durable unit of data, with an intention of lasting presence. This could be properly defined database, schema, or dbt project for that matter, that one unit owns end to end.

Often in smaller organisations, one domain would naturally be owned by one team - which is already a well-defined group with shared responsibilities and incentives. It can be ideal, but in the long term it might not, especially in larger organisations.

For us in the investment industry, equities would be a natural example regardless of embedded teams within, certain data products should last despite changing strategic focus or investment philosophies.

The unit should strive for data consistency regardless of the who is part of the owners. This could mean that it exist of one person from various teams, naturally by roles or responsibilities, but preferably defined upon the business functions. The fence isn’t the point. Survivability and durability is: when the reorg hits, the data domain still lives on with minimal impact.

Defining and building data domains based upon a well-defined unit improves the durability of a data, making the ownership and survivorship better and easier, consequently improving long-term insights.

Here is the promise: buy (or build) the platform, and your data problems go away. Here is what actually happens: your data problems change shape. You trade “I don’t know where this number comes from” for “I know exactly where this number comes from, and it takes forty minutes to find out why it’s wrong.” That’s not nothing. It’s also not what was promised.

Nobody sets out to create clutter.

The thing that makes a platform worth having is flexibility. You can plug in new sources, let teams self-serve, build their own pipelines, define their own models.

That’s also the thing that makes it a mess six months later. Nobody sets out to create clutter. It happens one reasonable decision at a time: one more source added without a clear owner, one more transformation nobody quite remembers the reasoning for, one more team that built their own version of a table that already existed somewhere else, because finding the existing one was harder than writing a new query.

This is basically the story of data mesh. The idea is good: distribute ownership to the teams closest to the data, let them move fast, stop bottlenecking everything through one central team. And it works, structurally.

What it does not do on its own is guarantee that anyone treats their piece of it with the same care. Distribute ownership without distributing standards, and you don’t get a mesh. You rather get a mess with better branding. Same clutter, just with an org chart that makes it look intentional.

None of this means platforms are a bad idea. I’d take a decent platform with known problems over an ad hoc mess of spreadsheets and one person’s Python scripts any day. But “we got a platform” is the beginning of the real work, not the end of it.

Once the data platform is live, the interesting questions start: who decides what counts as a trustworthy dataset, who’s allowed to publish one, and what happens when two teams build the same thing twice because nobody could tell them not to.

That’s what these next upcoming posts is about, really: the platform was never the hard part. The posts that follow get into why the org chart matters more than the tech stack, where the line between flexibility and governance should sit, and what a workable answer to “how do we scale this” looks like, with an actual example instead of a diagram.

This time it’s sort of different. One thought kept nagging at me that I’ve been very fortunate to live in this time of history.

Everyone in the world (with some exceptions) can post and can express into public debates and discussions, and together make atomic contributions forming this massive dynamic organism built on accumulated knowledge.

Ever since the inception of books, and amplified by the internet the last decades, the civilization has built on top of the prior giants. This is the backbone of our human success.

So, why not take part in this? I want to give back to that organism and what I hope could be valuable to others, even though it really requires going out of the comfort zone - regardless of how cliché it sounds.

So here I stand at the starting line - humbled by everyone who has plowed the way - taking a step into sharing both refined and unrefined thoughts. They might only be micro-contributions, but the goal remains: shedding a few more colors and making us slightly more enlightened.

You’re invited to follow along!

Best, Steffen