Why AI Infrastructure Startups Are Insanely Hard to Build

Infra has always been a tarpit idea. Google didn't start out as an "infra" company, and neither did Amazon or Facebook. In fact, the few "infra companies" that did start back then (companies like Godaddy) are minuscule compared to the aforementioned.

VC pouring money in LLM infra is legitimately crazy to me. It's clear as day that there will be winners of this AI cycle, but, as always, they will be companies that provide actual, real, tangible value. Making shovels works for huge companies like Nvidia or Intel, but it won't work for you. It's sad to see so much capital funneled in frameworks upon frameworks upon frameworks instead of fresh new ideas that could revolutionize the way we interact with our devices. I know it's a bit of a meme, but I'd rather see more Rabbit R1 and less LangChain.

Even OpenAI doesn't really have a product. Just throwing data at a bunch of video cards isn't value-generating in itself. We need a Dropbox or a Slack or an Instagram: something people love that makes their life easier or better.

Great article. I am not going to name names, but over the last one year, whenever there is a concept that became popular in Gen AI, thousands of startups pivoted to doing that. Many come from software background where the expectation was that if the code works on one dataset, it would work for everything. You can see this with 1/ Prompt engineering 2/ RAG 3/ and now, after Apple's WWDC, it's adapters.

Enterprises I have spoken to says they are getting pitched by 20 startups offering similar things on a weekly basis. They are confused on what to go with.

From my vantage point (and may be wrong), the problem is many startups ended up doing the easy things - things which could be done by an internal team too, and while it's a good starting point for many businesses, but hard to justify costs in the long term. At this point, two clear demarcations appear:

1/ You make an API call to OpenAI, Anthropic, Google, Together etc. where your contribution is the prompt/RAG support etc.

2/ You deploy a model on prem/private VPC where you make the same calls w RAG etc. (focused on data security and privacy)

First one is very cheap, and you end up competing with Open AI and hundred different startups offering it. Plus internal teams w confidence that they can do it themselves. Second one is interesting, but overhead costs are about $10,000 (for hosting) and any customer would expect more value than what a typical RAG provides. Difficult to provide that kind of value when you do not have a deep understanding and under pressure to generate revenue.

I don't fully believe infra startups are a tarpit idea. Just that, we havent explored the layers where we can truly find a valuable thing that is hard to build for internal teams.

I'm not sure I'm exactly at the edge of things, but I have 2 companies trying to setup regular meetings with me to be a beta customer. Both have promised I can help define a new product, but when I list my real problems... they aren't in the mission. Everyone wants to solve RAG (that's easy, don't need help) or they want to give me a gui I don't need, or wrap open source software like vllm. Or "solve privacy" (which usually comes in the form of masking... which surprise, that works for PII, but not PHI... I need the protected information).

Want to solve a real problem, help me create custom benchmarks, clean my data, get my small parameter model to reason better etc.

I work for a foundational AI company. I guess we're technically AI infra. We're inherently "narrowly" focused since our origin (which was well before the recent hype in past 2-3 years).

Our customers are really the type of AI infra companies being talked about in this article. And yea, the new ones I work with everyday are often a dime a dozen. A revolving door of small startups trying to make the same general purpose AI infra targeting other traditional "boring enterprise infra" companies.

The ones that I'm seeing get the most traction, have the best products, and best chances of success have zeroed in on specific niches and sub-industries. (Think AI infra that helps B2B2B companies where that last "B" is like Roofing companies and the value provided is helping Roofing companies easily and drastically scale their outbound and inbound marketing and sales.)

The startups I work with that make me scratch my head are the ones trying to build "disruptive" AI infra that does nothing different, provides nothing special, other than potentially nice UI/UX, and is liable to have their lunch eaten by either natural iterations and improvements of our own services they essentially just white label, or some other incumbent.

To me, it's like trying to create a new company to compete against Walmart and Target on groceries because they're too massive scale to win against "a well tailored customer experience" but then forgetting Costco, Aldi's, Trader Joes, and Whole Foods exist. And why would any of those aforementioned companies feel the need to acquire you rather than casually crush you as they go about their business either ignoring you as you wither or taking your good ideas and incorporating them into their own offering?

It's not impossible, just has to make sense and even then a certain degree of "the stars aligning" is required. Which is why there inevitably can only be a small group of winners out of this massive sea of hopefuls.

And I of course can only shrug my shoulders if asked if the AI infra startup I work at is differentiated, necessary, and lucky enough to be at the finish line with the survivors at the end. (We're finding our PMF and potential road to incumbency mainly with two-ish markets: old and new school enterprise infra and non-tech Fortune 500 type of companies.)

Way too many founders don't understand the impact of competing with cloud vendors.

Almost all enterprises have pre-committed budgets for cloud which means unless your product is FOSS it's going to be hard to convince someone to bet their business on it. Especially given that in this fundraising environment there is a 95% chance they won't be around in a year or two anyway.

It's going to be a brutal few years especially if we are heading into a period of diminishing returns in terms of LLM accuracy.

If I'm an application developer or manager, at any >100 person company, it normally doesn't fall into my remit to go out and pick a new company to contract with to provide services. Typically, it gets harder and harder to do that. Even with LLM stuff, we're contracting that through our existing relationship with Microsoft. When evaluating infra options, it therefore is a huge barrier to entry for most developers if there's a 'good enough' option on one of the main cloud providers

Great article, and pretty relevant to what I'm building (cloud developer tooling, including some genai, but also including non-AI tools + an application platform. Email me if interested.).

Obviously I'm not nearly as pessimistic about it. Zoom out for a sec and generalize to SaaS in general, not just AI infra (a subset of Saas) - all the arguments listed apply there too, except the data moat (which honestly doesn't matter to tons and tons of AI infra companies. That's more of an AI application problem). Now of course most startups are doing AI at least a bit, but in the past decade we've seen plenty of SaaS vendors compete with incumbents either head on or by carving out their own niche. In fact, two of the companies the author considers "incumbents" are arguably still challengers, but definitely were in this exact situation just a few years ago: Vercel and Databricks.

Also, competition from incumbents is hardly a deathknell. There's room for multiple products in some market segments - how many RDBMS companies are there? Competition from a huge incumbent in many ways comes with benefits, because it helps grow the overall market and awareness of the product space, including your own product.

I suppose according to this author I'm in the "application layer" even though really I'm in the AI-application-layer-now-but-not-later-layer, software-infrastructure-layer. And that's great because I actually do have experience in that specific application area. But honestly, saying "you ought to have expertise in your domain" is 1) duh 2) in the examples (llamaindex parsing/ocr, langchain llmops + agnetic stuff), there is clearly a big enough twist on doing it "but with AI" that the application/vertical is close to novel. Successful challengers create valuable businesses without prior deep expertise in their domain all the time and I don't really see how this is any different.

Basically, you could repeat this for any SaaS business. Starting a company is hard, but I don't know if AI infra is uniquely hard in the ways laid out.

The title is true. But, the arguments don't hold water for me. 12 years ago, I started a big data company. It looked similar for big data companies when Cloudera raised almost $1B in 2014. Too many people building data warehouses, especially in the cloud. I exited. Who knew that Snowflake and Databricks would emerge against the incumbents. Similarly, there will be winners in the AI infrastructure space. To win, you need to focus on your customers and delight them. Narrowing focus makes a lot of sense. Don't pay attention to the doom and gloom, or you'll never do a startup.

Good article, but what is the alternative? What can you build today as a software engineer that can have impact? Nothing seems to come close to AI / AI infra, even of its hard / risky / a moving landscape.

As someone who has semi-unwillingly worked in infrastructure and infrastructure consulting most of my career - we’ve never even really solved that problem, what on god’s green earth convinced you AI did?

I am genuinely curious.

It bugs me that all we are seeing in the vc-backed startup scene seems to be ai infrastructure startups. We got something close to ai and all people come up with is they want to be the next ai marketplace store or the millionth infrastructure startup that does exactly the same like their competitors. How boring.

It's as if all of the AI devex/infra companies are cargo culting the story of how the people that made the most money in the gold rush were the people who sold the tools.

The thing is that the tools were well understood and battle tested.

While I agree that AI infra startups are hard to build, I strongly disagree with the idea that they are harder than foundational or application layer startups. I think it boils down to what you know and what resources you can muster.

For instance, foundational AI startups are also ridiculously hard to build. You need an insane amount of funding, spend it pretraining models to stay competitive only to find that gains in hardware and model architecture make them obsolete within months plus there's no real guarantee that scaling will keep working.

Application layer startups are hard in a very different way, there's an insane amount of competition and new capabilities are emerging every few weeks. I have worked with a few AI girlfriend startups and they are really struggling with keeping apace and warding off ridiculous amount of competition.

I think it's really just YMMV. Of course, the deeper you get into the stack, the more monopolizing pressure there is. Is it hard to build AI infra startups? Yes 100%. Will there be very few winners? Yes. Is it harder than foundational or application layer startups? Depends on the founders' strengths. Is it Is it a lost cause? I really don't think so.

This is a well-written blog post. Thank you to share.

This part:

    > For AI infra startups to be “venture scale”, they will eventually need to win over enterprise customers. No question. That requires the startups to have some sustainable edge that separates their products from the incumbents’ (GCP, AWS, as well as the likes of Vercel, Databricks, Datadog, etc).

On the surface, I agree. But look at a parallel market segment: Cheap cloud hosting. Think: Linode (or any of its competitors). There are a bunch of cheap cloud providers who are more than 10 years old. They didn't all get bought out nor bankrupt by up-starts. Why? They must add just enough value to stay in business. Could we see something similar in the AI infra space? In fact, it looks more logical for the cheap cloud providers to try to build some AI infra -- low hanging fruit, to help with LLM training. (I am sure they already see GPU time.)

I've come across a number of these AI infra start-ups like Scale AI and Zerve and TBH I'm amazed they can do what they do with relatively small teams when you have Meta and Apple somewhat struggling in this area and buying rather than building themselves.

Did any of the startups in question actually ever want to build AI infrastructure or did they all pivot from Metaverse to Crypto to AI in the great pivotting of 2022.

Given VC's penchant for throwing cash at grifters in the latest hype space is it any suprise that some of the beneficiaries are looking for a quick exit before they have to do any actual work?

Yet there is little "AI" specific in this AI infrastructure startup challenge:

1) insane levels of competition towards any goal make relavant minor, secondary, traits that are not obvious before hand. Pure luck becomes more important.

2) excess market concentration (of which the tech sector is maybe the most egregious example) makes any new initiative harder. The more dominant and controlling the incumbents the harder to find a decent sized niche to grow.

3) selling to risk averse enterprizes / organizations is always an uphill battle that requires climbing a mountain of bureaucracy and regulation, only to eventually face random internal politics.

In the end the current craze will certainly produce a modified tech landscape. These recurring hypes always overpromise and underdeliver, but a cumulative effect is slowly happening.

In such stormy seas its hard to identify an optimal course and strategy. Riding every hype wave may sound silly but might work. On the other extreme, one may seek beacons indicating eventual stable land and try to navigate there.

Good luck

Good tips, especially the point about narrowing the scope. At https://Lemonfox.ai we started with a LLM, image and speech-to-text API. Now we are only focusing on the speech-to-text API as the other areas are already very crowded and there's a lack of innovation in the speech-to-text space.

Gen AI feels more and more like NFTs and blockchains, and overall, a lot like pre-2001-bubble (or more accurately post y2k).

A very exciting and expensive solution in search of an actual problem, that will ultimately find its way, commoditised, in a small niche, while adjacent technologies take the lead for productive use-cases.

Curious for discussion:

Does this logic also apply to industry-specific "AI Infra?," where the APIs are wrapping a service that solves a domain-specific problem using AI, rather than general purpose infra technology? And provides those APIs to other businesses within that industry?

In a nutshell:

If there's a goldrush, you get rich by selling shovels.

... unless there are already 200 shovel shops next to each other...

Selling shovels is a way to make money, but definitely not at venture scale.

“Insanely Hard” an overused term that is now a cliche. We don’t build ffmpeg wrappers as a service because it is easy.

Why does the author claim that Adept was acquired by Amazon? The linked article says they hired away the CEO and key staff.

Articles like this represent reminder that we are in the middle of a hype cycle. [1]

I don’t say that thinking that LLMs (really: Transformers and the corresponding scaling of compute around it) don’t represent a step change.

I say that because I am very sure that we are going to see a slope of enlightenment that results in products that improve the quality of human life.

[1] https://en.m.wikipedia.org/wiki/Gartner_hype_cycle