Skip to content

AI#

The Asymmetry

Two emails arrived on the same morning last week.

The first was four paragraphs long. It opened with background I already knew, wandered through a description of several problems without distinguishing which ones mattered, and ended mid-thought — no clear ask, no proposed next step. I read it twice and still wasn't sure what I was supposed to do. The sender had emptied their head into my inbox, and the work of organizing their thoughts was now mine.

The second was three sentences. It stated the problem, proposed a solution, and asked for my sign-off by Thursday. I read it once, replied in thirty seconds, and moved on.

Same medium — text in a rectangle on my screen. Completely different experience. One was like drawing in a breath and finding oxygen. The other was like drawing in a breath and finding nothing.

The Factoring Problem

There's an operation in mathematics called prime factorization. Given a large number, find the primes that multiply together to produce it. It's famously hard — so hard that modern cryptography depends on it. But here's the thing: verifying the factors is trivial. If I tell you that 7 × 13 = 91, you can confirm it in your head. Finding those factors in the first place is where the work lives.

The factoring asymmetry — finding the prime factors of 91 is hard; verifying that 7 × 13 = 91 is trivial

You Can Still Understand the Machine

There's a nostalgia among people who grew up with early personal computers — the Commodore 64, the Apple II, the TRS-80 — for the time when you could understand everything about your machine. The CPU had a few thousand transistors. The memory map fit on a single page. You could trace the flow of electricity from keystroke to screen pixel and predict exactly what would happen. You owned the whole thing, top to bottom.

Commodore 64 Evan-Amos, CC BY-SA 4.0, via Wikimedia Commons

Modern AI systems don't offer that same feeling of total mastery. But they're more understandable than most people assume — if you stop trying to grasp the whole thing at once.

The trick is to peel it apart, one layer at a time. Start at the top — the software system you interact with — and work your way down through the reasoning strategy, the language model, the network architecture, and finally the individual neuron. At each layer, the math is straightforward and the ideas are concrete. And somewhere on the way down, the thing that felt like digital alchemy starts to look like what it actually is: simple mathematical operations, repeated at extraordinary scale.

What Block Gets Right and Wrong About AI-Driven Organizations

Block recently published an essay arguing that AI will replace organizational hierarchy — that the span-of-control constraint governing every large organization since the Roman legions can finally be broken. The essay, introduced with an endorsement from Sequoia, spends considerable time on military history before arriving at Block's vision: a company organized as "an intelligence" rather than a hierarchy, where AI maintains a "world model" of operations and coordinates work that previously required layers of human management.

The piece is ambitious. It is also roughly 80% historical context, 15% vision, and 5% acknowledgment that none of this exists yet. Let's extract what's actually useful.

  • in AI, LLM, RAG
  • 5 min read

Revisiting the Limits of RAG: A Conversation with Claude

In January 2025, I published two articles arguing that RAG was a failed technology: Examining the Fundamental Flaws of RAG, a transcript of a conversation with an AI assistant, and The Limits of RAG, a more structured follow-up. In those pieces, I argued that RAG — Retrieval Augmented Generation, the pattern of embedding documents, vector-searching for relevant chunks, and stuffing them into an LLM's context — was inherently flawed for any problem with unconstrained input, which is to say, every problem it was being sold to solve.

Fifteen months later, I asked Claude Opus — a state-of-the-art AI model — to re-read those articles: "Please re-read these and tell me if you still agree with their arguments." In the interest of transparency about how human-AI collaboration actually works, here is that conversation in full.

The Cost of Software Is Now Zero

A survival rubric for software and SaaS entrepreneurs in the era of vibe coding.

In February 2025, we published The AI-Driven Transformation of Software Development. Our central thesis: AI would trigger a fundamental shift in the build-versus-buy calculus, accelerating a "Cambrian explosion of software" and driving development costs toward zero. We predicted that businesses would find building tailored solutions increasingly cost-effective and strategically superior to purchasing off-the-shelf software.

The thesis has played out. The cost of code is, for most practical purposes, zero.

What's Actually Happening Out There

We sat with two business owners last week. The conversations were different in detail but identical in conclusion: both had stopped buying software.

One is building a complete property management operating system: property records, CRM, fleet tracking, risk management, financials, task management, and more. Not a subscription he configured — a system his company owns outright, built for exactly how his operation works. He built it in two weeks — what would have cost $200,000 a year to rent from a vendor.

The other runs a retail chain. Someone on his team has been working through the software stack systematically — not one big build, but a rolling replacement of every tool they'd been renting. He's already cut $300,000 in annual costs. He's roughly halfway through. When the last subscription is gone, he's asked us to review the whole thing before it goes live — security, scalability, and production robustness.

Operators are replacing project management tools, CRMs, inventory systems, client portals — the entire layer of workflow software that SMBs have been renting for decades. Not because they became developers. Because describing software and building software are now the same thing.

The savings compound at exit. At a typical acquisition multiple, a $300,000 annual reduction in software costs adds over a million dollars to the sale price.

Now look at the same picture from the other side — the side trying to sell software to these operators.

One Million Vibecoders Writing the Same Thing

A massive crowd lined up for "Vibe Coders" and one person in line for "Users"

A million people are building ERP systems. A million people are building project management tools. A million people are building CRMs. They're all working on the same categories, pouring effort into software they intend to sell — and none of them have a market. Because anyone who wants that software will just build their own.

The vibecoders building products to sell are wasting their time. Their potential customers have the same tools they do.

The only vibecoders whose code actually gets used are the ones who are also the users: owner/operators building custom software for their own businesses. That ERP built specifically for one company's workflows, by the person running that company — it doesn't need to find a customer. It already has one.

This is the dividing line. Vibe coding is not a new software business model. It's the tool that lets operators stop being software customers.

The businesses in trouble aren't failing because they have bad products. They're failing because the people who used to buy from them have a better option: build it themselves, tailored to their exact needs, with no recurring subscription.

The Question That Follows

If code is free to produce, software businesses that sell code lose their moat.

The value proposition was never really the software itself. It was the arbitrage: someone already built this, so you don't have to pay a developer. That arbitrage is gone. The operator with a weekend and a capable AI assistant can now build exactly what they need, perfectly suited to their workflow, with no recurring subscription cost.

Not all software businesses face this. The ones selling code packaged as a product are in trouble. The ones that were always selling something else — using software as the delivery mechanism — are fine. Some are better than ever.

The question every founder needs to answer honestly: if code were free, would anyone still buy from us?

What Survives

Twenty years ago my colleague John Cage introduced me to Treacy and Wiersema's Value Disciplines. Operational Excellence, Product Leadership, Customer Intimacy — pick one to dominate, maintain threshold in the others. I've applied it to every strategic engagement since. Vibe coding just took one of the three off the table.

Operational Excellence. Competing on lowest cost and highest efficiency has been the dominant strategy for SMB SaaS. It's no longer defensible. When an operator can build exactly what they need at zero recurring cost, "cheaper than building it yourself" isn't a position.

Product Leadership survives — if the complexity is real. Feature-rich workflow software doesn't qualify. Genuine product leadership means ML models, optimization systems, domains that require years of specialized expertise to build correctly. A vibe-coded app can approximate a dashboard. It can't approximate a decade of algorithmic research.

Customer Intimacy not only survives, it wins. Anywhere the deliverable is judgment, accountability, or trusted expertise — with software as the delivery mechanism rather than the product. Cheap code helps these businesses. They deliver faster, operate leaner, and take on more clients with the same team. The operators winning here aren't the ones handing everything to AI — they're the domain experts who can supervise it. That's precisely why they're winning.

Two additional categories fall outside the disciplines but are equally defensible:

Regulatory and compliance moats. Healthcare software, financial systems, anything requiring liability acceptance, certifications, or audit trail requirements. A vibe-coded replacement might replicate the features. It won't replicate the compliance posture.

Infrastructure position. The picks-and-shovels layer that vibe-coded applications depend on: authentication, payments, deployment, APIs, databases. Network effects live here too — platforms where years of data and an embedded partner ecosystem make migration genuinely expensive. Vibe coding expands this market, not shrinks it.

The Rubric

Score your business across seven dimensions. Add them up.

Dimension 1 — Exposed 2 — Mixed 3 — Defensible
Value Delivery Software is the product. Customers pay for features. Software enables a service. Code and expertise blend. Judgment, trust, or accountability is the product. Software is delivery.
Switching Cost Data is portable. No integrations, no ecosystem. Meaningful friction: data history, integrations, learned workflows. Network effects or regulatory data residency. Migration is genuinely expensive.
Compliance Moat No requirements. Anyone can build a replacement. Compliance matters, but a determined operator could manage it. Certifications, liability acceptance, audit trails. Vibe coding can't satisfy these.
Problem Complexity Forms, dashboards, CRUD. Buildable in a weekend. Non-trivial integrations or moderate algorithmic depth. ML, optimization, real-time systems. Years of specialized expertise required.
Buyer Profile SMB operators — the people now building their own tools. Mid-market with some IT governance. Regulated enterprises, governments. Procurement and legal sit between you and replacement.
Layer End-user application for a specific use case. Platform with some application features. Infrastructure that vibe-coded apps depend on.
Proprietary Data / Content / IP No proprietary data or IP. Anyone starting from scratch would reach feature parity quickly. Some accumulated data advantage — user history, transaction data — but replicable with time and effort. Proprietary datasets, content licenses, or IP that cannot be recreated from scratch. The asset is the moat.

Reading Your Score

Total What it means
7–12 Pivot urgently. You're in Operational Excellence territory — the discipline vibe coding just ended.
13–17 Reinforce or reposition. You have assets but meaningful exposure. Identify which dimensions can be strengthened.
18–21 Press the advantage. You're operating in Customer Intimacy, Product Leadership, or infrastructure. Double down.

Two Examples

Monday.com scores a 10. It's a $10 billion company. It's also a work management application — forms, boards, and status columns with a clean interface. No compliance requirements. No proprietary data. No algorithmic depth that requires years to build. Its switching cost scores a 2 because workflows and integrations create some friction, but nothing that survives a determined replacement effort. The rubric doesn't care about revenue multiples. A tool called Zapta already lets teams feed in their Monday.com API token and vibe-code a custom replacement — database, authentication, and all — for $29 a month.

Stripe scores a 21. Every dimension is defensible, and most reinforce each other. The compliance posture is what creates the enterprise buyer. The enterprise buyer generates the transaction data. The transaction data trains the fraud models. The fraud models deepen the moat. A vibe coder building a payments app doesn't compete with Stripe — they depend on it.

The M&A market is already pricing this divergence in. Q1 2026 data shows that in vertical software acquisitions, revenue growth carries 2.4 times the predictive weight of EBITDA margins in explaining valuation outcomes. Buyers are paying for stickiness — which is another way of saying they're paying for defensibility.

What This Means

Most software businesses were built on the assumption that code was scarce. It isn't anymore.

The question in the middle of this article — if code were free, would anyone still buy from us? — isn't rhetorical. Run the rubric. If you're scoring in the 7–12 range, the answer is no, and your replacement isn't a competitor. It's your customer.

The AI-Driven Transformation of Software Development

1. Introduction: The Seismic Shift in Software Development

The software development landscape is undergoing a seismic shift, driven by the rapid advancement of artificial intelligence. This transformation transcends simple automation; it fundamentally alters how software is created, acquired, and utilized, leading to a re-evaluation of the traditional 'build versus buy' calculus. The pace of this transformation is likely to accelerate, making it crucial for businesses and individuals to stay adaptable and informed.

2. The Rise of AI-Powered Development Tools

For decades, the software industry has been shaped by a tension between bespoke, custom-built solutions and readily available commercial products. The complexity and cost associated with developing software tailored to specific needs often pushed businesses towards purchasing off-the-shelf solutions, even if those solutions weren't a perfect fit. This gave rise to the dominance of large software vendors and the Software-as-a-Service (SaaS) model. However, AI is poised to disrupt this paradigm.

Introduction to AI-Powered Automation

Large Language Models (LLMs) are revolutionizing software development by understanding natural language instructions and generating code snippets, functions, or even entire modules. Imagine describing a software feature in plain language and having an AI produce the initial code. Many are already using tools like ChatGPT in this way, coaching the AI, suggesting revisions, and identifying improvements before testing the output.

This is 'vibe coding,' where senior engineers guide LLMs with high-level intent rather than writing every line of code. While this provides a significant productivity boost—say, a 5x improvement—the true transformative potential lies in a one-to-many dynamic, where a single expert can exponentially amplify their impact by managing numerous AI agents simultaneously, each focused on different project aspects.

Expanding AI Applications in Development

Additionally, AI is being used for code review tools that can automatically identify potential issues and suggest improvements, and specific AI platforms offered by cloud providers like AWS CodeWhisperer and Google Cloud's AI Platform are providing comprehensive AI-driven development environments. AI is being used for AI-assisted testing and debugging, identifying potential bugs, suggesting fixes, and automating test cases.

Composable Architectures and Orchestration

Beyond code completion and generation, AI tools are also facilitating the development of reusable components and services. This move toward composable architectures allows developers to break down complex tasks into smaller, modular units. These units, powered by AI, can then be easily assembled and orchestrated to create larger applications, increasing efficiency and flexibility. Model Context Protocol (MCP) could play a role in standardizing the discovery and invocation of these services.

Furthermore, LLM workflow orchestration is also becoming more prevalent, where AI models can manage and coordinate the execution of these modular services. This allows for dynamic and adaptable workflows that can be quickly changed or updated as needed.

Human Role and Importance

However, it's crucial to recognize that AI is a tool. Humans will still be needed to guide its development, provide creative direction, and critically evaluate the AI-generated outputs. Human problem-solving skills and domain expertise remain essential for ensuring software quality and effectiveness.

Impact on Productivity and Innovation

These tools are not just incremental improvements; they have the potential to dramatically increase developer productivity, potentially enabling the same output with half the staff or even leading to a fivefold increase in efficiency in the near term, lower the barrier to entry for software creation, and enable the fast iteration of new features.

Impact on Offshoring

Furthermore, AI tools have the potential to level the playing field for offshore development teams. Traditionally, challenges such as time zone differences, communication barriers, and perceived differences in skill level have sometimes put offshore teams at a disadvantage. However, AI-powered development tools can mitigate these challenges:

  • Enhanced Productivity and Efficiency: AI tools can automate many tasks, allowing offshore teams to deliver faster and more efficiently, overcoming potential time zone delays.
  • Improved Code Quality and Consistency: AI-assisted code generation, review, and testing tools can ensure high code quality and consistency, regardless of the team's location.
  • Reduced Communication Barriers: AI-powered translation and documentation tools can facilitate clearer communication and knowledge sharing.
  • Access to Cutting-Edge Technology: With cloud-based AI tools, offshore teams can access the same advanced technology as onshore teams, eliminating the need for expensive local infrastructure.
  • Focus on Specialization: Offshore teams can specialize in specific AI-related tasks, such as AI model training, data annotation, or AI-driven testing, becoming highly competitive in these areas.

By embracing AI tools, offshore teams can overcome traditional barriers and compete on an equal footing with onshore teams, offering high-quality software development services at potentially lower costs. This could lead to a more globalized and competitive software development landscape.

3. The Explosion of New Software and Features

This evolution is leading to an explosion of new software products and features. Individuals and small teams can now bring their ideas to life with unprecedented speed and efficiency. This is made possible by AI tools that can quickly translate high-level descriptions into working code, allowing for quicker prototyping and development cycles.

Crucial to the effectiveness of these AI tools is the quality of their training data. High-quality, diverse datasets enable AI models to generate more accurate and robust code. This is particularly impactful in niche markets, where highly specialized software solutions, previously uneconomical to develop, are now becoming viable.

For instance, AI could revolutionize enterprise applications with greater automation and integration capabilities, lead to more personalized and intuitive consumer apps, accelerate scientific discoveries by automating data analysis and simulations, or make embedded systems more intelligent and adaptable.

Furthermore, AI can analyze user data to identify areas for improvement and drive innovation, making software more responsive to user needs. While AI automates many tasks, human creativity and critical thinking are still vital for defining the vision and goals of software projects.

It's important to consider the potential environmental impact of this increased software development, including the energy consumption of training and running AI models. However, AI-driven software also offers opportunities for more efficient resource management and sustainability in other sectors, such as optimizing supply chains or reducing energy waste.

Software will evolve at an unprecedented pace, with AI facilitating fast feature iteration, updates, and highly personalized user experiences. This surge in productivity will likely lead to an explosion of new software products, features, and niche applications, democratizing software creation and lowering the barrier to entry.

4. The Transformation of the Commercial Software Market

This evolution is reshaping the commercial software market. The proliferation of high-quality, AI-enhanced open-source alternatives is putting significant pressure on proprietary vendors. As companies find they can achieve their software needs through internal development or by leveraging robust open-source solutions, they are becoming more price-sensitive and demanding greater value from commercial offerings.

This is forcing vendors to innovate not only in terms of features but also in their business models, with a greater emphasis on value-added services such as consulting, support, and integration expertise. Strategic partnerships and collaboration with open-source communities will also become crucial for commercial vendors to remain competitive.

Commercial software vendors will need to adapt to this shift by offering their functionalities as discoverable services via protocols like MCP. Instead of selling large, complex products, they might provide specialized services that can be easily integrated into other applications. This could lead to new business models centered around providing best-in-class, composable AI capabilities.

Specifically, this shift is leading to changes in priorities and value perceptions. Commercial software vendors will likely need to shift their focus towards providing value-added services such as consulting, support, and integration expertise as open-source alternatives become more competitive. Companies may place a greater emphasis on software that can be easily customized and integrated with their existing systems, potentially leading to a demand for more flexible and modular solutions.

Furthermore, commercial vendors may need to explore strategic partnerships and collaborations with open-source communities to remain competitive and utilize the collective intelligence of the open-source ecosystem.

Overall, AI-driven development has the potential to transform the software landscape, creating a more level playing field for open-source projects and putting significant pressure on the traditional commercial software market. Companies will likely need to adapt their strategies and offerings to remain competitive in this evolving environment.

5. The Impact on the Open-Source Ecosystem

The open-source ecosystem is experiencing a significant transformation driven by AI. AI-powered tools are not only lowering the barriers to contribution, making it easier for developers to participate and contribute, but they are also fundamentally changing the competitive landscape.

Specifically, AI fuels the creation of more robust, feature-rich, and well-maintained open-source software, making these projects even more viable alternatives to commercial offerings. Businesses, especially those sensitive to cost, will have more compelling free options to consider. This acceleration is leading to faster feature parity, where AI could enable open-source projects to rapidly catch up to or even surpass the feature sets of commercial software in certain domains, further reducing the perceived value proposition of paid solutions.

Moreover, the ability for companies to customize open-source software using AI tools could eliminate the need for costly customization services offered by commercial vendors, potentially resulting in customization at zero cost. The agility and flexibility of open-source development, aided by AI, enable quick innovation and experimentation, allowing companies to try new features and technologies more quickly and potentially reducing their reliance on proprietary software that might not be able to keep pace.

AI tools can also help expose open-source components as discoverable services, making them even more accessible and reusable. This can further accelerate the development and adoption of open-source software, as companies can easily integrate these services into their own applications.

Furthermore, the vibrant and collaborative nature of open-source communities, combined with AI tools, provides companies with access to a vast pool of expertise and support at no additional cost. This is accelerating the development cycle, improving code quality, and fostering an even more collaborative and innovative environment. As open-source projects become more mature and feature-rich, they present an increasingly compelling alternative to commercial software, further fueling the shift away from traditional proprietary solutions.

6. The Changing "Build Versus Buy" Calculus

Ultimately, the rise of AI in software development is driving a fundamental shift in the "build versus buy" calculus. The rise of composable architectures means that 'building' now often entails assembling and orchestrating existing services, rather than developing everything from scratch. This dramatically lowers the barrier to entry and makes building tailored solutions even more cost-effective.

Companies are finding that building their own tailored solutions, often on cloud infrastructure, is becoming increasingly cost-effective and strategically advantageous. The ability for companies to customize open-source software using AI could eliminate the need for costly customization services offered by commercial vendors.

Innovation and experimentation in open-source, aided by AI, could further reduce reliance on proprietary software. Robotic Process Automation (RPA) bots can also be exposed as services via MCP, allowing companies to integrate automated tasks into their workflows more easily. This further enhances the 'build' option, as businesses can employ pre-built RPA services to automate repetitive processes.

7. Cloud vs. On-Premise: A Re-evaluation

The potential for AI-driven, easier on-premise app development could indeed have significant implications for the cloud versus on-premise landscape, potentially leading to a shift in reliance on big cloud applications like Salesforce.

There's potential for reduced reliance on big cloud apps. If AI tools drastically simplify and accelerate the development of custom on-premise applications, companies that previously opted for cloud solutions due to the complexity and cost of in-house development might reconsider. They could build tailored solutions that precisely meet their unique needs without the ongoing subscription costs and potential vendor lock-in associated with large cloud platforms.

Furthermore, for organizations with strict data sovereignty requirements, regulatory constraints, or internal policies favoring on-premise control, the ability to easily build and maintain their own applications could be a major advantage. They could retain complete control over their data and infrastructure, addressing concerns that might have pushed them towards cloud solutions despite these preferences.

While cloud platforms offer extensive customization, truly bespoke requirements or deep integration with legacy on-premise systems can sometimes be challenging or costly to achieve. AI-powered development could empower companies to build on-premise applications that seamlessly integrate with their existing infrastructure and are precisely tailored to their workflows.

Composable architectures can also make on-premise development more manageable. Instead of building large, monolithic applications, companies can assemble smaller, more manageable services. This can reduce the complexity of on-premise development and make it a more viable option.

Additionally, while the initial investment in on-premise infrastructure and development might still be significant, the elimination of recurring subscription fees for large cloud platforms could lead to lower total cost of ownership (TCO) over the long term, especially for organizations with stable and predictable needs.

Finally, some organizations have security concerns related to storing sensitive data in the cloud, even with robust security measures in place. The ability to develop and host applications on their own infrastructure might offer a greater sense of control and potentially address these concerns, even if the actual security posture depends heavily on their internal capabilities.

However, several factors might limit the shift away from big cloud apps:

The "As-a-Service" Value Proposition

Cloud platforms like Salesforce offer more than just the application itself. They provide a comprehensive suite of services, including infrastructure management, scalability, security updates, platform maintenance, and often a rich ecosystem of integrations and third-party apps. Building and maintaining all of this in-house, even with AI assistance, could still be a significant undertaking.

Moreover, major cloud vendors invest heavily in research and development, constantly adding new features and capabilities, often leveraging cutting-edge AI themselves. This pace of innovation in the cloud might be difficult for on-premise development, even with AI tools, to keep pace with.

Cloud platforms are inherently designed for scalability and elasticity, allowing businesses to easily adjust resources based on demand. Replicating this level of flexibility on-premise can be complex and expensive. Many companies prefer to focus on their core business activities rather than managing IT infrastructure and application development, even if AI makes it easier; the "as-a-service" model offloads this burden.

Large cloud platforms often have vibrant ecosystems of developers, partners, and a wealth of documentation and community support. Building an equivalent internal ecosystem for on-premise development could be challenging. Some advanced features, particularly those leveraging large-scale data analytics and AI capabilities offered by the cloud providers themselves, might be difficult or impossible to replicate effectively on-premise.

Cloud providers might also shift towards offering more granular, composable services that can be easily integrated into various applications. This would allow companies to leverage the cloud's scalability and infrastructure while still maintaining flexibility and control over their applications.

Therefore, a more likely scenario might be the rise of hybrid approaches, where companies use AI to build custom on-premise applications for specific, sensitive, or highly customized needs, while still relying on cloud platforms for other functions like CRM, marketing automation, and general productivity tools.

While the advent of AI tools that simplify on-premise application development could certainly empower more companies to build their own solutions and potentially reduce their reliance on monolithic cloud applications like Salesforce, a complete exodus is unlikely. The value proposition of cloud platforms extends beyond just the software itself to encompass infrastructure management, scalability, innovation, and ecosystem.

Companies will likely weigh the benefits of greater control and customization offered by on-premise solutions against the convenience, scalability, and breadth of services provided by the cloud. We might see a more fragmented landscape where companies strategically choose the deployment model that best fits their specific needs and capabilities.

8. The AI-Driven Software Revolution: A Summary

The integration of advanced AI into software development is poised to trigger a profound shift, fundamentally altering how software is created, acquired, and utilized. This shift is characterized by:

1. Exponential Increase in Productivity and Innovation:

AI as a Force Multiplier: AI tools are drastically increasing developer productivity, potentially enabling the same output with half the staff or even leading to a fivefold increase in efficiency in the near term.

Cambrian Explosion of Software: This surge in productivity will likely lead to an explosion of new software products, features, and niche applications, democratizing software creation and lowering the barrier to entry.

Rapid Iteration and Personalization: Software will evolve at an unprecedented pace, with AI facilitating fast feature iteration, updates, and highly personalized user experiences. This will often involve complex LLM workflow orchestration to manage and coordinate the various AI-driven processes.

This impact will be felt across various types of software, from enterprise solutions to consumer apps, scientific tools, and embedded systems. The effectiveness of these AI tools relies heavily on the quality of their training data, and the ability to analyze user data will drive further innovation and personalization.

We must also consider the sustainability implications, including the energy consumption of AI models and the potential for AI-driven software to promote resource efficiency in other sectors. These changes are not static; they are part of a dynamic and rapidly evolving landscape. Tools like GitHub Copilot and AWS CodeWhisperer are already demonstrating the power of AI in modern development workflows.

2. Transformation of the Software Development Landscape:

Evolving Roles: The traditional role of a "coder" will diminish, with remaining developers focusing on AI prompt engineering, system architecture, including the design and management of complex LLM workflow orchestration, integration, service orchestration, MCP management, quality assurance, and ethical considerations.

This shift is particularly evident in the rise of vibe coding. More significantly, we're moving towards a one-to-many model where a single subject matter expert (SME) or senior engineer will manage and direct many LLM coding agents, each working on different parts of a project. This orchestration of AI agents will dramatically amplify the impact of senior engineers, allowing them to oversee and guide complex projects with unprecedented efficiency.

AI-Native Companies: New companies built around AI-driven development processes will emerge, potentially disrupting established software giants.

Democratization of Creation: Individuals in non-technical roles will become "citizen developers," creating and customizing software with AI assistance.

3. Broader Economic and Societal Impacts:

Automation Across Industries: The ease of creating custom software will accelerate automation in all sectors, leading to increased productivity but also potential job displacement.

Lower Software Costs: Development cost reductions will translate to lower software prices, making powerful tools more accessible.

New Business Models: New ways to monetize software will emerge, such as LLM features, data analytics, integration services, and specialized composable services offered via MCP.

Workforce Transformation: Educational institutions will need to adapt to train a workforce for skills like AI ethics, prompt engineering, and high-level system design.

Ethical and Security Concerns: Increased reliance on AI raises ethical concerns about bias, privacy, and security vulnerabilities. This includes the challenges of handling sensitive data when using AI tools.

4. Implications for Purchasing Software Today:

Short-Term vs. Long-Term: Businesses must balance immediate needs with the potential for cheaper and better AI-driven alternatives in the future.

Flexibility and Scalability: Prioritizing flexible, scalable, and cloud-based solutions is crucial.

Avoiding Lock-In: Companies should be cautious about long-term contracts and proprietary solutions that might become outdated quickly.

5. Google Firebase Studio as an Example:

AI-Powered Development: Firebase Studio's integration of Gemini and AI agents for prototyping, feature development, and code assistance exemplifies the trend towards AI-driven development environments.

Rapid Prototyping and Iteration: The ability to create functional prototypes from prompts and iterate quickly with AI support validates the potential for an explosion of new software offerings.

In essence, the AI-driven software revolution represents a fundamental shift in the "build versus buy" calculus, empowering businesses and individuals to create tailored solutions more efficiently and affordably. While challenges exist, the long-term trend points towards a more open, flexible, and dynamic software ecosystem. It's important to remember that AI is a tool that amplifies human capabilities, and human ingenuity will remain at the core of software innovation.

9. Conclusion: A More Open and Dynamic Software Ecosystem

In conclusion, the advancements in AI are ushering in an era of unprecedented change in software development. This transformation promises to democratize software creation, accelerate innovation, and empower businesses to build highly customized solutions. While challenges remain, the long-term trend suggests a move towards a more open, composable, flexible, and user-centric software ecosystem, increasingly driven by discoverable services. Furthermore, the pace of these changes is likely to accelerate, making adaptability and continuous learning crucial for both businesses and individuals.

  • in AI, LLM, RAG
  • 5 min read

The Limits of RAG: Why It Fails in Unconstrained AI Applications

Introduction

RAG (Retrieval Augmented Generation) has gained popularity as a technique to enhance LLMs by retrieving information from external sources. However, this approach has significant limitations. This article argues that RAG, as it is currently conceived and applied, is fundamentally flawed for open-ended, unconstrained problems. While it may have niche applications in highly controlled environments, its inherent limitations make it unsuitable for the majority of real-world AI use cases. In many cases, RAG is inappropriately used when an agent-based approach would be more suitable. Model Context Protocol (MCP) offers a more promising way forward.

The Limitations of RAG

The core flaw of RAG goes beyond the "garbage in, garbage out" problem. The unconstrained nature of user input, especially in conversational interfaces, creates a fundamental challenge for retrieval systems. Even with vector search, which aims to capture semantic similarity, RAG struggles with nuanced queries and often disregards crucial metadata, leading to inaccurate or irrelevant results. The chat interface inherently encourages open-ended queries, creating an unbounded input space. Retrieval systems, even with adaptive learning, rely on the assumption that the space of possible queries is finite and predictable. When that assumption breaks, so does the system.

To understand RAG's limitations, it's helpful to categorize common failure scenarios:

Informational Retrieval Failures

While RAG is designed for this, it still fails when the information is nuanced, requires synthesis from multiple sources, or involves complex relationships.

Example: A question requiring understanding of cause-and-effect across documents.

Aggregate Query Failures

RAG struggles with calculations and summaries over a dataset.

Example: "What is the total revenue from product X in Q3?"

Temporal Query Failures

RAG's inability to handle time-based queries and reasoning.

Example: "Show me all the commits that Bob made between March 13th and March 30th, 2020."

Logical Reasoning Failures

While LLMs can exhibit some semblance of logical reasoning, their reliability is questionable. RAG's reliance on retrieved context can further hinder this capability, introducing noise and irrelevant information that throws off the LLM's reasoning process. Given the LLM's inherent limitations in this area, depending on RAG for logical reasoning is a risky proposition.

Example: "If all birds can fly and a penguin is a bird, can a penguin fly?"

Counterfactual Query Failures

LLMs can attempt counterfactual reasoning, but this is a cutting-edge and imperfect capability. RAG adds another layer of complexity, as the retrieved context may or may not be relevant to the counterfactual scenario. The results are often speculative and unreliable.

Example: "What would have happened if World War II had not occurred?"

Multimodal Query Failures

Multimodal queries pose a significant challenge for RAG. Consider the query, "Which animal makes this sound?" where the user vocalizes a kitten's meow. While a human easily recognizes the sound, current RAG systems struggle to process non-textual input. Even if the sound is transcribed, nuances like tone and pitch, crucial for accurate retrieval, are often lost. This highlights RAG's fundamental limitation in handling information beyond text.

Example: "Describe this image."

Business Logic/Policy Failures

RAG systems often fail to adequately incorporate business logic and policies. For example, a chatbot might incorrectly authorize the multiple use of a single-use coupon, leading to financial repercussions. Similarly, a RAG system could provide medical advice that violates healthcare regulations, potentially endangering patients. This is further exacerbated by the fact that the performance of a RAG system in the medical domain can be greatly enhanced with a taxonomy and metadata (i.e., a raw RAG search through medical publications vs. also having a full taxonomy and metadata linking medicines with diseases). This highlights a counterintuitive truth: taxonomies, ontologies, and metadata are more valuable in the age of LLMs, even though LLMs might seem to drive down the cost of producing them.

Furthermore, a RAG application might disclose personally identifiable information due to inadequate data filtering, resulting in privacy violations and legal issues.

Example: A chatbot incorrectly authorizing the multiple use of a single-use coupon.

These examples demonstrate a common thread: RAG struggles when queries require more than just simple keyword matching or semantic similarity. It lacks the ability to effectively utilize structured knowledge, such as taxonomies, ontologies, and metadata, which are often essential for accurate and reliable information retrieval.

Introducing Model Context Protocol (MCP)

Model Context Protocol (MCP) offers a new approach to providing LLMs with the context they need to function effectively. Unlike RAG, which retrieves context at query time, MCP standardizes how models declare their context requirements upfront. This proactive approach has the potential to address many of the limitations of RAG.

MCP as a Solution

MCP offers a more robust and future-proof way to provide context to LLMs. Consider an MCP service wrapped around a traditional SQL database. An LLM agent system, instead of relying on RAG to retrieve potentially irrelevant text snippets, can use MCP to precisely query the database for the exact information it needs. This approach offers several advantages:

  1. Constrained Input: By defining context needs upfront, MCP avoids the problem of unconstrained input. The LLM agent only queries for information that is known to be relevant and available.

  2. Query-Retrieval Alignment: MCP ensures that the query is perfectly aligned with the retrieval mechanism (e.g., a SQL query retrieves structured data from a database). This eliminates the "garbage in, garbage out" problem that plagues RAG.

  3. Structured Context: MCP facilitates the use of structured knowledge sources like databases, knowledge graphs, and semantic networks. This allows LLMs to access and utilize information in a more precise and compositional way, compared to retrieving large chunks of unstructured text.

  4. Reduced Complexity: By providing a standardized protocol for context acquisition, MCP reduces the need for ad-hoc patching and refinement that is typical of RAG systems.

The Power of Structured Knowledge

MCP's ability to leverage taxonomies, ontologies, and metadata is key to its potential. In contrast to RAG, which often struggles to extract meaning from unstructured text, MCP enables LLMs to interact with structured knowledge in a way that is both efficient and reliable. This is particularly important for complex queries that require:

  • Precise Definitions: Taxonomies and ontologies provide clear and unambiguous definitions of concepts, ensuring that the LLM is operating on a solid foundation of knowledge.

  • Relationship Understanding: Structured knowledge captures the relationships between concepts, allowing LLMs to perform complex reasoning and inference.

  • Contextual Awareness: Metadata provides additional context about data points, enabling LLMs to filter and retrieve information with greater accuracy.

Conclusion: The Future of Context

RAG, as it is currently conceived and applied, is fundamentally flawed for open-ended, unconstrained problems. Its reliance on query-time retrieval makes it inherently susceptible to the challenges of unconstrained input, query-retrieval misalignment, and the need for constant patching. MCP offers a promising alternative. By shifting to a proactive approach that defines context needs upfront and leverages structured knowledge, MCP has the potential to provide LLMs with the precise and relevant information they need to function effectively.

Further research and development of MCP and similar protocols are crucial for building robust and reliable AI systems that can truly understand and interact with the world. The future of LLMs and AI depends on our ability to move beyond the limitations of RAG and embrace more structured and controlled ways of providing context.