ChatGPT ruined it for application developers. Contrary to the hype, building and deploying AI applications has become harder, not easier. The percentage of companies that have seen ROI in their AI investments is still low. Find out why and what you can do to be among the few who have become successful.
ChatGPT stormed on to the scene by providing a chat interface in which users can ask questions and get meaningful answers within seconds. Its power was undeniable.
OpenAI’s user growth skyrocketed. However, ChatGPT fostered a mindset that a user can enter any random query into a chatbox and get an accurate answer 100% of the time. Many discovered quickly that the reality does not meet expectations and as a result, there were several instances where companies that followed this model had many failures that caused financial and reputational damage.
To combat these issues, many improvements were made, including the notion of agentic AI development. And yet, many applications still throw everything behind a chatbox, as companies scramble to replicate OpenAI’s success.
The problem with the open-ended chatbox
The range of questions that users can ask has no boundaries: This leads to agents having to do the heavy lifting of not only guessing intent but also figuring out which data source(s) to use to answer the question. This increases the odds of errors and hallucinations.
Implementing guardrails and securing applications becomes too complex: Following the previous point, if the agent is expected to answer any query from the user, the chances of data and PII leakage are extremely high. RBAC and other compliance policies increase in complexity.
Users don’t want to chat, they want solutions: Writing free-form text into a chatbox creates friction for users every time the agent fails to provide accurate results, forcing users to chat endlessly struggling to find the right words to describe what they want.
Rethinking the chatbox
The chatbox makes sense for companies such as ChatGPT and Google that have an army of engineers and enormous capital to develop and support a complex architecture behind a chat interface for what is essentially an open-ended AI search with Q&A capabilities. For many companies, however, broad AI search/Q&A does not represent their core strategic advantage.
In fact, AI strategy is not separate from product strategy. The goals are the same as they have always been: reduce friction and drive business outcomes. AI is simply a new tool that can be used to achieve these goals. Therefore, rather than copying OpenAI and Google’s business models, companies should be thinking in terms of what will make them successful. The most successful companies have used AI to automate the boring, repetitive, and inconsistent tasks so that they can either focus on their strategic advantages or unlock new business opportunities.
This requires moving away from using AI agents as “conversationalists” to using AI agents to complete discrete tasks with pre-defined goals and KPIs. These agents outperform general-purpose models by focusing on narrow, reliable business objectives. Examples of tasks include:
- Summarize cybersecurity incident reports
- Identify action items from a customer conversation
- Extract business risks from a 10-K
- Analyze market trends from the document
Task-specific agents have the following benefits:
- Prompts no longer need to include complex routing logic to determine the user’s intent. They can be tailored to the specific task, which reduces the complexity of the reasoning required.
- Simplify the organization, cataloging, and vectorization of the specific subset of data that is needed when the task is bounded.
- Clearly defined and bounded data is easier to lock down as the surface area for data leaks is reduced significantly, paving the path to establish a strong data governance practice.
- Data retrieval strategies for context engineering become simplified and less resource-intensive.
- Input and output guardrails used to protect from hallucinations and malicious usage are more effective because they can be designed to ensure that the agent only operates within the task boundary. With an open-ended application, bad actors may still circumvent the most sophisticated routing policies.
- Defining and tracking KPIs is easier when an application is bounded to a specific task.
- Token and infrastructure costs decrease significantly.
Tasks can also be chained together into workflows but the principle remains the same. Rather than having one “super-agent” that handles everything, applications can chain multiple agents together to perform specific tasks in the workflow. Effectively, we are evolving from conversational UI to full-scale applications powered by AI agents.
The human in the loop
A critical component of building task-driven applications is the human in the loop. While agents are extremely powerful and can be used in a larger variety of business use cases, they often fail to solve the last mile problem. Depending on the level of risk, having a human control loop is essential to handle edge cases and prevent catastrophic failures.
How you can get started
Start small. Analyze operations in your company and do the following:
- Identify a high-friction, repetitive task
- Define the domain and the data requirements for the task
- Establish the metrics and KPIs defining what good looks like for the task
- Create the input and output guardrails
- Run experiments with different settings, then evaluate the output before deploying to production
- Log and version all prompts, models, model settings, sources, and metrics used in every interaction
These are the lessons that we at Rotational have learned through building AI applications for our clients over the past five years. This is why we built Endeavor, an AI management platform, to help companies build cost-effective applications that drive productivity and unlock new revenue streams.
Image generated using Google Gemini
