TL;DR: Because it didn’t turn out the way we intended.
We spent several months building an internal AI product and stopped before launch. It was supposed to give any company a working data agent without custom development. The idea came from a client project we delivered shortly before—one that works exceptionally well. Yet turning that one-off build into something any team could set up for itself turned out to be a different job.
Company pages usually show the projects that worked. We’d like to share this one too, because it shows where an approach like ours breaks and why. Some of the problems came from the platform we built on. Others came from the market, which moved faster than we expected.
If you’re planning to build on a data platform’s built-in AI agent, the five problems below are worth checking against your own plans.
What we wanted to build
The idea was simple: take a use case we know for sure works in practice, add it to our workflows, and, after piloting, improve it and package it as an agent for commercial use.
The setup we started from
We built two AI agents on Databricks for one of our oldest clients, Rule Communication. Both agents are powered by Genie, Databricks’ built-in agent that turns plain-language questions into database queries. They have similar logic but different purposes:
- One answers the CEO’s questions about email campaigns, conversions, and other metrics.
- The other drafts replies for the support team based on past Intercom tickets.
The product idea
We wanted to create something similar for other teams: an AI-powered solution they could use for quick data processing that would work without custom development. Connect your integrations, share your documents, start asking questions, and Genie would do the data work.
One highlight we came up with was a simple interface on top of Databricks, since its native UI is built for engineers and is too complex for the average user. It’s full of SQL tools, database settings, and toggles most business users never need.
An important note: we knew it was a bit of a gamble. A small product on top of a big platform can lose its purpose very quickly. It’s enough for OpenAI or Anthropic to ship a single new feature that can serve a similar purpose, and your idea is done. And that’s exactly what happened later. We counted on specific data-processing needs staying niche, at least for a while. Large vendors started going after niche problems too.
Five reasons we stopped
Our team had an idea and wasn’t eager to give up with the first difficulties. We kept working on the project and solving the challenges that came up until too many piled up.
#1 Our interface couldn’t replace the native one
Databricks doesn’t let third-party clients, like our app, add users. Everyone has to register and log in through the Databricks system itself. Our interface could sit next to the native one, but it wouldn’t be a replacement.
Tracking became a problem too. Databricks keeps detailed logs of who added, wrote, or broke what. With our app in the middle, every action looked like it came from the app, not a specific user. It just looked like information overload, with zero specificity or value.
To solve this, we tagged each request with a user ID on our side, so we could still trace activity. Access to documents and resources still ran through a single account—the app’s, not each user’s personal account.
More workarounds were possible. But each one meant rebuilding a piece of Databricks. We’d planned to reuse a platform, but at some point, we ended up building a new one instead.
#2 We couldn’t say which AI model was answering
Databricks used to let you choose the model behind it: Claude Opus, Claude Sonnet, a Gemini model, or a Databricks model. Now the platform decides. Databricks’ Genie Code announcement says the agent isn’t powered by a single model. A Databricks employee confirmed on the community forum that customers can’t select one manually.
This automatic routing may lower costs over time. However, it leaves you with less control over the process. That interfered a bit with the initial idea. Automatic routing also makes the system less transparent. And when clients ask which model processes their data, we need a clear answer.
#3 Competitors caught up with Genie
Genie explores data and writes SQL from plain-language questions. That was our main reason to build on it. In September, OpenAI launched its own Data agent in ChatGPT Work. It connects to Databricks, Snowflake, BigQuery, Redshift, and other sources. Google made Conversational Analytics in BigQuery generally available even earlier, on July 1st. And we expect Anthropic to follow.
With several leading LLM vendors offering the same thing, our product needed another differentiator. The strongest one we could come up with was the security architecture that makes Databricks highly attractive to compliance-heavy European enterprises.
#4 Usage-based pricing was expensive
We decided to pilot the agent internally before turning it into a commercial offer—both for convenience and to make sure the solution is worth marketing. A custom agent on Databricks is billed by API calls and tokens. For our expected workloads, this came out higher than flat AI subscriptions. So it would be for a great share of the potential users.
The math could change in two cases:
- If AI vendors switch to usage-based pricing or raise subscription prices, the gap narrows.
- Teams with uneven usage pay less overall, because occasional users barely spend tokens.
We obviously don’t have any control over the first condition, and looking for very specific usage patterns makes zero sense.
#5 Data residency wasn’t a strong enough argument
With Databricks, you choose where your data is stored, such as your own AWS instance. Requests, however, are processed on Databricks’ servers by models Databricks hosts. We couldn’t see much of what happens there.
The data doesn’t go to Anthropic, OpenAI, or whatever LLM model it chooses to use for your request. Databricks guarantees it isn’t stored or used for model training. Enterprise AI plans from other vendors make similar commitments, so this didn’t set us apart.
What worked instead
Around the same time, we launched a much smaller internal project that actually did what we needed.
Vilmate holds ISO 27001 certification, which comes with a lot of documentation: policies, process descriptions, lists of people and devices, access records for tools and interfaces. After our last audit, we got a long list of required fixes and recommendations in Jira. Handling them all by hand took more time than we had.
So we set up a shared Claude agent. Its instructions describe how to work with the documents and where they live: a specific Google Drive folder and a specific Jira project. It runs in two modes:
- Change mode rewrites documents to close audit findings.
- Check mode reviews finished work.
We can ask an AI agent to open a Jira ticket, and it compares the ticket with the documents and recent changes. Then it comments in Jira or replies in the chat on whether the task is actually done.
This setup costs little more than a standard subscription and took a few weeks to configure. For documentation work, a data platform with usage-based billing would have been far more than we needed. With large volumes of business data, like Rule’s, the balance looks different.
Questions to ask before you build a custom AI agent
These are the questions we’d answer first if we started over. Your answers may lead somewhere different from ours.
- Can your users stay in your interface? Check whether the platform lets third-party apps add and manage users, and how its audit logs will identify them.
- Do you control the model? If clients or auditors ask which model processes their data, make sure you can answer and that the answer won’t change without notice.
- How unique is the core feature? List the vendors that offer something similar today, and the ones likely to soon.
- What will usage cost? Compare token-based pricing with flat subscriptions for your real usage pattern, including your lightest and heaviest users.
- How much data will the agent work with? Large volumes of structured data can justify a data platform. Documents and tickets may need much less.
- What does data residency actually cover? Find out where requests are processed, not only where data is stored.
Experiment with AI but keep expectations grounded
This isn’t a success story, but we wouldn’t call it a failure either. No one refers to AI and ML as “emerging technologies” anymore, but most companies are still discovering, learning, and experimenting with them—despite the AI race that is already going.
The lesson we learned is to validate use cases carefully: what works for your peers and competitors is not always relevant for you, even if it seems it does. With that in mind (and out of the company’s portfolio for now), we want to help businesses experiment and invest more carefully to focus on projects that drive value for their unique cases (even if those don’t seem so unique at first).
So, if you’re interested in discussing an AI feature, process automation, or another idea for custom development, contact our team.
