10/31/2024
Hot take but AI Assistants aren’t living up to the hype!
Despite the buzz, closed-domain AI assistants are falling short. Without reliable, context-aware responses, they’re not ready for serious business use.
Where AI Assistants Fail
Here’s a scenario from a well known sales assistant that’s out there today:
📊 User Query: “What’s the length of my average sales cycle?”
➡️ Assistant Response: “I calculated the average sales cycle length for your opportunities, but there are no results to show.”
The assistant can’t perform a computation. Why? Let’s break it down.
🛑 The Issue: Closed-domain AI assistants rely heavily on search-first algorithms, making them unsuitable for high-trust applications.
Consider a task like “Find all emails from last week that need follow-ups.” A search-based AI might skip important messages if they lack specific keywords, leaving critical follow-ups unnoticed. When this incomplete data is passed to the language model, the result is unreliable, making these assistants ill-suited for nuanced business queries.
✅ The Solution: Agentic query planning. Instead of rigid keyword search, assistants should gather all relevant emails and then use an LLM to classify follow-ups—just as a person would—ensuring accuracy.
PromptQL is currently available in Alpha ➡️ https://bit.ly/3C5c5Qi
We’re also releasing the Agentic Data Access Benchmark.
We built a dataset across 5️⃣ closed domains and put popular assistants to the test:
💡They could only handle the simplest questions.
⚠️ ~80% of real-world, medium-to-high complexity questions performed poorly.
Here's the full agentic data access benchmark ➡️ https://github.com/hasura/agentic-data-access-benchmark
We’re showcasing PromptQL today at our launch event happening now (31st Oct) !
Lots of demos in store and some comparisons too 🍿
Register here for Hasura Dev Day ➡️https://bit.ly/40lymmZ
