Answer Questions Beyond SQL introduced ontology-grounded reasoning: four reasoning types composing one model of your business. Making Structured Reasoning Accessible with AI showed agents making that workflow usable by your data team.
This post: how you know it works, and what makes it better over time.
No business question arrives alone. "Who are our high-value customers, by lifetime spend?" leads to "which suppliers do they transitively depend on?".
That leads to "which suppliers will likely delay next quarter?", which leads to "what's the minimum-cost routing plan that blocks the unreliable ones?", which leads, the day a hub goes offline, to "what changes, and why are the deltas asymmetric?".
Chains like this are hard for agents: each step needs knowledge of which data to use and what to trust – knowledge rarely captured in the database or the context. The SQL may compile, but the answer can still be wrong.
Agents are now the main users of RelationalAI. They've opened up ontology development and advanced reasoning to data teams, enabling them to answer new questions and work more efficiently – but only if their agents (Snowflake Cortex, ChatGPT, Claude Code, and more) are equipped with the right resources (context, skills, tools, examples) and can reliably deliver.
Our answer is the one the AI industry converged on: evals. Point benchmarks exist for pieces of this – text-to-SQL accuracy, single-task agent scores – but they don't measure what makes the chain above hard: context carried across steps, earlier answers consumed by later ones, messy data, a switch between reasoning approaches.
We measure the way customers experience RelationalAI: the workflow a user and their agent run together, over the representative data lifecycle supporting it. Because the system under test reasons over a shared ontology, we can grade the whole journey across one model of the business: the customers, suppliers, shipments, and rules the questions require. Each obstacle the first post named – the disposable formulation, the expertise bottleneck, the data disconnect – becomes something a run can pass or fail.
We demonstrate the evals on a full supply chain journey at enterprise scale. The test cases come from target business problems and the grades feed back into the source data, the ontology logic, and the agent and user resources.
Every enterprise evaluating agentic analytics should put seven questions to any vendor grading their own product, including us – and to your own harness if you're building the agent layer yourself:
(What we grade)
1. Do you grade whole workflows, or isolated queries?
(What we test)
2. Where do your test cases come from?
3. What data do you test on?
(How we score)
4. What does "good" mean for each person who will use this?
5. How do I know you're not grading yourself on a curve?
6. What are your goals? Do you score cost and time, or just success?
(What happens next)
7. What happens when a test fails?
And here's how we answer them.
Whole workflows, not isolated queries (question 1): we grade the customer workflow step by step, each step's output feeding the next.
A recent weekly run on a synthetic supply chain dataset and ontology:
From each run we report completion (the run executed end to end), success (the outcome checks passed), and runtime; the numbers point to improvements in the agent's resources, model and harness selection, and the underlying product.
The latest run surfaced two scale problems. A path question that behaved on a small graph timed out at production scale: enumerating routes explodes combinatorially on the full network. Tighter scoping in the prompt fixed the re-run; a weighted shortest-paths algorithm is on the roadmap.
Model training hit its own ceiling: defaults tuned for small graphs outgrew the available GPU, so we updated the skill guidance and the agent now self-corrects.
Both failures surfaced in run telemetry before any answer was wrong. A correctness-only eval would have recorded "unanswered," with no signal about why. With those fixes in, the journey passes end-to-end at enterprise scale.
Where do test cases come from, and what data do we run them on (questions 2 and 3)? You bring questions and user feedback, not evals: your questions become the eval set. Turning them into test cases grades the product on your context and replaces vibes with evidence.
A target use case inspires a template runbook – the ordered prompts a user would actually run, with expected outputs – which becomes eval test cases built on an ontology. Here are some examples:
The eval follows the agent's workflow as it generates and queries the ontology across rules, graph, predictive, and prescriptive reasoning.
A field issue becomes a regression case: reproduce it as a question with an expected outcome, fix it, keep the case so the failure cannot return silently. New cases slot into the journey where their dependencies live – a reported optimization failure joins the chain after the reliability tiers that feed it.
The suite is also seeded ahead of the product. Working backward from questions you need answered – including ones current capabilities can't yet handle – produces cases that fail by design, each naming the capability that would close it.
Eval realism climbs a ladder:
When any vendor shows you eval results, ask which rung they ran on.
Our eval ran on a large-scale and intentionally messy synthetic dataset inspired by our customers, so we can test representative use cases without touching customer data. Scale punishes approaches tuned on samples; mess – the duplicates, orphan keys, mixed formats, and inconsistent encodings every real warehouse carries – is injected deliberately and documented.
What "good" means (question 4): each stage is graded against the bar of the person it serves.
How do you know we're not grading ourselves on a curve (question 5)? Any vendor grading their own product has an incentive problem, so we grade under controls that make self-reported results worth trusting: the system runs blind, grading is by outcome, the product does the work, and every dimension scores separately.
The system under test sees only what a real user's session would contain – the base model, the public skills, its own prior outputs, live data – with expected answers and worked solutions withheld. Grading is by outcome: a query matches the expected result, a reachability or centrality result is correct, a rule fires on exactly the right entities, an optimization is optimal and non-trivial, a prediction beats the base rate on held-out data. Foundation artifacts like an ontology or a trained model are graded by what they enable downstream, since many valid shapes answer the same questions.
The answer must come from the product under pressure – a graph question answered by offloading to a general-purpose library fails, even when the numbers come out right. Every question in a run is scored on the triple – success, cost, and time – with retries and timeouts recorded beside the graded answer. Honesty is a gate: uncertainty stated plainly, nothing leaked from the withheld answers. A correct answer that blows its time or cost budget fails; a fast answer that leaked context fails. And every dimension scores separately: a single number never hides which one failed.
Which model and harness? The same suite settles it: hold the questions constant and swap the model, the agent harness, or the skill version, and the grades say which combination answers best, at what cost and time, for your workload.
The comparison extends to approaches, too. With the questions and expected answers held constant, the grades show what each stack lets you evaluate:
When a test fails (question 7), the graded journey becomes two improvement backlogs: yours and ours. Every failing run carries a root-cause label, so the backlog aggregates by failure mode instead of anecdote.
Your loop is what the grades tell you to enrich. Answers improve when the model has more to reason over, and a graded journey shows exactly where yours is thin.
Our loop is one harness across product layers. A failed step flags the layer that owns it – agent guidance, platform context, reasoning capability, engine performance – so every team can improve the product based on the same suite. Capability gaps work the roadmap backward from your questions: a new graph algorithm integrated, a new predictive method supported, sensitivity analysis added. The suite compounds: the journeys that grade guidance today become tomorrow's regression tests.
You can generate a starter ontology for your dataset using the RelationalAI skills in your coding agent. Our discovery skill then takes that ontology and returns reasoning questions that help solve your specified business problem. It flags data and ontology gaps and is a ready source of eval test cases.
You can also adapt a template to your domain and data. This is what a structured POC should be: a graded journey on your questions and your data, instead of a demo on cherry-picked ones.
The playbook:
When an answer disappoints, the grades say whether to enrich the ontology, feed it better data, or sharpen a definition where the workflow is thinnest.
Bring us three business questions your team can't answer today, and we'll run a graded journey on your data: your questions grounded in your schema, with answers scored on corectness and speed. Get in touch.
Already in the product? Start from a template runbook on your own schema, install the agent skills, and try the journey's question patterns against your data.
Start building with the Rel API, install the native app inside Snowflake, or get in touch about access to our ready-to-use Rel experience.
Rel is a decision agent aligned to your business, grounded in your semantic model, and powered by the advanced reasoners of the RelationalAI decision intelligence platform.
©2026 RelationalAI. All rights reserved.
