Conversational Analytics in Practice: Microsoft Fabric vs. a Self-Hosted Approach

Business analysts want answers. Natural-language Q&A, anomaly flags, forecasts, and charts, all from the same question, in a single interface, is what conversational analytics promises. The technology works. The harder question is what to build behind it.

Today there are two credible ways to deliver this: a fully managed platform like Microsoft Fabric, or a self-hosted stack you assemble yourself from open-source pieces and whichever LLM you choose. The right answer depends on the client sitting in front of you, their data residency requirements, their ops capability, their existing Microsoft footprint, and their tolerance for vendor lock-in.

So, we built the same capability both ways and benchmarked them against each other, using identical inputs:

• Same dataset: The M5 Walmart retail sales benchmark
• Same three business questions: Which stores have the highest total unite sales? Which SKUs have unusual sales patterns? Forecasting sales for the next 8 weeks?
• Same evaluation criteria: Accuracy, latency, precision/recall on anomalies, and forecast error

The Dataset

We used the M5 retail forecasting dataset, the standard academic benchmark for demand forecasting:

• 3,049 SKUs
• 10 stores across 3 states
• 58.3 million daily-sales rows‍
• 1,913 days covered (January 2011 to April 2016)

Both builds saw identical inputs. The dataset also generalizes further than retail. Swap "SKU" for "ATM" and "units sold" for "transactions," and the same anomaly-detection and forecasting techniques apply directly to predictive maintenance and telemetry-driven work.

Two Architectures, Same Job

Microsoft Fabric

Fabric ships less because Microsoft ships more. The whole build is a workspace containing a Data Agent, a KQL Database with a shortcut into a Lakehouse, and a handful of notebooks for statistical evaluation, all running on Microsoft-managed capacity.

The flow: a user asks a question in the Data Agent chat, Copilot (running on GPT-5.1) generates the KQL, the KQL Database executes it via a shortcut, and the result comes back narrated in plain English. Anomaly detection and forecasting run inside notebooks using KQL's native series decompose functions.

Data physically lives once, in OneLake. There's no moving it between storage and compute layers.

Self-Hosted

The self-hosted build is a React UI talking to a FastAPI backend, which hands off to Vanna, a text-to-SQL framework built on retrieval-augmented generation (RAG). Vanna retrieves context from a Chroma vector database (trained on the schema, documentation, and sample question-SQL pairs), asks an LLM to generate SQL, and executes it against Postgres. Anomaly and forecast questions route to dedicated Python services (PyOD and StatsForecast) instead of SQL. The final result passes through the LLM once more for a narrated answer, which the UI renders as markdown, a table, and an auto-generated chart.

Everything except the LLM endpoint lives in a single Docker Compose file. The whole stack ships as one artifact and runs anywhere Linux runs.

The architecture diagrams look different, but the shape of the response, a narration, a table, an auto-generated chart, was designed to be nearly identical across both systems. This is what lets us run an apples-to-apples benchmark instead of comparing two different products.

Results

Natural Language to Query

Both platforms hit roughly an 80% baseline on their own training. After giving both agents the same 21 training examples, Fabric edged ahead at 86.7% versus self-hosted’s 80%.

Both agents made the same early mistake. They misread “by month” as month-of-year, a fixable ambiguity with one training example. Fabric’s Data Agent obeyed the fix deterministically. Self-hosted’s Foundry endpoint had real run-to-run variance: the same training that fixed one question broke another, on identical inputs across three runs. That variance isa property of serverless LLM inference at temperature zero. When we’re advising a client, a system that responses predictably to tuning is worth more than as ix-point accuracy lead.

Anomaly Detection

Recall is close, within four points, but the two methods have very different profiles. Self-hosted's ECOD is a distribution-based method that casts a wide net: higher recall, including some inventory drops, at roughly 3x the noise. Fabric's line fit approach is more selective: it catches almost every spike, at higher precision, but is structurally blind to drops to zero on sparse retail data.

They're different tools tuned for different questions, and both were run untuned. Client-specific tuning would move either baseline.

Forecasting

Overall MAPE makes Fabric look considerably worse, 78.61% versus self-hosted's 60.40%. But that number is dragged up almost entirely by two SKUs where Fabric's single Holt-Winters model fails badly. On the other eight, Fabric's median per-SKU MAPE is 48.01%, actually better than self-hosted's overall number.

Self-hosted's three-model ensemble (Naive, AutoETS, AutoARIMA) smooths away outlier weeks that break a single-model approach. When you can afford ensembling, it wins on outlier-heavy series. On well-behaved series, Fabric's native forecast is competitive on its own.

Trade-offs

Self-Hosted

Pros: data residency (nothing leaves your network), egress-denied capable with Ollama on GPU, swappable LLM (a single environment variable), no per-query LLM cost lock-in, full control over algorithms and thresholds, ensemble forecasting for robust tails.

Cons: you own runtime, patching, and monitoring; CPU-only LLM latency is painful without a GPU; native security (SSO, row-level security) has to be built; alerting needs its own plumbing; Foundry endpoint variance erodes tuning; baselines need per-client work.

Microsoft Fabric

Pros: fully managed, no infrastructure to run; native security (Purview, RLS, ACLs); native anomaly and forecast functions in KQL; Copilot NL-to-KQL out of the box; native alerts and dashboards; GPT-5.1 responds predictably to tuning.

Cons: F-SKU capacity cost is fixed and high; vendor lock-in; no egress-denied option; data must live in OneLake; LLM choice is limited to Fabric's Copilot; single-model KQL forecasting lacks ensembling.

The real differentiator is who owns operational risk. Self-hosted trades your team's time for control. Fabric trades a fixed, often higher cost for someone else owning uptime, patching, and security surface. Most client conversations resolve faster once you frame it that way instead of walking through every row of the pros/cons table.

Decision Framework

Recommend self-hosted when:

• Data cannot leave the client's network
• The LLM must be swappable or open-weight
• The client has infrastructure and ops capability in-house
• Egress restrictions exist
• Cost-per-query predictability matters
• Forecasts need to hold up on outlier-heavy series

Recommend Fabric when:

• The client is already Microsoft-standardized
• Managed operations is a hard requirement
• Native security is a priority
• Time-to-value matters more than customization
• Data can, or already does, live in OneLake
• The team wants predictable, rule-driven tuning

Next Steps

If you're evaluating this for your own organization, here are some recommendations: ‍

• Start with your constraints, not the benchmark. Data residency and egress requirements will eliminate one option before accuracy numbers matter at all.
• Don't over-index on the headline accuracy gap. A six-point difference mattered less in our testing than which system responded predictably to tuning.
• Check whether your forecasting needs ensembling. If your data has outlier-heavy series, a single-model approach can produce a few catastrophic misses that a wide revaluation window would hide.
• Budget for what each option costs you. Fabric's cost shows up as fixed capacity pricing; self-hosted's shows up as engineering time for security, alerting, and tuning.

Need help figuring out what works for you?Bring us your limitations and we can map them to a couple of AI models todecide which will work best.