Real Time Intelligence for the Rest of Us with Microsoft Fabric
In the wake of the Build conference back in May, we highlighted how Microsoft Fabric was ramping up its real time intelligence capabilities. We’re now in November, and at its Ignite conference, Microsoft is significantly broadening Fabric’s reach, from a data analytics platform to simply a data platform that also includes transactional and operational use cases. The first database engine available as a workload in Fabric is unsurprisingly SQL Server, with more to come next year. We will come back to what this means in the months to come, but it goes to show how ambitious Microsoft is with Fabric.
Addressing real time scenarios is very much part of those ambitions, and Fabric’s Real-Time Intelligence (RTI) reached its General Availability (GA) milestone today. This includes Activator, one of the few workloads that didn’t go to GA at Ignite a year ago unlike Fabric as its underlying platform. It is now more scalable and can handle a broader variety of events.
Between Build and today, the RTI stack kept improving, with a slew on new connectors, an improved sample gallery, or better pause/resume capabilities in EventStreams. Another great recent addition has been Eventhouse platform monitoring via a read-only KQL database to report on query logs, ingestion metrics, and ingestion failures.
As of today, several other components of the RTI ecosystems are seeing another series of improvements. New Fabric event categories in the Real-Time Hub including OneLake events and job events tighten the integration between the speed/stream layer and batch/storage layer of the Lambda architecture. The lack of CI/CD support for RTI artifacts was a significant gap that we’re glad to see has now been addressed with Git support for Git and deployment pipelines. And OneLake query acceleration makes it much faster to blend reference dimensions with real-time streams.
The Real-time Dashboard (RDB) workload is still in preview, but its permission model is getting more granular. This functionality nicely complements Power BI reports, and both can be included in the new org apps.
Making Event-Based Data Actionable and Accessible
Fabric’s RTI offering stands out for its ease of use, lower barrier to entry, and more gentle learning curve than other streaming-based data platforms that can, frankly, be quite intimidating. This ethos is one of our favorite aspects of Fabric.
Elsewhere in Fabric, ease of use is sometimes offered with somewhat of a cost/performance tradeoff, as is the case with Dataflows Gen2. That is not the case here, as Fabric’s RTI bones are very sturdy. Fabric’s Eventhouse is grounded in Azure Data Explorer (ADX aka Kusto), the tool that Microsoft uses to store and query its own client telemetry data at massive scale. There’s a good reason why they’re called a “hyperscaler”!
This technology is not only scalable beyond what most organizations would ever need, it’s also remarkably cost-effective. Companies that shied away from event-based data ingestion because of cost and complexity concerns will do well to give it another look, and ask themselves what might be the opportunity cost of sitting on their data before acting.
What Can Happen in an Hour?
Nightly batches are OK to fuel your longer-term retrospective reporting, but they feel antiquated in light of what today’s technology can accomplish. Granted, incremental or otherwise partitioned processes can shorten that time to, say, a refresh every hour. But all sorts of events that may affect your operations – from natural or man-made disasters to more mundane order cancellations – don’t hold their breath to accommodate your delayed batch data processes.
When a container ship blocked the Suez Canal for six days back in 2021, the running losses were estimated to $400 million an hour, according to Lloyd’s List and CNBC. Even at the scale of a single importer, costs added up quickly. Less spectacular but nonetheless costly transportation rerouting is required in several places around the world pretty much on an ongoing basis.
Logistics is but one example. Other scenarios that benefit from an event-based approach to data include fraud detection and short-term supply and demand rebalancing, such as staffing needs, commodity trading, or yield management.
In particular, we are seeing a heightened interest from our client in labor-intensive businesses such as Quick Service Restaurants (QSR), in light of rapidly increasing minimal wages in states such as California. Companies in the United States are now looking much more closely at the mix between labor and capital, following the pattern already seen in countries such as Japan or France were vending machines replaced entry-level retail jobs long ago.
There are technical benefits too in ingesting data from streaming sources. If your systems are strained by massive batch refreshes, and if you’re slow in catching data errors, it would be worth looking more closely at the Lambda architecture.
Now What?
