Real-Time Intelligence, the next evolution of Real-Time Analytics and Data Activator
In today’s data driven world, businesses are increasingly turning to platforms that would help them derive actionable insights faster to enable informed decision-making. The sooner they have access to their data, the faster they can act and gain a competitive edge.
Microsoft Fabric was released for public preview in May 2023 and has been in GA (generally available) since November 2023. It really has been a game changer in how it has brought everything from data, reports, insights to AI under one SaaS platform. Tools that existed earlier in the Azure ecosystem were enhanced, some new concepts brought together to create a holistic platform that has made end to end BI simple and efficient.
Within Fabric, you have the Real Time Analytics experience that houses the Eventhouse (cutting edge database workspace that stores and manages event-based data in KQL databases), Eventstreams, and Data Activator (reflex items).
Customers have been using and leveraging Real Time Analytics in Fabric for multiple use cases across industries from inventory in retail, fraud analytics in finance, route optimization in transportation and logistics, predictive maintenance in manufacturing to name a few.
In the first edition of Real Time Analytics, with Eventstreams, you had the below connectors to get data into a Lakehouse, KQL database or reflex item (Data Activator).
iLink’s experience on Microsoft Fabric Real Time Analytics
At iLink, we had the opportunity to prove out conceptually a near real-time architecture leveraging Microsoft Fabric’s Real Time Analytics for a large Quick Service Restaurant (QSR) retail chain. The problem statement included a scenario to retrieve data in near real time from the point-of-sale (POS) devices in the restaurants and have analytics available in under 10 minutes.
Problem Statement
Their current architecture employed AWS Kinesis data streams routing Json messages to an S3 location using Glue jobs. This existing architecture had a current latency of 30-45 minutes depending on the time of day for volume surges.
The task cut out for Fabric Real Time Analytics was to retrieve data from S3 to begin with as there wasn’t an out of the box connector directly to Kinesis at the time of implementation.
The Solution
As a proof of concept, before the availability of a Kinesis connector, we simulated streaming from S3 into Eventhouse with the below process:
- Leveraging the power of shortcuts to external cloud sources like Amazon S3.
- Using Spark Structured Streaming notebooks to ingest real time data.
- Processing large volume of near real time data in KQL Db.
As a part of the target production implementation, we would be making use of the new kinesis connector that has been released in public preview today (May 21st).
Existing (As-Is) process
Simplified & Performant Fabric Real-Time Intelligence architected process (production target architecture)
The proof-of-concept solution simplified the entire architecture, and we were able to reduce the latency by over 80% and bring down the end-to-end data latency from 30-45 minutes to under 3 minutes.
Our Production implementation planning is underway.
New: Microsoft Fabric’s Real Time Intelligence
More and more customers are looking to get ahead of the curve by making use of real-time analytics. Due to the availability of various tools, customers tend to use applications that meet their specific needs from an application perspective. Over time this has led to a cacophony of tools only meeting their siloed purpose in the entire ecosystem.
Introducing the next edition of Real Time Analytics in Microsoft Fabric – Real Time Intelligence.
With Real Time Intelligence, Fabric extends the unified SaaS experience bringing the realm of streaming data to all the users in your organization to consume, analyze and act on this data to make faster informed decisions.
All of this is made possible with the Real-Time Hub, which is the one stop shop for all data-in-motion across your organization. It is the OneLake data hub equivalent for real time data. Every Microsoft Fabric tenant is automatically provisioned with the hub, with no extra resources to set up or manage.
Some key features of the hub:
- One stop shop for all data-in-motion for the entire organization.
- Multiple new connectors to consume data.
- The hub (river of data) is never dry.
- One copy events/streams for use with multiple real-time analytics engines.
- All Microsoft sources of real-time data like Event Hubs Namespace, IoT Hub show up automatically in the hub.
A whole new list of connectors is being made available to simplify data ingestion into the Real-Time hub. To begin with, below are the ones that are supported.
- Streaming data from other clouds: Google Pub/Sub, AWS Kinesis
- Kafka Clusters: Confluent cloud
- Database Change Data Capture (CDC) feeds: Azure SQL CDC, Postgres CDC, Cosmos DB CDC
- Microsoft streaming sources: Azure Event Hubs, Azure IoT Hub, CDC sources
- System events (Coming soon): both Azure system events (like Azure storage account events) and Fabric system events will be automatically generated and routed into Real-Time hub.
Drawing a parallel to shortcuts within Microsoft Fabric, the new connectors for real time data, makes Real-Time Intelligence the first cross cloud integrated streaming platform to unify real time data across your organization. Google Pub Sub and AWS Kinesis data stream connectors enable seamless data flow with a low-code, no-code experience.
We are excited about – using the new Kinesis connector to stream data directly into Eventhouse. This would eliminate the need for multiple components in the customer implementation architecture mentioned earlier and reduce the latency further. We’ve already tried out the connector in a test environment and with minimal configuration, we were able to stream data into Eventhouse.
Enhanced Eventstream
Event processing already existed with Real Time Analytics in Fabric, as a post processor once you specify a destination in the Enhanced Eventstream. With Real-Time Intelligence, you are now able to derive new events using event processing. This opens the eventstream as a new data source real time streaming which can be leveraged across the organization for multiple uses.
As you can see from the above screenshot, I’ve set up ingestion from a streaming source and used filters to route specific events to different KQL databases. I’ve also been able to do some minor transformations to cast source data into required data types and set up an aggregated derived stream as an output.
This derived stream is now available in Real-Time Hub to be used by other consumers as a stream source and get a jumpstart with their analysis on real-time data.
Real-Time Dashboards
Another promising feature is the Real-Time Dashboard, a user-friendly lightweight interface to create quick exploratory visualizations. It offers a high refresh frequency supporting a range of customization options catered to support low latency time series data.
The following visual types are supported:
- Bar Chart
- Scatter Chart
- Line Chart
- Pie Chart
- Markdown
- Anomaly Chart
- Map
- Area Chart
- Stat/Multi Stat
- Column Chart
- Time Chart
- Funnel Chart
In conclusion, mastering real-time analytics with Microsoft Fabric opens up endless possibilities for organizations to gain actionable insights and drive innovation. By leveraging the powerful capabilities of Real-Time Intelligence, businesses can build scalable, reliable, and high-performance real-time analytics solutions that deliver instant insights and drive business success.
Thank you for joining us on this journey to unlock the power of instant insights with Microsoft Fabric’s Real-Time Intelligence! This is just the beginning, stay tuned for more exciting features coming in this experience in the next few weeks!