OneLake: Microsoft Fabric’s Open Storage Architecture

Data practitioners encounter various challenges every day. Data is typically dispersed across several sources, in a variety of file types, and too often with questionable degrees of quality. It can take a lot of time to determine where data might be located and what the access rights are. Consequently, less time is spent on what really counts, namely using the data to make decisions.

To address the issues surrounding data fragmentation, Microsoft introduced a lake-centric and open architecture in their new Fabric offering. While each tool within Microsoft Fabric caters to specific requirements, they all share a common data foundation: OneLake, based on the increasingly popular “data lakehouse” paradigm.

The Data Lakehouse, A Pattern Aiming for the Best of Data Lakes and Warehouses

The lakehouse term and pattern were initially made popular by Databricks in the past few years as a way to bring governance and structure to the vast amounts of unstructured raw data typically loaded in data lakes. Lakehouses follow an Extract Load Transform (ELT) pattern also known as “schema on read”, as opposed to the ETL pattern associated with “schema on write” in traditional data warehouses.

This architectural shift has been driven by the explosion in the volume and variety of data and enabled by massive public cloud infrastructure investments. The lakehouse aims to combine cheap data lake storage with the sense of structure and the ability to run efficient aggregate queries associated with data warehouses.

Microsoft’s Take on the Lakehouse

“OneLake is the OneDrive for data and like OneDrive, OneLake is provisioned automatically with every Fabric tenant with no infrastructure to manage.”  -Microsoft

Microsoft already started supporting lakehouses years ago with Azure Synapse Analytics. OneLake now underpins Microsoft’s expanded vision of the lakehouse architecture. This unified, logical data lake supports lakehouses, warehouses and all other workloads.

comes automatically provisioned for each tenant and is the home for all your data, moving away from siloes and the need to copy data to multiple places where you need it. Under the hood, it is still Azure Data Lake Storage (ADLS) Gen2 where Parquet files are managed with Delta metadata.

This means that Fabric does not have vendor lock-in and that the data in your OneLake may be used with a wide range of tools and technologies. Fabric’s many compute engines rely on Delta/Parquet format, providing you with a stable basis and reducing the need for format changes.

You can also easily bring data from the outside without copying it. Shortcuts are embedded references within OneLake that point to other storage locations. The embedded reference makes it appear as though the files and folders are stored locally but in reality, they exist only in their original storage. Shortcuts can be updated or removed, but these changes don’t affect the original data and its source.

OneLake supports existing ADLS Gen2 APIs and SDKs giving you the ability to hook up existing applications and tools to a OneLake endpoint.

There are several ways to get your data into the OneLake:

• Manually with the OneLake file explorer

• Data Factory pipelines
• Notebooks
• Shortcuts are a way of creating a symbolic link to external storage locations and file paths (currently supporting ADLS Gen 2 and AWS S3 buckets)