15th November 2023
by

Microsoft Fabric Preview Projects: There’s a Different Fabric for Each Use Case, and That’s a Good Thing

Microsoft Fabric Preview Projects:
There’s a Different Fabric for Each Use Case
& That’s a Good Thing

Microsoft Fabric Preview Projects: There’s a Different Fabric for Each Use Case and That’s a Good Thing

Since Microsoft Fabric was released in public preview earlier this year, iLink has worked with our customers to develop end-to-end analytics projects using the new platform. We have had the opportunity to put the claims of an all-in-one enterprise-scale software as a service (SaaS) analytics platform to the test, and in this post, we’ll share what we’ve learned, what we’re excited about, and our overall impression. 

What is Microsoft Fabric? 

Microsoft Fabric is a single SaaS platform that combines seven core analytics workloads into one product, simplifying and unifying governance, security, administration, billing, navigation, monitoring, etc. It includes data engineering, data warehousing, data science, real-time analytics, Power BI, and a new tool to automate actions from data called Data Activator. The cherry on top of this multi-layered platform is Copilot, which promises to simplify the creation of Power BI reports, more easily provide insights, support the development of code, and more. 

What new tools and features does Fabric offer? 

Fabric takes many of the analytics products previously offered by Microsoft and brings them all together under one roof. These include Power BI, Azure Data Factory, Synapse Analytics, etc. However, Fabric is more than just a combination and rebranding of existing tools. A few of the new tools and features that are part of the service include: 

  • OneLake: OneLake (the OneDrive for data) is the storage layer and the foundation for all of the workloads in Fabric. OneLake is a multi-cloud data lake that is automatically provisioned for an entire tenant. Delta, the storage format for Fabric workloads is open, meaning that other tools beyond Fabric can easily access and work with data stored in the OneLake. Additionally, to reduce duplication of data, OneLake provides two key capabilities: shortcuts and mirroring. Shortcuts allow seamless connections between OneLake and external storage. Mirroring (in preview) provides near real time data duplication from other cloud data warehouses to Fabric, without needed to build pipelines. Finally, all of the Fabric workloads have been optimized to store and retrieve data quickly in the delta format, including Power BI. 
  • Direct Lake mode for Power BI datasets: In addition to import mode and DirectQuery mode for Power BI datasets, Fabric brings us the best of both worlds with Direct Lake mode. Direct Lake mode gives us the real-time capability of DirectQuery but with the speed of import. This can be a huge relief in those cases where import mode dataset refreshes become challenging, for example. 
  • Data warehouses and lakehouses: Storing data in a Fabric data warehouse and/or lakehouse within the familiar environment of Power BI allows both data engineering experts and non-experts to meet their data storage, transformation, and access needs in one place, without the need for complex set-up or configuration. It also provides many more methods and tools to access data used for Power BI reports beyond DAX and visualization tools. For example, when data is stored in a lakehouse and Power BI uses Direct Lake mode to access the data, other personas can choose to query it with T-SQL via the SQL endpoint or with Python via a Spark notebook. 
  • Single compute model for all workloads: With Fabric, you purchase a certain level of compute and all of the workloads come from the same pool. This simplifies planning for workloads and billing. Instead of creating different subscriptions and groups for each workload, you simply purchase an F SKU (stock keeping unit), and all the workloads draw from that compute. With the Fabric Capacity Metrics app, it’s possible to see how much of your compute is being used by each workload over time and modify accordingly. With both a pay-as-you-go model and a reserved instance pricing model, this has the potential to simplify monitoring and save money. 
  • Spark notebooks: Notebooks open all kinds of possibilities, from data engineering to data analytics and data science. Spark compute is incredibly fast and increasingly used for many workloads. Being able to combine notebooks with orchestration through data factory pipelines is one of many ways to bring in and transform data to OneLake. 
  • Semantic Link: Semantic link is a new feature that gives us a way to establish a connection between Power BI datasets and data science using Fabric notebooks. This opens many possibilities including documentation and governance scenarios through querying which datasets exist in a tenant, development of machine learning models using Power BI data, and data quality validation through querying of both Power BI datasets and source data within a single tool.  

What is semantic link? – Microsoft Fabric | Microsoft Learn 

What doesn’t change? 

Because all of the non-Power BI workloads were brought into the interface of Power BI, Power BI users have the home-court advantage of a familiar working environment. Power BI itself still functions as before, but now has many added features such as DirectLake mode. Fabric just brings new features, tools, and licensing options to the game. 

Lessons learned during the public preview 

From iLink’s implementations of Fabric with our customers and through our own internal tinkering, we’ve had the opportunity to kick the tires and get a look under the hood. We’re happy to share some of the things we’ve learned, including: 

  • DirectLake mode really is a game-changer for Power BI datasets. By being able to query billions of rows of data and get nearly instantaneous results without needing an imported dataset refresh, our customers gain simplicity at a massive scale. In cases where we might have needed to build out complicated incremental refresh policies, custom partitions, or aggregate tables in the past, DirectLake mode instead allows us to directly access the data we need. 
    Lessons learned:  
    • It’s also still very important to follow data modeling best practices to ensure optimal performance.  
    •  There are row limits in DirectLake datasets where the dataset will fall back to DirectQuery if they are passed. When these limits are published, it will be important to pay attention
  • Different workloads, particularly within engineering, have different levels of performance. For example, data transformation through Spark notebooks performs much better than through Dataflows Gen2 currently.  
    Lessons learned: 
    • With Fabric there is often more than one way to accomplish a goal. It’s important to test out various methods to determine which one is the right fit for your scenario based on factors such as cost, performance, and team skills.   
  • With the release of a greater range of skus available, there is a much more affordable entry point to get started with Fabric. The lowest price for an F SKU is $263 (in US West)/month in a pay-as-you-go scenario. With reserved instance pricing this will be even lower. Compared to the lowest Power BI Premium capacity of a P1 at $4,995/month, this opens Power BI premium and Fabric features to a much broader range of organizations. 
    Lessons learned: 
    • It’s critical to monitor your workloads with the capacity metrics app to ensure that you’re using the right capacity or capacities.   
    • A well-thought-out workspace and capacity strategy is a must if cost is important. 
       

Conclusion 

In working with Fabric for the past year, both while it was in private and public preview, iLink has had the opportunity to work with customers and the Fabric product team directly. Our successful implementations with both larger (Fortune 500) and smaller organizations has allowed iLink to become a Microsoft Fabric Featured Partner. This has given us a unique perspective on the product.  

From our experiences and conversations, we see great potential for Microsoft Fabric, both for new and existing customers. Microsoft understands that many customers have already implemented large-scale analytics projects and have an existing foundation to work with. We have seen that Fabric plays well with others in scenarios like this, but that it also has all the components to support an end-to-end analytics solution at a massive scale. What this means is that Fabric doesn’t require you to fit into a specific use case for it to work for you. Instead, all the Fabric building blocks can be put together to fit your unique needs.  

Fabric has the potential to increase developer and analyst productivity while reducing overall costs of ownership. On top of that, the introduction of Copilot within the various workloads of Fabric has the potential to be a true game changer. The 2023 “Total Economic Impact of Microsoft Fabric” Forrester report concludes that Microsoft Fabric will help increase productivity and reduce spend (by making it an end-to-end platform), leading to overall TCO reduction and much higher ROI. From what we’ve seen so far, we agree with their assessment. 

We’re excited to continue working with customers on Fabric implementations and feel confident that Fabric is going in the right direction. 

Contact us today to learn more about Microsoft Fabric and what it can do in your unique analytics scenarios. 

Stephanie Bruno

Stephanie Bruno

Business Intelligence Architect

Linkedin
About Author

With 10+ years of experience, Stephanie drives efficiencies by optimizing tabular models, migrating AAS cubes to Power BI datasets, and implementing CI/CD processes. She collaborates with cross-functional teams, presents actionable insights, and designs data models for key stakeholders. Her expertise and passion for business intelligence contribute to iLink Systems’ success in data-driven decision-making.

SHARE
Share on facebook Share on google Share on twitter Share on linkedin

Related Blog Posts

Best of tech trends in 2023
13th November 2023
by

Top Technology Trends of 2023: A Year in Review  

As we edge closer to the year 2023, the world of technology continues to evolve at an astonishing rate. We are witnessing a rapid growth of technological innovations that transform how we live, work, and interact with others, ranging from Artificial Intelligence and machine learning to the ever-growing need for cybersecurity and metaverse. 

In this blog, we’ll dive into the most impactful technology trends of 2023, exploring how they have redefined industries, reshaped businesses, and set the stage for a more connected and efficient future.  

Generative AI: AI Revolution  

ChatGPT is the most significant leap forward in the history of AI, launched in late 2022. Generative AI is one of the top critical trends in the 2023 Gartner Hype Cycle for Emerging Technologies. Unlike traditional AI, generative AI can enable the creation of new content in various forms, such as text, images, audio, video, and code autonomously based on training data and neural networks.   

It empowers businesses to revolutionize their product development, design, customer operations, and sales and marketing strategies in new and captivating ways. It can augment human capabilities in executing complex tasks, substantially increasing creativity and productivity across business and society.  

It covers a range of machine learning and deep learning techniques such as Transformer models, Generative Adversarial Networks (GANs), and Variational Auto Encoder (VAE) designed for specific challenges and tasks, including summarization, classification, and drafting. Since generative AI applications have been built using foundation models, companies have now developed fine-tuned Large Language Models (LLM), harnessing the power of pre-trained language models.  

Check out our Generative AI services.  

Blockchain and Web 3.0: Decentralized Architectures and Ecosystems  

In the world of ever-increasing mistrust, Blockchain and Web 3.0 have played a crucial role in rebuilding digital trust by decentralizing data.  Blockchain is a decentralized, secure, transparent digital ledger that records transactions without a mediator. This technology has disrupted several industries with the emergence of cryptocurrencies, including healthcare, finance, logistics, and supply chain management.  

In 2023, it has brought remarkable changes across a broad spectrum of industries with its ability to increase transparency and security, remove intermediaries, streamline operations, and reduce fraudulence in data management. Web 3.0 is the third generation of the internet, empowered by AI, machine learning, and blockchain. It decentralizes the authority to its users, providing more substantial ownership and increased control over their digital assets.  

Blockchain has the potential to build trust among stakeholders by establishing transparency and immutability of data stored on the blockchain. As we look forward to 2024, blockchain will continue to advance with decentralized Finance (DeFi) and Non-Fungible Tokens (NFTs).   

Top technology trends that influenced businesses 2023

Low Code or No Code AI: Next Generation Software Development  

In today’s competitive world, businesses must be agile, responsive, and resilient to meet customer needs and expectations. Low code and no code platforms are designed to simplify the modern software development process by enabling drag-and-drop interfaces for building software applications. These platforms empower businesses to develop and deploy functional prototypes, test designs, and customized AI models faster and more efficiently without extensive coding expertise.  

As a result, they can increase their agility and speed of software delivery by automating tasks, significantly improving accuracy while reducing costs making them remain competitive. Low code No code platforms such as Google’s AI provide potent tools and pre-built models for developing AI applications.   

Explore our Low code / No code engineering solutions.   

Digital Twin & 3D Printing: Bridging the Physical and Digital World  

Digital Twin and 3D printing are rapidly growing across several industries, including healthcare, manufacturing, automotive, aerospace, construction, and engineering. Digital Twin is a simulation software duplicating a physical object in a virtual space. It focuses on creating virtual models of organs, simulating surgical procedures, and developing personalized healthcare solutions. In the automotive industry, it helps to create realistic simulations of cars, from design to testing process, reducing time, costs, waste, effort, and carbon footprints.  

3D printing also outperforms the creation of digital simulations and prototyping. It can create customized and unique products that meet organizational needs. The healthcare industry is the most beneficial as it helps to develop prosthetics, dental implants, orthopedic implants, and bioprinting. Similarly, the automotive industry has benefited from 3D printing in various use cases, including producing complex parts, customized components, and prototypes.   

Metaverse: The Future of Digital Connection  

With the rapid advancement in networking and computations, the metaverse is considered an immersive incarnation of the internet that transforms how we interact with virtual environments in real-time. Metaverse is being created by combining Virtual Reality, Augmented Reality, and Blockchain technologies that provide a highly immersive experience to users that is distinguishable from the real world. Since the current version of Metaverse is a multitude rather than a single virtual space, it provides several tangible benefits ranging from video games to virtual operating rooms.   

More recently, enterprises have started to invest in the Metaverse platform due to its rising popularity, high ROI, and increased demand for virtual experiences. It helps employees streamline their operations while offering customers an immersive and unique experience. It drives deeper connections, more effective collaboration, cost-effectiveness, and enhanced personal productivity.  While it is still in the nascent phase, it is an exciting concept that shapes the future of virtual worlds and beyond.  

Trust Architecture and Digital Identity: A Secure Digital Future   

Trust architecture and digital identity are vital to building a secure and trustworthy digital economy. It encompasses zero-trust architecture, digital identity systems, and privacy engineering, enabling safe and reliable online transactions. Zero Trust Architecture (ZTA) provides a framework that manages and authenticates identities such as users, data, and assets in a digital environment. It enforces security policies consistently and proactively detects potential threats, including data breaches and cyberattacks.  

Privacy engineering focuses on reducing the risks related to data privacy, resource allocation, and privacy controls. It helps organizations enhance cyber resiliency in a distributed business environment while allowing legitimate users to access the appropriate resources. However, a robust architecture requires a collaborative effort between stakeholders, including governments, businesses, and technology providers. It also requires a well-planned strategy and governance to implement security tools to achieve desired business outcomes effectively.   

Robotic Process Automation: Cobots (Collaborative Robots)  

Every business must evolve with technologies to understand customer needs and market demands for continued growth and innovation. With the emergence of RPA, enterprises allow robots to take over repetitive and mundane tasks, optimizing employees’ performance by engaging them in high-priority tasks. It incorporates AI and machine learning capabilities and optimizes and speeds up complex tasks, improving efficiency and accuracy. By deploying RPA, businesses can drastically eliminate human errors, boosting productivity and reducing costs.   

RPA makes our business more agile and resilient by handling critical jobs faster and more precisely than humans or legacy systems. Cobots are a form of RPA, widely adopted by the automotive and electronics industries, that intend to work alongside humans to ensure safety in a collaborative workspace. Since it differs from autonomous robots, it has been deployed when industries need to prioritize safety, flexibility, and high ROI. However, RPA implementation requires careful planning, comprehensive training, and governance to work efficiently.  

Explore our intelligent automation and RPA services. Check out our dedicated case studies of RPA.   

Unleashing the Potential of Emerging Technologies with iLink Digital  

iLink Digital stands at the forefront of harnessing the power of emerging technologies with widespread business impact. With our pragmatic approach and expertise, we help organizations envision futuristic technology that shapes strategic business goals.   

If you’d like to connect and discuss more, please contact us!  

9th November 2023
by

Microsoft Fabric Vs. Snowflake: Unveiling the Powerhouses of Data and Analytics 

Microsoft Fabric Vs. Snowflake:
Unveiling the Powerhouses of Data and Analytics 

Microsoft Fabric Vs. Snowflake: Unveiling the Powerhouses of Data and Analytics 

In this data-driven world, businesses are constantly searching for the perfect data management and analytics solution that can cater to their unique needs. Among the top contenders are Microsoft Fabric and Snowflake – two behemoths in the SaaS realm for data management and analytics.  

But what distinguishes them? Let’s embark on a journey of exploration of the differences between these two entities and discover their potential. 

Architectures Showdown: Microsoft Fabric Vs. Snowflake 

At the heart of Microsoft Fabric lies a unified ecosystem designed to streamline data and analytics processes. Microsoft Fabric is empowered by leading-edge AI capabilities, including Co-Pilot, Chat GPT, and AI-driven insights, exemplifying a revolutionary leap in data and analytics solutions. Notably, Fabric adheres to the principle of segregating storage and computation, optimizing efficiency. It also boasts seamless integration with popular office applications and collaboration platforms such as Teams. 

With OneLake, Power BI, Azure Synapse, and Azure Data Explorer working harmoniously, users get the best of both worlds: tightly integrated analytics and a unified data lake. The integrated capabilities, powered by AI, amplify the innovation potential and informed decision-making like never before. 

The diagram below illustrates Microsoft Fabric’s architecture and services. 

Microsoft Fabric architecture 

On the other end of the spectrum, Snowflake is a SaaS service that is Cloud agnostic, has zero maintenance, limitless instant scaling, built-in query optimization, hybrid columnar storage, and decoupled compute and storage that scales independently. Those qualities have made Snowflake one of the favorite cloud data warehousing choices in the market so far.   

Snowflake follows the usage-based pricing model, and you only get charged for used compute (per second) and storage (average TB of data stored per month after compression). 

Benefits like performance and support for structured and semi-structured data, concurrency scaling and accessibility, seamless data sharing, high availability, and security first approach make Snowflake very popular in the market.  

Snowflake Architecture 

Microsoft Fabric prioritizes end-to-end integration and ease of use for users already familiar with Power BI and Azure data services. Meanwhile, Snowflake stands out for its managed cloud platform, data separation, flexibility, and robust security. 

Feature Showdown: Microsoft Fabric Vs. Snowflake 

Let’s dive in with a side-by-side comparison of the high-level features offered by both Microsoft Fabric and Snowflake: 

Sr. No  Features  Snowflake  Microsoft Fabric  Winner 
1  Use cases  Data Warehouse, Streaming, Data Engineering, Data Lake, Data Science, and Data Applications.  Data Engineering, Data Science and Machine Learning, Data Lake, Data Warehouse and BI.  Fabric: Broader set of services including leading BI product 
2  Compute  Limitless scaling with virtual warehouses.  Fully managed platform supporting multiple computes like Spark, Synapse SQL, and Kusto DB.  Snowflake:  Fabric needs to mature limitless compute. 
3  Storage  Hybrid columnar storage with managed micro-partitions with zero intervention needed for indexing and partitioning.  Fabric offers OneLake as a single, unified, logical data lake that is built on ADLS Gen2 leveraging open-source Parquet and Delta formats.  Both  
4  Monitoring   Resource monitors, control policies, and cost-monitoring options are available  Microsoft Fabric offers:  
1. Data Activator: “Monitors queries, datasets, and event streams for patterns, trigger alerts, and actions”.  
2. Monitoring hub, which serves as a central location to monitor various activities, including dataset refresh and Spark Job runs and is available for Power BI, Data Factory, Data Engineering, and Data Science. 		 Both  
5  Administration  Near-zero Administration is a SaaS platform.  Near-zero Administration is a SaaS platform.  Both  
6  Security and Compliance  Compliant with SOC 1 Type II, SOC 2 Type II, HIPAA, GDPR, ISO 27001, and FedRAMP.  Data is encrypted at rest and offers RBAC (Role-Based Access Control) for user and privilege management. Robust security includes MFA (Multi-Factor Authentication) and VPN connectivity.  Fabric offers applying Sensitivity labels to guard your sensitive content against unauthorized data access and leakage.   Additionally, data is encrypted at rest and offers RBAC (Role-Based Access Control) for user and privilege management. Robust security includes MFA (Multi-Factor Authentication) and VPN connectivity.  Snowflake: Due to support for virtual private Snowflake instances.  
7  Data Ingestion  With Snowpipes, Stream, Stages, External tables, and third-party tools like Mitillion, Fivetran, and the Native Copy command.  Data Pipelines, Data Flows, and Copy commands; cross-warehouse ingestion; and partners like DBT-Data Build Tool, Databricks, and more.  Both 
8  Support for building data apps  Streamlit is an open-source Python library that facilitates the creation and sharing of beautiful, custom web apps for machine learning and data science. It enables you to quickly build and deploy powerful data apps within minutes.  Custom data apps can be quickly created using Power BI.  Both 
9  Support for Notebook Environment  Python worksheets provide support for all DataFrame-based libraries, including but not limited to Pandas, NumPy, Boto3, Plotly, and more.  Fabric supports the Notebook environment for both data science and data engineering workloads.  Fabric: Snowflake depends on third-party tools (Jupyter) for notebook environments.   
10  Querying and Analysis  Snowflake supports Snowsight, Snowsql, Jupyter Notebook environment, etc. for querying data and analysis.  Fabric supports Notebook, SQL Endpoint (Spark), and SQL WH interfaces for connectivity.  Both 
11  Data Sharing  Snowflake supports both public and private live data sharing and data monetization capabilities using the marketplace.  Needs to leverage the “Azure Data Share” service for data sharing. Data sharing is not a default option with Fabric for now.  Snowflake: Advanced data-sharing features 
12  Cloning/Copying source datasets  Zero copy cloning is supported for most of the object types, including tables and schemas.   Only Zero copy cloning of tables is supported for now, and there are no other object types or schemas.  Snowflake: Advanced cloning options 
13  Reporting needs  Basic support for building dashboards and charts.  Fabric supports enterprise-grade reporting features supplied using Power BI.  Fabric: due to the inbuilt Power BI service 
14  Data Profiling Capability  Default Data profiling supports the first 1 million rows for every query result or output.  We need to develop custom scripts using languages like Python, R, or SQL to perform in-depth data profiling. Services like Azure Databricks, Data Pipelines, and Synapse Analytics can help with this too.  Snowflake Every select query result gets profiled by default. 
15  Pipeline and orchestration  Snow Tasks provides basic noncyclic orchestration and support for third-party tools like Airflow and Data Factory.  Data Factory, Data Flows, Copy Command, Pipeline Jobs.  Fabric as it provides native tools, flexibility, and integration ease 
16  Data support  Support structured and semi-structured data types like Avro, Parquet, JSON, etc.   Native integration with Azure Data Lake and Delta Lake makes it an ideal choice for structured and unstructured data.  Fabric:  As it supports unstructured data types. 

Wait, we’re not done yet! Let’s dig deeper into the trenches and explore the object-level comparison of these remarkable platforms. 

Sr. No  Snowflake Objects  MS Fabric- Equivalent Objects (Synapse) 
1  Tables  (Permanent, Transient, Temporary)  Tables (permanent and local temporary tables are supported). Global temporary tables and transient user tables in TEMPDB are not supported in Fabric. Primary and Foreign Keys are not supported. Tables are case-sensitive. 
2  Constraints and keys: Snowflake supports defining and maintaining constraints but does not enforce them, except for NOT NULL constraints, which are always enforced.  Constraints and keys: For warehouses, PRIMARY KEY, and UNIQUE constraints are only supported when NONCLUSTERED and NOT ENFORCED options are used. 
 
FOREIGN KEY is only supported when the NOT ENFORCED option is used. 
3  Standard SQL views are supported  Standard SQL views are supported 
4  Materialized views are natively supported.  Materialized views are only supported in Fabric real-time analytics and do not yet support fabric warehouse or Lakehouse. Instead of using materialized views, you can materialize the results of complex queries by storing them in tables. This approach allows you to pre-compute and store the results of expensive or frequently used queries, improving query performance for subsequent executions. 
5  External tables are supported natively in Snowflake.  External tables are supported only in Lakehouse, and you can create external tables directly on top of OneLake files. 
6  Secure views are used to share sensitive objects across different accounts.   The secure views option is not available or supported in Fabric yet. 
7  Capturing table CDC is made easy using table Streams in Snowflake. Below are different types: Delta (standard stream) Append Only stream Insert Only  In Fabric, Change Data Capture (CDC) is not natively supported as a built-in feature. However, you can implement CDC-like functionality using other Azure services.  
One possible approach is to use Azure Data Factory and Azure Event Hubs to capture, and process change data. 
8  Sequences are supported natively within Snowflake.  Sequence objects are not supported yet in Fabric. We need to build custom logic to create sequences in Fabric.  
9  In Snowflake, Procedures are supported with JavaScript, Python, and SQL Language.  In the Fabric warehouse -Stored Procedures are supported with only SQL language support. There is no support for other languages yet. 
10  Function, UDFs (support Java, JavaScript, and SQL).  In Fabric Scalar Functions, Table Valued functions are only available with SQL language support 
11  Stages (both internal and external Stage creation is supported).  Shortcuts for External Storage and Onelake/Lakehouse for Ad Hoc File Uploads. 
12  Schema’s creation is supported by Snowflake.  Schema creation is supported by Fabric. 
13  File Formats: All standard file formats are supported.  All standard file formats (CSV, ORC, Parquet, JSON) are supported. 
14  We can create database instances in Snowflake.  An equivalent construct in Fabric is called a “Warehouse”. 
15  Roles: Elaborate roles and role hierarchies can be created within Snowflake.  Roles are supported at the Fabric Workspace level. Available roles are: 
1. Admin  
2. Member  
3. Contributor  
4. Viewer. 
16  Resource monitors can be configured to manage virtual warehouse costs.  Microsoft Fabric, the Monitoring hub, serves as a central location to monitor various activities, including dataset refresh and Spark Job runs, and is available for Power BI, Data Factory, Data Engineering, and Data Science 
17  External function executions (AWS Lambda) are possible through Snowflake.  External function execution is not supported in the Fabric Preview Environment. 
18  Metadata access using the information schema.  The Information Schema option is supported in the Fabric Warehouse environment. 
19  Snowpipes can be created for the automatic loading of files from the external stage.  In Fabric, Structured Spark Streaming can be leveraged to load data files. 
20  In Snowflake Zero, copy cloning of databases, schemas, and other objects is fully supported.  In Fabric Zero, copy cloning of only tables is supported in Preview. 
21  Data Masking- Both row and column-level masking and tag-based masking are possible.  Object-level security can be managed using GRANT, REVOKE, and DENY syntax.  
Row-level security is currently not supported.  
Dynamic data masking is currently not supported. 
22  Search Optimization on tables.  Search optimization is not available as a service. 
23  Disaster Recovery: Up to 90 days of Time travel and 7 days of fail-safe.  Time travel is supported by Lakehouse Delta Tables only. 

Concluding Thoughts 

In conclusion, both Microsoft Fabric and Snowflake are robust data and analytics platforms, each with their own strengths and capabilities. Both are competitive platforms with their own pros and cons that would look a lot like each other in the future as both are trying to become end-to-end analytics platforms. 

The success of both platforms depends on your specific needs and requirements. There is no easy answer to which platform to choose without understanding the customer’s unique situation. It could be Snowflake, Fabric, or a combination of both, depending on the current state and where you want your future enterprise software landscape to be. 

Reach out to us today, and let’s make your data dreams a reality! 

At iLink Digital, we are passionate about empowering businesses with cutting-edge data solutions.  

Contact us and experience the power of data solutions tailored to your unique business needs. Request a personalized demo or free consultation with our team of experts. 

Digambar Jadhav

Digambar Jadhav

Senior Technical Architect DATA

Linkedin
About Author

“Bringing over 15 years of extensive experience in the field, Digambar Jadhav is a seasoned data architect specializing in modern cloud data technologies such as Snowflake, Databricks and Microsoft Fabric. With a versatile background in various roles within data warehousing projects, he excels in architecture design, solutioning, and offering comprehensive support in data engineering and data modeling. Known for leveraging cutting-edge cloud platforms, Digambar is committed to delivering innovative solutions and optimizing data ecosystems.”

SHARE
Share on facebook Share on google Share on twitter Share on linkedin

Related Blog Posts

7th November 2023
by

Dataflows Gen 2 vs Gen 1 

Microsoft Fabric has brought together and simplified all the components required for a complete analytics platform into one unified SaaS product. It provides multiple options for extracting, transforming, and loading (ETL) your data into its OneLake foundation, based on your specific needs and preferences. You can build Data Factory Pipelines to copy data and/or orchestrate other components, write Python in a notebook, or use Power Query in a Dataflow Gen 2, for example. 

In this entry, we dive into Dataflows Gen 2. We will help you understand the differences from the previous generation of Dataflows, how to use the new generation, and the options for migration of the previous dataflows to Gen 2. 

What is Dataflows Gen 2? 

Dataflows Gen2 is the new generation of dataflows included in Microsoft Fabric. They allow us to ingest and transform data using Power Query, just like in Gen 1. However, they add to the previous generation in several ways. Two of the major additions that we will discuss are (1) allowing the user to deliver the data to additional destinations, and (2) providing a simple option to append new data rather than being restricted to only overwriting or configuring incremental refresh. 

Gen 1 vs Gen 2 Differences 

A question that always comes to mind with a new version of any service or application is what the impacts are to existing processes; and the differences from previous versions. 

Dataflows Gen 2 has much to add when compared to Gen 1, so let’s check the main differences: 

Dataflows Gen 2 – Feature Overview 

There are only two points of contact in the above chart between the two generations of Dataflows: the Power Query authoring experience and the connectivity options which allow the user to access more than 140 data sources or custom connectors.  

Additionally, Dataflows Gen 2 has an in-built auto-save experience that allows us to continue the work from where we left off as it auto-saves into the cloud and no longer requires us to confirm the save at exit. Along with this feature, it is also available for background publishing that runs several validations. 

Data Destinations 

The improvement in the choice of destination of the data is one major difference between both generations. While on Gen 1, there is only internal/staging storage to be consumed in datasets (or a bring-your-own-lake scenario), Gen 2 allows us the option to select the destination for the transformed data, allowing us to save the data in: 

  • Fabric Lakehouse 
  • Azure Data Explorer (Kusto) 
  • Azure Synapse Analytics (SQL DW) 
  • Azure SQL Database 

With this new option, along with the use of pipelines, we can now automate the consumption of the output of dataflows, allowing us to perform tasks that can go from running a SQL query to running a Python workbook. 

Append or Overwrite 

With the new store locations, there is no need to use incremental refresh since the data is now stored in our different data sources to be consumed later. The new capabilities for storage are complemented by a new experience for refresh history and monitoring that allows us to have full-detailed information on our data refresh. 

Performance 

The fabric integration with the use of Lakehouse and Warehouse permits the achievement of an enhanced compute engine that improves the performance of the ETL process.  

Licensing 

Licensing for Dataflow Gen 2 only works with a Fabric capacity, while the previous version is available starting with the Pro License and above. However, for the tenants already in premium, Microsoft permits Fabric capacities without any need to update licensing, which means that Dataflow Gen 2 is currently available for all Power BI premium capacities. 

When to use Dataflow Gen2 or why to use it 

One of the questions that comes to mind with these two generations available is which one is the right fit for our requirements. Despite all the differences and improvements in the new generation, the answer lies in the licensing.  

Dataflow Gen 1 works with Power BI Pro, PPU and premium licenses, while Dataflow Gen 2 only works in a Fabric capacity. If you have Fabric licensing, having the latest version will significantly improve your performance and optimize the way you store and consume data. 

Another advantage of the new generation is the destination options, which allow us, using the SQL Endpoint, to create and/or edit a Power BI dataset that consumes the data and generates a report. This means that we can now have a start-to-end solution for the Fabric environment, from consuming data to doing the ETL and creating reports that can be shared across the organization. 

Storage and refresh setup on Dataflow Gen2 

One of the improvements on Dataflows is the storage and refresh setup. The new generation can help us determine the destination, whether it’s a Lakehouse or an Azure SQL database. This allows us to refresh the data by overwriting the previous data or appending it to our table. 

Dataflows Gen 2 – Destination Settings. 

This setup is simple and done within the workflow of the dataflow, and this changes the way the information is stored. This changes not only the refresh of the data but, depending on the selected option, can reduce the refresh time. 

With the usage of pipelines, the refresh is simplified and can be called at any time. In the previous versions, we had the use of scheduled refresh, which, based on your license, would be run a limited number of times. The new workflow allows a faster and timelier refresh and update of your data, adjusted to your business needs. 

How to migrate from Gen 1 to Gen 2 

Being based on the Premium licensing, most of the users already have several Dataflows Gen 1 implemented, and the upgrade to the new generation is simplified by the new concept of templates. 

Templates for dataflows are a package for the entire Power Query configuration. This includes all the definitions that are readable by all the Power Query environments, including Dataflow Gen 2, Dataflow Gen 1, and Excel. This can help us with a smooth and easy transition from our previous dataflows and include the information from our previous files in our dataflow information. 

The credentials and connections aren’t stored in the template file and need to be setup during the import step, adding to the security of the setup data. 

Currently there is no out-of-the-box solution for a bulk migration from Dataflow Gen1 to Gen2. If you need help in getting a bulk migration of your Dataflow to the new Gen, reach out so that we can help you achieve this. 

Conclusion 

With the powerful advantages of data destinations, appending options, and performance improvements, most organizations should consider migrating any existing Dataflows Gen 1 to Gen 2. The flexible Fabric licensing options provide several different configurations to allow you to enjoy these advantages. Reach out to us if you need help understanding licensing or to create a large-scale migration plan.  

Please contact us for a free data strategy briefing to learn how Fabric can help you accelerate, de-risk, and streamline your data and analytics strategy. 

How Top Industries can benefit most from Data Science & AI
5th November 2023
by

How Top Industries can benefit most from Data Science & AI 

Data Science and AI are more than just buzzwords. These technologies offer the precision, speed, and foresight needed in today’s hyper-competitive business landscape. From optimizing supply chains and enhancing customer experience to pioneering innovative products and services, Data Science and AI are catalyzing paradigm shifts across leading industries like retail, manufacturing, healthcare, banking, and finance.   

In this analytical deep down, let’s explore how top industries strategically deploy Data Science & AI to drive innovation, efficiency, and unparalleled business growth.    

Reimagining Retail and CPG Sector   

Retail and CPG are some of the most involved industries in embracing Data Science and AI technologies early in the game. From predictive analytics helping retailers forecast demand, identify trends, and make inventory decisions to generative AI breaking grounds in customer support, these technologies are making the retail sector smarter and more responsive. 

Computer Vision is helping them elevate the in-store customer experience, while the recommender system helps them personalize the shopping. Anomaly detection acts as a sentinel against fraud, ensuring transactional integrity. Machine Learning helps to optimize prices, hones customer segmentation, and delivers seamless automated customer service.   

iLink Digital helped one of the retail clients to do the same. Dig deeper into our work here.   

A Transformative Shift in the Manufacturing Sector  

In response to the pressing need to make the manufacturing sector safer and more sustainable, Data Science and AI have emerged as pivotal solutions. AI-powered predictive maintenance and asset optimization help manufacturers manage and maintain machinery in various phases. They can effectively forecast potential bottlenecks or equipment failures before they occur, resulting in reduced downtime, enhanced safety, cost savings, and increased equipment lifespan.   

Computer vision ensures unparalleled product quality through rapid inspections, ensuring quality and reliability at every phase of the production process. Advanced algorithms optimize complex supply chains and enhance energy management, facilitating high-speed production capability.  

AI further streamlines the production process, enables personalized manufacturing, and promotes workplace safety by handling hazardous tasks.  

Explore how iLink Digital helped one of the leading engineering firms inspect sewer pipes with Computer Vision.  

Revolutionizing Healthcare & LifeSciences 

AI-assisted predictive analytics plays a vital role in drug discovery and development, helping healthcare & pharmaceutical companies identify and select potential therapeutic molecules with a higher probability of success. It improves drug development by forecasting drug effectiveness in advance. Data science and AI have immense potential to increase the efficacy of personalized treatment, reduce R&D costs, improve clinical trials, control drug reactions, and predict toxicity.   

Computer vision helps doctors make more accurate and timely diagnoses by detecting early signs of abnormalities in medical images. Automating medical image processing will significantly improve precision medicine, assess patient risk factors, early intervention of diseases, and reduce physician workload, leading to better patient care and healthcare outcomes.    

A Case study on how our team helped one of the top veterinary hospital staffing with predictive modeling 

Reshaping Banking, Financial Services, and Insurance Sector

With advanced AI algorithms and machine learning capabilities, predictive analytics empowers BFSI to make accurate predictions and data-driven decisions across several areas, such as fraud detection and prevention, cash flow forecasting, credit risk assessment and management, personalized financial planning, algorithmic trading systems, pricing models, and sentiment analysis. It notably empowers banks to lend confidently by analyzing their customer’s credit histories, income levels, and financial behavior, ensuring well-informed credit decisions.  

Computer vision can automate and streamline many processes, including image-based fraud detection, automated check processing, insurance claims processing, and financial document processing. More specifically, facial recognition and biometrics help financial institutions ensure security across operations while reducing the risk of fraud by protecting personal and financial data and money against theft. 

iLink helped a leading UAE bank with its customer onboarding. Read more about our work! 

Reinventing Telecom Industry 

Data science and AI are widely used in the telecommunication sector in several areas, including network management and optimization, fraud detection, troubleshooting processes, customer segmentation, price optimization, customer sentiment analysis, customer churn prediction, and product development.  

By implementing predictive analytics, telecom can gain valuable insights to make accurate decisions on managing and optimizing their network resources constantly, resulting in uninterrupted network connection to their customers and helping to improve profitability, reduce risks, minimize downtime, enhance customer satisfaction, prevent financial losses, and boost operational efficiency.   

Computer vision can outperform well in image and object recognition by applying deep learning methods. In predictive maintenance and network optimization, computer vision can recognize patterns and detect anomalies at a work site by analyzing vast drone footage and sending situational alerts accordingly. Thus, it will increase equipment up-time with less human intervention.  

Learn about our work in the telecom Industry.

End Thoughts   

Despite its immense benefits, disruptive technologies have unique challenges, such as privacy and security, reliability and safety, data quality, transparency, and accountability. However, the top industries can benefit most from the incredible fusion of data science and AI. There is no one-size-fits-all solution; hence, building and deploying a scalable and resilient AI model based on requirements and objectives is crucial for organizations. As data science and AI evolve, businesses need to find innovative ways to embark on their presence in a highly competitive market.   

Interested in Leveraging the power of Data Science and AI?   

iLink Digital, a leading provider of Data Science and AI services, helps top industries to reinvent, reshape, and reimagine their operations that drive business growth and innovation. Don’t miss out on the chance to harness the power of predictive analytics and modeling- connect with iLink Digital now to embark on a transformative journey.    

Check out our services and request a demo. Begin your journey with us today!    

7 Guided Strategies for Microsoft Power Platform Implementation
30th October 2023
by

6 Guided Strategies for Microsoft Power Platform Implementation 

The Microsoft Power Platform has become a popular choice for organizations looking to accelerate their digital transformation journey. With the platform’s low-code or no-code applications, businesses can quickly automate and digitize their operations. However, implementing the power platform requires careful planning and execution to derive maximum benefits. 

To achieve desired business outcomes, organizations need to adopt a guided strategy, vision, and best practices. This approach can help businesses capitalize on the power platform’s capabilities to improve their agility, scalability, efficiency, collaboration, and security while achieving their business goals and objectives.

What is Microsoft Power Platform? 

Microsoft Power is a unified platform that combines with other Microsoft platforms, such as Microsoft 365, Microsoft Dynamics 365, and Azure for analytics, design, and development. It empowers organizations to quickly automate and digitize their end-to-end business operations that deliver value to stakeholders and accelerate business growth. This powerful platform can be widely customized to adapt to other tools to create custom applications that fulfill the growing demands of modern business.  

Learn more about our Power Platform Capabilities.  

Steps for Effective Microsoft Power Platform Implementation 

#1: Identifying the Business Needs and Objectives 

Identifying the business needs and objectives helps to determine which platform components to implement and use according to the business objectives. Power platforms intend to accelerate automation by identifying the business problems that were previously solved manually.  

The proper definition of the existing problems makes them easier to resolve, which could save time, money, and resources. The potential pitfalls, such as complaints, gaps, and inefficiencies, must be rectified in the current work environment to make businesses remain competitive. 

Eventually, capturing the business needs will help companies define their business goals and plans by engaging key stakeholders, conducting deep research, and prioritizing requirements, allowing them to make better decisions, anticipate trends, and identify opportunities effectively.   

#2: Measuring Business Value: Align with KPIs and OKRs  

Power platform solutions focus on aligning with KPIs (Key Performance Indicators) and OKRs (Objectives and Key Results) to achieve the company’s vision and strategic goals. KPIs aim to monitor overall performance, whereas OKRs set organizational goals and align with the team to measure the business values that drive significant impact and innovation. These metrics will be regularly tracked and analyzed to assess our progress and identify areas of improvement. 

As such, the power platform contributes to the organization’s success by evaluating the effectiveness of existing solutions, functions, and processes in terms of efficiency, cost, and performance. Most importantly, seeking feedback from stakeholders will catalyze modifying the existing business solutions to meet business needs better and exhibit a return on investment to stakeholders. 

#3: Establishing a Center of Excellence (CoE) 

The Power Platform Center of Excellence (CoE) is a set of components and tools that helps organizations regulate maintenance workflows, reports, and best practices, ensuring organic growth and innovation while maintaining the necessary level of governance and control. 

It empowers organizations to develop a strategy for adopting the Microsoft Power Platform that comprises Power BI, Power Apps, Power Pages, Power Automate, and Power Virtual Agents. It provides an intuitive, collaborative, and resilient platform that includes several low-code tools that help organizations create effective and flexible solutions.  

A CoE must be deliberately designed and customized by each organization based on their needs and preferences, which include geographical and organizational silos with the same business goals to share knowledge and success. 

A successful Center of Excellence is not only managed by tools alone but also focuses on people, communication, requirements, and processes. The CoE kit consists of clear rules, frameworks, and best practices, allowing individual developers to leverage the experience of experts. However, Microsoft Power Platform requires a mature investment strategy, considering the multiple aspects of governance, training, and support for numerous functions and roles involved across organizations.   

#4: Admin and Governance: Assigning Admins and Defining Roles and Responsibilities 

Assigning admins and defining their roles and responsibilities is an essential step towards successfully establishing CoE in the Power Platform. We must include the following roles and responsibilities while establishing the CoE:, 

  • Microsoft Power Platform Admin Team: The admin team should be accountable for establishing an environment strategy, setting up data loss prevention (DLP) policies, and managing users, capacity, and licensing. Connectors, integration, and migration have been extensively used to make data available to developers. 
  • Microsoft Power Platform Nurture Team: This team is entitled to nurture our community of makers to ensure the usage of tools the organization has already invested in to achieve the business goals by organizing various events, training, and hackathons. 
  • Automation and Reusable Components: Microsoft Power Platform tools support extensive automation by approving licensing requests from end users for utilizing unused and highly used resources. It inherently adopts reusability across organizations by sharing standard templates and reusable components to get added benefits more quickly. 

#5: Establishing Low-Code Strategy using Power Platforms 

In the epoch of digital transformation, enterprises have started to leverage the incredible potential of low-code/no-code strategy, where automation is imperative. The paradigm shift towards low-code engineering ensures that the Microsoft Power Platform strategy is aligned with the organization’s goals. 

The low code platform paves the way for the automation of core repetitive tasks using drag-and-drop tools with templates, making developers focus on higher-value activities. It allows companies to rapidly build and deploy custom business applications in a modern agile environment without the need for technical expertise. 

Most notably, integrating a low-code strategy can significantly reduce the turnaround time for project development. This helps organizations drive new revenue and streamline the business operations that meet customer needs, which will automatically enhance customer satisfaction.  

Gartner predicted, “A rise in business technologists and a growing number of enterprise-wide hyper-automation and composable business initiatives will be the key drivers accelerating the adoption of low-code technologies through 2026.”  

It enables organizations to reach a high level of digital competency with concrete action plans, which makes them stay ahead of the curve. In a nutshell, understanding the methodologies, delivery models, frameworks, and organizational change management plans of Power Platforms is crucial while implementing a low-code strategy. Hence, we must equip ourselves with the latest tools and technologies to implement low-code strategies. 

#6: Executive Sponsorship 

Successfully adopting the Power platform always depends on acquiring executive sponsorship and leadership buy-in, especially in large organizations. It is essential to identify an executive sponsor by considering their roles and activities by applying top-down and bottom-up approaches. They have expertise in various use cases, such as formulating a strategic vision for Power Platform, prioritizing resources, staff allocation, fund approval, decision making, and expanding CoE.  

Interested in Harnessing the Power of Power Platform?   

Being a trusted Microsoft Gold Level Partner, our iLink Digital team of Power Apps developers has excelled in rapidly developing low-code to no-code business applications that help companies automate and improve their business processes with greater efficiency to achieve desired business outcomes. 

Here are some case studies you can explore showcasing our latest work in Power Platform:

If you want to leverage Power Apps for your company, connect with our experts today to embark on the journey of Microsoft Power Platform. 

Let’s talk!

    Open Chat

    Etiam magna arcu, ullamcorper ut pulvinar et, ornare sit amet ligula. Aliquam vitae bibendum lorem. Cras id dui lectus. Pellentesque nec felis tristique urna lacinia sollicitudin ac ac ex. Maecenas mattis faucibus condimentum. Curabitur imperdiet felis at est posuere bibendum. Sed quis nulla tellus.

    ADDRESS

    63739 street lorem ipsum City, Country

    PHONE

    +12 (0) 345 678 9

    EMAIL

    info@company.com

    PURCHASE