What it is?

An approach to data management, that is based on consumer driven, late binding of loosely coupled , domain centric data sources.

Data products where data is published rather than handed off , and ownership persists.

Business Domains (Producers) publish their data
Analytics Domains (Innovators) add value to data
Users (Consumers) fetch data and integrate as and when needed

Decentralized ownership of data

Distributed architecture to supports big data following domain driven design strategies
Domain definition that encompasses ownership, roles and interaction model

Federated and global governance

Enabled by well managed metadata

Self service infrastructure for domains to leverage the data platform and tooling

Why Data Mesh ?

Often there are solutions which are based around centralized data management principle. Reinforce business ownership while ensuring transparency & alignment . Execute iteratively to drive near term value while building a foundation for scale

User Perspective of ‘we cannot scale’
Publishing Data: how do I describe and host my data?
Discovering Data: where can I find data about
Understanding Data: what is the business meaning
Trusting Data: is the data ‘clean’?
Consuming Data: can I get access?

Critical Success Factors-

Ownership of Domains

Data ownership and custody must me established and agreed

Data published by each node must be clean, consistent, and documented
No free lunch: data must be prepared for publication
Need for human readable documentation and/or machine readable meta data.

Discoverability : How do users (consumers) learn what’s available?
Search or browse?

Implementation of technical interface
REST API, publicly accessible relational DB instance, general query API, streaming, pub/sub?

Understand granularity of data available from each node
Security and access control must be implemented by every node

Data Mesh Principles

Data as a Product
Product mindset, delivering useful data that are trusted and reusable in variety of use cases
Decentralized Ownership
Shifting rom centralized data management to domain driven ownership model
Self service Data Platform
Enable producers publish their data, consumers greater ease of access and promoting a vibrant marketplace
Federated Governance
Federated execution and centralized enablement, leveraging codified standards and automated processes

Now that we understand what and why data mesh is significant and is creating a lot of buzz. Let us now see how Snowflake platform empowers to create data mesh seamlessly

Building data mesh on Snowflake

Snowflake is a cloud-based data warehousing platform that provides a highly scalable and flexible environment for storing and analyzing large volumes of data. It is well-suited for implementing a Data Mesh architecture due to its support for multi-tenancy and data sharing.

With Snowflake, organizations can create separate accounts for each data domain, allowing them to manage their own data and control access and permissions. At the same time, they can share data across domains using Snowflake’s secure data sharing capabilities, enabling collaboration and reducing data duplication.

Elastic compute capabilities : Snowflake supports separate resources for different teams or workloads. Features like workload isolation, avoiding resource contention, multiple compute clusters accessing the same or different data, RBAC governance, etc.. are enablers for data mesh.

Governance in Snowflake : Data can easily be governed and federated within Snowflake. Here if we part it in 3 different sections, Snowflake has solutions for these as shown below.
Part 1: Know your data. Features of Snowflake used are Object tagging, data classification, etc..
Part 2 : Protect your data. Features of Snowflake used are Row access policy, dyanamic data masking, external tokenization, etc..
Part 3 : Unlock & monitor. Features of Snowflake used are Resource monitors, Snowsight dashboards.

Snowflake, a distributed platform : Snowflake is a distributed but connected platform that avoids silos and enables distributed teams to share data in a governed and secured manner.

Use Case

1. Global Asset Management Company Accelerates Digital Transformation

To bring investment processes to a scalable, digital environment and enable whole office solutions , this large asset management company partnered with Other Organization and Snowflake to modernize its data and analytic capabilities and build the data ecosystem for the future on the Cloud.

Summary :

In summary, Snowflake and Data Mesh are an perfect fit. Snowflake provides individual domain teams with easy access to storage, compute power, performance, scalability, security, and governance. The data domain (architect) teams utilize these mentioned resources to build data products that they can easily share with other data domain teams.