Are you excited / on the fence / downright sceptical about a data mesh?
Not long after a new buzzword makes the mainstream, a train of hype often follows 🚇.
How do we decode the top notch promotion vs. the value vs. the pitfalls?
As Bill Inmon said in his note to Piergiuseppe, today a lot of vendors have taken the role of industry leadership and turned it into a role of marketing and selling their products.
The data mesh is no exception. It is comprised of four core principles:
- Domain-oriented decentralized data ownership and architecture
- An example would be a team responsible for a data producing domain, e.g. customer data, and making that data accessible via APIs could also be responsible for providing the corresponding historical data & analytics for that domain
- Data as a product
- Behind the scenes: code + data & metadata + infrastructure
- From a data product consumer perspective, abstracted using e.g. APIs
- Self-serve data infrastructure as a platform
- The mesh’s underlying data platform / infrastructure needs to provide a very broad set of common, easy-to-use capabilities and tools, ranging from storage to access control to lineage to cataloguing
- Federated computational governance
- To help with interoperability of independent data products in terms of the scope of domains, common technical standards, compliance and security.
The list reads well, however each of these areas offer both benefits and challenges.
So where to begin?
A great starting point is an article from Chris Jackson.
It reviews the motivation for a data mesh, it’s principles, proposals, and highlights that fundamental challenges persist.
If you wish to build your understanding of the data mesh concept along with it’s challenges and risks, then this article is a must.
As Chris states:
“The best architectures aim to be as simple as possible whilst meeting the business needs”.
Need Snowflake training? You’ve come to the right place.