NextGen Data Platforms: The Data Mesh
As enterprises become more agile, centralization appears more and more as a thing of the past world, a waterfall world.As businesses become more complex, a major challenge became about marrying big data, self-service discovery, experimentation, compliance, and governance, while providing a clear path from data experimentation to production.Data Mesh is bringing to data engineering many of the concepts of agile software engineering to solve the above challenges.
Domain Driven architecture based on the data mesh paradigm enables platforms with decentralized access.A data mesh architecture distributes data ownership across domain teams, with federated governance and decentralized data products.The higher degree of centralization in this model addresses concerns like:
- Allowing data producers and data consumers to enforce and act upon data contracts;
- Allowing to check the degree to which the data are logically connected and mutually consistent;
- Allowing master data management to run more efficiently;
- Reducing costs by sharing compute resources between different teams.
At the same time, it applies several best practices from data mesh, such as data product thinking, domain-oriented data owners, self-serve functionality and data governance.