14:00 - 14:45  |  Fast Data

Data Engineering Patterns & Architecture. The future: Microservices Patterns with Streaming

Thursday 24 September 2020


With Industry 4.0, several technologies are used to have data analysis in real-time. Maintaining, organising, and building this, on the other hand, is a complex and complicated job. Over the past 30 years, we saw several ideas to centralise the database in a single place, as the true source of data, have been implemented in companies, such as Data Warehouse, NoSQL, Data Lake, Lambda & Kappa Architecture. On the other hand, Software Engineering has been applying ideas to separate applications to facilitate and improve application performance, such as microservices. The idea is to use the MicroService patterns on the data and divide the model into several smaller ones, and a good way to split it up is to use the model using the DDD principles. That's how I try to explain and define DataMesh & Data Fabric. I go through the last 30 years of data architecture and I combine with a simplification of the concept and idea of ​​the Data Mesh & Data Fabric to show the idea that; Yes, Streaming is everywhere; Yes, Streaming is a database; Yes, Microservice patterns; Yes, we are already in the Serverless era; Yes, we are already in the era of native Kubernetes; Yes, divide to conquer; Yes, Kubernetes Patterns is the plus.

If you have already registered, login using your email and password.