The Usual Suspects of a microservices ecosystem: Kafka, Kubernetes and Cloud

Hopefully you didn’t miss the cloud infrastructure train during the last few years. Managed to finally install your event-driven architecture? What about microservices? Still processing data in batches? Figured out how to deal with all those Docker containers? Relax, take a deep breath. The endlessly emerging heap of architectural patterns and corresponding technologies can be quite overwhelming. Keeping in mind the actual business cases you’re trying to solve; how can you keep your head above water and evolve your IT landscape to be future-proof?

At kuori we firmly believe in ecosystems of intercha­­ngeable components and therefore assembled the core of our platform (we playfully call it the heart of kuori) based on technologies enabling modularity and composability. In this article we give insights into the heart of kuori and illustrate how your business cases can truly profit from such a setup.

The heart of kuori

Containerization

Having majorly matured during the past two years, containerization plays a key role at kuori. Portability and mobility are the characteristics based on which this architectural pattern is able to provide us the same level of modularity as the world-famous Lego bricks. For us, Docker brings the flexibility to add, replace or remove components on the fly, securely let components interact with each other, and seamlessly promote components to different environments. One crucial actor for this to succeed are two talented hands capable of constructing fairytale castles out of the available bricks. The leading container orchestrator, Kubernetes, has the potential to fulfil this role.

Cloud

Once we are capable of constructing the castles of our dreams (or better: the castles of our customer’s dreams), we need a way of shipping them to multiple locations. Kubernetes brings infrastructure-agnostic deployments to the table or at least obscures most of the pains when deploying to various environments. We prefer to have our castles in the cloud. Deploying to the cloud facilitates simpler integration between the ecosystem and the underlying infrastructure, e.g. managed storage solutions provide the means for stateful applications inside the platform.

All major cloud platforms offer managed Kubernetes, however, for us as a platform builder it is important to have a full ‘vanilla’ experience. This allows us to fully understand the underlying mechanisms of the technology and enables us to work with other Kubernetes flavours. Therefore our internal clusters run on GCP, AWS, and Azure which in contrast to platforms like Openshift and Cloud Foundry don’t add an extra layer on top of the Kubernetes CLI.

Event-driven

Lastly, being ‘your data platform provider’ kuori builds castles to move data around. The same way our castles are constantly reshaped along the way, we crave the freedom to let data travel in various directions. Setting up new ingestion flows, transforming data to a new model, storing intermediate results in a new data repository, or publishing undiscovered data all on the go, delivers us the power to smoothly adapt to the regularly changing requirements.

To achieve the potential of such a pluggable data architecture, some kind of message broker is typically placed in the middle. Amongst others the popular candidates are established platforms like RabbitMQ and ActiveMQ and the new kids on the block: Apache Pulsar and NATS. We placed our (substantiated) bets on Apache Kafka. Its publish-subscribe pattern realizes the possibility to introduce new applications without disrupting the existing data flows. Connecting repeated cycles of applications indicating they are interested in some pieces of data (subscribe), performing actions on that data, and announce the results (publish) creates an open space of possibilities.

Benefits in a VUCA world

In today’s era of volatility, there is no other way but to re-invent. The only sustainable advantage you can have over others is agility, that’s it. Because nothing else is sustainable, everything else you create, somebody else will replicate. Jeff Bezos

Sure, exploiting Kubernetes to go to the Cloud and having Apache Kafka in place to handle the evolving data flows doesn’t fill the bill entirely. But it’s a starting point. More so, it’s a starting point that forms the foundation on which the solutions to your business cases can grow. In a world of rapidly changing needs and uncertain requirements such an accelerator grants companies to start small, fail often, and improve continuously.

It may well be the best advice we give our clients: “Aim for the moon, shoot for the stars”. Start small. Ecosystems focused on modularity give you the means to explore and begin implementing isolated use cases that could instantly deliver real business value. Evolve your platform as you grow and extend it with new components and data flows whilst developing new cases one by one.

Being able to rely on a robust backbone should encourage you to embrace failure. Stop over-analyzing every requirement and start building. As requirements change and new insights emerge, everything is in place to modify the set of components and tweak the corresponding data flows accordingly.

Conclusion

It was our pleasure to share with you how we at kuori prepare ourselves for what the future brings. Being able to efficiently adapt to change gives us to confidence to focus on delivering value fast. How does you company deal with these kind of problems? Does our approach sounds familiar? Let us know!