My original post is written at Grab Engineering blog https://engineering.grab.com/kafka-connect Grab’s real-time data platform team a.k.a. Coban has written about Plumbing at scale, Optimally scaling Kakfa consumer applications, and Exposing Kafka via VPCE. …

My original post is written at Grab Engineering blog https://engineering.grab.com/zero-trust-with-kafka Introduction Grab’s real-time data platform team, also known as Coban, has been operating large-scale Kafka clusters for all Grab verticals, with a strong focus on ensuring a best-in-class-performance and 99.99% availability. Security has always been one of Grab’s top priorities and…

1.Motivation Previously, when we connect to TLS listener in Kafka, we have to prepare the keystore and truststore in JKS format. JKS stands for Java Keystore which means it used explicitly for Java world. However, the rest of the world have adopted a common format i.e PEM Recently, KIP-651 proposes additional…

Is Cluster Autoscaler alone enough ? Cluster Autoscaler (CA) is part of the Kubernetes ecosystem that allows automatical addition or removal or nodes in a cluster based on resource request from pods. You may have heard of Horizonal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA) but let’s save those topics for another day :P In…

What I have learnt from running Debezium Connectors with Kafka cluster at scales Why do we need transactional outbox? In the realm of microservices, often, application will have its own datastore. It can be a MySQL or PostgreSQL database that the application can persist data into. At the same time, the same application will want to…

Recently, I’m really lucky to have a chance to work with Kafka and have a deeper understanding on how it works. Hence, this post will be about client.dns.lookup , one of the many nuances that you have to deal with when working with, maintaining kafka cluster. What is it about? It is one of…