Skip to main content

Deploying Apache Kafka on Qubinets

Deploying an application like Apache Kafka on Qubinets involves several steps. Qubinets provides a platform to manage and orchestrate Kubernetes clusters, making it simpler to deploy and manage applications in a cloud-native environment. Here is a detailed tutorial to guide you through the process:

Prerequisites​

  1. Qubinets Account: Ensure you have an account on Qubinets. Sign up if you don’t have one.
  2. Kubernetes Cluster: You need an existing Kubernetes cluster or you can create a new one using Qubinets.
  3. kubectl: Ensure kubectl is installed and configured to interact with your Kubernetes cluster.
  4. Helm: Install Helm, which simplifies the deployment of applications on Kubernetes.

Step 1: Create a Kubernetes Cluster on Qubinets​

  1. Login to Qubinets: Go to the Qubinets website and log in with your credentials.
  2. Create a Cluster:
    • Navigate to the "Clusters" section.
    • Click on "Create Cluster".
    • Choose the appropriate configuration for your cluster (e.g., cloud provider, node size, number of nodes, etc.).
    • Follow the prompts to create the cluster. This may take several minutes.

Step 2: Configure kubectl​

  1. Download Configuration File:
    • Once the cluster is created, download the kubeconfig file from the Qubinets dashboard.
    • Save the kubeconfig file to a secure location on your machine.
  2. Set KUBECONFIG Environment Variable: 
    export KUBECONFIG=/path/to/your/kubeconfig
  3. Verify kubectl Access: 
    kubectl get nodes
    • This should list the nodes in your cluster, indicating that kubectl is correctly configured.

Step 3: Install Helm​

  1. Download Helm: Follow the installation guide on the Helm website for your specific operating system.
  2. Initialize Helm: 
    helm repo add bitnami https://charts.bitnami.com/bitnami
    helm repo update

Step 4: Deploy Apache Kafka using Helm​

  1. Search for Kafka Chart: 
    helm search repo kafka
  2. Install Kafka: 
    helm install my-kafka bitnami/kafka
    • Replace my-kafka with a name of your choice for the release.
    • This command deploys Kafka with default settings.

Step 5: Customize Kafka Deployment​

  1. Create a values.yaml File:
    • Customize Kafka settings by creating a values.yaml file.
    replicaCount: 3
    config:
      log.dirs: /bitnami/kafka/data
      default.replication.factor: 3
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
    persistence:
      enabled: true
      size: 10Gi
      storageClass: standard
  2. Deploy with Custom Values: 
    helm install my-kafka -f values.yaml bitnami/kafka

Step 6: Verify the Deployment​

  1. Check Pods: 
    kubectl get pods
    • Ensure that the Kafka pods are running.
  2. Check Services: 
    kubectl get svc
    • Verify that the Kafka services are created and accessible.

Step 7: Access Kafka​

  1. Port Forwarding:
    • Forward a local port to the Kafka service.
    kubectl port-forward svc/my-kafka 9092:9092
    • This forwards port 9092 on your local machine to the Kafka service.
  2. Connect to Kafka:
    • Use a Kafka client to connect to the Kafka broker.
    kafka-console-producer.sh --broker-list localhost:9092 --topic test
    kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning

Step 8: Manage and Scale Kafka​

  1. Scale the Deployment:
    • Increase or decrease the number of replicas.
    kubectl scale statefulset my-kafka --replicas=5
  2. Upgrade the Deployment:
    • Apply new configuration changes using Helm.
    helm upgrade my-kafka -f values.yaml bitnami/kafka

Step 9: Backup and Restore​

  1. Backup:
    • Use a tool like kafka-backup to create a backup of Kafka topics.
    kafka-backup --topics test --backup-dir /path/to/backup --broker-list localhost:9092
  2. Restore:
    • Use the same tool to restore Kafka topics from a backup.
    kafka-restore --backup-dir /path/to/backup --broker-list localhost:9092

Step 10: Monitor Kafka​

  1. Prometheus and Grafana:
    • Deploy Prometheus and Grafana to monitor Kafka.
    • Install Prometheus:
    helm install prometheus bitnami/prometheus
    • Install Grafana:
    helm install grafana bitnami/grafana
    • Configure dashboards to monitor Kafka metrics.

Conclusion​

Deploying Apache Kafka on Qubinets using Kubernetes and Helm simplifies the process of managing a robust and scalable streaming platform. By following these steps, you can efficiently deploy, manage, and monitor Kafka on your Kubernetes cluster.

Remember to regularly update your Helm charts and Kubernetes configurations to ensure security and performance optimizations. Happy deploying!