DC/OS Apache Kafka is an automated service that makes it easy to deploy and manage Apache Kafka on Mesosphere DC/OS, eliminating nearly all of the complexity traditionally associated with managing a Kafka cluster. Apache Kafka is a distributed high-throughput publish-subscribe messaging system with strong ordering guarantees. Kafka clusters are highly available, fault tolerant, and very durable. For more information on Apache Kafka, see the Apache Kafka documentation. DC/OS Kafka gives you direct access to the Kafka API so that existing producers and consumers can interoperate. You can configure and install DC/OS Kafka in moments. Multiple Kafka clusters can be installed on DC/OS and managed independently, so you can offer Kafka as a managed service to your organization.


DC/OS Kafka offers the following benefits of a semi-managed service:

  • Easy installation
  • Multiple Kafka clusters
  • Elastic scaling of brokers
  • Replication for high availability
  • Kafka cluster and broker monitoring


DC/OS Kafka provides the following features:

  • Single-command installation for rapid provisioning
  • Multiple clusters for multiple tenancy with DC/OS
  • High availability runtime configuration and software updates
  • Storage volumes for enhanced data durability, known as Mesos Dynamic Reservations and Persistent Volumes
  • Integration with syslog-compatible logging services for diagnostics and troubleshooting
  • Integration with statsd-compatible metrics services for capacity and performance monitoring

Quick Start

Install a Kafka cluster. If you are using open source DC/OS, install a Kafka cluster with the following command from the DC/OS CLI. If you are using Enterprise DC/OS, you may need ...

Install and Customize

About installing Kafka on Enterprise DC/OS In Enterprise DC/OS strict security mode, Kafka requires a service account. In permissive, a service account is optional. Only someone wi...


Run the following command from the DC/OS CLI to uninstall Kafka. Alternatively, you can uninstall Kafka from the DC/OS web interface. More information about uninstalling DC/OS serv...


Changing Configuration at Runtime You can customize your cluster in-place when it is up and running. The Kafka scheduler runs as a Marathon process and can be reconfigured by chang...

Connecting Clients

The only supported client library is the official Kafka Java library, i.e., org.apache.kafka.clients.consumer.KafkaConsumer and org.apache.kafka.clients.producer.KafkaProducer. Oth...


Add a Broker Increase the BROKER_COUNT value via the DC/OS web interface as in any other configuration update. Upgrade Software In the DC/OS web interface, destroy the Kafka schedu...

API Reference

REST API Authentication REST API requests must be authenticated. This authentication is only applicable for interacting with the Kafka REST API directly. You do not need the token ...


The Kafka service will be listed as “Unhealthy” when it detects any underreplicated partitions. This error condition usually indicates a malfunctioning broker. Use the ...


Configurations The “disk” configuration value is denominated in MB. We recommend you set the configuration value log_retention_bytes to a value smaller than the indicat...