}}

Kafka

Preview Updated: March 28, 2017

DC/OS Apache Kafka is an automated service that makes it easy to deploy and manage Apache Kafka on Mesosphere DC/OS, eliminating nearly all of the complexity traditionally associated with managing a Kafka cluster. Apache Kafka is a distributed high-throughput publish-subscribe messaging system with strong ordering guarantees. Kafka clusters are highly available, fault tolerant, and very durable. For more information on Apache Kafka, see the Apache Kafka documentation. DC/OS Kafka gives you direct access to the Kafka API so that existing producers and consumers can interoperate. You can configure and install DC/OS Kafka in moments. Multiple Kafka clusters can be installed on DC/OS and managed independently, so you can offer Kafka as a managed service to your organization.

Benefits

DC/OS Kafka offers the following benefits of a semi-managed service:

  • Easy installation
  • Multiple Kafka clusters
  • Elastic scaling of brokers
  • Replication for high availability
  • Kafka cluster and broker monitoring

Features

DC/OS Kafka provides the following features:

  • Single-command installation for rapid provisioning
  • Multiple clusters for multiple tenancy with DC/OS
  • High availability runtime configuration and software updates
  • Storage volumes for enhanced data durability, known as Mesos Dynamic Reservations and Persistent Volumes
  • Integration with syslog-compatible logging services for diagnostics and troubleshooting
  • Integration with statsd-compatible metrics services for capacity and performance monitoring

Install and Customize

Kafka is available in the Universe and can be installed by using either the web interface or the DC/OS CLI. Prerequisites Depending on your security mode in Enterprise DC/OS, you m...

Uninstall

Run the following command from the DC/OS CLI to uninstall Kafka. Alternatively, you can uninstall Kafka from the DC/OS web interface. More information about uninstalling DC/OS serv...

Configure

Changing Configuration at Runtime You can customize your cluster in-place when it is up and running. The Kafka scheduler runs as a Marathon process and can be reconfigured by chang...

Connecting Clients

The only supported client library is the official Kafka Java library, i.e., org.apache.kafka.clients.consumer.KafkaConsumer and org.apache.kafka.clients.producer.KafkaProducer. Oth...

Managing

Add a Broker Increase the BROKER_COUNT value via the DC/OS web interface as in any other configuration update. Upgrade Software In the DC/OS web interface, destroy the Kafka schedu...

API Reference

REST API Authentication REST API requests must be authenticated. This authentication is only applicable for interacting with the Kafka REST API directly. You do not need the token ...

Troubleshooting

The Kafka service will be listed as “Unhealthy” when it detects any underreplicated partitions. This error condition usually indicates a malfunctioning broker. Use the ...

Limitations

Configurations The “disk” configuration value is denominated in MB. We recommend you set the configuration value log_retention_bytes to a value smaller than the indicat...