Job Scheduling

Stable Updated: March 16, 2017

This document is a simple overview of material described in greater
detail in the Apache Spark documentation here and here.


Spark on Mesos supports two “modes” of operation: coarse-grained mode
and fine-grained mode. Coarse-grained mode provides lower latency,
whereas fine-grained mode provides higher utilization. More info

Coarse-grained mode

“Coarse-grained” mode is so-called because each Spark executor is
represented by a single Mesos task. As a result, executors have a
constant size throughout their lifetime.

  • Executor memory: spark.executor.memory
  • Executor CPUs: spark.executor.cores, or all the cores in the
  • Number of Executors: spark.cores.max /
    spark.executor.cores. Executors are brought up until
    spark.cores.max is reached. Executors survive for duration of the
  • Executors per agent: Multiple


  • We highly recommend you set spark.cores.max. If you do not, your
    Spark job may consume all available resources in your cluster,
    resulting in unhappy peers.

Fine-grained mode

In “fine-grained” mode, each Spark task is represented by a single
Mesos task. When a Spark task finishes, the resources represented by
its Mesos task are relinquished. Fine-grained mode enables
finer-grained resource allocation at the cost of task startup latency.

  • Executor memory: spark.executor.memory
  • Executor CPUs: Increases and decreases as tasks start and
  • Number of Executors: Increases and decreases as tasks start
    and terminate.
  • Executors per agent: At most 1


The following is a description of the most common Spark on Mesos
scheduling properties. For a full list, see the Spark configuration
and the Spark on Mesos configuration page.

property default description
spark.mesos.coarse true Described above.
spark.executor.memory 1g Executor memory allocation.
spark.executor.cores All available cores in the offer Coarse-grained mode only. DC/OS Spark >= 1.6.1. Executor CPU
spark.cores.max unlimited Maximum total number of cores to allocate.