}}

Job Scheduling

Stable Updated: January 13, 2017

This document is a simple overview of material described in greater
detail in the Apache Spark documentation here and here.

Modes

Spark on Mesos supports two “modes” of operation: coarse-grained mode
and fine-grained mode. Coarse-grained mode provides lower latency,
whereas fine-grained mode provides higher utilization. More info
here.

Coarse-grained mode

“Coarse-grained” mode is so-called because each Spark executor is
represented by a single Mesos task. As a result, executors have a
constant size throughout their lifetime.

  • Executor memory: spark.executor.memory
  • Executor CPUs: spark.executor.cores, or all the cores in the
    offer.
  • Number of Executors: spark.cores.max /
    spark.executor.cores. Executors are brought up until
    spark.cores.max is reached. Executors survive for duration of the
    job.
  • Executors per agent: Multiple

Notes:

  • We highly recommend you set spark.cores.max. If you do not, your
    Spark job may consume all available resources in your cluster,
    resulting in unhappy peers.

Fine-grained mode

In “fine-grained” mode, each Spark task is represented by a single
Mesos task. When a Spark task finishes, the resources represented by
its Mesos task are relinquished. Fine-grained mode enables
finer-grained resource allocation at the cost of task startup latency.

  • Executor memory: spark.executor.memory
  • Executor CPUs: Increases and decreases as tasks start and
    terminate.
  • Number of Executors: Increases and decreases as tasks start
    and terminate.
  • Executors per agent: At most 1

Properties

The following is a description of the most common Spark on Mesos
scheduling properties. For a full list, see the Spark configuration
page
and the Spark on Mesos configuration page.

property default description
spark.mesos.coarse true Described above.
spark.executor.memory 1g Executor memory allocation.
spark.executor.cores All available cores in the offer Coarse-grained mode only. DC/OS Spark >= 1.6.1. Executor CPU
allocation.
spark.cores.max unlimited Maximum total number of cores to allocate.