The Elastic Server can process Spark tasks concurrently and queue additional Spark tasks. You can configure concurrency properties to enable or disable queuing and set the maximum number of concurrent Spark tasks.
The following Elastic Server properties define queuing and concurrency:
allow_queuing
Allows the Elastic Server to queue Spark tasks. The default value is set to true.
If this property is set to false, the Elastic Server processes all submitted Spark tasks concurrently. If you set the maximum number of concurrent tasks on the Elastic Server to a value that is higher than the default concurrency, increase the Java heap size accordingly. Otherwise, the Secure Agent process might crash.
The Elastic Server processes data preview and SQL ELT optimization jobs immediately without queuing them.
max_concurrent_jobs
Maximum number of concurrent Spark tasks that the Elastic Server can process. When the Elastic Server reaches the maximum number of concurrent Spark tasks, additional Spark tasks can be queued.
By default, the maximum number of concurrent Spark tasks depends on the cloud platform. The Elastic Server uses the following defaults for fully-managed and self-service clusters:
- 500 concurrent Spark tasks per 2 GB of Java heap space on AWS
- 375 concurrent Spark tasks per 2 GB of Java heap space on Google Cloud
- 250 concurrent Spark tasks per 2 GB of Java heap space on Microsoft Azure
A local cluster can run 15 concurrent Spark tasks, and the concurrency can't be changed.
Note that the Java heap size is set to 2 GB by default. If you set the maximum number of concurrent tasks to a value that is higher than the default, increase the Java heap size using the default ratios.
For example, if you want to run 500 concurrent Spark tasks on Microsoft Azure, increase the Java heap size to 4 GB. If the Java heap size remains at 2 GB, the Elastic Server will process a maximum of 250 concurrent Spark tasks and the remaining 250 Spark tasks can be queued.
The Elastic Server uses an upper bound of 1,000 Spark tasks per 2 GB of Java heap space. If you set the maximum number of concurrent tasks to a value that is higher than 1,000 without increasing the Java heap size above 2 GB, the Elastic Server fails to start.