RE: How to configure number of executors, executor cores and executor memory settings for a Spark application created by Diyotta?

How to configure number of executors, executor cores and executor memory settings for a Spark application created by Diyotta?

TonyP Beginner Asked on July 10, 2019 in Spark Process Platform.

Good question

on July 10, 2019.
Add Comment
1 Answers

Executor configurations can be set up from 2 different places in Diyotta Studio

  • Spark Data Point
  • Spark Data Flow

On spark data point, the configuration is available under “Runtime Properties”.

RE: How to configure number of executors, executor cores and executor memory settings for a Spark application created by Diyotta?

Spark application created for this datapoint will be with the configuration set in this window. All data flows that use this data point will make use of the same configuration. However, this behavior can be overridden by choosing “Run as separate Spark application” from the data flow properties and then setting the configuration from the Data Flow level “Runtime properties”. This option creates separate Spark application per data flow each with their own configuration.

RE: How to configure number of executors, executor cores and executor memory settings for a Spark application created by Diyotta?

Expert Answered on July 10, 2019.
Add Comment

Your Answer

By posting your answer, you agree to the privacy policy and terms of service.