RE: Does Diyotta provide options to log the Spark execution details to its history server?

Does Diyotta provide options to log the Spark execution details to its history server?

RaviP Beginner Asked on July 15, 2019 in Spark Process Platform.
Add Comment
1 Answers

Diyotta enables spark event logs by adding below properties from “Runtime Properties->Run Property Type->Spark UI”

1)spark.eventLog.enabled (Whether to log Spark events, useful for reconstructing the Web UI after the application has finished.)

2)spark.eventLog.dir (Base directory in which Spark events are logged, if spark.eventLog.enabled is true. Within this base directory, Spark creates a sub-directory for each application, and logs the events specific to the application in this directory. Users may want to set this to a unified location like an HDFS directory so history files can be read by the history server.)

3)spark.eventLog.compress (Whether to compress logged events, if spark.eventLog.enabled is true.)

RE: Does Diyotta provide options to log the Spark execution details to its history server?

Expert Answered on July 15, 2019.
Add Comment

Your Answer

By posting your answer, you agree to the privacy policy and terms of service.