RE: Does Diyotta provide options to log the Spark execution details to its history server?
Diyotta enables spark event logs by adding below properties from “Runtime Properties->Run Property Type->Spark UI”
1)spark.eventLog.enabled (Whether to log Spark events, useful for reconstructing the Web UI after the application has finished.)
2)spark.eventLog.dir (Base directory in which Spark events are logged, if spark.eventLog.enabled is true. Within this base directory, Spark creates a sub-directory for each application, and logs the events specific to the application in this directory. Users may want to set this to a unified location like an HDFS directory so history files can be read by the history server.)
3)spark.eventLog.compress (Whether to compress logged events, if spark.eventLog.enabled is true.)