Screenshot of Delight, a free hosted cross-platform to track your Spark applications. Image by Author.
Introducing Delight, an open-source project providing a free & cross-platform monitoring dashboard for Apache Spark.
For “live” Spark applications, accessing the Spark UI — the open-source monitoring interface of Apache Spark — is easy, as it is served directly by the Spark driver on port 4040.
But once an application has completed, accessing the Spark UI requires setting up a Spark History Server, which takes a bit of work. In this article, we’re going to show you how to access the Spark UI for completed Spark applications with very little effort by leveraging a free open-source monitoring project called Data Mechanics Delight (see the github page).
