PerfGazer, Performance Gazer for Apache Spark

This repository contains the PerfGazer Spark Listener.

PerfGazer is a configurable Spark Listener that allows to retrieve important stats about Spark SQL queries, jobs and stages in a post-mortem way. Users should attach it as a listener to the SparkSession. It will cause to report certain lines in the logs, which can be interpreted afterwards.

Use cases

The use-cases that this library is intended to address:

• allow to do post-mortem analysis of Spark SQL queries, jobs and stages programmatically
• measure Spark jobs / stages accumulated in-executor durations
• identify jobs that take the longer cumulated execution time (as measured in executors)
• identify Spark jobs that have spill
• monitor certain SQL metrics like amount of files read, pruned, ...
• investigate predicate pushdowns and their effectiveness on data skipping
• connect to any monitoring system to expose certain metrics (spill, files read, ...)
• ...

Why not just the Spark UI?

There are some problems with the analysis of execution stats from the Spark UI:

• the process is mostly manual (UI navigation)
• it is often slow (takes time to load the UI)
• has a limited sql queries / jobs retention (so stats data is often purged for large applications)
• not made for analytics (i.e. how many of my SQL queries used 'BroadcastHashJoin'?)

---

Documentation

For setup instructions, usage guides, and more, visit the documentation site.

For AI agents

• llms.txt — index of all documentation pages
• llms-full.txt — full documentation as a single Markdown file
• perfgazer-schema.json — data model schema (SQL view definitions) as structured JSON

The documentation site is versioned. Replace latest in the URLs above with dev for the development version or a specific release tag (e.g. v0.1.0) to access that version's docs. Available versions can be found on the Releases page.

Authors

• Mauricio JOST
• Generoso PAGANO
• Bruno JOUBERT
• Thierry ACCART
• Sergei DOLGOV
• Mathieu TRAMPONT