Apache Spark 4.0.0

Author: jaceklaskowski

docs/index.md

@@ -10,11 +10,13 @@ hide:
 
 Welcome to **The Internals of {{ book.title }}** online book! 🤙
 
-I'm [Jacek Laskowski](https://pl.linkedin.com/in/jaceklaskowski), a Freelance Data(bricks) Engineer specializing in
+I'm [Jacek Laskowski](https://www.linkedin.com/in/jaceklaskowski), a Freelance Data(bricks) Engineer 🧱 specializing in
 [Apache Spark](https://books.japila.pl/apache-spark-internals/) (incl. [Spark SQL](https://books.japila.pl/spark-sql-internals/) and [Spark Structured Streaming](https://books.japila.pl/spark-structured-streaming-internals/)),
 [Delta Lake](https://books.japila.pl/delta-lake-internals/),
-[Databricks](https://www.databricks.com/),
-and [Apache Kafka](https://books.japila.pl/kafka-internals/) (incl. [Kafka Streams](https://books.japila.pl/kafka-streams-internals/)) with brief forays into a wider data engineering space (e.g., [Trino](https://trino.io/), [Dask](https://www.dask.org/) and [dbt](https://www.getdbt.com/), mostly during [Warsaw Data Engineering](https://www.meetup.com/Warsaw-Data-Engineering/) meetups).
+[Unity Catalog](https://books.japila.pl/unity-catalog-internals/),
+[MLflow](https://mlflow.org/),
+[Databricks](https://github.com/jaceklaskowski/learn-databricks)
+with brief forays into a wider data engineering space (mostly during [Warsaw Data Engineering](https://www.meetup.com/Warsaw-Data-Engineering/) meetups).
 
 I'm very excited to have you here and hope you will enjoy exploring the internals of {{ book.title }} as much as I have.
 

mkdocs.yml

@@ -7,7 +7,7 @@ repo_name: spark-internals
 repo_url: https://github.com/japila-books/apache-spark-internals
 edit_uri: edit/main/docs/
 
-copyright: Copyright © 2023-2024 Jacek Laskowski
+copyright: Copyright © 2025 Jacek Laskowski
 
 theme:
   name: material
@@ -106,34 +106,39 @@ extra:
     home: https://metrics.dropwizard.io
     api: https://www.javadoc.io/static/io.dropwizard.metrics/metrics-core/4.2.7
   hadoop:
-    version: 3.3.4
-    docs: https://hadoop.apache.org/docs/r3.3.4
-    api: https://hadoop.apache.org/docs/r3.3.4/api
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L126
+    version: 3.4.1
+    docs: https://hadoop.apache.org/docs/r3.4.1
+    api: https://hadoop.apache.org/docs/r3.4.1/api
   jackson:
     api: https://fasterxml.github.io/jackson-databind/javadoc/2.12
   java:
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L117
+    version: 17
     api: https://docs.oracle.com/en/java/javase/17/docs/api/java.base
+    spec: https://docs.oracle.com/javase/specs/jls/se17/html
   scala:
-    version: 2.13.8
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L173
+    version: 2.13.16
     binary_version: 2.13
-    api: https://www.scala-lang.org/api/2.13.8
+    api: https://www.scala-lang.org/api/2.13.16
     docs: https://docs.scala-lang.org/
     spec: https://www.scala-lang.org/files/archive/spec/2.13
   social:
     - icon: fontawesome/brands/github
       link: https://github.com/jaceklaskowski
-    - icon: fontawesome/brands/twitter
-      link: https://twitter.com/jaceklaskowski
+    - icon: fontawesome/brands/bluesky
+      link: https://bsky.app/profile/books.japila.pl
     - icon: fontawesome/brands/linkedin
       link: https://linkedin.com/in/jaceklaskowski
     - icon: fontawesome/brands/medium
       link: https://jaceklaskowski.medium.com
-    - icon: fontawesome/brands/mastodon
-      link: https://fosstodon.org/@jaceklaskowski
+    - icon: fontawesome/brands/twitter
+      link: https://twitter.com/jaceklaskowski
   spark:
-    version: 3.5.2
-    api: https://spark.apache.org/docs/latest/api/scala
-    github: https://github.com/apache/spark/tree/v3.5.2
+    # https://github.com/apache/spark/blob/v4.0.0/pom.xml#L29
+    version: 4.0.0
+    github: https://github.com/apache/spark/tree/v4.0.0
     jira: https://issues.apache.org/jira/browse
   tags:
     DeveloperApi: developerapi

requirements.txt

@@ -1,6 +1,6 @@
 git+https://${GH_TOKEN}@github.com/squidfunk/mkdocs-material-insiders.git
 mkdocs-minify-plugin>=0.7.2
-mkdocs-git-revision-date-localized-plugin>=1.2.2
+mkdocs-git-revision-date-localized-plugin>=1.4.7
 mkdocs-git-revision-date-plugin>=0.3.2
 mkdocs-awesome-pages-plugin>=2.9.2
 mkdocs-redirects>=1.2.1