The Internals of Apache Spark Online Book

The project contains the sources of The Internals of Apache Spark online book.

Tools

The project is based on or uses the following tools:

• Apache Spark

• MkDocs which strives for being a fast, simple and downright gorgeous static site generator that's geared towards building project documentation

• Material for MkDocs theme

• Markdown

• Visual Studio Code as a Markdown editor

• Docker to run the Material for MkDocs (with plugins and extensions)

Previewing Book

Custom Docker Image

This project uses a custom Docker image (based on Dockerfile) since the official Docker image includes just a few plugins only.

Build the custom Docker image using the following command:

docker build -t macros-material .

Building Book

Run the following command to build the book.

docker run \
  -it \
  -p 8000:8000 \
  -v ${PWD}:/docs macros-material \
  build --clean

Consult the MkDocs documentation to get started and learn how to build the project.

$ docker run --rm -it -p 8000:8000 -v ${PWD}:/docs squidfunk/mkdocs-material --help
Usage: mkdocs [OPTIONS] COMMAND [ARGS]...

  MkDocs - Project documentation with Markdown.

Options:
  -V, --version  Show the version and exit.
  -q, --quiet    Silence warnings
  -v, --verbose  Enable verbose output
  -h, --help     Show this message and exit.

Commands:
  build      Build the MkDocs documentation
  gh-deploy  Deploy your documentation to GitHub Pages
  new        Create a new MkDocs project
  serve      Run the builtin development server

Online Editing

Start mkdocs serve (with --dirtyreload for faster reloads) as follows:

docker run \
  -it \
  -p 8000:8000 \
  -v ${PWD}:/docs macros-material \
  serve --dirtyreload --verbose --dev-addr 0.0.0.0:8000

You should start the above command in the project root (the folder with mkdocs.yml).

Use mkdocs build --clean to remove any stale files.

No Sphinx?! Why?

Read Giving up on Read the Docs, reStructuredText and Sphinx.