Create testable, reproduceable documentation with Jupyter notebooks

Overview

Deploy to GitHub Pages

Create Testable, Reproduceable Docs and Blogs With Notebooks

Never copy and paste code into documentation again!

👉 See a live example of a post made with notebooks 🌐 🚀

Background

This repo provides an example of how to create blog posts and/or documentation with Docusaurus using Jupyter Notebooks. When you create documentation with notebooks:

  • Your documentation is fully testable with unit tests
  • No more copy / pasting code into your blog posts and docs
  • The output of code is always up to date, since it is generated by actually running code, rather than copying and pasting it from somewhere else.
  • You can author technical posts and docs in a WYSIWYG manner, which allows you to iterate faster.
  • You can still write markdown posts if you want to.

Getting Started

This project assumes some familiarity with static site generators. It is also very helpful to gain some familiarity with Docusaurus and it's general features.

1. Fork this repo

To create your own blog or docs site, you can start by forking this repo, or creating a new repo by using this one as a repository template.

This repo has both blog and docs, but you might want to enable blog only mode or docs only mode

2. Edit settings.ini

After that, you should change the follwing lines in the settings.ini file. The keys are described below:

[DEFAULT]
nbs_path = .
recursive = True
tst_flags = notest
-user = outerbounds
+user = <your github username>
-doc_host = https://outerbounds.github.io
+doc_host = https://<your github user name>.github.io or your custom domain
-doc_baseurl = /nbdoc-docusaurus
+doc_baseurl = /<name of your repo> or path
-module_baseurls = metaflow=https://github.com/Netflix/metaflow/tree/master/
+module_baseurls = <your python module>=https://github.com/a/path/tree/master/

The definintions of these fields are as follows:

  • nbs_path: the top most in your directory that contains notebooks to be converted to markdown files. The default value for this ., which just means the root of the repo.
  • recursive: set to True if you want to recurisvely find all notebooks under nbs_path, and False otherwise. Defaults to True
  • tst_flags: special cell comment that will allow yout skip cell execution and test. For example, adding the comment #notest in a code cell would result skip cell execution for that cell. You can add mulitple values in this field seperated by a pipe |
  • doc_host: this is the domain where your docs are served. If your docs are served on GitHub Pages, this is typically your-username.github.io
  • doc_baseurl: the path on your domain that has the docs. This is usually the root / so doesn't normally need to be changed.
  • module_baseurl: This is optional for a situations where you want to document python apis. This allows documentation to include links to source code.

3. Install Dependencies

  1. Before you get started, you need to make sure you have node and npm installed.

  2. Next, create an isolated python environment using your favorite tool such as conda, pipenv, poetry etc. Then, from the root of this repo run this command in the terminal:

    make install

Note that this references requirements.txt, which you will have to update appropriately anytime you include addtional python dependencies in your documentation.

Using Notebooks

Notebook Development Setup

  1. You need to open 3 different terminal windows (I recommend using split panes), and run the following commands in three seperate windows:

    Note: we tried to use docker-compose but had trouble getting some things to run on Apple Silicon, so this will have to do for now.

    Start the docs server:

    make docs

    Watch for changes to notebooks:

    make watch

    Start Jupyter Lab:

    make nb
  2. Open a browser window for the authoring guide in the docs. You may have to hard-refresh the first time you make a change, but hot-reloading generally works.

Tutorial

To go through the tutorial, open the rendered authoring guide in one window, and the notebook docs/nb.ipynb in another window. Study the notebook cells and the corresponding rendered page carefully. You will see many options for:

  • How to deal with scripts vs interactive code cells
  • How to run bash commands / scripts
  • How to setup front matter for page configuration
  • Hiding/Showing cell inputs, outputs or both
  • Selectively displaying Metaflow logs
  • How to setup tests
  • How to Author API docs

After you complete the tutorial, you should run through this list and make sure you know how to do all of the above tasks.

Running Tests & Updating Notebooks

To test the notebooks, run make test from the root of this repo. This will execute all notebooks in parallel and report an error if there are any errors found.

Skipping tests in cells

If you want to skip certain cells from running in tests because they take a really long time, you can place the comment #notest at the top of the cell. You can add additional flags by adding pipe-delimited values into the tst_flags field of settings.ini.

Update all notebooks

You can refresh all notebooks in place with make update. This runs all notebooks programatically, skipping over cells with #notest (or other tst_flags flags you set) and saves the result to the same filename. This will also re-generate the markdown files from the updated notebooks.

The reason for providing make update seperately from make test is so you can have the option of first testing your notebooks without overwriting them.

Ignoring Skip Flags

We have previously discussed that you can skip tests or notebook refreshes in specific cells with the flag #notest (or other flags you set in settings.ini). You can force these cells to be run or refreshed with the --flags argument. For example, let's say we want to run tests or refresh notebooks and do not want to ignore cells with the #notest flag:

  • For testing: nboc_test --flags #notest
  • For refreshing notebooks: nbdoc_update --flags #notest

Hotkeys For Jupyter

People complain about "state" in Jupyter. This can be easily avoided by frequently restarting the kernel and running all cells from the top. Thankfully, you can set a hotkey that allows you to do this effortlessly. In Jupyter Lab, go to Settings then Advanced Settings Editor. Copy and paste the below json into the User Prefences pane. If you already have user-defined shortcuts, modify this appropriately.

{
"shortcuts": [
    {
        "command": "notebook:restart-run-all",
        "keys": [
            "Ctrl R",
            "R"
        ],
        "selector": "body",
    },
    {
        "command": "notebook:restart-and-run-to-selected",
        "keys": [
            "Ctrl R",
            "S"
        ],
        "selector": "body",
    },
...
}

Using Markdown

You do not have to use notebooks to update documentation. You can use markdown as you would normally do with Docusaurus. This is a good option when there isn't much code in the document you are trying to write.

Running the documentation locally

  • Clone this repo
  • cd nbdoc-docusarus
  • make install
  • make docs

Any saved changes that you make to the .md files in the docs or blogs directory will automatically be reflected at the local preview page.

Automatic publishing

Any pushes to the master branch will automatically publish to the live documentation pages . The publishing uses GitHub Actions and Github pages. You can see the progress of the publish action by going to the Actions tab of this repository.

About

We use nbdoc as a notebook converter and testing utility, as well as docusaurus for the static site generator. This technology is built using nbdev, a literate programming system.

You might also like...

Open Source projects are a project to improve your JavaScript knowledge with JavaScript documentation, design patterns, books, playlists.

Open Source projects are a project to improve your JavaScript knowledge with JavaScript documentation, design patterns, books, playlists.

It is a project I am trying to list the repos that have received thousands of stars on Github and deemed useful by the JavaScript community. It's a gi

Aug 14, 2022

Support documentation for Decrypt's applications, token, and community.

Website This website is built using Docusaurus 2, a modern static website generator. Installation npm install Local Development npm start This command

Dec 8, 2022

Automatically generated documentation for the Valorant API endpoints the client uses internally.

Valorant API Docs To read documentation and get started, see Docs This is a project designed to automatically document Valorant endpoints based on a J

Dec 25, 2022

Maintain your API development, documentation and experimentation all within your codebase.

Maintain your API development, documentation and experimentation all within your codebase.

dostman Maintain your API development, documentation and experimentation all within your codebase. This project is a new release! Feel free to contact

Mar 11, 2022

Website for the Io Language. Includes documentation

Website for the Io Language. Includes documentation

TheIoLang Website Contributors ✨ Thanks goes to these wonderful people (emoji key): Dillon Barnes 💻 🖋 🎨 🚇 anime fanatic 📆 This project follows th

Dec 22, 2021

Swagger UI is a collection of HTML, JavaScript, and CSS assets that dynamically generate beautiful documentation from a Swagger-compliant API.

Swagger UI is a collection of HTML, JavaScript, and CSS assets that dynamically generate beautiful documentation from a Swagger-compliant API.

Introduction Swagger UI allows anyone — be it your development team or your end consumers — to visualize and interact with the API’s resources without

Dec 28, 2022

A multipurpose transporting app that contains, loads databases, documentation and more.

TransportingApp This project was generated with Angular CLI version 12.0.3. Development server Run ng serve for a dev server. Navigate to http://local

Jan 21, 2022

A magical vite plugin that helps you to generate and manage documentation website.

vite-plugin-book A magical vite plugin that helps you to generate and manage documentation website. ⚠️ This project is still WIP. It's a MVP now. A bu

Dec 20, 2022

Link your documentation to the relevant code files

Link your documentation to the relevant code files

Jul 19, 2022
Comments
  • Utilizing nbdoc for existing docusaurus pages

    Utilizing nbdoc for existing docusaurus pages

    I want to support using jupyter notebooks for creating docs in my existing docusaurus repo. The README suggests cloning this repo, but where could I find steps to configure it to my existing repo?

    opened by shenoynikhil 0
Owner
Outerbounds
Building the modern, human-centric ML infrastructure stack.
Outerbounds
An obsidian plugin that allows code blocks executed interactively in sandbox like jupyter notebooks. Supported language rust、kotlin、python、Javascript、TypeScript etc.

Obsidian Code Emitter This plugin allows code blocks executed interactively like jupyter notebooks. Currently, support languages: Rust Kotlin JavaScri

YiiSh 38 Dec 28, 2022
Transpile TypeScript to Espruino flavor of JavaScript for ESP8266/ESP32 microcontrollers template project. Writing safe, maintainable, and testable code is the projects goal.

ESP-TypeScript Transpile TypeScript to Espruino flavor of JavaScript for ESP8266/ESP32 microcontrollers template project. Writing safe, maintainable,

Spencer Kittleson 152 Dec 20, 2022
Shikhar 4 Oct 9, 2022
An Inkdrop plugin to synchronise notebooks and notes with Todoist projects and tasks.

inkdrop-todoist-sync Todoist Sync is an Inkdrop plugin that allows synchronising of Inkdrop notebooks and notes with Todoist projects, sections and ta

Manuel Steiner 5 Aug 23, 2022
Chrome Extension To Reveal Observable Notebooks As Quarto QMD {ojs} Blocks & provide downloads of FileAttachments and zipped Quarto project

reveal-qmd Chrome Extension To Reveal Observable Notebooks As Quarto QMD {ojs} Blocks Usage: git clone [email protected]:hrbrmstr/reveal-qmd In Chrome (e

boB Rudis 20 Nov 29, 2022
Random Fractals stash of Observable Data Tools 🛠️ and Notebooks 📚 in ES Modules .js, .nb.json, .ojs, .omd, .html and .qmd document formats for Data Previews

Random Fractals stash of Observable Data Tools ??️ and Notebooks ?? in ES Modules .js, .nb.json, .ojs, .omd, .html and .qmd document formats for Data Previews in a browser and in VSCode IDE with Observable JS extension, Quarto extension, and new Quarto publishing tools.

Taras Novak 14 Nov 25, 2022
WhyProfiler is a CPU profiler for Jupyter notebook that not only identifies hotspots but can suggest faster alternatives.

Introduction WhyProfiler is a CPU profiler for Jupyter notebook that not only identifies hotspots but can suggest faster alternatives. It is powered b

Robusta 44 Dec 5, 2022
Interactive Text Annotation for Jupyter Notebook/Lab

Interactive Text Annotation for Jupyter Notebook/Lab Perform entity extraction inline without leaving your notebook. Iteratively label and train a mod

null 8 Dec 26, 2022
This extension allows you to create Markdown files which are automatically transformed to in-game documentation for your Add-On.

Markdown Doc Generator An extension for bridge editor which makes creating docs in minecraft easier, by generating json-ui from markdown View extensio

Freddie 7 Dec 31, 2022
API Documentation Browser

DevDocs — API Documentation Browser DevDocs combines multiple developer documentations in a clean and organized web UI with instant search, offline su

freeCodeCamp.org 30.5k Jan 9, 2023