Notebook Programming

Notebook Programming#

Table of Contents#

Jupyter#

[ d ] Jupyter Notebooks in VS Code

  • [ h ][ d ][ g ][ w ] Jupyter

    • [ d ] Get Started

    • [ d ] Architecture

  • [ h ][ d ][ g ][ w ] IPython

    • [ d ] Kernels for Conda environments

  • [ h ][ d ][ g ][ w ] Jupyter Lab

    • [ d ] Displaying Hidden Files

    • [ d ] Jupyter Extensions

  • [ h ][ d ][ g ][ w ] Jupyter Notebook

    • [ d ] The Jupyter Notebook

    • [ d ] Edit Mode and Command Mode

    • [ d ] Command Mode Shortcuts

    • [ d ] Importing Jupyter Notebook as Modules

    • [ d ] Troubleshooting Jupyter

    • [ d ] Troubleshooting Jupyter: Windows

  • [ h ][ d ][ g ][ w ] Jupyter Server

  • nbformat

    • [ d ] The Jupyter Notebook Format

  • [ h ] Binder

  • [ d ] ipyparallel

  • [ d ] ipywidgets

  • [ d ] Voila

Online

Config#

macOS

  • ~/.ipython

  • ~/.jupyter

Jupyter Notebook Shortcuts in Visual Studio Code#

Open the command pallette (Cmd-Shift-P (macoS) or Ctrl-Shift-P (Windows)) and search for Open Keyboard Shortcuts.

Shortcut

Symbol

Action

Alt-Down

⎇↓

Move the current cell below the cell below (i.e., swap them) (Windows)

Alt-Enter

⎇↵

Run the current cell and create a new cell below (Windows)

Alt-Up

⎇↑

Move the current cell above the cell above (i.e., swap them) (Windows)

Cmd-F

⌘F

Filter (macOS)

Ctrl-Enter

⌃↵

Run the current cell

Ctrl-F

⌃F

Filter (Windows)

Ctrl-S

⌃S

Save the notebook

Down

↓

Select the cell below the current cell

Enter

↵

Switch to Edit Mode

Esc

Switch to Command Mode

Opt-Down

⌥↓

Move the current cell below the cell below (i.e., swap them) (macOS)

Opt-Enter

⌥↵

Run the current cell and create a new cell below (macOS)

Opt-Up

⌥↑

Move the current cell above the cell above (i.e., swap them) (macOS)

Shift-Enter

⇧↵

Run the current cell and advance to the next cell

Shift-L

⇧L

Toggle line numbering for the entire notebook (when in command mode)

Shift-V

⇧V

paste above ???

Shift-LeftClick

Select consecutive cells

Up

↑

Select the cell above the current cell

Command Mode

  • a create a new cell above the current cell

  • b create a new cell below the current cell

  • c copy the current cell

  • j select the cell below the current cell

  • k select the cell above the current cell

  • l enable/disable line numbering in a code cell

  • m convert the current cell to a markdown cell

  • v paste the previously copied cell below the current cell

  • y convert the current cell to a code cell

  • x or dd delete the current cell

  • z undo previous cell operation

Managing Jupyter Kernels#

List Jupyter kernels.

jupyter kernelspec list

1 - Activate the Conda environment that you want to make a Jupyter kernel.

conda activate my_conda_env

2 - Notice how the default Jupyter kernel path changes to the activated Conda environment. This is why the default Jupyter kernel “Python 3 (IPyKernel)” depends on the underlying Conda environment.

jupyter kernelspec list

3 - Make your Conda environment a new Jupyter kernel.

python -m ipykernel install --user --name my_conda_env

4 - Verify the successful creation of the new Jupyter kernel.

jupyter kernelspec list

Delete a Jupyter kernel.

jupyter kernelspec remove my_conda_env

Command Reference#

jupyter --version jupyter lab --version
jupyter notebook --version

jupyter lab

  • jupyter lab -h How do I use command jupyter lab?

  • jupyter lab --version What is the Jupyter Lab version?

  • jupyter lab Launch Jupyter Lab.

  • jupyter lab list Which Jupyter Lab servers are currently running?

  • jupyter lab path or jupyter lab paths Where are the Jupyter Lab configuration directories located in the filesystem?

  • jupyter lab build -h How do I use command jupyter lab build?

  • jupyter lab clean -h How do I use command jupyter lab clean?

  • jupyter lab --notebook-dir=.

jupyter lab --ContentsManager.allow_hidden=True &>> .jupyter.log &

jupyter labextension

  • jupyter labextension -h How do I use command jupyter labextension?

  • jupyter labextension --version What is the version?

  • jupyter labextension list Which Jupyter Lab extensions are currently configured?

jupyter notebook

  • jupyter notebook -h How do I use command jupyter notebook?

  • jupyter notebook --version What is the Jupyter Notebook version?

  • jupyter notebook Launch Jupyter Notebook.

  • jupyter notebook list Which Jupyter Notebook servers are currently running?

  • jupyter notebook --notebook-dir=.

jupyter nbextension

  • jupyter nbextension -h How do I use command jupyter nbextension?

  • jupyter nbextension --version What is the version?

  • jupyter nbextension list Which Jupyter Notebook extensions are currently configured?

jupyter

  • jupyter -h How do I use command jupyter?

  • jupyter --version What is the Jupyter version?

  • jupyter --config-dir Where is the Jupyter configuration directory located in the filesystem?

  • jupyter --data-dir Where is the Jupyter data directory located in the filesystem?

  • jupyter --runtime-dir Where is the Jupyter runtime directory located in the filesystem?

jupyter kernelspec

  • jupyter kernelspec -h How do I use command jupyter kernelspec?

  • jupyter kernelspec --version What is the version?

  • jupyter kernelspec list Which kernels are currently available?

jupyter server

  • jupyter server -h How do I use command jupyter server?

  • jupyter server --version What is the Jupyter Server version?

  • jupyter server list Which Jupyter servers are currently running?

  • jupyter server extension list or jupyter lab extension list Which Jupyter Server extensions are currently configured?

ipython

  • ipython -h How do I use command ipython?

  • ipython -V What is the IPython version?

  • ipython locate Print the path to the IPython directory.

Exporting Jupyter notebook files as python files#

[ d ] Visual Studio Code’s Export your Jupyter Notebook: Export

Jupyter Lab: File -> Export Notebook As... -> Export Notebook to Executable Script.

Jupyter Notebook: File -> Download as -> Python (.py).

Google Colab#

Resources#

  • [ d ] Overview of Colaboratory Features

  • [ d ] Markdown Guide

  • [ d ] Charts in Colaboratory

  • [ d ] External data: Local Files, Drive, Sheets, and Cloud Storage

  • [ d ] Getting started with BigQuery

  • [ d ] Forms

  • [ d ] Data Table

  • [ d ] Accessing files

  • [ d ] Advanced Outputs

  • [ d ] Altair

  • [ d ] BigQuery

  • [ d ] Google Drive

  • [ d ] Saving Data to Google Cloud Storage

  • [ d ] Forms snippets

  • [ d ] Importing libraries

  • [ d ] Google Sheets

Apache Spark, PostgreSQL, & Google Colab#

#####
#
#     APACHE SPARK + POSTGRES + GOOGLE COLAB
#
#     Find the latest versions of
#        Spark & Hadoop:  https://spark.apache.org/downloads.html (https://www.apache.org/dist/spark/)
#        Postgres driver: https://jdbc.postgresql.org/

#####
#
#     CONFIGURE VERSIONS
#
import os
os.environ['HADOOP_VERSION']   = hadoop_version   = 'hadoop3'
os.environ['SPARK_VERSION']    = spark_version    = 'spark-3.5.0'
os.environ['POSTGRES_VERSION'] = postgres_version = 'postgresql-42.6.0'
#
#     it should not be necessary to modify code below this point
#
####################

# Install Java
! apt install openjdk-11-jdk-headless > /dev/null
os.environ['JAVA_HOME']  = '/usr/lib/jvm/java-11-openjdk-amd64'

# Install Spark
! wget https://www.apache.org/dist/spark/$SPARK_VERSION/$SPARK_VERSION-bin-$HADOOP_VERSION.tgz
! tar xf $SPARK_VERSION-bin-$HADOOP_VERSION.tgz
os.environ['SPARK_HOME'] = f'/content/{spark_version}-bin-{hadoop_version}'
! pip install findspark

# Install Postgres driver
! wget https://jdbc.postgresql.org/download/$POSTGRES_VERSION.jar

# Install AWS's Boto3
! pip install boto3

import boto3
import findspark
findspark.init()
from   getpass     import getpass
from   pyspark.sql import SparkSession

spark = SparkSession.builder \
  .appName('CustomerReviews') \
  .config('spark.driver.extraClassPath', f'/content/{postgres_version}.jar') \
  .getOrCreate()
spark

Terms#

  • [ w ] Google Colab

  • [ w ] Interactive Computing

  • [ w ] Jupyter

  • [ w ] Literate Programming

  • [ w ] Notebook Interface