28 Jupyter Notebook Tips, Tricks, and Shortcuts

The Jupyter interface.

5. IPython Magic Commands

The

# This will list all magic commands

Available line magics:

I recommend browsing the documentation for all IPython Magic commands as you’ll no doubt find some that work for you. A few of my favorites are below:
6. IPython Magic – 
You can manage environment variables of your notebook without restarting the jupyter server process. Some libraries (like theano) use environment variables to control behavior, 
# Running 
env: OMP_NUM_THREADS=4
7. IPython Magic – 

Note that using 
# this will execute and show the output from
# all code cells of the specified notebook


8. IPython Magic – 
This will replace the contents of the cell with an external script. You can either use a file on your computer as a source, or alternatively a URL.
# Before Running

# After Running
# 
Hello World!
9. IPython Magic – 
The 
data = 'this is the string I want to pass to different notebook'

Stored 'data' (str)
Now, in a new notebook…

this is the string I want to pass to different notebook
10. IPython Magic – 
The 
one = "for the money"
two = "for the show"
three = "to get ready now go cat go"

one three two
11. IPython Magic – Timing
There are two IPython Magic commands that are useful for timing – 


CPU times: user 21.5 ms, sys: 14.8 ms, total: 36.3 ms Wall time: 11.6 s

import numpy

The slowest run took 7.29 times longer than the fastest. This could mean that an intermediate result is being cached.
100000 loops, best of 3: 5.5 µs per loop
12. IPython Magic – 
Using the 

Writing pythoncode.py

import numpy
def append_if_not_exists(arr, x):
 if x not in arr:
 arr.append(x)def some_useless_slow_function():
 arr = list()
 for i in range(10000):
 x = numpy.random.randint(0, 10000)
 append_if_not_exists(arr, x)
13. IPython Magic – 
Using 

26324 function calls in 0.556 seconds 
Ordered by: internal time 
ncalls tottime percall cumtime percall filename:lineno(function)
10000 0.527 0.000 0.528 0.000 :2(append_if_not_exists)
10000 0.022 0.000 0.022 0.000 {method 'randint' of 'mtrand.RandomState' objects}
1 0.006 0.006 0.556 0.556 :6(some_useless_slow_function)
6320 0.001 0.000 0.001 0.000 {method 'append' of 'list' objects}
1 0.000 0.000 0.556 0.556 :1()
1 0.000 0.000 0.556 0.556 {built-in method exec}
1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects}
14. IPython Magic – Debugging with 
Jupyter has own interface for The Python Debugger (pdb). This makes it possible to go inside the function and investigate what happens there.
You can view a list of accepted commands for pdb here.

Automatic pdb calling has been turned ON
--------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
in ()
5 raise NotImplementedError()
6
----> 7 pick_and_take()
in pick_and_take()
3 def pick_and_take():
4 picked = numpy.random.randint(0, 1000)
----> 5 raise NotImplementedError()
6
7 pick_and_take()
NotImplementedError:
> (5)pick_and_take()
3 def pick_and_take():
4 picked = numpy.random.randint(0, 1000)
----> 5 raise NotImplementedError()
6
7 pick_and_take()
ipdb>
15. IPython Magic – High-resolution plot outputs for Retina notebooks
One line of IPython magic will give you double resolution plot output for Retina screens, such as the more recent Macbooks. Note: the example below won’t render on non-retina screens
x = range(1000)
y = [i ** 2 for i in x]
plt.plot(x,y)
plt.show();



16. Suppress the output of a final function.
Sometimes it’s handy to suppress the output of the function on a final line, for instance when plotting. To do this, you just add a semicolon at the end.

# Here you get the output of the function
plt.hist(x)
(array([ 216., 126., 106., 95., 87., 81., 77., 73., 71., 68.]), array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ]),
<a list of 10 Patch objects>)
# By adding a semicolon at the end, the output is suppressed.plt.hist(x);


17. Executing Shell Commands
It’s easy to execute a shell command from inside your notebook. You can use this to check what datasets are in available in your working folder:
!ls *.csv
nba_2016.csv titanic.csv pixar_movies.csv whitehouse_employees.csv
Or to check and manage packages.
!pip install numpy !pip list | grep pandas
Requirement already satisfied (use --upgrade to upgrade): numpy in /Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages pandas (0.18.1)
18. Using LaTeX for forumlas
When you write LaTeX in a Markdown cell, it will be rendered as a formula using MathJax.
This:
$P(A \mid B) = \frac{P(B \mid A)P(A)}{P(B)}$
Becomes this:

Markdown is an important part of notebooks, so don’t forget to use its expressiveness!
19. Run code from a different kernel in a notebook
If you want to, you can combine code from multiple kernels into one notebook.
Just use IPython Magics with the name of your kernel at the start of each cell that you want to use that Kernel for:









i is 1
i is 2
i is 3
i is 4
i is 5
20. Install other kernels for Jupyter
One of the nice features about Jupyter is ability to run kernels for different languages. As an example, here is how to get and R kernel running.
Easy Option: Installing the R Kernel Using Anaconda
If you used Anaconda to set up your environment, getting R working is extremely easy. Just run the below in your terminal:
conda install -c r r-essentials
Less Easy Option: Installing the R Kernel Manually
If you are not using Anaconda, the process is a little more complex. Firstly, you’ll need to install R from CRAN if you haven’t already.
Once that’s done, fire up an R console and run the following:
install.packages(c('repr', 'IRdisplay', 'crayon', 'pbdZMQ', 'devtools'))
devtools::install_github('IRkernel/IRkernel')
IRkernel::installspec() # to register the kernel in the current R installation
21. Running R and Python in the same notebook.
The best solution to this is to install rpy2 (requires a working version of R as well), which can be easily done with pip:
pip install rpy2
You can then use the two languages together, and even pass variables inbetween:


array([1], dtype=int32)
import pandas as pd df = pd.DataFrame({
'Letter': ['a', 'a', 'a', 'b', 'b', 'b', 'c', 'c', 'c'],
'X': [4, 3, 5, 2, 1, 7, 7, 5, 9],
'Y': [0, 4, 3, 6, 7, 10, 11, 9, 13],
'Z': [1, 2, 3, 1, 2, 3, 1, 2, 3]
})


Example courtesy Revolutions Blog
22. Writing functions in other languages
Sometimes the speed of numpy is not enough and I need to write some fast code.
In principle, you can compile function in the dynamic library and write python wrappers…
But it is much better when this boring part is done for you, right?
You can write functions in cython or fortran and use those directly from python code.
First you’ll need to install:
!pip install cython fortran-magic


myltiply_by_2(23.)
Personally I prefer to use fortran, which I found very convenient for writing number-crunching functions. More details of usage can be found here.


compute_fortran([1, 2, 3], [4, 5, 6])
There are also different jitter systems which can speed up your python code. More examples can be found here.
23. Multicursor support
Jupyter supports mutiple cursors, similar to Sublime Text. Simply click and drag your mouse while holding down Alt.

Multicursor support.
24. Jupyter-contrib extensions
Jupyter-contrib extensions is a family of extensions which give Jupyter a lot more functionality, including e.g. jupyter spell-checker and code-formatter.
The following commands will install the extensions, as well as a menu based configurator that will help you browse and enable the extensions from the main Jupyter notebook screen.
!pip install https://github.com/ipython-contrib/jupyter_contrib_nbextensions/tarball/master !pip install jupyter_nbextensions_configurator !jupyter contrib nbextension install --user !jupyter nbextensions_configurator enable --user

The nbextension configurator.
25. Create a presentation from a Jupyter notebook.
Damian Avila’s RISE allows you to create a powerpoint style presentation from an existing notebook.
You can install RISE using conda:
conda install -c damianavila82 rise
Or alternatively pip:
pip install RISE
And then run the following code to install and enable the extension:
jupyter-nbextension install rise --py --sys-prefix jupyter-nbextension enable rise --py --sys-prefix
26. The Jupyter output system
Notebooks are displayed as HTML and the cell output can be HTML, so you can return virtually anything: video/audio/images.
In this example I scan the folder with images in my repository and show thumbnails of the first 5:
import os
from IPython.display import display, Image names = [f for f in os.listdir('../images/ml_demonstrations/') if f.endswith('.png')]
for name in names[:5]:
 display(Image('../images/ml_demonstrations/' + name, width=100))





We can create the same list with a bash command, because magics and bash calls return python variables:
names = !ls ../images/ml_demonstrations/*.png names[:5]
['../images/ml_demonstrations/colah_embeddings.png',
'../images/ml_demonstrations/convnetjs.png',
'../images/ml_demonstrations/decision_tree.png',
'../images/ml_demonstrations/decision_tree_in_course.png',
'../images/ml_demonstrations/dream_mnist.png']
27. Big data analysis
A number of solutions are available for querying/processing large data samples:

ipyparallel (formerly ipython cluster) is a good option for simple map-reduce operations in python. We use it in rep to train many machine learning models in parallel
pyspark
spark-sql magic 

28. Sharing notebooks
The easiest way to share your notebook is simply using the notebook file (.ipynb), but for those who don’t use Jupyter, you have a few options:

Convert notebooks to html files using the File > Download as > HTML Menu option.
Upload your .ipynb file to Google Colab.
Share your notebook file with gists or on github, both of which render the notebooks. See this example.

If you upload your notebook to a github repository, you can use the handy mybinder service to allow someone half an hour of interactive Jupyter access to your repository.


Setup your own system with jupyterhub, this is very handy when you organize mini-course or workshop and don’t have time to care about students machines.
Store your notebook e.g. in dropbox and put the link to nbviewer. nbviewer will render the notebook from whichever source you host it.
Use the File > Download as > PDF menu to save your notebook as a PDF. If you’re going this route, I highly recommend reading Julius Schulz’s excellent article Making publication ready Python notebooks.
Create a blog using Pelican from your Jupyter notebooks.

What are your favorites?
Let me know what your favorite Jupyter notebook tips are.
At Dataquest, our interactive guided projects use Jupyter notebooks to building data science projects and get a job in data. If you’re interested, you can sign up and do our first module for free.
I also recommend the links below for further reading:

IPython built-in magics
Nice interactive presentation about jupyter by Ben Zaitlen
Advanced notebooks part 1: magics
and part 2: widgets
Profiling in python with jupyter
4 ways to extend notebooks
IPython notebook tricks
Jupyter vs Zeppelin for big data
Making publication ready Python notebooks.

                    

                
jupyter notebook
Tutorials