Customizing IPython - Magics (8) "IPython in Depth, SciPy2013 Tutorial, Part 3 of 3"

Customizing IPython - Magics (8) "IPython in Depth, SciPy2013 Tutorial, Part 3 of 3"¶

На 90-й минуте (третьего) видеоролика речь зашла о настройке пользовательских %magic. Начинвется пост с примера "%timeit A=numpy.random.random((1000,1000))", потом определяются "Define %tic and %toc magics, which can be use for simple timings"... Затем "load a notebook into memory using IPython.nbformat."... После этого можно было оперировать объектом notebook, как dictionary.... Потом "Now I can run all of the code cells with get_ipython().run_cell"... И наконец "write and register an %nbrun line magic to run a notebook"

IPython extends Python by adding shell-like commands called magics.

In [1]:

%lsmagic

Out[1]:

Available line magics:
%alias  %alias_magic  %autocall  %automagic  %autosave  %bookmark  %cd  %clear  %colors  %config  %connect_info  %debug  %dhist  %dirs  %doctest_mode  %ed  %edit  %env  %gui  %hist  %history  %install_default_config  %install_ext  %install_profiles  %killbgscripts  %less  %load  %load_ext  %loadpy  %logoff  %logon  %logstart  %logstate  %logstop  %lsmagic  %macro  %magic  %man  %matplotlib  %more  %notebook  %page  %pastebin  %pdb  %pdef  %pdoc  %pfile  %pinfo  %pinfo2  %popd  %pprint  %precision  %profile  %prun  %psearch  %psource  %pushd  %pwd  %pycat  %pylab  %qtconsole  %quickref  %recall  %rehashx  %reload_ext  %rep  %rerun  %reset  %reset_selective  %run  %save  %sc  %store  %sx  %system  %tb  %time  %timeit  %unalias  %unload_ext  %who  %who_ls  %whos  %xdel  %xmode

Available cell magics:
%%!  %%HTML  %%SVG  %%bash  %%capture  %%debug  %%file  %%html  %%javascript  %%latex  %%perl  %%prun  %%pypy  %%python  %%python3  %%ruby  %%script  %%sh  %%svg  %%sx  %%system  %%time  %%timeit  %%writefile

Automagic is ON, % prefix IS NOT needed for line magics.

In [2]:

import numpy

In [3]:

%timeit A=numpy.random.random((1000,1000))

100 loops, best of 3: 11.8 ms per loop

In [4]:

%%timeit -n 1

A=numpy.random.random((1000,1000))
b = A.sum()

1 loops, best of 3: 15.8 ms per loop

Defining your own magic¶

As we have seen already, IPython has cell and line magics. You can define your own magics using any Python function and the register_magic_function method:

In [1]:

ip = get_ipython()

In [2]:

import time

def sleep_magic(line):
    """A simple function for sleeping"""
    t = float(line)
    time.sleep(t)

In [3]:

ip.register_magic_function?

In []:

Type:       instancemethod
String Form:<bound method ZMQInteractiveShell.register_function of <IPython.kernel.zmq.zmqshell.ZMQInteractiveShell object at 0x0000000007E76A90>>
File:       c:\users\kiss\anaconda\lib\site-packages\ipython\core\interactiveshell.py
Definition: ip.register_magic_function(self, func, magic_kind='line', magic_name=None)
Docstring:
Expose a standalone function as magic function for IPython.

This will create an IPython magic (line, cell or both) from a
standalone function.  The functions should have the following
signatures: 

* For line magics: `def f(line)`
* For cell magics: `def f(line, cell)`
* For a function that does both: `def f(line, cell=None)`

In the latter case, the function will be called with `cell==None` when
invoked as `%f`, and with cell as a string when invoked as `%%f`.

Parameters
----------
func : callable
  Function to be registered as a magic.

magic_kind : str
  Kind of magic, one of 'line', 'cell' or 'line_cell'

magic_name : optional str
  If given, the name the magic will have in the IPython namespace.  By
  default, the name of the function itself is used.

In [4]:

ip.register_magic_function(sleep_magic, "line", "sleep")

In [5]:

%sleep 2

In [6]:

%sleep?

In []:

Type:       Magic function
String Form:<function sleep_magic at 0x00000000080A1518>
Namespace:  IPython internal
File:       c:\users\kiss\documents\github\ipython-in-depth\notebooks\<ipython-input-2-eaf3ef465508>
Definition: %sleep(line)
Docstring:  A simple function for sleeping

Exercise¶

Define %tic and %toc magics, which can be use for simple timings, e.g. where

for p in range(1,4):

    N = 10**p

    print "N=%i" % N

    %tic

    A = np.random.random((N,N))

    np.linalg.eigvals(A)

    %toc

each %toc will print the time since the last %tic. Create separate tic and toc functions that read and write a global time variable.

In [7]:

%load soln/tictocf.py

In [8]:

import time

def format_time(dt):
    if dt < 1e-6:
        return u"%.3g ns" % (dt * 1e9)
    elif dt < 1e-3:
        return u"%.3g µs" % (dt * 1e6)
    elif dt < 1:
        return u"%.3g ms" % (dt * 1e3)
    else:
        return "%.3g s" % dt

def tic(line):
    global t0
    t0 = time.time()

def toc(line):
    global t0
    print (format_time(time.time() - t0))

ip = get_ipython()
ip.register_magic_function(tic)
ip.register_magic_function(toc)

In [9]:

import numpy as np
import sys
for p in range(1,4):
    N = 10**p
    print "N=%i" % N
    sys.stdout.flush()
    %tic
    A = np.random.random((N,N))
    np.linalg.eigvals(A)
    %toc

N=10
481 ms
N=100
761 ms
N=1000
24.5 s

Cell Magic¶

Cell magics take two args:

1. the line on the same line of the magic
2. the cell the multiline body of the cell after the first line

In [10]:

def dummy_cell_magic(line, cell):
    """dummy cell magic for displaying the line and cell it is passed"""
    print "line: %r" % line
    print "cell: %r" % cell

ip.register_magic_function(dummy_cell_magic, "cell", "dummy")

In [11]:

%%dummy this is the line
this
is the
cell

line: u'this is the line'
cell: u'this\nis the\ncell'

In [12]:

def parse_magic_line(line):
    """parse a magic line into a name and eval'd expression"""
    name, values_s = line.split(None, 1)
    values = eval(values_s, get_ipython().user_ns)
    return name, values

parse_magic_line("x range(5)")

Out[12]:

('x', [0, 1, 2, 3, 4])

Excercise¶

Can you write and register a cell magic that automates the outer iteration, timing a block for various values of a particular variable:

In [13]:

%load soln/scalemagic.py

In [14]:

def scale_magic(line, cell):
    """run a cell block with a variety of input values"""
    name, values = parse_magic_line(line)
    ns = get_ipython().user_ns
    for v in values:
        assignment = "%s=%r" % (name, v)
        print assignment
        ns[name] = v
        sys.stdout.flush()
        %tic
        exec cell in ns
        %toc
ip = get_ipython()
ip.register_magic_function(scale_magic, "cell", "scale")

In [15]:

%%scale N [ int(10**p) for p in range(1,4) ]

A = np.random.random((N,N))
np.linalg.eigvals(A)

N=10
5 ms
N=100
128 ms
N=1000
27 s

In [16]:

%%scale N [ int(2**p) for p in np.linspace(6, 11, 11) ]

A = np.random.random((N,N))
np.linalg.eigvals(A)

N=64
30 ms
N=90
112 ms
N=128
580 ms
N=181
478 ms
N=256
1.1 s
N=362
2.62 s
N=512
7.59 s
N=724
15.2 s
N=1024
61.8 s
N=1448
53.3 s
N=2048
279 s

Executing Notebooks¶

We can load a notebook into memory using IPython.nbformat.

In [21]:

import io
import os

from IPython.nbformat import current

In [22]:

def load_notebook(filename):
    """load a notebook object from a filename"""
    if not os.path.exists(filename) and not filename.endswith(".ipynb"):
        filename = filename + ".ipynb"
    with io.open(filename) as f:
        return current.read(f, 'json')

In [23]:

nb = load_notebook("Sample")

A notebook is just a dictionary with attribute access for convenience.

In [24]:

nb.keys()

Out[24]:

[u'nbformat', u'worksheets', u'metadata', u'nbformat_minor']

In [25]:

nb.worksheets[0].keys()

Out[25]:

[u'cells', u'metadata']

In [26]:

cells = nb.worksheets[0].cells
cells

Out[26]:

[{u'cell_type': u'heading',
  u'level': 1,
  u'metadata': {},
  u'source': u'A sample notebook'},
 {u'cell_type': u'code',
  u'collapsed': False,
  u'input': u"print 'hello'",
  u'language': u'python',
  u'metadata': {},
  u'outputs': []},
 {u'cell_type': u'code',
  u'collapsed': False,
  u'input': u'%matplotlib inline\nimport matplotlib.pyplot as plt\nimport numpy as np\nplt.plot(np.random.random(100))',
  u'language': u'python',
  u'metadata': {},
  u'outputs': []},
 {u'cell_type': u'markdown',
  u'metadata': {},
  u'source': u'A function for displaying the summary of a notebook object.\nIt prints a simple summary, such as:\n\n```\n  1   markdown cells, total:   4 lines\n  5       code cells, total:   4 lines\n  1    heading cells, total:   1 lines\n```'},
 {u'cell_type': u'code',
  u'collapsed': False,
  u'input': u'def nb_info(nb):\n    """display a summary of the contents of a notebook"""\n    cell_counts = {}\n    cell_lines = {}\n    \n    for cell in nb.worksheets[0].cells:\n        cell_type = cell.cell_type\n        count = cell_counts.setdefault(cell_type, 0)\n        lines = cell_counts.setdefault(cell_type, 0)\n        cell_counts[cell_type] = count + 1\n        try:\n            content = cell.source\n        except AttributeError:\n            content = cell.input\n        cell_lines[cell_type] = lines + len(content.splitlines())\n    \n    for cell_type in cell_counts:\n        print "%3i %10s cells, total: %3i lines" % (cell_counts[cell_type], cell_type, cell_lines[cell_type])\n',
  u'language': u'python',
  u'metadata': {},
  u'outputs': []}]

We can see all the cells and their type

In [27]:

for cell in cells:
    print
    print '----- %s -----' % cell.cell_type
    if cell.cell_type == 'code':
        print cell.input
    else:
        print cell.source

----- heading -----
A sample notebook

----- code -----
print 'hello'

----- code -----
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
plt.plot(np.random.random(100))

----- markdown -----
A function for displaying the summary of a notebook object.
It prints a simple summary, such as:

```
  1   markdown cells, total:   4 lines
  5       code cells, total:   4 lines
  1    heading cells, total:   1 lines
```

----- code -----
def nb_info(nb):
    """display a summary of the contents of a notebook"""
    cell_counts = {}
    cell_lines = {}
    
    for cell in nb.worksheets[0].cells:
        cell_type = cell.cell_type
        count = cell_counts.setdefault(cell_type, 0)
        lines = cell_counts.setdefault(cell_type, 0)
        cell_counts[cell_type] = count + 1
        try:
            content = cell.source
        except AttributeError:
            content = cell.input
        cell_lines[cell_type] = lines + len(content.splitlines())
    
    for cell_type in cell_counts:
        print "%3i %10s cells, total: %3i lines" % (cell_counts[cell_type], cell_type, cell_lines[cell_type])

Now I can run all of the code cells with get_ipython().run_cell

In [28]:

for cell in cells:
    ip = get_ipython()
    if cell.cell_type == 'code':
        ip.run_cell(cell.input, silent=True)

hello

And we can now use the function that was defined in that notebook:

In [29]:

nb_info(nb)

  1   markdown cells, total:   8 lines
  3       code cells, total:  20 lines
  1    heading cells, total:   1 lines

Exercise¶

Can you write and register an %nbrun line magic to run a notebook?

%nbrun Sample

In [30]:

%load soln/nbrun.py

In [32]:

%nbrun Sample

hello

The common way to make your magics reusable is to write an Extension, so let's give that a try.

Посты чуть ниже также могут вас заинтересовать