path: root/software/python.page

---
format: rst
toc: no
...

======================
Python
======================

iPython tricks
--------------

Use the ``cpaste`` command to copy in blocks of raw python code, even if the
indenting is a little weird. 

Style
-------
`Python PEP-008 <http://www.python.org/dev/peps/pep-0008/>`_: Style Guide for
Python Code

`pylint <http://pypi.python.org/pypi/pylint>`_, a Python syntax checker. Very
verbose, use pylint -E (errors only) or at least ``pylint -r no`` (no report).
Eg, ``pylint -r no file.py -d W0614 -d C -d R``.

For docstring documentation, refer to
`PEP-257 <http://www.python.org/dev/peps/pep-0257/>`_ and the `Sphinx
documentation
<http://packages.python.org/an_example_pypi_project/sphinx.html>`_;
specifically, document script functionality in a top level (above imports,
below any hashbang) docstring.

Use leading ``#:`` style comments to document important non-object/non-function
element definitions (eg, static variables) in a way that will get pulled out
into Sphinx.  Use "Google-style" function argument/return documentation instead
of "Sphinx style". For example::

    def public_fn_with_googley_docstring(name, state=None):
        """This function does something.

        Args:
            name (str):  The name to use.

        Kwargs:
            state (bool): Current state to be in.

        Returns:
            int.  The return code::

                0 -- Success!
                1 -- No good.
                2 -- Try again.

        Raises:
            AttributeError, KeyError

        A really great idea.  A way you might use me is

        >>> print public_fn_with_googley_docstring(name='foo', state=None)
        0

        BTW, this always returns 0.  **NEVER** use with :class:`MyPublicClass`.

        """
        return 0


autopep8 is a tool to automatically pep8-ify a file. Use it like::

    sudo pip install autopep8
    autopep8 --in-place --select=W293,W191,W291 *.py

pep8radius is sort of similar, but only applies to code that you are going to
commit (using VCS info).

Packaging
-----------

Flask app packaging advice, including ``MANIFEST.in`` and non-PyPi dependancy
advice: http://flask.pocoo.org/docs/patterns/distribute/

Use ``console_scripts`` in ``setup.py`` to install system-wide scripts:
http://packages.python.org/distribute/setuptools.html#automatic-script-creation

For debian packaging, use `stdeb <http://pypi.python.org/pypi/stdeb>`_
(via `stackoverflow thread <http://stackoverflow.com/questions/7110604/standard-way-to-create-debian-packages-for-distributing-python-programs>`_).

For notes on pip vs. setup.py dependencies:
https://caremad.io/blog/setup-vs-requirement/

"Fucking" String Encoding
---------------------------
(str/unicode errors in python are very prevalent and give me the rage)

The ``codecs`` package has some helpers; see for example
``open(f,mode,encoding)``.

ASCII
----------------
'ord' is the function that takes a single ASCII character and returns the value
number (as an int).

RunSnakeRun
-------------

Runsnake seems to be unmaintained... `snakeviz` is the new thing?

Example session::

    $ python -m cProfile -o ./dump.profile myscript.py --script-option blah
    $ # run to completion or Ctrl-C, then
    $ runsnakerun ./dump.profile
    # or
    $ snakeviz ./dump.profile

nosetests
-------------

NOTE: by default I now use pytest instead of nose

To do minimal tests without wrapping everything in a class, import assert
functions from nose.tools, eg::

    from nose.tools import assert_raises, assert_equal

To do interactive pdb debugging, simply::

    $ nosetests --pdb
    # or sometimes:
    $ nosetests --pdb-failures


pdb
-------
To debug a script (interactive prompt on exception)::

    python -m pdb myscript.py

or in-line, always at a particular point::

    import pdb; pdb.set_trace()

Use ipdb (debian: python-ipdb) instead of pdb to get a nicer IPython prompt.

Python 3 Porting
-------------------

To help port and support both Python 2.7 and 3.0+, start with an import::

    from __future__ import absolute_import, division, print_function

str/bytes/unicode is indeed the major porting challenge. Using bytearrays
helps. Use ``b'asdf'`` style byte array definitions for most low-level
constants.

``struct.unpack()`` wants doesn't allow ``bytearray()``; use ``bytes()``
instead.

Make sure ``rase Exception ("Message here")`` style is used everywhere, instead
of ``raise Exception, "Message here"``.

There was some change in comparison between None and integers, which makes
``if args.verbose > 0`` break. Use ``if args.verbose and args.verbose > 1``
instead.

Debian Default Python
----------------------

Keep in mind that this can break some system things (eg, arandr, some cups
things).

To make Python 3.4 the default for `python` system-wide, do something like:

    sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.4 2

For an individual user you can also just create a `python` alias. See also:

http://linuxconfig.org/how-to-change-from-default-to-alternative-python-version-on-debian-linux


Nice Packages
--------------

subprocess/pexpect: https://github.com/kennethreitz/delegator.py

time/datetime: https://github.com/crsmithdev/arrow

tsv: https://github.com/kennethreitz/tablib

simple HTML scraping: https://github.com/michaelhelmick/lassie

sqlite3dbm is a library to back a python dict with sqlite3 on disk


pytest
-----------

pytest sort of "just works" if you put test files under `./tests/`. If you want
crude file imports, skipping directories, and including `test_*` functions from
any python file (not just those with `test_*.py`, install `pytest-pythonpath`
and create a `pytest.ini` like:

    [pytest]
    python_paths = .
    python_files = *.py
    norecursedirs = .svn _build tmp*

Need to mock? <https://blog.fugue.co/2016-02-11-python-mocking-101.html>

Debugging Memory Usage
------------------------

Most helpful tools I found were `psutil` and `pympler` (both need to be
installed).

    import os, psutil
    process = psutil.Process(os.getpid())
    print(process.memory_info().rss)
    # ... do some stuff ...
    print(process.memory_info().rss)

and

    from pympler import tracker
    tr = tracker.SummaryTracker()
    tr.print_diff()

    # ... do some stuff ...
    tr.print_diff()


Canonical Timestamp
--------------------

As a terse one-liner (with `datetime` imported):

    f"{datetime.datetime.utcnow().isoformat()[:-3]}Z"