Virtual Python on Astrophysics Macs

This document describes how to use MacPorts's Python libraries, and to install other packages into your own library directory, using a virtual-Python setup to layer packages of your choice on top of our centrally-build MacPorts Python and its library set.

PLEASE NOTE: This replaces the previous advice, which included use of mysterious environment variables and Activations, and the file .pydistutils.cfg in the home directory. Said advice is now deprecated.

It also replaces use of the Python libraries in SciSoft and our Python Astropack, neither of which have been updated for some years. These are in the process of being replaced by more up-to-date libraries in our MacPorts build where available; specialist Python libraries such as pyraf can be installed into your virtual Python using pip, as described below.

Caveats

  • We assume that you are using an Apple desktop system in Physics (primarily Astrophysics) with our centrally-distributed build of MacPorts, or an Apple laptop with a reasonably up-to-date copy of our MacPorts build (if in doubt, check for the presence and datestamps of files in /opt/local such as Datestamp). Readers on Linux are invited to help us document the differences (principally locations of files and directories) by e-mailing to ITsupport at Physics, or by adding comments at the end of this page.

  • This also assumes you are using Python at the shell prompt. Those using integrated development environments such as Eclipse or PyCharm should have equivalent facilities available to them; please help us document them (see above).

  • The sections below about Python 2.7 are intended only for those users who are using packages which have not yet been converted to Python 3. Python 2.7 should be regarded as legacy for most purposes: it will reach end-of-life in 2020, after which it will receive no security updates. By that time, all of the third-party Python library suites will have been either converted or declared "abandonware" .... and newer packages (and new features of existing ones) are already appearing only for Python 3.

Please also see:

Suggestions for upgrading your code to Python 3 will appear in due course.

Prerequisites

Dispose of or disable Old News:

  • Don't use any Activation for Python or SciSoft in your shell.

    • If you've used one in your current shell, start a fresh one, and work in that.

    • If (heaven forfend) you've got such things in your shell startup files, comment out the offending line(s), start up a fresh shell, and start again from the top in that one.

  • Ensure that your PYTHONPATH environment variable is empty (test:

    printenv | grep -i python

    returns nothing in a fresh shell).

  • Ensure that there is no file called .pydistutils.cfg in your home directory. (If necessary, rename it aside.)

  • Ensure that the directory Library/Python in your home directory is missing or empty; rename it aside if necessary. (There may be other directories implicated in the same way: ~/.local has been sighted on occasion.)

  • Final check: say python2 or python3 (as appropriate) at the shell prompt, then say to Python:

    import sys
    for p in sys.path: print p

    .... plus a blank line, and verify that all the paths named are under /opt/local (ie in MacPorts).

Failure to clean out Old News of this sort will confuse the Python installation process in the virtual Python setup you're about to create or use. Symptoms have been seen to include pip successfully installing packages somewhere that Python cannot reach, or picking up previously-installed Python packages that should have been left behind, with (ahem) hilarious consequences.

First-time preparation steps: Python 3.5

At a fresh shell prompt, use the commands:

cd $HOME
python3.5 -m venv --system-site-packages myVirtualenv-3
source $HOME/myVirtualenv-3/bin/activate.csh (users of csh)
source $HOME/myVirtualenv-3/bin/activate (users of bash)
python3 -m pip install --upgrade pip setuptools

This is slightly more complicated with MacPorts's Python-3 setup than will appear under instructions for (eg) Linux. My thanks to Russell Jones for valuable clues.

First-time preparation steps: Python 2.7 (legacy)

At a fresh shell prompt, use the commands:

cd $HOME
virtualenv-2.7 --system-site-packages myVirtualenv-2
source $HOME/myVirtualenv-2/bin/activate.csh (users of csh)
source $HOME/myVirtualenv-2/bin/activate (users of bash)
pip install --upgrade pip setuptools

Preparing for next time

The above is only used in full the first time. To simplify the startup process for the next time, you can add these aliases to your shell startup files, for use in subsequent shell sessions:

  • .cshrc:

    alias vpython3-init 'source ~/myVirtualenv-3/bin/activate.csh'
    alias vpython2-init 'source ~/myVirtualenv-2/bin/activate.csh' # (legacy)

  • .bashrc:

    alias vpython3-init='. ~/myVirtualenv-3/bin/activate'
    alias vpython2-init='. ~/myVirtualenv-2/bin/activate' # (legacy)

Second (and subsequent) times:

Use the appropriate alias (vpython3-init or vpython2-init, as required), and check that the shell prompt has changed to reflect the change. If you now say:

which python # (either Python 3 or Python 2)
which python3 # (be explicit)
which python2 # (legacy)

.... you'll see a Python instance named which is inside your virtual environment.

PyRAF: the Extra Step

In the special case of PyRAF, the installation requires an instance of IRAF to be active, for which the version in SciSoft is not suitable. Please see:

.... about how to get IRAF on your side before proceeding. This includes instructions on how to set up and activate the freestanding 64-bit IRAF as well as your virtual Python.

Installing and using Python packages

To install (say) the PyRAF package into the virtual environment, activate your virtual Python and/or IRAF if and as necessary, then the appropriate one of:

python3 -m pip install pyraf
pip install pyraf # (legacy)

You can then verify that your Python package has been installed by starting Python, and saying at its prompt:

import pyraf
pyraf.__file__ # (a pathname inside your virtual environment)
pyraf.__version__ # ('2.1.14' at the time of writing)

.... or (in PyRAF's case) by saying at the shell prompt:

which pyraf

.... and verifying that its pathname is within your virtual-Python directory.

Third-party Python scripts

A script intended to be invoked by name at the shell prompt may well have the "flash-bang" line at the beginning, to tell it which (in this case) Python interpreter to use, eg one of:

#!/usr/bin/python # (WRONG: system Python)
#!/opt/local/bin/python3 # (WRONG: not portable)

This is Bad News: it hard-wires use of one particular Python interpreter (Apple's, or MacPorts's, respectively) into the program, which is likely to break portability. In third-party Python scripts or examples, you may even see:

#!/usr/local/bin/python # (WRONG: not there, here)

.... or some other variant which assumes you've installed some version of Python manually. (This may be true for your laptop, but scripts written there would break when run on your desktop system.)

To properly generalise such scripts, edit the first line to instead say the appropriate one of:

#!/usr/bin/env python3
#!/usr/bin/env python2 # (legacy)

This will cause the script to use the first appropriate Python executable on $PATH, which (when it's active) will be the one in your virtual environment .... unless perchance $PATH gets pruned or preempted by something in between, as integrated development environments are wont to do.

Leaving the virtual Python environment

If you wish (for whatever reason) to revert to using the system Python, exit from Python if necessary, then say (at the shell prompt):

deactivate

Note that the shell prompt has been changed back, as saying:

which python # (should show the system Python)
which python3 # (should complain or return nothing)

.... will confirm.

Cloning a virtual Python environment

Virtual Pythons have absolute pathnames stitched in, so they can't be moved or copied as-is; but they can be recreated without excessive effort. The first part is done inside the virtual Python in question, using the appropriate one of:

python3 -m pip freeze -l | tee ~/requirements-3.txt
pip freeze -l | tee ~/requirements.txt # (legacy)

(The second is necessary within a virtual Python 3 built as shown above, as pip will not be available directly via your $PATH.) This yields, both in the named file and to your screen, a list of everything installed by pip into that virtual environment, eg:

d2to1==0.2.12.post1
pyraf==2.1.14
stsci.distutils==0.3.7
stsci.tools==3.4.3

To add that set of packages into another virtual Python setup elsewhere, deactivate the old virtual Python, activate the new one (creating it if necesary), then saying in it:

python3 -m pip install --upgrade -r ~/requirements-3.txt
pip install --upgrade -r ~/requirements.txt # (legacy)

The requirements file documents your required set of extra packages in a near-executable form. If you're keeping all your own Python programs under version control, it would be wise to add this file to your saved set as a record of any support libraries you may have found necessary, and to keep it up-to-date. Please bear in mind that the underlying system packages, which aren't mentioned here, may well differ between your desktop, your laptop, and whatever Linux servers you're using. If in doubt, use separate names for them (eg requirements-desktop.txt, requirements-glamdring.txt etc) .... and let us know of any differences you may find, as they may indicate holes in our Python coverage.

Categories: Astro software | Astrophysics | IRAF | PyRAF | python | virtual Python