Virtual Python on Astrophysics Macs

This document describes how to use MacPorts's Python libraries, and to install other packages into your own library directory, using a virtual-Python setup to layer packages of your choice on top of our centrally-build MacPorts Python and its library set.

PLEASE NOTE: This replaces the previous advice, which included use of mysterious environment variables and Activations, and the file .pydistutils.cfg in the home directory. Said advice is now deprecated.

It also replaces use of the Python libraries in SciSoft and our Python Astropack, neither of which have been updated for some years. These are in the process of being replaced by more up-to-date libraries in our MacPorts build where available; specialist Python libraries such as pyraf can be installed into your virtual Python using pip, as described below.


  • We assume that you are using an Apple desktop system in Physics (primarily Astrophysics) with our centrally-distributed build of MacPorts, or an Apple laptop with a reasonably up-to-date copy of our MacPorts build (if in doubt, check for the presence and datestamps of files in /opt/local such as Datestamp). Instructions for Linux systems are gradually being interpolated; readers on Linux are invited to read along, and to help us document the differences (principally locations of files and directories) by e-mailing to ITsupport at Physics, or by adding comments at the end of this page.

  • This also assumes you are using Python3 or iPython3 at the shell prompt. Those using integrated development environments such as Eclipse or PyCharm should have equivalent facilities available to them; please help us document them (see above).

  • The sections below about Python 2.7 are intended only for those users who are using packages which have not yet been converted to Python 3. Python 2.7 should be regarded as legacy for most purposes: it will reach end-of-life in 2020, after which it will receive no security updates. By that time, all of the third-party Python library suites will have been either converted or declared "abandonware" .... and newer packages (and new features of existing ones) are already appearing only for Python 3.

Please also see:

  • IRAF in a virtual Python, about the extra hoops necessary to jump through for installing and using PyRAF. PLEASE NOTE that PyRAF, and the IRAF on which it depends, are best regarded as all but abandonware.

Suggestions for upgrading existing Python-2 code to Python 3 will appear in due course.


Dispose of or disable Old News:

  • Don't use any Activation for Python or SciSoft in your shell.

    • If you've used one in your current shell, start a fresh one, and work in that.

    • If (heaven forfend) you've got such things in your shell startup files, comment out the offending line(s), start up a fresh shell, and start again from the top in that one.

  • Ensure that your PYTHONPATH environment variable is empty (test:

    printenv | grep -i python

    returns nothing in a fresh shell).

  • Ensure that there is no file called .pydistutils.cfg in your home directory. (If necessary, rename it aside.)

  • Ensure that the directory Library/Python in your home directory is missing or empty; rename it aside if necessary. (There may be other directories implicated in the same way: ~/.local has been sighted on occasion, and under Linux.)

  • Final check: say python3 (or python2 if you must) at the shell prompt, then say to Python:

    import sys
    for p in sys.path: print (p)

    .... plus a blank line, and verify that all the paths named are under /opt/local (ie in MacPorts) for macOS, or /usr/lib under Linux.

Failure to clean out Old News of this sort will confuse the Python installation process in the virtual Python setup you're about to create or use. Symptoms have been seen to include pip successfully installing packages somewhere that Python cannot reach, or picking up previously-installed Python packages that should have been left behind, with (ahem) hilarious consequences.

Zeroth step: create a personal directory on the local hard drive

We have found the Hard Way that virtual Python setups and network-connected home directories don't play nice. We now suggest that you create a directory under /Data, and work in that instead. In what follows, LOGNAME is your login name (so in my case I'd use /Data/carter); substitute as appropriate.

cd /Data
mkdir -p LOGNAME
cd /Data/LOGNAME

Side notes:

  • mkdir -p won't complain if the directory already exists.
  • If /Data doesn't exist, or doesn't let you create anything in it, please let us know.

First-time preparation steps: Python 3.5 and up

At a fresh shell prompt, use the commands:

cd /Data/LOGNAME
python3 -m venv --system-site-packages myVirtualenv-3
source /Data/LOGNAME/myVirtualenv-3/bin/activate.csh (users of csh)
source /Data/LOGNAME/myVirtualenv-3/bin/activate (users of bash)
python3 -m pip install --upgrade pip setuptools wheel

(As usual, replace LOGNAME with your login name.) This is slightly more complicated with MacPorts's Python-3 setup than will appear under instructions for (eg) Linux. My thanks to Russell Jones for valuable clues.

First-time preparation steps: Python 2.7 (legacy)

At a fresh shell prompt, use the commands:

cd /Data/LOGNAME
virtualenv-2.7 --system-site-packages myVirtualenv-2
source /Data/LOGNAME/myVirtualenv-2/bin/activate.csh (users of csh)
source /Data/LOGNAME/myVirtualenv-2/bin/activate (users of bash)
pip install --upgrade pip setuptools

(As usual, replace LOGNAME with your login name.)

Preparing for next time

The above is only used in full the first time. To simplify the startup process for the next time, you can add these aliases to your shell startup files, for use in subsequent shell sessions. Please note that each listing consists of two lines, neither of which will fit in this margin, and will have to be stitched together again as shown. (Please see the worked example at the end of this page. As usual, replace LOGNAME with your login name.)

  • .cshrc: the middle of the line sbould read "-init 'source" (leave only one space).

    alias vpython3-init \
        'source /Data/LOGNAME/myVirtualenv-3/bin/activate.csh'
    alias vpython2-init \
        'source /Data/LOGNAME/myVirtualenv-2/bin/activate.csh'
        # (legacy)

  • .bashrc: the middle of the line should read "-init='. /Data" (leave only the equals sign).

    alias vpython3-init=\
        '. /Data/LOGNAME/myVirtualenv-3/bin/activate'
    alias vpython2-init=\
        '. /Data/LOGNAME/myVirtualenv-2/bin/activate'
        # (legacy)

Second (and subsequent) times:

Use the appropriate one of:

vpython2-init # (legacy)

Now check that the shell prompt has changed to reflect the change:

which python # (either Python 3 or Python 2)
which python3 # (be explicit)
which python2 # (legacy)

.... you'll see a Python instance named which is inside your virtual environment.

PyRAF: the Extra Step

In the special case of PyRAF, the installation requires an instance of IRAF to be active, for which the version in SciSoft is not suitable. Please see:

.... about how to get IRAF on your side before proceeding. This includes instructions on how to set up and activate the freestanding 64-bit IRAF as well as your virtual Python.

PLEASE NOTE: IRAF is woefully out-of-date, and is not long for this world. Most or all of what IRAF could do can now be done by (eg) AstroPy.

Installing and using Python packages

THIS SECTION WILL BE OVERHAULED at some point, to use a different example package. Please see the note in the previous section.

That said, to install (say) the PyRAF package into the virtual environment, activate your virtual Python and/or IRAF if and as necessary, then the appropriate one of:

python3 -m pip install pyraf
pip install pyraf # (legacy)

You can then verify that your Python package has been installed by starting Python, and saying at its prompt:

import pyraf
pyraf.__file__ # (a pathname inside your virtual environment)
pyraf.__version__ # ('2.1.14' at the time of writing)

.... or (in PyRAF's case) by saying at the shell prompt:

which pyraf

.... and verifying that its pathname is within your virtual-Python directory.

Problems with numpy

Under Linux in particular, attempts to update (eg) astropy by the obvious incantation will also pull in a newer version of numpy which is suboptimal. (For the Advanced Player: Debian's system numpy links to Debian's copy of OpenBLAS, which is well-tuned, while the newer one uses its own libopenblasp, which seems not to be. Similar problems may exist with MacPorts, whose numpy now hook into OpenBLAS; scipy certainly does this, as SciPy on macOS no longer supports Apple's accelerator libraries.)

The following workaround has been seen to work:

python3 -m pip install --no-deps --ignore-installed astropy

The first argument (--no-deps) tells pip to not follow the chain of dependencies, so that the resulting astropy will link against the system version of numpy; the second (--ignore-installed) tells pip to ignore any local copies of astropy.

Third-party Python scripts

A script intended to be invoked by name at the shell prompt may well have the "flash-bang" line at the beginning, to tell it which (in this case) Python interpreter to use, eg one of:

#!/usr/bin/python # (WRONG under macOS: system Python)
#!/opt/local/bin/python3 # (WRONG under Linux: macOS-specific)

This is Bad News: it hard-wires use of one particular Python interpreter (Apple's, or MacPorts's, respectively) into the program, which is likely to break portability. In third-party Python scripts or examples, you may even see:

#!/usr/local/bin/python # (WRONG: not there, here)

.... or some other variant which assumes you've installed some version of Python manually. (This may be true for your laptop, but scripts written there would break when run on your desktop system.)

To properly generalise such scripts, edit the first line to instead say the appropriate one of:

#!/usr/bin/env python3
#!/usr/bin/env python2 # (legacy)

This will cause the script to use the first appropriate Python executable on $PATH, which (when it's active) will be the one in your virtual environment .... unless perchance $PATH gets pruned or preempted by something in between, as integrated development environments are wont to do.

Leaving the virtual Python environment

If you wish (for whatever reason) to revert to using the system Python, exit from Python if necessary, then say (at the shell prompt):


Note that the shell prompt has been changed back, as saying:

which python # (should show the system Python)
which python3 # (should complain or return nothing)

.... will confirm.

Cloning a virtual Python environment

Virtual Pythons have absolute pathnames stitched in, so they can't be moved or copied as-is; but they can be recreated without excessive effort. The first part is done inside the virtual Python in question, using the appropriate one of:

python3 -m pip freeze -l | tee ~/requirements-3.txt
pip freeze -l | tee ~/requirements.txt # (legacy)

(The second is necessary within a virtual Python 3 built as shown above, as pip will not be available directly via your $PATH.) This yields, both in the named file and to your screen, a list of everything installed by pip into that virtual environment, eg:


To add that set of packages into another virtual Python setup elsewhere, deactivate the old virtual Python, activate the new one (creating it if necesary), then saying in it:

python3 -m pip install --upgrade -r ~/requirements-3.txt
pip install --upgrade -r ~/requirements.txt # (legacy)

The requirements file documents your required set of extra packages in a near-executable form. If you're keeping all your own Python programs under version control, it would be wise to add this file to your saved set as a record of any support libraries you may have found necessary, and to keep it up-to-date. Please bear in mind that the underlying system packages, which aren't mentioned here, may well differ between your desktop, your laptop, and whatever Linux servers you're using. If in doubt, use separate names for them (eg requirements-desktop.txt, requirements-glamdring.txt etc) .... and let us know of any differences you may find, as they may indicate holes in our Python coverage.

A worked example: moving from home directory to /Data/carter

Let us assume that I've got Python-3.5 in a virtual environment, set up (following advice on the previous version of this page) under my home directory, and need to move it to /Data on the local hard disk. This would be necessary, for example, when moving from macOS 10.12 (Sierra) to 10.13 (High Sierra):

  • Network-connected home directories are not supported under 10.13 and upward.
  • There may well be macOS-specific code embedded into the older virtual environment.

For the purposes of this exercise, I'll also pretend I'm using tcsh.

source ~/myVirtualenv-3/bin/activate.csh python3 -m pip freeze -l | tee ~/requirements-3.txt deactivate

(At this point, I would upgrade the system in question to High Sierra.)

mkdir -p /Data/carter mv ~/myVirtualenv-3 ~/myVirtualenv-3.old cd /Data/carter python3 -m venv --system-site-packages myVirtualenv-3 source /Data/carter/myVirtualenv-3/bin/activate.csh

(A new virtualenv, activated for the first time, currently empty.)

python3 -m pip install --upgrade -r ~/requirements-3.txt

(.... and it's populated.)

Now I edit my .cshrc and .bashrc to update the alias to read (respectively):

alias vpython3-init 'source /Data/carter/myVirtualenv-3/bin/activate.csh' alias vpython3-init='source /Data/carter/myVirtualenv-3/bin/activate'

.... and start a fresh shell before continuing.

Categories: Astro software | Astrophysics | IRAF | PyRAF | python | virtual Python