OBSOLETE: Python and SciSoft under MacOS X 10.6 and later

MacOS X 10.6 (Snow Leopard) and Lion (10.7) come with versions 2.5, 2.6 and 2.7 of Python; the default version is v2.6 for Snow Leopard, and v2.7 for Lion and Mountain Lion. The Snow-Leopard-and-later-compatible versions of SciSoft now extend the system Python, rather than using their own.

WARNING: The Apple port of SciSoft (which always had the unofficial nature) has not been updated since 2013; ESO's SciSoft page has described it as "no longer available" since at least June 2016, whatever other websites may still say. In particular, this port's Python executables and libraries are woefully out-of-date.

We are in the process of replacing SciSoft on Apple Macs, piece by piece. More up-to-date versions of SciSoft's Python libraries are available from our build of MacPorts, with the sole known exception of PyRAF (for which, since IRAF is an increasingly minority sport, a virtual-Python setup is appropriate). If you find otherwise, please let us know (at itsupport (at) physics), and we'll extend our MacPorts build, or add extra freestanding packages, as necessary.

We reserve the right to completely remove this port of SciSoft from the Astro build of macOS at any time. Tread with care. Continue at your own risk.

'Yere be dragons ---

Caveats:

  • The system Python uses the system's version of Tcl/Tk to do graphics display; and said Tcl/Tk uses only Apple's graphics interface, which cannot be tunnelled over SSH. This makes it hard to use (eg) matplotlib, thence the plotting facilities of pylab, remotely. Please see below for ways to work around this.

  • If you migrate from Snow Leopard to Lion (or Mountain Lion), you may have to reinstall any Python libraries from original sources, in order to allow for the difference in default Python versions.

  • SciSoft 2013.4.1 is now the standard for Lion-and-newer desktops and laptops, and sports Python 2.7 libraries. Please help us discover any shortcomings therein, for possible additions to the Python AstroPack.

Please see also:

Python's search path

Say the following, at the interactive Python (or ipython, or PyRAF) prompt:

import sys
for p in sys.path: print p

.... plus an empty line; or say at your shell prompt:

pypaths

.... which runs the above script under the default version of Python. This gives you an ordered listing of which directories Python looks into and which "eggs" are searched, and in which order, in response to something of the form import foo. (Having one directory per line is much easier on the eye than the traditional print sys.path, and makes checking of path-order problems a snip.)

Trivia note: Python won't put a directory into sys.path unless it actually exists and contains something substantive. This can be confusing.

Nontrivial note: It may or may not matter if you see references to Pythons 2.6 and 2.7 in the same pathname: some Python libraries built under Python 2.6 run under Python 2.7 without problems, but others very pointedly do not. Please let us know (at the usual address) of any such refuseniks you may discover.

Annoyance note: Python's easy_install (see below) won't install things into a directory which does not yet exist.

To directly check which version of a library will be loaded, say something of the following form at the interactive Python (or ipython, or PyRAF) prompt:

import foo
print foo.__file__
print foo.__version__

The first print statement will give you the full pathname of the file which has been imported, which may include the short Python version number. The second will often yield version information (but don't be disappointed if it yields complaints). Alternatively, saying

pypaths foo

.... at the shell prompt does this for you as a Python script, and silently swallows any version-info errors for you.

Speed hint: You don't need your Python libraries to be in the current directory; and if you're working under your network home directory, your file read/write access will be slower than it need be. The solution comes in two parts:

In addition to speeding up your code, this will save you cluttering up your Python source directories with data, or your data directories with Python source. If you need to constantly tweak your Python code while you're doing the analysis with it, you can always have two or more terminal (or xterm) sessions open at the same time with different current directories, and edit in one while doing the analysis in another.

Using SciSoft's Python libraries

At the shell prompt, say:

Activate scisoft

This adds SciSoft's Python library directory to the end of the environment variable $PYTHONPATH, the eventual upshot of which is to add its "eggs" towards the front of sys.path, and its directories a little further down.

Problems with matplotlib

To get matplotlib to display plots, you may need to first create the file ~/.matplotlib/matplotlibrc containing one line from the following set:

backend : MacOSX # Known to fail under 10.5
backend : GTKAgg # Known to fail under 10.6
backend : TkAgg  # The traditional standard

The Advanced Player can, of course, adjust this to taste, and is hereby invited to share his or her findings by e-mailing IT Support at the usual address, or in ]the Comments section at the foot of this page. Help with checking matplotlib in the current version of SciSoft under the new system setup is particularly welcome.

SSH vs matplotlib:

The display backends in the newer Scisoft builds work only with the local display (see the caveats at the head of this page). Attempts to tunnel matplotlib plots over SSH tend to yield error avalanches, including the giveaway:

_RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL.

.... somewhere under all the rubble. One way round this is to specify non-display backends:

backend : ps  # Yields PostScript
backend : pdf # Yields PDF
backend : agg # Yields PNG

You will see nothing on your screen when you say plot(whatever), but then saying savefig('foo') will produce the file foo.ps, foo.pdf or foo.png in your current directory, which you can view separately.

PyRAF vs matplotlib:

  • When matplotlib is imported, it sets its backend once and for all.
  • When PyRAF is imported, it forcibly imports its own version of matplotlib, which expects to be using the TkAgg backend.
  • Upshot: If you import matplotlib and PyRAF in that order, PyRAF's plotting will be at liberty to throw a fatal and mysterious error.

Known fixes:

  • Don't do that, or:
  • Set your backend to TkAgg to avoid the conflict.

My thanks to Sam Doolin for this information.

The Python AstroPack

The current (July 2011 and July 2012) builds of SciSoft have some minor shortcomings, including:

  • DateUtils and ATpy were found missing.
  • The copy of the Python Imaging Library doesn't know how to handle JPEG or fonts.
  • PyRAF has assorted problems, which are fixed in a more recent version. If you get an error avalanche including ....

    IrafError: IRAF graphics WCS is singular!

    .... about halfway down, then using the AstroPack is for you.

To get round this, and to update one or two other items (eg pytz), the Python AstroPack has been created: this extends SciSoft's Python library set, in the same way that SciSoft extends the system's. (We hope to add in SSH-tunnel-compatible backends for matplotlib at some point, as and when we work out how to do so without breaking anything else.)

The AstroPack is now available from the Astrophysics category in Self-Service, named "SciSoft: Python AstroPack" to group it with its parent suite. Thereafter, you can say either of:

Activate astropack
Activate scisoft

at the shell prompt to Activate it: this performs a sub-Activation of scisoft-no-astropack (see below) for you, and fiddles with $PYTHONPATH in such a way that the AstroPack's libraries take precedence over SciSoft's. See /Local/astropack-osx10.x/README after installation for more information, and:

Activate help astropack

.... to see the contents and version manifest.

We may or may not make this part of the standard Astro setup at some point. If you have installed the AstroPack, but wish to Activate SciSoft without it, say either of:

Activate scisoft --no-astropack
Activate scisoft-no-astropack

.... in a clean shell session, and let us know (at the usual e-mail address), as needing to do so would constitute a bug.

Using your own Python library directory

.... for your own Python source, or for third-party pure-Python libraries.

MacOS X's system Python has certain user-specific and Apple-centric directory names hardwired into itself. Amongst other things, if the directory Library/Python/2.7/lib/python/site-packages/ (or Library/Python/2.6/lib/python/site-packages/ for those using MacOS X Snow Leopard) exists under your home directory, this should appear at the end of sys.path without needing to do anything beyond creating the directory. In particular, the traditional mucking-about with the environment variable $PYTHONPATH, which may be necessary under Linux, isn't required here, and the previous standard pathname (starting with a directory such as ~/.python or ~/.lib) seems to have fallen by the way.

UPDATE WARNING: Something seems to have changed, but I've yet to ascertain quite what. In the meantime, putting ~/Library/Python/2.7/lib/python/site-packages/ at the head of $PYTHONPATH (as detailed in Python's search path, revisited below) places this directory before any other directories in sys.path, and its eggs before all others; and all appears to be well again.

All assistance humbly accepted. In particular, I don't really know which of:

~/Library/Python/2.N/site-packages/
~/Library/Python/2.N/lib/python/site-packages/

.... is supposed to be used, and under which circumstances.

Any Python code you put into the above directory will be visible to Python without further ado. To keep things clean, however, the wise user would segregate his or her personal code from that from other sources, which requires a few more steps. For example, to install a set of code called "foo", provided as pure-Python sourcefiles:

mkdir -p ~/Library/Python/2.7/lib/python/site-packages
cd ~/Library/Python/2.7/lib/python/site-packages
mkdir foo
cp -pr ~/Downloads/foolib/* foo/
echo "foo" >>foo.pth

The last line causes ~/Library/Python/2.7/lib/python/site-packages/foo/ to appear in sys.path before ~/Library/Python/2.7/lib/python/site-packages/, where Python can see such things as __init__.py and act accordingly. Again, you won't need to tinker with $PYTHONPATH to make this happen.

Adding other third-party Python libraries

If you wish to use a third-party library which is available as an "egg", first create a file called .pydistutils.cfg in your home directory, with the following contents:

[install]
install_lib = \
 ~/Library/Python/$py_version_short/\
    lib/python/site-packages
install_scripts = ~/bin

(This is three lines: the middle one has had to be folded twice, once at each backslash, to get it to fit in this margin. Please see the attached downloadable file, whose name has been mangled horribly by Drupal's security mechanisms, which contains the above text unwrapped. Those editing for themselves should remove both each backslash and any spaces either side of it.)

Next, download the "egg" file, eg foo-v42.0-py27.egg, to (eg) your Downloads directory. Then, at the shell prompt, say:

mkdir -p ~/Library/Python/2.7/lib/python/site-packages
mkdir -p ~/bin
easy_install ~/Downloads/foo-v42.0-py27.egg

This should copy foo-v42.0-py27.egg into your site-packages directory, and additionally create an instance of easy-install.pth in the same directory which tells Python to use this "egg" whenever you say import foo. You can verify this for yourself with:

pypaths
pypaths foo

It's also often possible to use easy_install to install (eg) foo-v42.0.tar.gz if the "tarball" in question contains pure Python; under certain mysterious circumstances, it will install it as an "egg". If you need to do the python setup.py build|install dance, eg if compiling something with C or Fortran is involved and/or you need to use non-default settings, check with python setup.py --help where Python thinks it's going to place the installation, and adjust to taste. Please see also:

Portable Software

Python's search path, revisited

If you're trying to use the above mechanism to replace an outdated or badly-configured version of a library already present in SciSoft, you'll need to check that your version appears first in sys.path, using the suggestions in the section Python's search path at the head of this page. If the order's wrong, now is the time to add your site-packages directory to the head of $PYTHONPATH, by something of the form:

setenv PYTHONPATH \
  ~/Library/Python/2.7/lib/python/site-packages:\
  $PYTHONPATH

(Again, this is a single line, folded twice at the backslashes. You'll need to remove all the spaces except for those around PYTHONPATH.)

This builds up $PYTHONPATH in an incremental manner. That way, you don't have to know or care what the previous value might have been, and you're not leaving yourself a hostage to fortune if and when said value changes (eg if you move your files to another operating system). If this is to go in your .cshrc, or in some other shell script before $PYTHONPATH has been set, instead use:

setenv PYTHONPATH \
  ~/Library/Python/2.7/lib/python/site-packages

(Examples of this style of coding appear many, many times in /Local/bin/init/*.csh, together with code to detect whether the variable in question is empty and to then employ the appropriate incantation.)

PLEASE NOTE: This is not a universal panacea: every "egg" will hoist itself above any non-egg entry in sys.path other than that for the current directory. If you are a sufficiently Advanced Player, you may be able to fix up sys.path in a more appropriate manner, but strictly on the basis that if you break Python, you get to keep both pieces.

Assistance, please ....

Python is (let's say) a young language, and things change at a rate of knots. If you find anything in this page which is out-of-date, misleading, outright wrong, or missing, please let us know at the usual e-mail address, or add a comment in the Comments section at the foot of this page.

FileSize
dot_pydistutils_std.txt108 bytes

Categories: Apple | Astro software | Astrophysics | Mac | OS X | python | scisoft