Care and Feeding of Shells

CONTENTS: one Klein bottle.
Please open this packing case with the crowbar which you will find inside it.

Something I've noticed people do, in all innocence, is to put frequently-used initialisation commands and environment-variable settings into their shell startup files (.tcshrc, .cshrc and .login for users of tcsh, or .profile, .bash_profile and .bashrc for those of us who prefer the Bourne-Again shell). I've done it myself over the years, and suffered the consequences. I offer the following guidelines to help others learn from my mistakes, and some clean-start guidelines and examples for cases of extreme emergency.

Please see also:

The Problem:

Executive summary:

"Tuning" a program or suite by setting environment variables blurs the boundary between that software and your running shell. If this piece of software requests or requires the setting of environment variables, doing so causes said software to "leak out" into your shell, in ways which may quite possibly conflict with the corresponding "leakage" for that program.

Putting commands and shell-variable settings in startup files can pose several related problems:

  • Each such command or variable setting doesn't just "tune" the software in question: it also tends to specialise your shell to do this set of things. If you later need to use a different set of commands to do that (whether instead, or as well), then the shell adaptations to do this can get in the way.

    • Egregious example: Once upon a time, to save having to manually tell an e-mail client to send carbon copies to myself, I put my e-mail address in the environment variable CC in my own shell startup files. Shortly thereafter, I discovered the make utility had forgotten how to compile C programs.

  • Certain commands, notably those used by Application Setup, emit comments to the "standard output" (usually your terminal screen) and/or the "standard error" (also your screen). If said commands are in shell startup files, the resultant shell is not "clean": this can break certain remote-access methods such as scp, sftp and rsync, often with mystifying symptoms.

    • Tip of the Day: if scp or rsync fails, try:

      ssh machinename "true"

      If you see anything between giving your password and getting your (local) shell prompt back, that's where your problem is. (My thanks to David Glover for this tip.)

    • Underlying problem: the ssh client software being used as a transport layer is expecting to see version information from the SSH daemon to help it do its job. If it sees anything else first, it assumes it's not talking to an SSH daemon, and gives up. (This is actually a safety mechanism masquerading as a bug. The resemblance is agreed to be uncanny.)

  • If in all innocence you put something in a shell startup file which prevents the shell from starting up, this will (ahem) prevent you from starting up a fresh shell session to fix the problem. See the one Klein bottle comment above. (This is much more deadly if a GUI is not directly available, eg while using ssh from elsewhere.)

    • If you do need to edit your .tcshrc (or whatever), then test it in a different window, eg by firing up a fresh Terminal or xterm session, before you exit from the shell in which you've been editing the startup files, while you've got the chance to correct any problems before they become fatal.

  • Underlying metaproblem: Each software producer has a hard enough time getting their own software working in isolation on every supported platform, without also checking (and working around) potential conflicts with every other program or suite in every possible combination. It's inevitable that they tend to develop tunnel vision.

Best Practices:

If you wish to use a particular set of shell commands and associated variable settings frequently, make a wrapper: put the commands and settings in a file, put a "flash-bang" line right at the top to tell it which shell to use, eg one of:

  • #!/bin/sh
  • #!/bin/tcsh

.... then make that file executable (man chmod will be your friend), and run the shellfile. That's how various things we put in /Local/bin work. Some of them (eg IRAF) use Application Setup internally, which is an idea I stole from Alison Crocker.

If you wish to use the same set of shell initialisation commands frequently, put them in a file, and source that file on a need-to-use basis. For example, I habitually say

source ~/.majoc

at my first shell prompt, every time I log on (and within each new shell window), to pick up a whole slew of nonstandard settings. That way, if I alter this code in ways which would blow my shell out of the water, I get the chance to start up a shell without said code to remedy the situation. Having to change my $HOME many times over the years has taught me to live out of a suitcase, and that the inside of said suitcase is not the best place to put the spare set of keys.

If you wish to save typing, you're at liberty to make that sort of thing an alias, and put the alias in your startup file. For example:

source /Local/bin/Setup.csh
alias mystartup 'source $HOME/mystartup.csh '
alias myscisoft 'Activate scisoft ' (# NB: now well and truly Deprecated)

If you do no more than that, your login shell will be "clean", and your (possibly complicated) set of startup instructions is all in one place.

Beware of causing an alias to invoke itself, directly or indirectly: this can cause your shell to hang. (If alias foo calls alias bar which in turn calls foo, the shell may not be able to detect that foo is still an alias. Boom.) Adding "my" to the alias name, or appending "-x" or "-init", is one way to avoid this sort of confusion. The wrapper script emacs-x, documented more fully in Editors available in Astrophysics, follows this pattern, which also permits you to get at Apple's EMACS without special pleading.

Second-Best Practices:

If you really must put active code (basically anything but aliases) in your shell startup files, try this:

source /Local/bin/Setup.csh
source $HOME/mystartup.csh

This keeps your startup code in one place, and can be commented out quickly in emergencies. (Bear in mind the "clean shell" problem .... read on.)

source /Local/bin/Setup.csh
Activate scisoft

Much the same, but more limited. Also the Activation emits a comment to your standard output, so your shell won't be "clean" enough to satisfy (eg) scp without further work.

if ($?prompt) then
source /Local/bin/Setup.csh
Activate scisoft
endif

Said further work: The portion inside the if/endif statements only gets executed under an interactive shell. (The test in the if statement is true if the primary shell prompt exists and is nonempty, which is the case for interactive shells but not noninteractive ones.)

if [ -n "${PS1:-}" ] ; then
source /Local/bin/Setup.bash
Activate scisoft
fi

The Bourne-Again shell equivalent, which explicitly tests whether the primary shell prompt is nonempty. (The extra punctuation round PS1 is an example of defensive programming: it avoids throwing an error if the variable doesn't exist.)

WORST PRACTICES (Danger, Will Robinson):

Please do not stitch into your .tcshrc the raw shellcode behind any Activation, still less the initialisation code for (eg) IDL, SciSoft or Starlink.

  • This can easily render your shell "unclean", or can break other software or your entire shell session in a wide variety of interesting ways, sometimes by delayed action.

    • Fallout and other collateral damage can range from the subtle (eg from real life, a single missing character caused the man utility to fail to find its own manual pages) to the spectacular (eg preventing your shell from working at all: need I say "one Klein bottle" again?).

  • Commenting out twenty or more lines, especially if they've been scattered through your .tcshrc at different times over many months, is an extremely fiddly and painful process. Trust me on this.

    • It's even more painful dissecting a single absolute $PATH statement whose far end disappears into the mists, and whose nearer portions are smothered in cobwebs. If instead you build up long environment variables like this in an incremental manner:

      # 25th March 2000: for programs foo and bar.
      setenv PATH $PATH:$HOME/millennium/bin

      .... then you can also add comments, as shown, to remind yourself in a decade's time why you added this bit here, and that bit there.

  • Even if you've correctly second-guessed what the Activation (or other initialisation code) did yesterday, that by no means guarantees that your guess will remain correct tomorrow, when the Activation, the code underneath it and/or your circumstances have changed.

    • This includes copying existing shell startup files to (or from) your laptop without modification, even if they're both Departmental machines.

Remember: in automatic startup files, defining aliases is OK, but DIRECT EXECUTION CAN BE FATAL.

Clean start

If you (or something you install, or an rsync from another system) manage to mangle your shell startup files beyond all hope of recovery, the first thing to do is to rename them aside, by saying:

cd
mv .cshrc .cshrc.broken # for all appropriate values of .cshrc

.... then start up a fresh shell without them. (That is, if you can get at the shell prompt. If you can't, but can log onto the console GUI, see the end of this section for a suitable Self-Service script.)

As a guide to what to do next, here are the contents of the relevant standard shell startup files which people are given when they first log onto MacOS X in Astrophysics.

  • .cshrc
    Please note:
    • tcsh uses .tcshrc if it can, and .cshrc if it must; so if you've created .tcshrc, that'll hide .cshrc's contents from it, and they won't be used. So don't do that.

    # ~/.cshrc

    if ( $?prompt ) then
    # .... an interactive shell:
    #
    # If you don't reference the Astro Application Setup
    # You risk breaking a LOT OF THINGS for yourself
    #
    # You have been warned!
    if ( -r /Local/bin/Setup.csh ) then
    source /Local/bin/Setup.csh
    endif

    # Add your own startup commands here, for use in interactive
    # logins.
    endif

    # WARNING: if you put anything which emits strings to the
    # console after the above 'endif', it will break ssh, rsync etc.

  • .bashrc

    Please note:

    • bash reads .bash_profile or .profile for "login" shells, and .bashrc for non-"login" shells; quite what constitutes a "login" shell is slightly mysterious at best. If your desired additions must always be executed, put them at the end of .bashrc, and arrange that .profile always sources it (see below).

    # ~/.bashrc

    # Cease sourcing this file if we're not in an interactive shell.
    # Failure to do so can break ssh, rsync etc.
    if [ -z "${PS1:-}" ] ; then
    # .... shell not interactive:
    return
    fi

    # If you don't reference the Astro Application Setup
    # You risk breaking a LOT OF THINGS for yourself
    #
    # You have been warned!

    if [ -r /Local/bin/Setup.bash ] ; then
    . /Local/bin/Setup.bash
    fi

    #
    # Add what you want down here, for use in interactive logins.
    Example append $HOME/bin to $PATH, if dir exists and isn't yet in $PATH.
    if [ ! -d $HOME/bin ] ; then
    true
    elif echo $PATH | grep $HOME/bin >/dev/null 2>&1 ; then
    true
    else
    export PATH=$PATH:$HOME/bin
    fi

  • .bash_profile

    Please note:

    • Any code in .profile will be ignored by bash if .bash_profile exists.

    • bash reads .bash_profile or .profile for "login" shells, and .bashrc for non-"login" shells; quite what constitutes a "login" shell is slightly mysterious at best. If you always need the contents of .bashrc, .bash_profile or .profile needs to include the following code.

    # ~/.bash_profile
    #
    # .bashrc should include the call to the Astro modifications
    # scripts
    # if you don't call those scripts, things WILL break
    #
    # You have been warned!

    if [ -r $HOME/.bashrc ] ; then
    . $HOME/.bashrc
    fi

    #
    # Add what you want down here, for use in interactive logins.
    # Example: append $HOME/bin to $PATH, if dir exists and isn't
    # yet in $PATH.
    #
    # if [ ! -d $/bin ] ; then
    # true
    # elif echo $PATH | grep $HOME/bin >/dev/null 2>&1 ; then
    # true
    # else
    # export PATH=$PATH:$HOME/bin
    # fi

Read-only (and inactive) copies of these files will eventually appear in /physics/share/skel/ on MacOS X systems in Physics. (For Linux systems, the equivalent files, as distributed with Debian and Ubuntu, are in /etc/skel/ ; but you will need to add the Linux equivalents for the Applications Setup stanzas for yourself.)

For readers on Astrophysics desktop systems, there is now a script in the Uninstall category of Self-Service which renames all your shell startup files aside (by appending a timestamp), then instantiates clean versions of the above files as shown. If you've done heavy customisation, it is then your responsibility to amend these clean versions if and as necessary, avoiding of course whatever it was broke your shell, and testing the while by launching a fresh terminal session each time without exiting the current one. The script in question can be made available more widely on request.

Categories: Apple | Astro software | Astrophysics | Development | HOWTO | Mac | OS X