Screen: managing multiple programs over a single ssh connection

If you're running a command-line program which takes a fortnight to run to completion, you don't need to leave yourself logged on for a fortnight. Using the screen facility, you can start up your program, detach the "screen session", log off, log on remotely a week later (from, for example, a cybercafe in the middle of the Lake District), reattach your session from there, and examine the progress of your long-running program. For that matter, you can start your session while logged on remotely (from, for example, a cybercafe in the middle of the Lake District), detach it from there, and leave it running on your system without having to leave yourself logged on remotely for a fortnight (from, for example, a cybercafe in the middle of the Lake District).

Other benefits:

  • Jobs started in the background in the more traditional UNIX manner, eg with nohup or via at or batch, have to run completely "hands off": any and all input and output needs to be via files on disc, and needs to be specified in advance.

    • With screen, you can interact with your program when you need to, eg for keyboard-entered startup parameters or mid-flight course corrections, as if you were using Terminal or xterm.

  • While you're logged on, you can turn your back on your long-running job; it needn't be taking up your valuable desktop space, or clamouring for your even more valuable attention, unless and until you wish it to.

  • You can do more than one thing in the same screen session: create a second screen "window" (subsession), and run something else within it, in a manner familiar to users of tabbed browsers.

    • If you are (or will be) logged on remotely (from, for example, a cybercafe in the middle of the Lake District), this all happens over the same SSH connection, which is much less of a strain on your laptop and the network than N separate logins. See also the next point.

  • If you're logged on remotely without using screen, and the network (or your laptop) crashes, your job would normally also perish; similarly if the SSH gateway decides, however mistakenly, that you haven't been active enough, and times out the session.

    • If the job is running under screen, and your connection to the system dies, screen will detach your session for you, and your job continues running, blissfully unaware that you've gone away without notice.

  • Logging off permits your system to do the sort of housekeeping jobs which it can't do while you're logged on. This includes some system updates.

    • Admittedly, the Law of Inanimate Objection applies: the more important an update is, the more likely it is to require a reboot. Please see also the Caveats.

Caveats:

  • You can't run Aqua-based programs from within screen, any more than you can run them remotely from within an SSH session.

  • You may or may not be able to start up X11-based programs from within screen. Even if you can, you can successfully detach the screen session, but the act of logging off will terminate the X11 connection, which in turn may kill the program. This rather defeats the object of using screen in the first place.

    • Beware on this point: some programs which ought to be able to run without X11 may detect the presence of an X11 session from which screen happens to have been launched, and behave (in)appropriately. Caveat emptor; and test before relying on it.

    • If you have problems with this, try SSHing into the target system from a direct login on another system (eg the laptop you intend to take on holiday to the Lake District), using the -x flag to disable X forwarding.

  • This is a simple program, not a UPS, nor can it turn a single system into a high-availability cluster. Using screen can't protect you from system crashes, power cuts, third party, fire, theft, acts of $DEITY, or other causes of rebooting or total system lossage. These are not unknown.

    • For On OsX (Macintosh) machines, the automatic Central Physics operating-system update mechanism always forces a reboot under MacOS X 10.8 (Mountain Lion) and later; and some security updates are sufficiently urgent (eg due to attacks being seen in the wild) that we have to apply them to all systems Right Now and with extreme prejudice. This is agreed to be regrettable.

    • For better defence against malign fate, use of the Linux clusters is strongly advised:

How?

Starting up a session

  • At the command line, type:

    screen -R -d

    This starts up a session if none already exists, or reattaches to one if it does, running a copy of your favourite shell.

    • If you need to do any Activations, do not do so outside screen. The current implementation of the Python AstroPack in particular will object violently at attempts to Activate it more than once. (This is a good example of why putting an Activation in one's shell startup file is a Bad Thing.)

  • For various security-related reasons, what you get first isn't a login shell, and may be missing important elements from $PATH. To rectify this, type:

    tcsh -l

    or

    bash -l

    to taste: this gives you a login shell within the non-login shell which screen first thought of. (This is agreed to be confusing. If you start other new screen "windows", as mentioned in the next section, you'll need to do this in each subsession you start.)

  • Take whatever preparatory steps you need to take (eg Applications Setup commands) to prepare your shell to run your program. (This, again, would need to be done afresh in any other new subsession you may start.)

    • Under central Linux machines, some additional magic is required to maintain authentication to the back-end disk servers. Please see:

      Long jobs under Linux

  • Start your program running, and interact with it if and as necessary.

Using multiple windows

  • To create a second "window" in the same session of screen, type

    ^A ^C

    (control-A control-C). This starts up a second "window", containing a fresh instance of your shell.

    • You will need to set this shell up in the same way as in the first subsession, using tcsh -l and any required Activation(s) in that order.

  • To switch to the next "window":

    ^A ^N

    (control-A control-N, no space).

  • To close the current "window", see End Of Play below.

Detaching and reattaching the session

  • Detach by typing

    ^A d

    (that's control-A, then d). Your programs will continue to execute, and the last screenful of the output will be kept for you, awaiting your reattachment.

    • Some programs (as mentioned in Caveats above) can survive in a detached screen session, but expire if you log off completely. If you're trying this for the first time, or are otherwise of a nervous disposition, make the test more rigorous: log off, and log back on again immediately. This will save you the frustration of logging on a week later (from, for example, a cybercafe in the middle of the Lake District), only to find that your program expired when you logged off a week before.

  • Reattach by typing

    screen -R -d

    (yup: same incantation) at the command line. Any output from the program to the screen(s) will be waiting for you, and you can interact with it (or them) if and as necessary.

  • Lather, rinse and repeat.

End of play

  • When you've completely finished, exit from a single subsession by typing

    exit

    You will usually need to do this twice (once for the inner login shell, and once for the outer non-login shell), unless you have used the exec trick mentioned in Tips below.

    • Repeat if and as necessary in your other subsessions.

  • Once the last subsession has ended, screen itself will exit (and tell you so), and you'll be back at a non-screen shell prompt.

Tips, and further reading

Tips and Tricks:

  • Control-D at the beginning of a line is treated by the shell as equivalent to the exit command; one inadvertent keystroke can ruin your entire screen (sub)session. If you type

    set ignoreeof=7

    at the command line (or

    IGNOREEOF=6

    for us bash users), your shell will ignore the first six consecutive Control-Ds, which is a reasonable safety-catch. This is worth doing anyway to protect your normal interactive shell, and can happily go in your shell startup files. Saying man tcsh (resp: man bash) will be your friend here.

    • Aside for tech-heads: Unless subverted by the IGNOREEOF mechanism, Ctrl-D at a keyboard is translated to end-of-file. See man stty, and meditate on the implications of standard input being a file, even when it's a keyboard.

  • For the Advanced Player: If you replace

    tcsh -l

    by

    exec tcsh -l

    in the startup phase, the non-login shell screen awarded you is replaced by a login shell. You need then only say

    exit

    once per subsession at the end of play. (Us bash users can make the obvious substitution.)

More:

For more information, please see:

  • http://mdxi.collapsar.net/docs/screen/
    A tutorial on using screen. This includes how to create multiple windows in the same screen session and switch between them, and how (if you're determined to use multiple screen sessions) to reattach to the right one.

  • The manual page:

    man screen

    .... at the command line, for (very) full information, including what the command-line invocation and the commands within screen actually mean.

Categories: Apple | Astro software | Astrophysics | HOWTO | Mac | OS X | Theory