ASTR 511 (O'Connell) Lecture Notes

INTERACTIVE COMPUTING
AND IMAGE PROCESSING


Saturn-3 Images


I. INTRODUCTION

Everyone knows that the science that can be done at a given time is circumscribed by the available resources. In astronomy, we often regard telescopes as the essential infrastructure.

But data taking and analysis resources are just as important, often more so, in determining what we can do. The development of astronomy over the last 50 years has been shaped by the capabilities of computers to a much larger extent than is often recognized.


II. HISTORY OF COMPUTERS IN ASTRONOMY

  • 1800-1920: Women

  • 1920's-50's: Mechanical calculators (arithmetic)

  • 1960's:

      Mainframe computers (e.g. IBM 704, 360; CDC)
      Batch processing only (i.e. no interactive control)
      Hollerith punch card programming
      Data storage: punch cards or magnetic tape
      Fortran is dominant language
      UNIX developed (1969)
      First Internet experiments (1969)
      VICAR: JPL planetary imaging reduction system
      CITRAN (Caltech), JOSS (MIT)
        (Experimental interactive time-sharing systems, running on mainframes; printer terminal I/O)

What is this picture?

  • 1970-75:

      Minicomputers (e.g. PDP) used in data acquisition
      HP & TI hand calculators appear (full function/programmable)
      Interactive data analysis using Tektronix graphics terminals
        E.g. NRAO, KPNO spectroscopy packages
        1-bit plots but no greyscale images

  • 1975-80:

      Fortran 77
      Internet protocols (TCP/IP) developed
      PC's appear, but little used professionally
      Early systems for interactive 2-D astronomical image processing:
        IRAF (KPNO), AIPS (NRAO), IDL (NASA)
        Use stand-alone greyscale (256 level) or multiframe (color) TV monitors

  • 1980-85:

      DEC VAX Minicomputers widely used
        Virtual memory allows processing of large images
        VAX's offer multiple shared terminal access; cards eliminated
        Greyscale/color terminals
      UNIX versions proliferate (Berkeley BSD, Sun, other)
      Standardized transportable astronomical data formats (FITS)
      File exchange protocols (e.g. FTP) established
      Email available on limited-access networks (government)
      STScI develops STSDAS (IRAF-based) to reduce HST data
      IBM PC's used as desktop terminals; limited graphics
      X-windows software developed (MIT)
      Supercomputers (e.g. Cray) are specialized, expensive systems

  • 1985-90:

      Mainframe computers/operating systems begin to fade
        1988: UVa Cyber mainframe allows maximum of 64 files per user(!) in disk storage
      SUN, Silicon Graphics microcomputers gain popularity
        Desktop workstations include full graphics/image displays
        High capacity disk drives begin to drop rapidly in price
      VAX/VMS operating system cedes ground to UNIX
      TeX typesetting software invented by Donald Knuth
      DAOPHOT (stand-alone Fortran program for stellar photometry)

  • 1990-99:

      UNIX systems dominate in astronomy
      LINUX (PC-based UNIX) appears
      Fortran 90/95 modernize the standard computational language
      Networked workstations/PC's linked to form supercomputers
      Microsoft Windows operating system appears and dominates non-scientific market
      Portable laptop computers become popular
      Graphical User Interfaces (GUI's) become widespread in applications software
      C, C++ in wide use
        (Note that the Baroque programming style of these languages is totally unsuited to the earlier punch-card environment.)
      Internet transforms computer communications
        Email becomes universal
        World Wide Web protocols established (led by physical scientists)
        Efficient browsers, search engines (e.g. Google)
        Astronomical public internet data centers established (e.g. CDS, NED, ADC)
        Public literature databases established (e.g. ADS, astro-ph)
        Internet data transfer rivals or replaces transfer by magnetic tape
        Open source software proliferates
      IRAF, STSDAS, AIPS, IDL become more sophisticated & robust
      CIAO (Chandra X-ray data reduction)

  • 2000--  :

      Search engines & literature databases become essential to routine work
      Paper-based publications fade
        Electronic publications become versions of record
      Electronic presentations become standard (PowerPoint, PDF)
      Laptops become ubiquitous
      Lower cost LINUX/PC systems take lead over UNIX for research
      Apple adopts UNIX for Macintosh OS-X and reaps ~75% of the astronomy laptop market
        Outlook poor for Windows in research
      Large databases established:
        Sky surveys (e.g. SDSS, 2MASS, POSS, NVSS)
        Telescope archives (e.g. STScI/MAST, Chandra, NOAO, NRAO)
      Plans for new facilities push envelope of computing power (e.g. ALMA, LSST)
      International Virtual Observatory planned


III. WIDELY-USED CURRENT SYSTEMS IN ASTRONOMY

A. Operating Systems

    UNIX
    LINUX and Apple OS-X
    Windows
      Mainly for communications, text, presentations
B. Languages

    General purpose: Fortran, C, C++, IDL
    Scripting: Perl, Python

C. Multipurpose Interactive Astronomical Data Reduction Packages

    UVOIR: IRAF, STSDAS, MIDAS
    Radio: AIPS, AIPS++
    X-ray: CIAO, TARA

D. Special Purpose Software (interactive and not)

    Image Display: SAOImage, DS9, ATV
    Point Source Photometry: DAOPHOT, DOPHOT
    Plotting: SuperMongo, PGPLOT
    Source ID/Classification: FOCAS, SExtractor
    X-ray Model Spectra: XSPEC
    Math/Statistics: Numerical Recipes (C, C++, Fortran)

E. Interactive Mathematical Packages
    Mathematica, Maple, MATLAB


Recent evolution:

    Dramatic improvements in computer hardware and software have made it possible to manipulate very large amounts of data rapidly and efficiently. However, the critical qualitative change in the conduct of astronomy has come about through access to powerful interactive computation and display, which permits understanding of numerical data in ways that were possible only for analytic functions 20 years ago.

===> In the last 20 years, interactive computing has produced a fundamental transformation in the conduct and scope of astronomical research.


Choices?
    If you become an observational astronomer, you have essentially no choice but to become familiar with IRAF/STSDAS (UVOIR), AIPS (radio), CIAO (X-ray), etc. because these provide the primary standardized data reduction procedures. The main decision, therefore, is what other systems are most profitable to learn.


IV. IDL

The "Interactive Data Language" is distributed by Research Systems, Inc.

Unique status as a fully interactive, high-level language with a large library of astronomer-written utilities.

Nearest thing to a "universal" astronomical data system because:
  1. It is currently used across the entire EM spectrum from gamma ray to radio wavelengths. This is unlike most other astronomical software, which is usually confined to one wavelength regime.

  2. It potentially duplicates the functionality of almost any other astronomical software, from simple arithmetic computations and graphics to full-blown GUI-based data analysis packages.

      Don't be misled: the capability is there, but you may have put in the effort to make it work, depending on the application.

Places to look for a quick introduction:

Key advantages over Fortran, C, etc:

    Fully interactive with embedded graphics and I/O device drivers
    Dynamic memory allocation
    Accelerated computation on arrays
    Large number of built-in interactive functions & utilities
    Large body of public, easily-implemented astronomical utilities
    Writing code with interactive programming/iteration greatly
          enhances efficiency & reliability
    Pre-compilation not required

Key advantages over MATLAB, etc:

    Full high-level programming features
    Oriented toward image processing
    Large suite of astronomical-oriented utilities (e.g. FITS file I/O,
          coordinate systems, astrometry, databasing, etc)

Key advantages over IRAF, etc:

    Full high-level programming features; not simply a package of
          pre-defined, specialized programs
    Intended for user customization, adaptation, extension
    Active data stored in RAM, not as files
          (minimizes use of cumbersome file names and
          greatly facilitates computations)
    Greater versatility, transparency, and user control
    Journaling, command recall/edit, & other convenience features
          for increased efficiency
    Source code for applications routines available and modifiable on demand
    Writing code with interactive programming/iteration greatly
          enhances efficiency & reliability
    Pre-compilation not required
Key disadvantages:
    IDL is an interpreted rather than compiled language:
      Compiled languages execute more rapidly and use memory more efficiently. The differences will be invisible for most computations up to moderate scale. But for large-scale computing, Fortran and C are a better choice.
    No equation-solving capability (use Mathematica/Maple there).


V. THE ROLE OF IDL

  • Data analysis and visualization (as opposed to reduction)

    • IDL is not a recommended alternative for routine data reduction tasks (e.g. for CCD mosaics) where IRAF, AIPS, etc., offer powerful, reliable, and convenient packages. These have gone through many iterations, and most user needs have been accommodated.

    • But the multitude of specialized analysis procedures needed following data reduction requires versatility, extended capabilities, and customization not offered by standard packages. IDL is optimized for this role.

  • Bridging gaps in standard packages

    • IDL is now being used to convert and analyze NRAO GBT data in ways not possible with AIPS++

    • Convenience of IDL interface now recognized in effort to develop Python-based "PYRAF" in order to make IRAF "more like IDL"

  • Any moderate-scale computation requiring file manipulation or generation of numerical tables, graphics displays, or images

    • There might be dozens of such applications arising in a single project that would be more cumbersome to program in a compiled or scripting language, especially if graphics is required

    • Like a "super hand-calculator," IDL is instantly available for a wide range of such applications

  • Enhancing scientist efficiency

      The main concern of working scientists in approaching software should be how much of their own time elapses before they get a given result. Speed of execution is secondary, whereas the speed of coding/configuring and testing software is primary.

      Although we tend to think of computers in the context of large-scale "pipeline" computations and applications which are CPU/memory-bounded, these are really specialties. Less than 10% of the overall computing effort in astronomy (measured in person-hours) involves such problems.

      Instead, the software effort of most astronomers is devoted to moderate-scale computations, interactive data inspection and evaluation, special cases not supported by large applications packages, production of graphics for publications, and so forth.

      This environment places a premium on software convenience, transparency, modularity, portability, and versatility---exactly where IDL excels.

      The intrinsic capabilities of IDL coupled with its interactive environment and extensive user code libraries greatly enhance scientist efficiency. A given result can be reached using IDL in a small fraction of the time it would take to do the equivalent programming in other high level languages.

      IDL supplants graphics-display software such as SuperMongo or PGPLOT. It duplicates much, though not all, of the mathematical functionality of MATLAB, Maple, and Mathematica. It can do much of what can be done with PERL. Finally, it duplicates all the functionality of Fortran or C and is to be preferred unless the speed of computation is really important.

  • As an educational tool


VI. IDL DEMOS

  • A demo of interactive computing and image processing will be shown in class
  • A package of standard IDL demos is supplied by RSI. Start from the UNIX prompt and type idldemo.
  • An IDL Tutorial designed for this class introduces IDL's basic features and its use in image processing. Other sites offering IDL tutorials are linked to the Astronomy User's Library.


VII. GENERAL TIPS FOR DEALING WITH COMPUTERS

  • Computers are tools, not science
  • Despite appearances, computers have not increased the capacity of the
          human brain to absorb and process information
  • Don't trust and always verify
  • If it's important, read it from paper

      [I have yet to meet anyone who can proof-read accurately
      from a computer screen. I am not sure why this is,
      but it is not a comforting observation.]

Reading and lab assignments for this lecture

Web links:
Previous Lecture Lecture Index Next Lecture

Last modified September 2015 by rwo

Text copyright © 2000-2015 Robert W. O'Connell. All rights reserved. These notes are intended for the private, noncommercial use of students enrolled in Astronomy 511 at the University of Virginia.