Please cast your votes for Sorcerer's Apprentice (long msg)

[mcw@cs.rice.edu (Michael Wirth)]

Some of you may have seen this information before (since I already posted
it on comp.lang.clos).  If so, I apologize for the redundancy.  But the
recent postings here on Application Frameworks and events at Apple led me
to conclude that this is an appropriate time to post this to
comp.lang.lisp.mcl and the oodl-sig mailing list, and to ask for your
support.

For the past three years or so, I directed a small R&D team at Petroleum
Information Corp. in Houston which developed a MCL class library and
toolkit (framework?), called Sorcerer's Apprentice.  A synopsis of SA
follows below. You can also find a description of an earlier version of SA
as a case study in David Taylor's new book, "Object-Oriented Information
Systems:  Planning and Implementation," John Wiley & Sons, 1992, pp.
326-328.

Unfortunately, due to the downturn in the domestic oil business, my group
was disbanded, we were all laid off, and SA was shelved.  My former
employer, PI, appears to have no desire to continue to develop and market
SA or applications built with it.  But they're not prepared to give the SA
code away either.

I think that SA would be an excellent (and substantial) starting point for
a MCL framework.  You can read the description below to see for yourself if
the functionality it contains is of value to you.  I can vouch for the
quality of its design and implementation, mainly due to the genius of Chris
Wakefield (Newpoint Microsmiths, Houston, D0966@applelink.apple.com), who
designed and wrote almost all of the toolkit code.  Incidentally, Chris is
also an expert MacApp programmer, having written a commercial MacApp-based
product (the Analyzer part of INIT Manager).  Therefore, many good ideas
from MacApp found their way into SA.

I can think of no better use for SA than for it to be made openly available
to the MCL community (presumably through Apple's sponsorship) and to form
the basis for a MCL application framework.  If you think SA might be of
value to you, I encourage you to communicate that fact to Harvey Alcabes,
MCL product manager (Alcabes@applelink.apple.com) and also Alice Hartley
(Alice@cambridge.apple.com).  If there are some parts of SA that are of
more interest than others, please communicate that fact, too.

DISCLAIMER:  I have no financial interest in SA, but I do have a
significant intellectual and emotional investment in it.  Besides, I no
longer have access to the code myself, and would love to see it be made
available so I can again write full graphical, data-based MCL applications
in a week. I also think it would be a shame if such a significant piece of
work were to be lost to the MCL community when the need is so apparent.
Therefore, I hope you'll pardon this blatant appeal for a write-in campaign
to the MCL community :-)

I, and I'm sure Chris and Cleland Early (71321.3443@CompuServe.com), our
other team member, stand ready to answer questions and help in any way we
can.  OK, I'll get off my soapbox now.

Mike Wirth
17519 Hidden Forest Circle
Spring, TX   77379
(713) 370-2965 (office)
(713) 370-3372 (home)
mcw@cs.rice.edu

---------------------------------------------------------------------------

Synopsis of the Sorcerer's Apprentice Class Library and Toolkit
===============================================================

Sorcerer's Apprentice (SA) is a class library and toolkit written in the
Common Lisp Object System (CLOS) by a small team at Petroleum Information,
Houston, over the last three years.  Using SA, five substantial data browsing
and editing applications have been written which combine relational database
storage of data with point-and-click access to all objects from a map-based
interface.  Descriptions of the design philosophy behind SA, the class library
and toolkit, and the applications follow.

Design Philosophy of Sorcerer's Apprentice
===============================================================
Geoscientists spend up to 80% of their time being data detectives -- finding,
collecting, transforming, organizing and integrating data -- before they can
use it.  The Sorcerer's Apprentice class library and toolkit was built to take
advantage of emerging computer technology, empower these end users and wage a
frontal attack on the many forms of complexity they face:
* Complexity in organization:
  Use maps as a simple visual interface and integrating framework for
  point-and-click access to data.
* Complexity in access:
  Provide an easy-to-use front-end to data stored in multiple local and remote
  data repositories.
* Complexity in data diversity:
  Use object technology to conquer the thousands of variations in geoscience
  data representations between the various data types and domestic vs.
  international standards.
* Complexity in data volume:
  Provide an effective and efficient front-end to heterogeneous computer
  networks which store huge volumes of information.
* Complexity in data structure and interrelationships:
  Use object modeling to provide the end users with conceptual models in their
  own terms, which facilitate understanding and navigation through complex    
  data interrelationships.
* Complexity due to constant change:
  Provide a solid framework which optimizes extensibility, flexibility and
  reuse of components.
* Complexity in media and presentation:
  Based on an integrated object model, provide multiple views of data ranging
  from detailed numeric reports to conceptual visualization presentations.
* Complexity in application use:
  Provide a framework within which applications can evolve from stand-alone,
  ad hoc islands of processing to methods fully integrated into the object
  model.

The guiding principle in the the development of the SA toolkit has been to
build a solid, robust foundation, rather than to populate it with features,
and to stress test that foundation with real-world data and applications.

Sorcerer's Apprentice Class Library and Toolkit
===============================================================
SA was developed using Macintosh Common Lisp and the Oracle RDBMS on Macintosh
computers, but was designed to be portable.  In fact, a test port of the
toolkit and the SWIS application to the Sun SPARCstation using Motif/X-Windows
was demonstrated in November, 1991.

SA consists of about 31K lines of code (including 2K lines of low-level C
code) and over 200 classes divided into six major components:
** A portable Graphical User Interface (GUI) manager.
** Tools for maps, map layers and spatial data handling.
** An object-based interface to SQL and the Oracle RDBMS.
** An object-to-RDBMS mapping system.
** Data visualization tools.
** Development tools.

A portable Graphical User Interface (GUI) manager.
------------------------------------------------------
With over 80 classes, the GUI manager in SA provides a windowing environment
in the style of MacApp and LispView (as opposed to CLIM or Garnet).  Built on
the low-level windows of MCL and QuickDraw on the Mac (or the Motif widgets
and XLIB calls on Unix workstations), it provides:
* Regular and floating (tool palette) windows.
* Hierarchical nested panes with:
  - Scrolling, pan and zoom
  - Local coordinate systems with extended addressing (i.e., fixnums) to allow
    very large scrolling panes (e.g., a map of the world zoomed to treetop
    level)
  - Scaled floating point world coordinates
* A full complement of displays and controls including: system buttons, icon
  buttons, text buttons, radio buttons and groups, check boxes, sliders, etc.
* Event handlers for mouse, keyboard and other events and commands,
  coordinated with a tree-structured command chain for nested panes.
* Extensive support for scrolling lists, including:
  - Simple lists of text items
  - Nested outlines with indented expanding/collapsing sub-outlines (using
    System 7 twist-down triangle icons, as in the Finder)
  - "Panelets," list pane cells which can contain a complex mixture of
    displays, controls and text, the format of which repeats from cell to cell
* Popup menus.
* Text-editor panes.
* Dialog boxes with tabbing between data entry panes.
* Mouse trackers and tools for: point-and-click item selection, pull-down
  selection rectangles, free-form selection regions, polygon selection 
  regions, hand grabbers to scroll the view, magnifying glasses for zooming,
  etc.
* Printer support.
* Status indicator panes (e.g., thermometers).
* A framework for the (future) construction of a graphical interface builder.

Tools for maps, map layers and spatial data handling.
------------------------------------------------------
With over 40 classes, SA provides an extensive framework for building GIS
applications.  The facilities provided include:
* Variable map detail based on map scale and user selection; zoom in to get
  more detail.
* In-memory quad-tree indexes (with planned extension to paged indexes).
* Spatial element handling for points, lines, polylines and polygons,
  including:
  - Spatial element caching from disk storage
  - Spatial intersection testing
  - Display of spatial elements
  - Capable of handling tens of thousands of objects on a Mac II
* Map layer administration and interface to object-mapping facilities.
* Support for both vector and raster based maps.
* Point-and-click object selection.
* Tools for semi-automatic map annotation placement.

An object-based interface to SQL and the Oracle RDBMS.
------------------------------------------------------
With just 6 classes, SA provides an object-oriented interface to the Oracle
RDBMS, including:
* Query objects to execute SQL commands and return the results of queries,
  a row at a time or all at once.
* Loader objects for efficient bulk loading of data.
* A simple, generic "SQL form" (pane) for display and editing of table data.

An object-to-RDBMS mapping system.
------------------------------------------------------
One of the more interesting parts of SA is the object mapping (OM) facility,
which allows the user to establish a mapping from object classes and
attributes to relational tables and columns.  Combined with "place holder"
(lightweight surrogate) objects, this provides some of the benefits of object
data bases, while allowing easy access to the vast repositories of data that
now exist in relational data bases.  Using OM, the application programmer and,
ultimately, the user deal only with objects.  As objects are accessed (by
direct links, spatial selection, or OM-select queries, as in the Itasca
OODBMS), SQL queries are automatically issued and the objects are fully
instantiated.  The OM code consists of over 30 classes and includes:
* Base support for mapping classes and their attributes onto relational tables
  and columns.  Types of attributes supported include:
  - Base data values directly storable in relational columns, or which can be
    represented as text strings.
  - References to other objects.
  - References to sets of other objects.
* An OM-Editor window which is designed for point-and-click browsing and
  modification of object-to-RDBMS mappings.  The RDBMS schema is automatically
  loaded and made available for use. (Write-back of mapping modifications not
  operational yet.)
* Example uses of OM for geopolitical GIS entities (e.g., countries, states,
  counties), hypermedia objects (e.g., images and QuickTime movies, stored as
  files but referenced through the RDBMS).
* A framework which can be extended to other types of data repositories
  including object data bases, hierarchical data bases, flat files, etc.

Data visualization tools.
------------------------------------------------------
One of the primary purposes of SA is to enable applications which provide easy
access to, browsing and visualization of complex data.  To support this,
visualization tools were added on an as-needed basis.  The goal, over time,
was to supplant these tools with an inter-application communication facility
(IAC) so that SA could concentrate on navigating through complex data
structures and feeding other specialized visualization applications.  The
current SA toolkit contains over 20 classes in areas such as:
* Generic browser panes for objects (similar to MCL inspector windows).
* Framework for specialized object browsers which take advantage of the SA GUI
  capabilities.
* A 2-D charting package including bar charts, line charts, linear and log
  scales, etc.
* Image panes for display of:
  - Any Mac PICT file (raster and/or vector)
  - Special tiled images (to support browsing through very large images, e.g.,
    raster scanned maps) with provision for extension to compressed images.
  - Mac QuickTime movies with sound.
  - Planned extension to full TIFF support.

Development tools.
------------------------------------------------------
Several software development tools were added to SA as required.  Comprising
over 20 classes, these tools include:
* A multi-paned class, attribute and method documentation browser (Source code
  browsing was left to the MCL FRED editor, which could be easily linked to
  the documentation browser).
* A multi-paned browser for PI's Model 204 (hierarchical DBMS) data
  dictionary.
* A framework for and initial tests of a graphical interface builder.

Applications built with Sorcerer's Apprentice
===============================================================
GRID Connection
------------------------------------------------------
Built for PI's internal use, GRID (Gas Reference Information Database)
Connection is a map-based data entry tool.  In production use for over two
years, it allows the user to enter, browse and validate natural gas pipeline
information, which is stored in an Oracle data base.  It provides:
* County level maps throughout the USA and province level in Canada.
* Point-and-click entry and editing of pipeline nodes and segments.
* Cross-linking with GRID code numbers for nodes and facilities (e.g., 
  compressor, interconnects, gas plants).
* Linkages to related data (e.g., production statistics on gas wells tied to
  a particular gas plant, displayed as scrolling tables and charts).

Executive Information System (EIS)
------------------------------------------------------
Built for a client oil company, EIS is a multi-media application which allows
managers to browse quickly through both background information and daily
reports on drilling activity.  Background information typically includes color
raster scanned images of maps, structure maps, seismic profiles, lithology
displays, time-depth curves, etc.  As textual and numeric information arrives
from the drilling sites on a daily basis, it is stored in a central Oracle
data base and from there can be viewed immediately in specialized, multi-paned
scrolling forms, as 2-D charts for depth and cost data, and as superimposed
well paths on seismic sections.  A tool palette is provided so that users can
simply add buttons to images which function as hypermedia links (saved in the
Oracle data base).

Risk Analysis System
------------------------------------------------------
Also built for an oil company, the Risk Analysis system provides a map-based
interface to historical drilling statistics and reserves data on over 22,000
fields world-wide, grouped into over 400 basins, and spanning over 75 years,
all contained in an Oracle data base.  Using point-and-click basin selection
and dialog-box parameter setting, the user can select a portion of this data,
which can then be plotted in over a dozen different statistical analysis
charts.

The initial implementation of this application took about 10,000 lines of
code. A rewrite which takes advantage the object mapping facility contains
about one-third as much code.

Seismic and Well Information System (SWIS)
------------------------------------------------------
Designed as a generic, map-based data browsing application for most of the
data types that PI markets, SWIS was the driving force for most of the recent
enhancements to the SA toolkit:
* Object mapping to support easy inclusion of new data types and map layers
  (e.g., natural gas pipelines and facilities were added in an afternoon).
* Variable map detail based on map scale to support views from full-world
  level down to sub-county level.
* Efficient quad-tree indexing, caching and display of spatial entities which
  allowed over 14,000 wells and hundreds of other objects to be displayed
  simultaneously on a Mac IIci with good interactivity.
* Both generic and specialized browsers, as required, for all data types.

The initial version of SWIS handles geopolitical objects (from country down to
county level), seismic lines, wells and natural gas pipelines.  It took
approximately one man-week to program and contains only about 1,000 lines of
code!

IDEA System
------------------------------------------------------
Under the International Drilling Exchange Agreement, PI provides a
clearinghouse for economic and technical data on new international exploration
wells (provided by the participating oil companies).  The IDEA system was
built as an experiment to test the adaptability of the SA toolkit and various
human interface techniques for presenting the IDEA data.  Starting from a
color-coded map of the world, the user can bring up browser windows for each
country showing a small country map, basic petroleum and geopolitical
statistics, and an annotated inventory of IDEA wells.  By clicking on a well
identifier, the user can create a detailed browser for well information that
includes a selection of several panes including:
* Scrolling tables of drill bits used, casing data, mud weights, depth vs.
  time, etc.
* Charts of depth vs. time, etc.
* Forms containing cost data, etc.

The IDEA system was implemented in about two man-weeks and contains about
2,700 lines of code, mostly in the user interfaces.  As a further experiment,
the IDEA system was also successfully combined with SWIS by essentially just
merging the application code for the two systems.