Mario Latendresse

I am a member of the Science and Technology Advancement Team at FNMOC (Monterey, California), U.S. Navy.

Contents

• RegReg: a parser generator based on tagged regular expressions
• The 4 days Scheme course
  Metcast
  • Installation instructions of Metcast fastCGI
  • Introduction
  • The Metcast client request language, MBL
  • Implementation
  • Metcast fastCGI
  • Metcast fastCGI benchmarks
  • Evaluation of Scheme systems for Metcast

The package is available as a gzip tar file (version 1.1b): RegReg.tar.gz

This tool is written in the Scheme programming language. It generates parsers based on tagged regular expressions. For example, it can be used to construct WMO bulletin decoders.

RegReg HTML documentation

RegReg PDF documentation

A presentation made in October 29th, 2002, at the International Lisp Conference, San Francisco, covering Metcast fastCGI and RegReg applied to meteorological decoders, is available here: in PostScript, in PDF. A paper, titled RegReg: a Lightweight Generator of Robust Parsers for Irregular Languages to appear in the proceedings of WCRE 2003, Novembre 2003, shows an application of RegReg in reverse engineering.

The slides, in PDF, of a "demo" presentation, for October 21th, 2003, is available here.

This package allow Scheme code to be run as a fastCGI application under the mod_fcgi module available for Apache. It is composed of a proxy written in C, basic input and output functions written in Scheme to communicate with that proxy, and a general code structure for the Scheme application. The Scheme application can be run localy or on remote computers.

You need the fastCGI development kit to compile the proxy. It can be obtained from the fastCGI web site: fastCGI development Kit (external link).

Here is a tar of the code files: fastCGI_Scheme.tar

The files contain important explanations. They should be read before attempting to use this code.

If you have any questions regarding the use of this package, please contact me at mario dot latendresse dot ca at metnet dot navy dot mil. (As usual, replace ' dot ' for '.' and ' at ' for '@')

Here are the transparencies that were used during the four days course given at FNMOC in February 2002.

Transparencies in PostScript

Transparencies in PDF

I recommend printing the R5RS document for more details on Scheme; although it is not intended to be pedagogical. Dorai Sitaram wrote a nice pedagogical introduction to Scheme.

Dorai Sitaram Scheme introduction is available as PDF or HTML.

Local copies of these documents:

R5RS in PostScript

R5RS in PDF

Dorai Sitaram, Teach Yourself Scheme in Fixnum Days in PDF

Links to further information

Testing the Metcast server

The Scheme language

The Metcast Decoders

The Metcast server was originaly designed and written by Oleg Kiselyov, at FNMOC.

This documentation presents the modifications made on the code of the Metcast server. Among other things, these modifications increase the performance of the Metcast server. Some technical details of the implementation is also described. For all technical details not related to the fastCGI, the authoritative documentation is at Spawar/JMV-TNG.

The Metcast server runs under a Scheme interpreter, that is, most of the Metcast code is written using the Scheme language. The Scheme interpreter runs as a CGI application under Apache and until recently exclusively on a modified version of Gambit Scheme. (Some upcoming modifications will make it portable to several Scheme interpreters, hopefully under PLT Scheme and Bigloo, probably on many others.)

There is also some documentation on the MBL language, the query language used by the Metcast clients. It explains how the language and the implementation can be extended to handle new products and features. The ease with which this can be done demonstrates the flexibility of the Scheme language.

A client request, to Metcast, is formulated using a specific language, the MBL language. Its syntax is relatively simple: it uses a well formed parenthesis expression, known as a S-expr. This a common syntax of Lisp and Scheme expressions. Here is an example:

(example1
  (bounding-box 50 -77 42 -20)
  (st_constraint (st_country_code "CN")) 
  (products (METAR)))

This request, fomulated by the client, is identified by the token `example1'. It contains three sub-expressions. The first one specifies the region of the request, the second one constrains the request to stations from Canada, and the last one identifies the product, that is Metar.

General syntax of MBL

In general, a MBL request has the form:

(id parameter1 ... parametern (products product1 ... productn))

(product-name  parameter1 ... parametern) 

The list of parameters can be empty, as shown above in example1.

As can be seen from the syntax, a request may contain several products. The parameters specified at the top level of the request apply, unless otherwise instructed by a local parameter, to each product. These are global parameters. Each product may have its own local parameters. They only apply to the product which specifies them. Some products have mandatory parameters, a request without them cannot be answered. The list of parameters, product names and their mandatory parameters are presented below.

Note: unknown parameter names will be silently ignored, without causing an error. But an unknown product name will generate an error message.

Product identification

The product names are divided in categories. Here are the list of product names and the categories they belong to.

• Synoptic: MSL, METAR, SYNSEA, UAR, BTSC, TAF, SIGMET
• Imagery: imagery
• Gridded: grib, pressure.
• WMO messages: METMSG
• Miscellaneous: GlobalWarnings, Channel

Parameters

A parameter is a S-expr of the form

  (parameter-name arg1 ... argn)

The argument types and values depend on the parameter.

Bounding-box parameter

(bounding-box lon-W lat-N lon-E lat-S) 

This is the mostly used parameter has it is required by most products. The four parameters can be strings, numbers, symbols, or a mix of these.

Examples

  (bounding-box -75.0 40 -70 35.0)
  (bounding-box 75W 40N 70W 35N) 
  (bounding-box "75W" "40N" 70W 35.0) 

Longitude strings have the form <number><d> where <d> is the letters W, w, E, or e. The letters w and W specify a west longitude, the letters e and E specify an east longitude.

Latitude strings have the form <number><d> where <d> is one of the letters N, n, S, or s. The letters n and N specify a north latitude, the letters s and S specify a south latitude.

West longitudes are negative, up to -180, inclusive. East longitudes are positive, up to 180, inclusive. South latitudes are negative, up to -90, inclusive. North latitudes are positive, up to 90, inclusive.

Symbols, like 70W or 35N, are translated into strings and interpreted as above.

Constraints parameter

A st_constraint parameter is a S-expr of the following form

(st_constraint constaint1 ... constraintn)

• (st_country_code string)
• (st_name string)
• (call_id string₁... string_n)
• (block_id string₁... string_n)

Parsing MBL

The dynamic environment

A module registers, into the dynamic environment, the services it can provide. This is done at the time the module is loaded in the Scheme interpreter. It is done by associating a symbol with a closure. When parsing a sub-expression of a MBL request, which should be a list, the symbol at the head is used to retrieve the associated service.

Database communication

The Metcast server communicates with the Informix database server through two pipes: an output pipe to send the SQL request to the Informix database and an input pipe to read the answer from it. The input pipe is specified by the global Scheme variable DB:PIPE-FROM-SQL and the output pipe by DB:output-port. The input pipe is a special node generated by the Unix mknod utility. The output pipe actually contains a command to execute the Informix program dbaccess. It is something similar to

"| dbaccess -e DB:NAME - 1>&2"

The file db-util.scm defines the function open-data-base which opens those two pipes. It is called when the file is loaded in the Scheme interpreter. Thereafter, these remain opened, until the server dies.

Two functions can be used to actually send the SQL requests and process the answers. These are DB:for-each and DB:assoc-val.

If no results are expected from the database server, the function Db:imperative-stmt should be used.

The function DB:make-sql-stmt-buffer can be used to build up and execute a SQL request.

It is assumed by all these functions that the database server is running.

Benchmark results, presented in the subsections below, quantify the fastCGI performance.

Some fastCGI Metcast benchmarks

FastCgiConfig -restart -restart-delay 5 -idle-timeout 120
              -maxProcesses 30 -maxClassProcesses 30 -initial-env none 

<Location /cgi-bin/start_server_fcgi_proxy>
 SetHandler fastcgi-script
 #perhaps other authorization directives
</Location>

fastCGI applications under Apache

One of the technique is to use dynamic loadable Apache modules. Apache must be built with the mod_so module. To do so configure an Apache make file by

./configure --enable-module=so

The mod_fastcgi.so module, which controls fastCGI applications, should be placed in the libexec directory (usually at /usr/local/apache/libexec). It is dynamically loaded when Apache needs to run fastCGI applications.