Computing Responses

• Classes of Computed URLs
• Server Interfaces
• Advanced Input Management
• Issues in Response Functions

Writing Common Lisp functions that compute responses to incoming HTTP requests is the main feature of the CL-HTTP. This page documents the supported interfaces that allow you to do this. In conjunction with the HTML Synthesis tools, these facilities allow you to dynamically compute web pages.

Response functions compute a reply to the HTTP methods GET or POST. Before returning HTML to the client, they must arrange for an appropriate status code and appropriate headers to be returned, usually via the macro http:with-successful-response. A response function becomes accessible only after an associated URL is exported with http:export-url. A number of working examples of response functions are available in http:examples;exports.lisp.

Classes of Computed URLs

Four general classes of URLs that compute responses to requests.

HTTP-COMPUTED-URL merely returns some HTML in response to the HTTP GET method. It receives no arguments from the client.The HTML is computed by a response function which takes two arguments: the requested URL and the STREAM to the client.

HTTP-FORM computes the response to the HTTP POST method. It performs any side effects and returns HTML to the client based on submitted values. The response function receives three arguments: the requested URL, the STREAM to the client, and a QUERY-ALIST of values. The macro http:bind-query-values is normally used to decode and bind the contents of the QUERY-ALIST as local variables within the response function.

HTTP-COMPUTED-FORM is just like HTTP-FORM except that it also computes the response to the HTTP GET method by synthesizing the HTML fillout form which the client may later POST back to the server. The form function receives two arguments: the requested URL and the STREAM to the client. In order to synthesis the form for the client, the form function needs to use html:accept-input to emit HTML input types (user queries] within the scope of the macro html:with-fillout-form.

HTTP-SEARCH computes a response to the HTTP GET method based on arguments passed in via the search suffix of the URL following the "?". The response function typically uses arguments from the search suffix accessed by url:search-keys to compute a returned HTML page and perform any side effects. It may also refer to search database with url:search-database. At times, the response function may wish to access the parent of the search URL with url:search-parent, for example to issue a new search anchor. The arguments to a search URL can carry only a relatively small amount of data compared to POSTing to a form because the URL standard limits URLs to 1024 characters.

Server Interfaces

When CL-HTTP receives an HTTP request, it creates an instance of the class SERVER from which all information relevant to the transaction can be accessed. This section documents the program interfaces to the server object. Among other things, these interfaces allow response functions to:

• Return status codes and headers to the client;
• Access HTTP headers sent by the client;

Variables

• HTTP:*SERVER*: A variable that is always bound to the server instance handling the current transaction.

• HTTP:*SERVER-CLASS*: Advanced applications may set this variable to a specialization of standard server class in order to customize the behavior of CL-HTTP.

Functions and Macros

• HTTP:ALLOW-USER-ACCESS-P: Use this function to decide which real resource will be accessed based on the identity of the user accessing a controlled resource. For example, people in the developers group see more internal details than regular users.

• HTTP:BIND-QUERY-VALUES: Use this macro with a the response function to a POST method in order to bind the incoming values of a form to local variables. The local variables will be the same name as the query identifiers used on the input types in the actual form.

• HTTP:CURRENT-USER-AGENT: Provides a convenient interface to identifying the client accessing the resource. This useful,for example, in conjunction with http:user-agent-capability-p for deciding in your response function whether to use the Netscape table extensions.

• HTTP:DEBUG-SERVER: Allows errors during HTTP transactions to enter the debugger by turning off most condition handling. Challenging bugs in response functions succumb most easily when debugging from the runtime environment of an interactive LISP debugger.

• HTTP:GET-HEADER: Provides marginally faster access to the value of header in the current http transaction.

• URL:EXPIRATION-UNIVERSAL-TIME: returns the universal time when a URL should expire. It should be used to compute the expires argument for the macros http:with-successful-response and http:with-conditional-get-response.

• HTTP:REDIRECT-REQUEST: Use this function to issue a redirect to another URL.

• HTTP:SEND-MAIL-FROM: Use this function to send email from a response function. Normally, you will want to specify the from field as http:*server-mail-address*. Lisps other than Symbolics Genera, need to initialize smtp:*network-mail-host* and enumerate any secondary mailer in smtp:*store-and-forward-mail-hosts*. For automatic bug reporting, see http:report-bug.

• HTTP:WITH-SERVER-INTERFACE-ENVIRONMENT: Binds a series of variables useful for writing response functions. It actually creates symbol-macros that macroexpand at compile time into the correct form to obtain the desired information. Dynamic values are shown in the example of the Server Interface. The CL-HTTP Server Interface 1.0 is largely analogous to CGI 1.1 (available via http:with-cgi-environment), except it produces values that are more useful to the Lisp programmer and it renames some variables for better mnemonics. Additionally, one may use the methods on the server objects for more advanced applications, including ones in which the SERVER class itself is specialized (See HTTP:*SERVER-CLASS*).

• NS1.1:WITH-SERVER-PUSH-RESPONSE: This macro allows a response function to keep the open the connection and replace displayed URIs for clients supporting Netscape 1.1 extensions. It use a multipart MIME body that is sent in blocks. The macro takes care of sending the appropriate status code and relevant headers back to the client, and thus, does not require use of html:with-successful-response or variants.

• HTTP:WITH-SUCCESSFUL-RESPONSE: This macro should wrap around all code in a response function. It takes care of sending the appropriate status code and relevant headers back to the client.

• HTTP:WITH-CONDITIONAL-GET-RESPONSE: When the computation can return the same result as the one cached by a client or a proxy server, wrap this macro around all code in a response function instead of http:with-successful-response. The generic function url:expiration-universal-time is used to compute the value for the expires argument.

Advanced Input Management

The server comes with a facility that simplifies the presenting and accepting user data which is called the W3P Presentation System. This facility relieves the programmer from calling http:accept-input directly and (re-)writing validation code for each element of form data received. Instead the programmer invokes generic operations on presentation types that automatically make the correct calls to http:accept-input and automatically process the returned values. The presentation types form a lattice of types describing all the data types either presented to or received from the user. Applications can easily extend the presentation system by defining new types, presenters, and parsers. It is strongly recommended that you learn to use W3P because it will help you reduce code duplication and speed the development of interactive Web applications as it simplifies and streamlines your response functions.

Issues in Response Functions

1. Collisions Across Threads: Because CL-HTTP is multi-threaded, it can respond to multiple client requests simultaneously. This makes it important to ensure that response functions to not collide while accessing a shared resource. Reading the same data is fine. But, writing the same data can cause unpredictable results and trouble. If you need to write shared data from a response function, you must to arrange to synchronize the accesses to shared resources by the different HTTP connections using process tools available in your Common Lisp.

2. Efficiency: Response functions need to be as efficient as possible in order to service your users swiftly and maximize the load that your server can carry, given its endowment in processor speed and physical memory. See the Efficiency Heuristics for ideas on how to keep your code nimble.

3. Conformant HTML: Normally, the macros html:with-html-document, html:with-document-preamble, and html:with-document-body should be used to establish the status of the resource returned to the client and to delineate the components of the resource. Clients, especially SGML aware ones, and servers are increasingly reliant on this information.

4. Evolution: Because the HTML standards are evolving, you may wish to always use the HTML synthesis tools to emit HTML. This way, each time the standards change (every 4-6 months), you merely use the next version of the synthesis tools, rather than revising all your code. Defining higher level abstractions above the CL-HTTP synthesis tools can allow you to slot in new features to implement those abstractions as new and more powerful versions of of HTML become available.

5. Standing Search URLs: Often it is desirable to have both search url and fillout form interfaces to a computed response by the same function.. The fillout form can provide a more user-friendly environment while the search url can be inserted in documents to run a prespecified operation with one click. This idea is applied by /cl-http/find-documentation? and /cl-http/find-documentation.html whose implementation is available in http:examples;documentation.lisp.

6. Carrying State Across HTTP Transactions: Sometimes it is desirable to associate state with uses of resources. Examples of most of the techniques below are available in http:examples;exports.lisp.

   • Hidden Fields: html:accept-input supports the input type hidden, which carries state with the form without displaying it on the client side. Just like normal input types, the macro http:bind-query-values can rebind the hidden value when the form is returned. When using hidden fields, http:write-to-armor-plated-string and http:read-from-armor-plated-string protect Lisp data from damage by HTML and HTTP escaping in transit.

   • Client-Side Cookies: Some browsers support client-side cookies to store state across HTTP transactions and sessions. To set cookie values, the primitive function http:make-set-cookie-header-value creates values for the SET-COOKIE. The top-level macro http:set-cookie-http-headers expands into possibly multiple calls to http:make-set-cookie-header-value, ensures that the client handles cookies, and returns a header plist that can be passed in as the additional headers argument to http:with-successful-response, or relatives. To inspect cookie values, http:with-cookie-values allows provides an interface for accesses cookies transmitted by the client.

   • HTML Meta Info: Outside forms, html:declare-meta-info allows header information to be passed back to a client hidden in the HTML markup.

   • Maintaining the Connection: Server push allows the server to maintain dynamic state because it keeps client connection open. Beyond these approaches, maintenance of state on the server side requires a backing store, for example a transaction-controlled, persistent-object database.

7. Displaying Pages in Specific Browser Window: The content of URLs can be displayed in named windows on a number of clients. TARGET arguments to HTML generation forms provide on way to do this. It is also possible to send the :window-target header with a string naming the window in which to display. This header can be sent as an additional header for http:with-successful-response and related response macros. The function ns2.0:client-target-window-http-headers constructs the header to send only when the client supports the feature.