CL-HTTP Development Projects

Last updated December 12, 2006

Overview

This page describes a set of development projects that you could contribute to the CL-HTTP community. You will work with an international team of CL-HTTP developers, and perhaps learn some interesting things about advanced Common Lisp programming as you win recognition and build your reputation in the Lisp community.

Please be a good citizen and contribute something, no matter how small, to improve these tools for yourself and other users. Not only did somebody else write this server, client, web walker, proxy, full-text search engine, mail hyperarchive, and numerous extensions, they also empowered you with the source code. Why not invest a little effort in return? In the end, you will help yourself by showing that the Lisp culture can produce shared resources and motivating the major developer to continue providing high-quality advanced software to you without charge.

Before contributing code, you may wish to consult the guidelines and efficiency considerations.

If you are working on some code for CL-HTTP, please send a paragraph (or a URL) to www-cl@csail.mit.edu (or bug-cl-http@no-spam.cl-http.org if you would prefer less notice) explaining what you are doing, your time frame, and how to contact you. That way, people will know what you are doing and can contact you if they have ideas, suggestions, or want to help out. It can also help avoid duplicated effort. Have a look at the WWW-CL archives to see what other people are doing.

Project Ideas

CL-HTTP.org Infrastructure

Bug Tracking: Implement a bug tracking system for CL-HTTP.
Source Control based on Subversion: Develop a framework for locating CL-HTTP source code in Subversion that provides access control according to user capabilities and handles all aspects of the CL-HTTP code development cycle.
Wiki for CL-HTTP: Allow registered user to attach comments to CL-HTTP.org Web pages.
User Registry: Develop a framework for registering users and enabling capabilities for them on the CL-HTTP.org site.
Bulletin Board System: Develop a implementation that allows users to chat with each other without using email addresses susceptible to junk mail.
Implement RSS: Distribution anouncements and CL-HTTP news over RSS.
Contributor Access: Develop a framework for CL-HTTP contributors to access their contribution directories and make updates.
Next Generation Web Site: Based on a new dynamic Web site generation framework, design and implement a new CL-HTTP Web site to showcase this technology.

Server Enhancements

Secure Sockets Layer (SSL): Test the SSL implementation on ports supporting it. Write documentation for how to use it. Develop appropriate logging for SSL accesses and errors. Add the connect method based on RFC 2817.
Sessions Implementation: Develop cookie-based session objects in order associate state with users as they traverse a Web site.
Portable Lisp Substrate: Test and extend the port template developed by Karsten Poeck (knpk@nospam.cl-http.org). Rewrite existing ports using the framework so that that CL-HTTP runs across platforms with minimal platform-specific code.
Certificate Management: Develop OpenSSL interfaces and a Lisp-based framework for managing a certificate authority, including issuing certificates and keys for servers and users.
WebDav: Implement WebDav for CL-HTTP. The Jigsaw implementtion may serve a a useful example.
Protocol Conformance
- HTTP 1.1 Digest Authentication: Update the current digest implementation to conform with the latest draft of the digest specification (proposed standard digests).
- New Cookies: Implement the new cookie extension specification for HTTP.
- File Upload: Make sure file upload uses fast buffer level copying on all platforms.
- HTTP 1.1 Details: Make sure all advanced and obscure features of HTTP 1.1 are implemented.
- HTTP 1.1 Proxying: Test HTTP proxying on all platforms and complete any remaining details to maximize efficiency and flexibility.
- Content Negotiation: Implement the HTTP 1.1 content negotiation specification. (see: RFC 2068). Implement deflate/inflate and integrate them with the server and client to enable more efficient data transfer.
- Multiple Languages & Character Sets
- HTTP Features: Implement the HTTP feature extension that allows clients and servers to advertise the set of features that they support in a standardized way.
XML Generation & Parsing: Provide efficient tools for generating and parsing XML. This should include RDF and some RDF schemas as a high priority.
SOAP Binding: Implement the relevant SOAP specifications so that CL-HTTP can act as both a SOAP server and client.
Extended Directory Listings:
Cascading Style Sheets: Implement style sheet generation tools (CSS2) for integration with HTML 4.0 generation facilities.
Performance Tools: Use the log file replaying tools to exercise a server and develop new code for:
- Metering: Develop higher level metering facilities atop the current facilities that capture computation expended on a per URL basis.
- HTML Display of Performance Statistics: Write HTML displays to facilitate use of metering information to enhance URL performance.
- Tracing processes: Show Execution Process.
Efficiency Enhancements
- Resource URLs: Uninterned URL are created with increasing frequency by search URLs, proxying, and other applications, these should be resourced to minimize consing.
- In-Memory Data Caching Facility:
  - Develop caching policies for the pathname datacaching facility.
  - Develop directory-oriented caching policy that is invoked via a keyword on directory export.
  - Where applicable, provide areas to hold cache data structures and GC policies for optimal performance.
  - Integrate the caching proxy with the datacache facility.
- Server Metering: Use TCP metering tools to test server and client latency, throughput and efficiency. Then, implement performance enhancements while preserving abstraction.
- Double-Buffered Asynchronous Transfer: Develop platform-specific facilities, where applicable, for multi-buffered asynchronous TCP transfers. This improves performance by queuing new buffers TCP to ship, and thus, freeing Lisp to continue computing rather than blocking waiting for TCP to complete delivery of the current buffer.
Compressed Page Transfer Encoding
- Provide GZIP compression of static files in a manner analogous to CRLF encoding.
- Implement Deflate Compression and uncompression
- Integrate with Server & Client
- Support
  - Compressed Static Pages
  - On-the-fly Compression
  - On-the-fly Uncompression
Implementation Enhancements
- Declarative Header Parsing/Generation
- Declarative URL Parsing/Generation
- Escape Characters In Exported Pathnames
- Factor Code Into Component Modules
Portable Dynamic Form Processing: Use the W3P presentation system to replace CLIM in the Dynamic Forms module. Make Dynamic Forms run portably. The Dynamic Form processing facility is a high-level object-oriented form processing facility that predates HTML form and allows very high level coding of forms with maximal code reuse. It has been used in COMLINK for 5 years and drove all the forms used over SMTP and HTTP on the White House Publications system 1992-2000.
Ajax Support: Implement the Asynchronous JavaScript and XML (Ajax) (based on HTML 4.0, Document Object Model, XML HTTP request objects) dynamically display and interact with information displayed by CL-HTTP. See Ajax frameworks from Google, Dojo toolkit.
Proxy Service: Add intelligent caching to proxy implementation (see the CERN server). This is pretty important for many companies that operate through firewalls. It is also very handy for downloading URLs to a laptop for disconnected use
Differentiate Server Log Event Classes: Extend CL-HTTP to differentiate log event class according to errors, URL export/unexport, server start/stop.
Log Analysis:

Static & Dynamic Log Analysis: Develop tools for analyzing CL-HTTP logs and creating Web-accessible reports.
Efficient Storage Format: Develop a storage format that requires no parsing or analysis when loading log files into memory. The format should be modular so that only logfiles in a time interval need to be examined.
KDTREE Implementation: Revise the Common Lisp KDTree implementation for high efficiency needed in logging applications.

Site Management Tools: Develop a change management facility and incorporate it into tools for managing the content of Web site. Contact Rainer Joswig for existing code and further suggestions.
Personalized Content Generation: Analyze session activity and provide flexible query mechanisms to personalize content.
Test Suite Framework: Develop a test infstructure that allows code to be tested at the granularity of functions, methods, libraries.
Custom Logging: Develop a variety of custom log formats that yield useful information about HTTP operations. Useful ideas include logging:
- Form submissions and responses
- Metering information for URLs
- Proxy operation
- Precomputed server analyses
Portable Remote CLOS Interface: Develop an object interface using Inter-Language Unification (ILU) from Xerox's Palo Alto Research Center. Develop interfaces to readily available persistent object stores and CORBA.
CLIM Server Window: Overhaul the existing CLIM window so that:
- It runs on multiple platforms
- It offers a full set of relevant commands
- It edits user access control information
- It offers a variety of useful displays, including:
  - Realtime log display
  - Performance statistics
  - Metering of URL access times
Digital Signatures:
- Develop a framework for digitally signing static and dynamic resources.
- Ensure portability DSS signature code (available from John Mallery).
- Devise key management system.
- Document secure operation of digital signature facility.
Secure HTTP: Implement HTTP tunneling using after-download cryptography. The first step involves implementing a digest authenticated tunnel that allows the client to know with complete certainty that the server received the HTTP message in tact and processed the request successfully.
Security Hardening: Robustify CL-HTTP against malicious users, viruses. Extend IP black lists to referers and user agents. Detect and filter request patterns generated by viruses or malicious users. Ignore buffer overflow attacks via URLs.
Distribution Source Compare: Compare all directories and files across CL-HTTP versions and find differences in file and unused files.
Cruise the source code and look for bugs, incomplete features, better numerical declarations, inefficiencies and write patches to fix them.

Server Documentation Projects

Wiki-based Documentation: Develop a Wiki-based documentation and tutorial system for CL-HTTP.
Source Code Coloring, Cross-reference & Annotation: Develop a framework for source code coloring, cross-reference & annotation that can be applied to the CL-HTTP sources and Web pages.
Source Overview & Source Metric Tools: Develop various tools to browse CL-HTTP code (by class, package, system, files, etc.). Compute and display various source metrics (complexity, package access, module layering, style checker, overlong methods and functions, test coverage, documentation completeness, etc.
Extend the online manual or tutorial by writing one or more sections that explains how to do something. The manual should be written in HTML and should explain things with live working examples. We need to target these audiences:
- Naive users who just want to serve web pages with minimal fuss.
- Novices who want to Learn Lisp
- Lisp-capable people who want to write their own response functions and hook up their own systems but do not want to learn more about CL-HTTP than it takes to solve their problem.
- Lisp Programmers who want to Learn the Server
  - Extend and annotate the example file examples.lisp. Improve the comments to make this example more of a live tutorial. Add additional examples illustrating any previously undocumented features.
  - Improve documentation strings and comments in source code: Give priorities to those areas people are most likely to use or find confusing.
- Wizards who want to understand the details of CL-HTTP and the Web so they can experiment with new extensions to CL-HTTP and the Web. There may be a category of wizards who are not lisp hackers and we should have a way to reach them.
Overview Server & Architecture
Interfacing Existing Applications
- CLIM To Web
- GGIs To LISP
Create icons and gifs for use spicing up CL-HTTP generated pages and advertising the technology.
Dynamic FAQ: Implement a facility for capturing, storing, searching, and displaying FAQs. The facility should provide ultra-targeted answers to user queries based on an inventory of questions and answers,
Graphical Class Hierarchy Browser: Implement a Web-based interface for browsing CLOS class hierarchy and viewing associated documentation strings. Use this to document CL-HTTP class structure. The interface should utilize Lisp's introspection capabilities, using basic (widely supported) Meta-Object Protocol calls to generate and update the tree structure. Pick a few standard starting points, such as the url class. Tree layout algorithms could be combined with the GIF generation code or Java to provide a nice graphical front-end.

Client Projects

Client HTTP 1.1 Conformance: Identify all the client MUSTs in the http 1.1 specification. Prioritize them. Implement them in the order of priority.
Make the client RPC facility faster and more robust

Web Walker Projects

Multithreaded: Use the task queue facility to make the Web walker operate asynchronously with multiple threads.
Persistence: Develop a persistent storage model for use with the Web walker.
CLIM Interface: Develop a CLIM interface to operate the Web Walker. It should include facilities for:
- Defining Activities
- Defining Constraints
- Monitoring Operation
- Debugging Errors
Web Site Maintenance: Develop an Web walker application for finding all the dead urls on a web site.
Generate VRML views of Sites
Develop Facilities for Periodic Site Walking: This involves developing persistent actions that awake a certain times to execute a Web Walking operation. The operations and any predicates need to be store persistent as well. See: server realms (file or OODB storage) and COMLINK persistent actions.
URL Reminders: Improve the URL reminder application from Rainer Joswig.
XML Parsing: Integrate an XML parser.
Index Web Structures: Use LambdaVista to index and retrieve Web structures.

HyperArchive

Archive Migration: Develop management policies and associated implementations that migrate messages to archives at specified intervals or archive sizes.
MIME Support: Add MIME support so that MIME multipart messages can be viewed using appropriate helper applications.

Database Projects

Flatfile Database: Develop a database building from the specialized code used by the Mail HyperArchive.
Generic API: Provide a generic database API so that platforms can specialize into various native database backends. Generalize some of the SQL interfaces to Lisp so they can work across multiple platforms and provide a clean object layer with transaction and extensible databased Lisp types.

LambdaVista

Documentation: Write documentation strings for major functions and datastructures.
Document Universes: Abstract the document-universe data structure into CLOS mixins.
Web Walker Support
Documentation Search: Full text search in CL documentation.
Substring Search via Thesaurus
Easy Document Indexing With Fragment Support
WAlS Interface
WordNet Interface: Implement an interface to WordNet designed to enable higher level applications atop a tookit of WordNet operations.
- Utilize to improve indexing & retrieval performance of LambdaVista

Platform-Specific Projects

ACL Platform

ACL JKF: Bring port up to HTTP 1.1 conformance.
Test CL-HTTP on 64 bit CL: Bring up CL-HTTP on a 64bit platform running 64bit ACL and verify that it works correctly.

CMUCL Platform

Production Quality: Bring the x86 port up to HTTP 1.1 conformance and full production quality.

LispWorks Platform

Multiple Network Addresses: Extend port management infrastructure to allow HTTP service on multiple network addresses.
FTP Methods: Integrate FTP functionality to enable the port to act as an HTTP proxy gateway for FTP. Implement the following methods:
- HTTP::FTP-COPY-FILE-TO-HTTP-STREAM: Used by proxy
- HTTP::FTP-DIRECTORY-INFO: used by proxy and client
- HTTP::FTP-COPY-FILE: Used by client
Control Panel: Hack up the CAPI control panel by Rainer Joswig to perform more functions.
Port Authentication to Postgres:Port database authentication facility by John Mallery from MySQL to PostGres.
Start-up for Production Web Site: Develop an initialization script that launches CL-HTTP on port 80 automatically at boot time on Mac OS X server.
Implement HTTPS: Use LispWorks SSL streams to implement HTTPS for the server, client, and proxy.
Stress Test: Stress test the LispWorks port to find any weaknesses and correct for production quality.
Precompiled Distribution: Develop a precompiled version of CL-HTTP that will run in LispWorks Personal Edition. People should be able to dowload a small package and run the demonstration Web site.

MCL Platform

OpenTransport Polish: Tune the MCL OpenTransport interface. correctly. See: ``Technote 1145 Living in a Dynamic TCP/IP Environment'' which describes how to write OpenTransport code that correctly handles multiple IP addresses, dial-up links, sleep and wakeup on PowerBooks, modem disconnection, and user reconfiguration.

OpenMCL Platform

Initial Port: Develop a CL-HTTP port for OPenMCL starting from the MCL port. Requires writing a new version of the HTTP stream code.

Symbolics Platform

Mail HyperArchive: Recognize epsilon encodings and render them in corresponding HTML.
Concordia Interface: Develop a Web interface that provides access to the Symbolics document examiner.

Extensions, Add-Ons & Applications

Higher-Level HTML Synthesis
- Cliche & Abstractions
  - Lisp To HTML Translator
  - Graphics
  - Command Processor
- Automatic Generation of Navigational Links
- Graphics For Space Filling (e.g., Backgrounds)
Java Synthesis
- Java Presentations For Client
- Java Primitives Composable From LISP
Indexing & Search For Web Regions
- Gatherers On The Server
- Brokers For Delivery To Clients
- Specification For Collecting & Distributing Site Indices. See work by Harvest group.
Server Collaboration Site
- Register Servers
- Register People & Interests & Abilities
- Share Extensions
- Advertise Extensions
- Peer Review Extensions
- Automatic Patch Distribution
Code Sharing and Distribution

Build Web Collaboration Systems For Sharing Code: Provide a forms interface that allows people to submit code or extensions to CL-HTTP and retrieve submissions of others. Incorporate a peer review system that allows people to rate software along various dimensions.
Java Library: Use the facility to:
- Capture Java applets from users.
- Document applet APIs.
- Locate applets based on search criteria.
- Invoke applets dynamically.

Mobile Lisp Code
- Secure Lisp Listener
- Safe Common Lisp Package
- No Side Effects
- Bounded Computation Time & Space
- Fasl Streams To Clients
Modern Common Lisp Defsystem: This will provide superior system management that is not wedged in the past, i.e., based on pathnames. It should accept URNs and provide a patching facility and version control system at least as good as the Lisp Machine.
- Start from CLOS-based portable defsystem
- URN fragments provide relinearization capability.
- Tacking of code dependency enables configurable loading
- Tree shaking provides small application images.
Synchronized Multimedia: Implement the current standards for synchronized multimedia Web display.
- Emit Streamed Audio
- Emit Streamed Video: Implement facilities for emitting streamed video using standard formats.
Integrate CLIM Graphics: Port the SRI CLIM generation tools to CL-HTTP and portable lisp.
MacroMedia Shockwave Generation: Develop a facility for generating Shockwave on the fly.
SMTP Mailer: Extend the current portable SMTP transmission code into a full-fledged mailer. Abstract the storage componet so that a database can serve as the backend. Use the mailer as an SMTP gateway for the server.
NNTP Gateway: Extend the NNTP code (available from John Mallery) to gateway between NTTP and the server. Ensure portability of implementation.