As a programmer who spend a good portion of his times developing web applications I'm always on the lookout for new frameworks or approaches. This is especially true for web application frameworks, as there is plenty of room for improvement in the web world. In production settings I have rolled my and used struts. In a demo environment I've played with PLT's web server and similar packages.
It was with that background that I saw the announcement for SISCweb. As the name suggests, it's a web application framework built on top of the SISC Scheme implementation. I downloaded the samples and played with them, and hit me - this could be really special. Why? Using SISCweb you get access to any Java library, hosting by any standard J2EE servlet container, access to a full Scheme implementation, and a powerful approach to developing web applications. This seemed like a very impressive combination, but I wanted to more than think that - I wanted to know that. So I decided to write a significant web application using SISCweb as a proof of concept. I figured this would give me an opportunity to see if my hunch about SISCweb and its advantages, was correct.
Choosing an application as an example turned out to be pretty easy, mostly because it had already had been done. That's exactly what SUN did when they published the Java Pet Store. They wanted to demonstrate Java's viability as a web applicatio platform, and I wanted to do the same thing for Scheme. Others have followed this tradition, leading to a .net, ColdFusion, Flash versions of the Pet Store.
It turned out that having a running version of the Pet Store was a significant help during the development of the application. I didn't need to figure out the requirements of my test application, or guess at how a certain function worked - I could just play with a pre-existing application.
The Pet Store is also a nice application to use as an example because of it's realative complexity. It requires the programmer to do things such as manage server side state and execute transactions. These details regularly go into a real life web application, while samples can easily forgo them.
So, with a fresh emacs buffer and a browser window pointing to a version of the Pet Store, I began.
There are two overriding principals that I hoped to meet with development of a Scheme Pet Store. The first is that I wanted to develop a web application that looked as little like a web application as possible, and second, I wanted to support the concept of seperation of concerns.
Developing a web application in a new framework is always a little tricky. I have found that it is easiest to spend some time focusing on what the shape of the application will be more so than any of the details. For example, in struts, the shape of the application consists of Action and Form objects, JSP views and a configuration file to tie it all together. It was hardly this clear to me when I started developing the pet store in SISCweb.
My first plan was to develop SISCweb as a series of small web programs, connected together by URIs. For example, the section of the site that allows you to view pets by category would be a simple program that would be invoked by a URI that had as a single argument the category name. In this world view, a web application is a collection of small web scripts and the URIs that hook them together. I discussed this with the SISC user community and in the end they steered me to a different approach.
The approach suggested, and the one I finally settled on was ridiculously simple: a web application is just a collection of functions. These functions can invoke each other using the standard function call mechanism. The fact that these function calls are spread across multiple HTML pages is pretty much irrelevant. After focusing on this approach, I realized that I could develop a web application that had a relatively small amount of web'ness associated with it. For example, the only URI the Scheme Pet Store concerns itself with is the top level one which kicks off the main function. After this initial function is invoked, the control proceeds as normal.
This behavior of calling a main function is very similar to how any C or
Java program would be called. As a programmer you define a special
function typically named main which is magically invoked by the
operating system. The same is true for the Scheme Pet Store, there is a
top level function named store which is invoked as a servlet,
however, after this point, all functions are called in a normal fashion.
The shape of the Scheme Pet Store goes beyond simply specifying that functions invoke each through a normal process. A second component of the chosen approach is the type of values that a function returns. In general, functios return SXML. This SXML is convereted through a seperate process into HTML and rendered to the user. However, the SXML value from these functions may just as easily be converted to another format, or used in other compuations.
For example, the view shopping cart function may return a value such as described below.
(cart (item (price 3.45) (description "Foo") (quantity 3)) (item (price 9.99) (description "Bar") (quantity 1)))
I have frequently referred to the above approach as a Single Web Application methodology. This is because an entire website, such as the pet store, is thought of as a single application. This application is no different than MS-Word or Emacs, in that it is a complex application providing a variety of services.
The main benefit that of this single application approach is that many issues relating to the complexity web applications simply go away. For example, you don't need to worry about storing state in a user's session, or if a value stored in a request paremeter will be lost during an HTTP 302 redirect, simply because that's not part of the vocabulary of the system. A value is available if it is passed in as a function argument or if it is visible in lexical scope.
A second key goal I had for the Pet Store application was demonstrating Scheme's ability to build well abstracted systems. From my experience, this abstraction is most benenficial when isolating parts of the system from each other. I wanted to show that a DBA could develop database queries, which an application programmer could use and the graphic designer of the system should be able to give the application a facelift without bothering the application programmers or DBA.
I accomplished this goal by viewing key parts of the system as being split into proceducer concerns and consumer concerns. For example, a producer concern for the query system may be the actual SQL statement used, while a consumer concern may simply be the ability to execute that query and review the results. I provided this seperation of concerns for all of the following areas, including: database queries, layout, forms and navigation. The seperation of concerns always revolves around some resource, be it a specific query, form, layout or other entity.
The sections below describe some of the specific features of the ways that I seperated concerns. In general, the goal is always to try to give the producer flexibility and the consumer simplicity.
Take the query system as an example. The producer can provide arbitrarily complex queries that take in any number of arguments and return any number of rows. The consumer however, needs to only know the name of the query, what arguments it expects and what the return values will be. The consumer can remain ignorant in the details of the query and the producer can remain obliviously to where the query is used.
While developing the Scheme Pet Store I played the role of the application programmer, DBA and designer. However, I still found the separation of concerns mechanism to be quite useful. I could put a simple query in place, make use of it in the application and then go back and improve it without disturbing the application. SISCweb provides a mechanism whereby Scheme definitions can be updated on the fly, which meant that I could play all of the above roles without ever needing to restart the server.
XXX: I don't think I talk enough about SISCweb and I'm using it. Where should that go?
In this section I describe some specific modules used in the Scheme Pet Store. I hope this section will serve to demonstrate some of the key decisions I made, as well as give some valuable clues to those who are interested in reading the Scheme Pet Store source code.
One common theme that will quickly become clear is the role pre-exisiting 3rd party libraries played in the development of the Scheme Pet Store. Quite a bit of the difficult work of developing a web application had already been done. For example, Java provides JDBC as a method for accessing a database, and SXML provides a standard for representing XML in Scheme. In many cases these technologies were simply wrapped up in a module to expose them in a such a way that the goals outlined above could be met. This has the additional benefit of making a large amount of the the code in the Scheme Pet Store non-controversial and well tested.
I have noted how SISCweb gives programmers the opportunity to develop web applications without having to worry about many of the details of the web. While that is true, one issue that must always be kept in mind is that in the end, a browser needs to render the output of our functions. In this section I describe how I developed a seperations of concerns framework for handling the layout of the Scheme Pet Store.
Scheme's ability to support XML is remarkable. In fact, the ability to easily generate and manipulate XML and HTML is just as impressive as SISCweb's magic handling of function calls. There has been quite a bit of work done in the Scheme community to map XML to Scheme and vice versa. SISCweb made use of the SXML representation which is one of the more common approaches out there. Consider how the following is represented in both XML and Scheme.
<cart count="1"> <item> <name>Cat</name> <price>3.94</price> </quantity>3</quantity> </item> </cart>
(cart (@ (count "1")) (item (name "Cat") (price "3.94") (quantity "3")))
SISCweb extends the standard SXML format by allowing numbers to appear in the structure, as well as strings. In the Scheme Pet Store this approach has been extended even further, allowing arbitrary Scheme values to appear in the XML tree. Consider how a form might be represented in Scheme versus XML.
<form action="something.do"> <field name="q" size="20"/> <submit/> </form>
`(form (@ (action ,(lambda () ...))) (field (@ (name "q") (size 20))) (submit))
In the above case, rather than naming an action assocaited with the form one simply provides the implementation of the action itself.
In earl iterations of the Scheme Pet Store each function was responsible for outputting well formed HTML. Initially this approach held some promise because of the ease in which Scheme allows you to generate HTML. Consider the example below:
(page "Search for a Pet" `(ul ,(map (lambda (found) `(li ,found)) (search-for-pets query))))
XXX: Talk about consumers and producers here?
In the above case the function page takes in two arguments, a
title and the body of the page. The second argument to page is well
formed HTML, which makes use of quasiquote. This expands to the
following HTML:
<html> <head> <title>Search for a Pet</title> </head> <body> <ul> <li>Saltwater fish</li> <li>Freshwater fish</li> </ul> </body> </html>
The main issue with this approach is that it doesn't allow for the seperation of concerns that I was looking to attain. The application developer would be responsible for choosing the HTML tags used to render the page, rather than a designer. In an ideal world, the application programmer could return a value from the function which contained the data to be rendered, but not the actual rendering instructions. Using Scheme I was able to accomplish this.
The package I developed is named domain-ml, which stands for domain
markup language. The approach I had in mind was that functions would
return XML that made sense to themeselves, they would be domain specific
results. For example, a domain specific representation of the above may
be:
`(query-results ,(map (lambda (found) `(row ,found)) (search-for-pets query)))
When a domain-ml fragment of XML needs to be rendered it looks for a transformation function to convert it to HTML. This transformation function is seperate from the domain-ml use. Thus, our seperation of concerns is accomplished. The application programmer can think in terms of a domain specific markup language. The designer simply needs to construct a function to convert the domain-ml format into HTML.
The actual transformation of domain-ml into HTML is accomplished by
using the sxml-match package. This package allows the programmer
to trivially capture patterns and re-write SXML into a new flavor of
SXML. It performs the same function as XSLT but is more brief. The
sxml-match makes use of the same ellipses construct that
define-syntax does. Using the ellipses notation one can easily
decompose and compose XML as well as match complex patterns.
Consider the following conversion function, which converts our
query-results into valid HTML.
(define (query-results->html doc) (sxml-match doc ((query-results (row ,value) ...) (page "Query Results" `(ul (li ,value) ...)))))
In the above case query-results is converted into a ul tag with
nested li tags.
While I found the use of sxml-match to be an example of Scheme almost
reading my mind, there may be other programmers who do not like this
notation. If that's the case, all one needs to do is to redefine the
implementation of the translation functions, and activity that can be
done without affecting the functionality of the system.
It is recommended that one become familiar with both SXML and
sxml-match before attempting to understand the Scheme Pet Store.
One of the most common requirements of a web application is to communicate with a relational database. Because the Scheme Pet Store is written in SISC, it can easily take advantage of the JDBC API that is standard with Java. For connection pooling, I made use of the Apache Commons DBCP package. For an actual database implementation I chose to use PostgreSQL, though I could have chosen any database vendor which provides a JDBC interface.
While JDBC accomplishes the difficult work of communicating with a
database implementation, it's interface from Scheme is not particularly
friendly. SISCweb helps to correct this by putting a Scheme friendly
interface on JDBC by provding an sql module. The SISCweb API is a
collection of procedures and macros that make using JDBC a much simpler
task.
Much like my orginal approach to the Pet Store involved having functions work directly with HTML, I also had functions embed SQL directly. It did not take long for me to realize how poor a choice this was, as it was a clear violation of the seperations of concerns principal I was striving for.
The approach I settled on is very similar to how I handled the issues of layouts described above. At the highest level, queries are registered by name with the system. The query author is the producer in this situation, and as such is responsible for all the details of the query. This includes the actual SQL statement to use, the specification of the returned columns and a conversion function to turn user supplied arguments into SQL values. The consumer in this scenario, needs to know the same information that would be required if he wanted to invoke a function created by the producer. This includes the name of the query, the arguments, and the expected return values.
From the consumer's view, the query framework is made up of a series of macros. I chose to use macros, instead of functions, because they allowed me to capture patterns that functions could not. This was very much inspired from SISCweb's query framework which also makes use of macros. Consider the following query:
(map-query ctx find-products-by-category-name (category) (product-id name) (display (format "I found: ~a (~a)" name product-id)) `(row ,product-id ,name))
The above code executes the query named
find-products-by-category-name. The first list after the name is the
arguments to invoke the query with, in this case we have a single
argument category. The next list is a list of variables which
will be bound to result set columns. In this case, the result set has
two variables available, product-id and name. The rest of
the statements are simply executed for every row returned in the query,
and make the variables product-id and name available for
use.
Not only does the above query hide the details of the query to be run,
but it also hides details such as the names of the columns that are
returned, and how to convert category into a argument to be handed
to the query. The map-query also takes care of destructuring each
row's values from a result set into scheme variables, so the programmer
doesn't have to. This is not only convenient, but saves the programmer
from a whole class of errors.
Along with the map-query function, a let-query function is
provided. This behaves like a let statement, but pulls a single row
the database to initialize variables.
HTML Forms were another aspect of the Scheme Pet Store that I felt cried out for a particular framework. I wanted to make HTML forms a resource that could be used by name, just like a query or a layout. The producer of a form would specify the exact fields, the validation and any conversion functions. The consumer of a form would simply look up the form by name, and provide functions to be invoked when the form succeeds or fails.
Using this strategy, the many details of exactly what fields to collect, how they should be rendered, and how they should be validated, can all be handled behind the scenes. Consider how one might use a search form.
(use-form ctx 'search (lambda (ctx query) (run-search query)) (lambda (ctx query) (show-oops-page (format "The query: ~a is invalid" query))) "")
The look and feel as well as what type of validation will take place are
all details the consumer need not bother with. However, the consumer is
required to provide a success and failure function. These functions
will be invoked with the form values as their arguments, effectively
destructuring the form request object into local variables . In other
words, the form above had a single input box, which was being stored in
the query parameter. If the form had two arguments, then the
success and failure functions would take in two arguments in addition to
the context variable.
The above strategy makes use of Continuation Passing Style (CPS). While CPS programming can often be very difficult to follow, I found the above strategy to be quite clean. When compared to a framework such as struts I find the above strategy to be easier to follow. The basic question, "what happens when I hit this submit button" is answered just by looking at the place the form is used, and not by looking up some name in a lengthy XML configuration file. This provides a degree of locality to forms, which makes them easier to reason about.
The CPS approach of handing in both success and failure functions makes it so that you simply can't cheat and not handle the case of form failure. Below is another example usage of a form.
(define (ask-for-birthday ctx month year) (use-form ctx 'ask-for-birthday (lambda (ctx month year) (store-data month year) (say-thanks)) (lambda (ctx month year) (ask-for-birthday ctx month year)) month year))
In this scenario, if we succeed in asking for a birthday, we run a
function that stores and continues the transaction. However, if we do
not have a valid birth date, then we simply invoke the
ask-for-birthday function again. In this case we assume that the
validation code has added error messages explaining why the birthday is
invalid.
The form code and use of CPS helped to further goal of not having a web centric web application. Web forms, just like any other construct, are wired together with functions and SISCweb insures that the correct function is invoked on your behalf.
The producer side of the form framework is fairly straightforward. The author of the form can make use of the layout framework above, so that forms are defined in a custom markup language and converted using translators into HTML.
At the lowest level, the validation framework makes use of CPS much like the consumer side of the framework does. In general, a validator is handed a success and failure function. If the validation passes, success should be called, if the form fails to validate the failure function should be invoked.
I implemented a set of high level validators so that the details of continuations would be hidden. An unexpected benefit of making use of continuations, is that I was able to change the behavior of the validators without changing their implementation.
For example, the typical implementation of the failure function is to
stop validation and call the form's failure function. I setup a
validator named try-all that invokes a series of validators, but
sends in a different implementation of the failure function. In
try-all's case, when the failure-function is invoked, it continues
to invoke the rest of its validators, however, it insures that any
future call to success or failure, will result in the failure of the
form. The end result is that all the validators are tried, and may
possibly succeed or fail, showing the user a collection of errors
messages, not just the first error.
Consider the following validation example:
(list (try-all (field-is-a-number? "month" "Birthday Month") (field-is-a-number? "year" "Birthday Year")) (fields-validate '("month" "year") (lambda (ctx month year) (birthday-sanity-check month year))))
In this example, we have provided a list of validators. We have three
criteria we would like to check, mainly that the month and
year fields contain numbers and that the function
birthday-sanity-check passes. I have grouped the first two checks
together with the try-all so that both checks will be conducted
before giving up and failing the form. If the user were to leave the
form totally blank, he would see an error about both the month and year
not being filled in. The third validator demonstrates the ability to
run arbitrary checks for a form.
As mentioned above, the form framework not only validates arguments, but
also has the opportunity convert raw form values into more sophisticated
data structures. I made use of this functionality while dealing with a
shipping address form. The shipping form has quite few fields
individual fields, though they can be thought of as one
shipping-address structure. The form
framework took responsibility for collecting all of these individual
fields and put stored them in the shipping-address structure.
This simplified the success function for the form, because it took in a
shipping-address object and processed it, rather than having to deal
with many raw values.
There are many parts of the Scheme Pet Store which I am quite impressed with. In general, SISC and SISCweb are terrific environments to produce applications in. However, it is important to outline a couple areas where I ran into issues. I look forward to getting feedback from the Scheme community on how to address these.
Performance and scalability are simply areas I did not delve into. I have no evidence for how the Scheme Pet Store's performance compares to the Java Pet Store or to any of the other implementations out there. This is an area I am hoping to explore and understand at some point. Volunteers anyone?
XXX: Did I say enough about how I used SISCweb?
As mentioned above, the Scheme Pet Store is implemented as a single web application. All links, and forms have URLs that are from the user's perspective non-sensical. They are simply a resource that SISCweb manages, much like a garbage collector would manage memory addresses. This is a nice feature from the programmer's perspective, and does improve security to some degree. However, this turns out to be an issue when someone either bookmark's a link or tries to send the link to a friend.
Imagine a scenario where a user is browsing the application and notices a specific pet a friend would be interested in. The user cuts and pastes the URL into a mail note, only to have his friend unable to access the URL because the URL contains information available only to the sender . This problem is sometimes referred to as deep linking, because the issue relates to how you link into an application. This is not simply a problem in the SISCweb world, both Flash and many AJAX applications suffer from this same drawback.
As a stop gap measure I have taken a similar approach to solving the problem that Google Maps takes. Because their application is AJAX based, the map you are looking at on the page may not correspond to the URL in the address bar. Their solution was to provide a "link to this page" button. When this button is pressed, the page reloads, and places the URL that currently represents this map in the address bar. In the case of the Scheme Pet Store, pages that can be linked to directly, have a "show link to this page" button, that when pressed, shows a full URL to the page. This URL can then be thought of as a direct link to that page.
The implementation of the direct linking turned out to be made quite
simple in SISCweb. The top level function that is invoked by SISCweb is
actually a dispatching function and dispatches to a specific location in
the application, or to a default starting page. The implementation of
this dispatching function is reminiscent of the main function in C
or Java. This is yet another example of thinking in terms of functions
and not web pages.
While I have implemented a workaround, I'm hardly convinced that the deep linking problem is solved. There are still questions about bookmarking links and how long a user's state should be available for them to resume that have not been fully explored.
The Scheme Pet Store currently represents a proof of concept. I feel as though it demonstrates some of the areas in which Scheme can shine as a web development environment. In general, I'm very pleased with my experiment and do feel as though my hunch that SISCweb is a uniquely powerful tool is accurate.
The next level to take the Scheme Pet Store to is to turn it into a reference application. This was one of SUN's primary goals for producing the Java Pet Store, and would no doubt be of huge benefit to the Scheme community. In order to accomplish this goal the Scheme and programming languages community needs to provide feedback on the Pet Store. What patterns and concepts can be introduced, improved upon, and most importantly, taken away?
The most common feeling I had while developing the Scheme Pet Store was not that of trying to deal with a rigid structure, in fact it was the complete opposite feeling - I almost had too much flexibility. Should the application be based on an existing framework? Should the application make heavy use of macros or functions? Should the application be object oriented, procedural, logic based, or something inbetween? How much code should be written in Scheme, in Java or in a domain specific language? I am quite interested to hear what kind of direction we can get on answering these and other questions.
Once input has been provided, I can imagine developing improved versions of the application, as well as conducting neccessary performance benchmarks.
Please contact me at ben@amazingmedia.com if you are interested in learning more about the Scheme Pet Store, have any feedback, or need help running this application.
This project would not be possible without the SISCweb and SISC authors, or the help of those on the sisc-user mailing list.
$Id: writeup.tex,v 1.3 2005/08/29 17:24:17 ben Exp $