Monday, August 17, 2009

Introducing SHP 0.1: Scheme Hypertext Processor - A Tutorial

SHP v0.1 has just been released, and this is a tutorial for using SHP.

Introduction

SHP is a PHP/JSP-like framework for PLT Scheme web-server. Instead of developing in servlets, you develop in shp scripts that are dynamically loaded and evaluated. The main benefits for developing in shp scripts instead of servlets are:
  • instant refresh of changes - you do not need to reload servlets manually anymore
  • file-based dispatching and URL mapping - you get URL mapping to the underlying script location, which takes out a big chunk of URL mapping work if you like to have "pretty" URLs
As SHP's underlying platform is PLT Scheme and web-server, you get retain all of the benefits of developing in scheme instead of in php as well, including:
  • writing in xexpr and use quasiquotes instead of html snippets and PHP SSIs
  • scope safety - no global or superglobal variables roaming around somewhere, and every script is compiled into its own functions, so all variables will either need to be parameters or explicitly passed as arguments.
  • much safer language than PHP - you'll never have a mysterious variable magically appearing due to typos


License

SHP is released under LGPL.

Prerequisites

SHP depends on PLT Scheme version 4.2.1 or later. Make sure you have it installed.

Installation

SHP is easy to install within PLT Scheme, just open either DrScheme or mzscheme and type the following in REPL:

(require (planet bzlib/shp))

PLT Scheme automatically takes care of downloading the latest version. If you want to download a specific version, just do

(require (planet bzlib/shp:<major>:<minor>))

and substitute <major> & <minor> for the appropriate planet version numbers.

Example Site

SHP comes with an example site - you can browse and take a look at the source code to get a sense of how to develop in SHP. In general, you can just think of developing SHP as developing with PHP (if you have such experiences).

The example is located under the example/ subpath of the installed location of SHP package. If you go to the example directory you will find the following:
  • web.ss - for starting a single servlet web server
  • servlet.ss - the servlet wrapper for SHP
  • test-servlet.ss - an example servlet that is being "wrapped" by SHP
  • shp - the example shp script directory; contains the scripts


You can study all of the source code in the example directory for developing your own sites. The rest of the tutorial goes through the process and introduce the currently available features of SHP.

To start the example site, just do the following:

(require (planet bzlib/shp/example/web))

And point your browser to http://<site>:8001/.

Setting Up a Site

If you have been developing web-server servlets then you already know how to get started. Use the following as your start procedure:

(define start
(make-shp-handler ))


make-shp-handler generates your start procedure for you. You can specify the following parameters:

  • path: the main path of your shp script directory
  • #:default: the path to the "default" script, which by default points to "index.shp" (like index.html or index.php).
  • #:not-found: the path to the "not found" script. Unlike #:default, there is currently only one "not found" script per site (not per directory).
  • #:required: an optional script for holding a list of common required modules
  • #:topfilter: an optional script for handling pre & post processing for all of the requests.


We'll discuss the rest of the parameters shortly.

Developing Your First Script

It is easy to develop a hello world example - the following is your first script, call it index.shp:

;; index.shp
;; the first hello world example:
`(p "Hello world - this is an shp script")

Save the above to your SHP script directory, and refresh your browser.

Quasiquotes and Script Evaluations

Since this is a scheme xexpr - you can use quasiquotes to include snippets of scheme code for dynamic execution. Below we add the current timestamp to our hello world example:

;; index.shp
;; the first hello world example:
`(p "Hello world - this is an shp script and the timestamp is: "
,(number->string (current-seconds)))

Each script gets compiled into one scheme procedure.

Requiring External Modules & #:required

The goals for SHP is to simplify the writing of presentational layers in web-server, not to replace modules, which is probably one of the most advanced module systems available on the market. Hence, the best practice is to do most of your logic and business layers in PLT Scheme modules, and then require the modules from within SHP scripts. The following requires scheme/string into the hello world example so we can use string-join.

;; index.shp
;; the first hello world example:
(require scheme/string)
`(p "Hello world - this is an shp script and the timestamp is: "
,(number->string (current-seconds))
"."
(br)
"The pathinfo are: "
,(string-join '("abc" "def" "ghi") "/"))

All of the required modules are available to all of the scripts. But the evaluation order still applies, so if you call a script that's expecting a module is already available, without first calling the script that loads the module, then you'll get a compilation error. To solve the issue of evaluation order, you can put all of the require statements into a single script, and pass that into make-shp-handler via the #:required parameter.

(make-shp-handler <path> #:required <required-script-path> ...)

The require statement differ from regular PLT Scheme in one regard - it does not handle the extra require options such as only-in, prefix-in, etc. So make sure you only have require statement with module paths.

Handling Not Found

The #:not-found option for make-shp-handler takes in a script that'll get executed when user enters in an invalid URL that does not resolve into any particular script. This option is currently required. Below is an example of a "not found" script:

;; this script gets executed when user enters in an invalid URL
`(p "Sorry we did not find what you are looking for. Please check your URL and try again")
Accessing Environment Variables

The following environment (request, response, cookies, etc) are available for this release:
  • ($uri) - returns the url of the request
  • ($pathinfo) - return the extra path attached to the script.
  • ($header <key>) - return the header value in string, or false if the header is not found
  • ($query <key>) - return the first value (either in uri query or form post) in string, or false if not found
  • ($query* <key>) - return the value as a list - null if no value found. This is useful for retrieve multiple values
  • ($status) or ($status <new-status>) - retrieve or sets the status, which must be one of the following values - 'continue, 'switching-protocols, 'ok, 'created, 'accepted, 'non-authortative-information, 'no-content, 'reset-content, 'partial-content, 'multiple-choice, 'moved-permanently, 'found, 'see-other, 'not-modified, 'use-proxy, 'temporary-redirect, 'bad-request, 'unauthorized, 'payment-required, 'forbidden, 'not-found, 'method-not-allowed, 'not-acceptable, 'proxy-authentication-required, 'request-timeout, 'conflict, 'length-required, 'precondition-failed, 'request-entity-too-large, 'request-uri-too-long, 'unsupported-media-type, 'request-range-not-satisfied, 'expectation-failed, 'internal-server-error, 'not-implemented, 'bad-gateway, 'service-unavailable, 'gateway-timeout, 'version-not-supported
  • (header! <key> <value>) - sets a response header with key & value
  • ($content-type) or ($content-type <new-content-type>) - get or sets the content-type of the response, defaults to text/html
  • ($redirect <new-url> <status>) - redirect to the new url. The status should be one of the following - 'found, 'see-other, or temporary-redirect (default). The headers are automatically passed along with the redirect.
  • ($headers) or ($headers <new-headers>) - this parameter holds the underlying headers. Since we already have ($header <key>) and (header! <key> <val>), direct use of ($headers) is discouraged, except to set it to empty list.
  • (cookie-ref <key>), (cookie-set! <key> <val>), (cookie-del! <key>) - for manipulating cookies
  • ($cookies) - this is the underlying parameter that holds all of the cookies. Similar to ($headers) its direct use is generally discouraged


Including Other Scripts

An important feature in PHP is the ability to include other scripts for execution, and in SHP we can also do the same. To do so, let's include a secondary script called foo.shp in our hello world example:

;; foo.shp
;; returns the value of a + b
(:args a b)
(number->string (+ a b))

As you can see - foo.shp takes in two arguments, created via the :args directive, a and b, add the two values and return the results. Let's call foo.shp from within index.shp:

;; index.shp
;; the first hello world example:
(require scheme/string)
`(p "Hello world - this is an shp script and the timestamp is: "
,(number->string (current-seconds))
"."
(br)
"The pathinfo are: "
,(string-join '("abc" "def" "ghi") "/")
(br)
;; call /foo.shp
,(include! "/foo.shp" 5 10))

include! handles the magic of calling the script (which takes an absolute path as if mapped from the URL). Since each script gets compiled into its own function and has its own scope, there are no shared variables and variables must be either created as parameters (so they are available to all scripts) or passed as parameters in the include! call.

SHP automatically ensures that any scripts that requires arguments (like foo.shp) cannot be called directly by the user, and this provides an added bonus of hiding library scripts that PHP cannot do!

Toplevel Filters for Managing Global Parameters

The best place to hold any global parameterizations is with the #:topfilter option of make-shp-handler, which takes a filter script path.

A filter is a function that takes one parameter, which is a compiled form of the script. The following is a template for a filter:

;; example filter template
(:args inner) ;; inner is the handler
(parameterize ((parameter-a (some-value)) ...)
(inner))

Thus you can use #:toplevel to initialize and control any parameters (let's say a database connection handle) so all of the scripts have access to them. Currently make-shp-handler can only handle one filter, but in the future we can have multiple filters, which will open up the possibilities of using filters!

Interoperating with web-server and servlets

Since the idea of SHP is to be a script language for rapidly developing UIs, it's important that it can operate well within the context of the existing infrastructures. Besides hooking into web-server's servlet system, it currently have a couple of other features for interacting with web-server.

Punting the request back to web-server

If you would like the web-server instead of SHP to handle a particular request - just issue (punt!). This comes in handy if a particular URL maps to a static file, and you want to serve it through web-server.

Example - we want to punt all of the /scripts/ path to web-server, then put the following scripts to /scripts/index.shp:

;; /scripts/index.shp
(punt!)

Now if you have a scripts directory that maps to /scripts/ in your web-server document root, those files will be served instead.

Wrapping Around Servlets

If you already have substantial development done in servlets, you can call it through shp as well, so you do not have to rewrite everything to shp scripts. To do so is also simple, just have a particular script call (servlet <module-path>), like what we have in the example site:

;; servlet.shp
(servlet! "test-servlet.ss")

The <module-path> value can be a path that's resolvable by the underlying dynamic-require. And as long as the result value can be wrapped (i.e. an xexpr), you do not need to do any additional modification.


That's it for now - folks! Would love to hear back on any thoughts, etc. Happy SHP'ing.

Packaging SHP and Releasing to Wild

At this particular point there is enough basic functionalities in SHP that it's time to release it to the wild.

Releasing a package in PLT Scheme is a very straight forward process, which we will follow:
  1. organize all of the files into a single directory
  2. create an info.ss file to describe the metadata of the package
  3. use planet create to generate the PLT package file
  4. test the package file to ensure it is functional - use planet fileinject to install the package, and planet remove to deinstall the package
  5. submit the package to http://planet.plt-scheme.org
Specifically, SHP is developed on PLT Scheme 4.2.1 and due to the differences of the functionalities available in web-server from previous versions, SHP will not be compatible with older web-server versions without work, so we'll only support SHP for 4.2.1 onwards.

SHP will be released with the next post, which will include a tutorial for using SHP that summarizes all previous posts so far.

Friday, August 14, 2009

Servlet Support (wrapping servlets)

It would be nice to have the script able to call existing servlets, so they can be integrated easily.

The basic idea would be like:

;; foo.shp
(servlet "servlet.path")


Let's see how it would work:

(define (servlet! path (request ($request)) #:start (start 'start))
((dynamic-require path start) request))

Pretty straight forward!

The procedure allows loading of a servlet via dynamic-require, and it also offers the potential for you to rewrite the request before passing it to the servlet.

Now dynamic-require does not reload the servlet module when it changes, so that would be something we'll have to deal with in the future.

One thing you should make sure is that the servlet does not return an actual response object, as currently the code will not be converting the response object back so it can be wrapped by outer scripts. This is something we can attempt to address in the future. This means that if your servlet calls redirect-to, it will not work (you'll have to call redirect!).

Redirect Support

Perhaps we would want to control redirects from within the scripts - ideally something like below should work:

(redirect! url)

To enable such handling is straight forward:
(define (redirect! url 
#:status (status 'temporary-redirect)
#:headers (headers '()))
(define (header-helper h)
(cond ((header? h) h)
((pair? h) (make-header (string->bytes/utf-8 (car h))
(string->bytes/utf-8 (cdr h))))
((cookie? h)
(cookie->header h))))
(redirect-to (if (url? url) (url->string url) url)
(case status
((found) permanently)
((see-other) see-other)
((temporary-redirect) temporarily))
#:headers (map header-helper headers)))

The wrapper over redirect-to simplifies type conversion between url and string, and pairs of string to header, as well as cookie to header, so you can do the following:

(redirect! '/another/path' #:headers `(("test" . "me") ,(cookie-ref! "session")))

As redirect-to only returns a response/full struct, we need to raise the struct so we can escape the calling stack if the script is being included. And let's catch the object at the request handling:

(define (handle-request server request)
(parameterize (($pathinfo ($pathinfo))
($server server)
($request request))
(parameterize (($cookies (init-cookies!)))
(eval-script-if-changed! (shp-handler-required server))
(with-handlers ((response/basic?
(lambda (r) r)))
(make-response (include! (request-uri request)
#:topfilter (shp-handler-topfilter server)
#:partial? #t))))))


Now we can do a redirect from within the scripts! ;)

Uploaded Files Support

In PHP there is a $_FILES superglobal variable, which contains information about the uploaded file that have been stored to the temporary folder. In PHP, files and the rest of the form variables are treated separately.

In web-server, the uploaded files are treated just the same as rest of the form variables, i.e. you can access them through $query. The advantage of this approach is that you do not have think about file separately, but the disadvantage is that web-server will hold all of the uploaded files in memory, so the actual total upload size will be limited to the available RAM.

We are not going to solve that particular limit for web-server right now, instead, we want to think about whether it makes sense to have a $files parameter separately from $query, so it appears more similar to PHP.

I think at this moment it makes no sense to treat files the same way as PHP. So we'll keep them in $query.

Enabling Cookie Support

Allright - time to add more environment support, and this time it's the cookies.

web-server already provide cookie support, so it's just a matter of wrapping around them to get started.

Let's first define a parameter to hold cookies - this parameter will be populated at the start of the request, and be used to generate cookie headers during response time.

(define $cookies (make-parameter '()))

We will also use our own structure, since in web-server there are two types of cookie objects (client-cookie and cookie):

(define-struct cookie (name value domain path max-age comment secure?))


Extracting Cookies

web-server put handling of parsing request-cookies under web-server/http/cookie-parse, let's wrap around it so we have access to cookies:

;; convert ws:client-cookie (web-server cookie) to cookie
(define (ws:client-cookie->cookie c)
(make-cookie (ws:client-cookie-name c)
(ws:client-cookie-value c)
#f
#f
#f
#f
#f
))
;; extract the cookies from the request object
(define (init-cookies!)
(map (lambda (c)
(let ((c (ws:client-cookie->cookie c)))
(cons (cookie-name c) c)))
(ws:request-cookies ($request))))

;; parameterize cookie with extraction
(define (handle-request server request)
(parameterize (($pathinfo ($pathinfo))
($server server)
($request request))
(parameterize (($cookies (init-cookies!)))
...)))

So once we are inside the scripts we have access to the cookies from the request.

Manipulating Cookies


We will want to access cookie values, set cookies, and delete cookies, as follows:


(define (cookie-set! name value
#:domain (domain #f)
#:path (path #f)
#:max-age (max-age #f)
#:comment (comment #f)
#:secure? (secure? #f))
($cookies (cons (cons name (make-cookie name value domain path max-age comment secure?))
($cookies))))

(define (cookie-ref name)
(let ((kv (assoc name ($cookies))))
(if kv (cdr kv) #f)))

(define (cookie-del! name)
(let ((kv (assoc name ($cookies))))
($cookies (if kv
(filter (lambda (c)
(not (equal? kv c)))
($cookies))
($cookies)))))

Generating Cookies for Response


Finally - we will send the cookies in $cookies parameter through the response object back to the client:

(define (cookie->header c)
(ws:cookie->header (ws:make-cookie (cookie-name c)
(cookie-value c)
#:domain (cookie-domain c)
#:path (cookie-path c)
#:max-age (cookie-max-age c)
#:comment (cookie-comment c)
#:secure? (cookie-secure? c))))

(define ($cookies->headers)
(map (lambda (kv)
(cookie->header (cdr kv)))
($cookies)))


The IE Bug

If you like to set path to "/", IE has an issue with handling "/" in the cookie path; it prefers / instead. So we will have to do a manual conversion to solve this problem:

(define (cookie->header c)
(define (helper c)
(ws:make-header #"Set-Cookie"
(string->bytes/utf-8
(regexp-replace #px"Path=\\\"/\\\""
(ws:print-cookie c)
"Path=/"))))
(helper
(ws:make-cookie (cookie-name c)
(cookie-value c)
#:domain (cookie-domain c)
#:path (cookie-path c)
#:max-age (cookie-max-age c)
#:comment (cookie-comment c)
#:secure? (cookie-secure? c))))

Now we can enjoy cookies in all its glory of good, bad, and ugliness like other platforms :)

Thursday, August 13, 2009

Intermission - Refactoring Path and Dispatch Related Procedures

At this time we have multiple procedures that overlap in terms of functionality for path and dispatching, and it is probably a good time to take a look at how to refactor them.

The following are path related procedures:
  • normalize-path: convert a path to the underlying system path, depends on path->segments and shp-handler-path
  • url->path-segments: converts the url into path segments; overlapping partly with path->segments
  • normalize-partial-path: takes the segments and convert into an underlying path; depends on shp-handler-path
  • segments->path: converts the segments back to the underling path; calls normalize-partial-path
  • url->shp-path: wraps around url-path->segments and segments->path
And dispatch related procedures:
  • eval-script-if-changed: evaluate the script if it is changed; takes in script struct
  • shp-handler: evaluate the script from top level, takes in nothing
  • envoke-topfilter-if-available: call the top filter if available - takes in a procedure
  • include! - calls the code, takes in a path
The dispatching procedures appear even more overlapped than the path procedures. Let's start somewhere to refactor them.

First - url-path->segments can be refactored to:

;; url->path-segments - returns an underlying path based on input url (duplicate with normalize-path)
(define (url->path-segments url (default "index.shp"))
(filter (lambda (path)
(not (equal? path "")))
(map path/param-path (url-path url))))

But now url->path-segments looks a lot like path->segments, so the two can be refactored:

(define (path->segments path)
(filter (lambda (path)
(not (equal? path "")))
(if (url? path)
(map path/param-path (url-path path))
(regexp-split #px"\\/" path))))

(define (url->shp-path url)
(segments->path (path->segments url) #t))

Change the name of normalize-partial-path to segments->partial-path to more correctly state its purpose.

The difference between include!, shp-handler, eval-script-if-changed, and envoke-topfilter-if-available are basically their signatures, one takes in a path, the other takes a request, then one takes a script structure, and the last takes a procedure. It would be great to "normalize" them.

Note shp-handler and envoke-top-filter-if-available depend on include!, so include! should be the base of our refactoring. Let's see if we can push redundant features into include!.

Since include! takes a path into path->segments, which now also takes an url, we can reduce the code in shp-handler if we can fold envoke-topfilter-if-available in as well.

(define (include! path #:topfilter (topfilter #f) . args)
(define (helper topfilter)
(let ((proc (evaluate-script (segments->path (path->segments path) #f))))
(if topfilter
(topfilter (lambda () (apply proc args)))
(apply proc args))))
(helper (if topfilter
(evaluate-script (segments->path (path->segments topfilter) #f))
topfilter)))

So we can now get rid of envoke-topfilter-if-available, and modify shp-handler as such:

(define (handle-request server request)
(parameterize (($pathinfo ($pathinfo))
($server server)
($request request))
(eval-script-if-changed! (shp-handler-required server))
(make-response (include! (request-uri request)
#:topfilter (shp-handler-topfilter server)
#:partial? #t))))

handle-request is extracted out of the struct, which now becomes:

(define-struct shp-handler (path default not-found required topfilter)
#:property prop:procedure
(lambda ($struct request)
(handle-request $struct request)))

You'll notice that handler-request calls include! with an extra partial? parameter, which will allow a partial path match, so include! now looks like:

(define (include! path
#:topfilter (topfilter #f)
#:partial? (partial? #f)
. args)
(define (helper topfilter)
(let ((proc (evaluate-script (segments->path (path->segments path) #f))))
(if topfilter
(topfilter (lambda () (apply proc args)))
(apply proc args))))
(helper (if topfilter
(evaluate-script (segments->path (path->segments topfilter) partial?))
topfilter)))

Notice that partial? is only applied to the script and not to the toplevel filter, which needs a full path match.

At this time only eval-script-if-changed has not been combined. A simple way of handling the issue is to allow include! to take in script struct as well:

(define (include! path
#:topfilter (topfilter #f)
#:partial? (partial? #f)
. args)
(define (helper topfilter)
(let ((proc (evaluate-script
(if (script? path)
(script-path path)
(segments->path (path->segments path) partial?)))))
(if topfilter
(topfilter (lambda () (apply proc args)))
(apply proc args))))
(helper (if topfilter
(evaluate-script (segments->path (path->segments topfilter) #f))
topfilter)))

(define (eval-script-if-changed! script)
(unless (not (file-exists? (script-path script)))
(let ((timestamp (file-or-directory-modify-seconds (script-path script))))
(when (> timestamp (script-timestamp script))
(set-script-timestamp! script timestamp)
(include! script)))))

Finally - we can abstract (segments->path (path->segments ...)) into its own procedure:

(define (resolve-path path (partial? #t))
(segments->path (path->segments path) partial?))

(define (include! path
#:topfilter (topfilter #f)
#:partial? (partial? #f)
. args)
(define (make-script path partial?)
(evaluate-script (if (script? path)
(script-path path)
(resolve-path path partial?))))
(define (helper topfilter)
(let ((proc (make-script path partial?)))
(if topfilter
(topfilter (lambda () (apply proc args)))
(apply proc args))))
(helper (if topfilter (make-script topfilter #f) topfilter)))


Allright! Now the code looks a lot better! We can move onto adding more features!