SHP 0.2 and JSMGR 0.1, HTTP 0.1 Are Now Available

SHP 0.2 is now available through planet, along with JSMGR 0.1 and HTTP 0.1. SHP 0.2 adds a number of enhancements on top of SHP 0.1 toward making SHP simple to use:

• simpler startup
  
• default and configurable chrome - so you can switch the UI chrome depending on the page
  
• proxy integration - a built-in http/s proxy so you can use AJAX with ease
  
• JSMGR, which allows you to develop javascript and css in modular fashion but yet serve them combined and compressed; it is optionally available (you'll have to require it) in SHP
  
• HTTP client abstraction, which include https protocol - this is the basis of proxy
  

All code are released under LGPL.

Simple Startup

Now to start a new site - you just need to do:

;; startup.ss or startup.scm
(require (planet bzlib/shp/start))
(start-shp-server! <path> #:htdocs <htdocs-path> ...)

The following parameters are available:

• path: this is the root directory for your SHP scripts
  
• #:port: the port of the server - default to 8080
  
• #:default: this is the path of the default script for each directory - defaults to "index"
• #:not-found: the not-found script for the site (within the SHP root directory) - default to #f as it is optional
  
• #:required: the required script for the site - default to /include/required - the site will function if the script does not exist within the SHP directory
  
• #:topfilter: the topfilter script for the site - default to #f. If it is specified it must exist or an error will occur
  
• #:chrome: the chrome script for the site - default to #f. If specified it must exist.
  

Default and Configurable Chrome

Instead of using topfilter to factor out the common chrome, you can now specify a default chrome (through the startup) and dynamically change chrome within the scripts (NOTE - there is only one chrome for a given request).

To specify the default chrome - do it when you startup the shp server via the #:chrome parameter. Make sure the chrome script does exist.

To disable chrome within the script, issue the following:

;; inside your shp script
($chrome #f)

To change chrome to another chrome script, issue the following:

;; inside your shp script
($chrome "/new/chrome/path")

A chrome script takes in one parameter, which is the actual results evaluated of all of the inner scripts (so it differs from the topfilter's parameter, which is a procedure). Remember chrome is evaluated last of all scripts, so please ensure there aren't evaluation order dependencies between chrome and other scripts.

Proxy Integration
A HTTP proxy is built-in by default in SHP to help you integrate with other web services (your own or a third party's). All you need to do is use the following script:

;; /proxy script
(proxy!)

And it'll automatically convert the pathinfo into the target URL, and pass the rest of the data in query string or post to pass it to the URL.

Headers are currently filtered, and only headers with the prefix of "bzl-" are passed through (with the "bzl-" prefix stripped).

This is designed for using with AJAX, where there is issue of the same-origin policy, so you can query web services residing in different domains without issues. It is not a full web browser proxy (yet).

Proxy is built on top of bzlib/http, which will be described later.

JSMGR Integration

An optional integration is available with JSMGR, which helps you to develop javascript and css scripts in modular fashion but allows you to serve them in combined and compressed mode.

To use JSMGR, add the following to your required script:

(require (planet bzlib/jsmgr/shp)) 

If you have not previously downloaded JSMGR - the above will do it for you the first time you run the code in SHP script, but since JSMGR is quite large as it includes YUI compressor, it's probably best to do so in DrScheme or mzscheme REPL.

Then you can serve js and css with the following code:

;; either /js or /css script
(js/css! #:base <your-javascript-or-css-path-here>)

And you can use helpers to generate the URL for you:

;; in your shp scripts
(script* <js1> <js2> ...) ;; generates a <script> xexpr
(css* <css1> <css2> ...) ;; generates a <link> xexpr

They generate the block by utilizing a known url path (i.e., /js for javascript and /css for css) - if you decide to use a different url you'll have to reset the following in your topfilter script:

;; topfilter script
(parameterize ((js-base-path "/new/js/url")
                        (css-base-path "/new/css/url"))
...)

JSMGR is distributed with YUI compressor, which is licensed by Yahoo! under BSD. I do not believe there are license compatibility issues, but let me know if that is not the case.

You can switch to a different compressor of your choice, as long as you implement a compress! procedure that has the following signature:
(-> path-string? path-string? any). The first is the path to the actual script (need to exist), and the second is the path for the compressed script (need not exist).

JSMGR also comes with a servlet-util module that allow you to integrate JSMGR into a regular servlet rather than SHP. This is not a well tested feature, however, so let me know if you run into issues.

;; your servlet
(require (planet bzlib/jsmgr/servlet-util))
(define start (make-start <base-path-to-the-javascript-or-css-directory>))
(provide start)

HTTP Client Abstraction

SHP proxy depends on bzlib/http, which will be automatically installed if you install SHP. You can use it as a replacement for the net/url's get-impure-port and post-impure-port. Besides working with the http and file protocol, http-get and http-post also work with https protocol.

(require (planet bzlib/http/client))
(http-get <url> <list-of-headers>)
(http-post <url> <data> <list-of-headers>)

The list of headers is a list of string pairs so the headers are easy to construct. The data for http-post is a list of bytes, so you'll have to manually construct the values for now.

bzlib/http also depends on bzlib/net, which provides additional utilities to deal with network code. This module will currently not be supported so I won't document how it can be used.

That's it for SHP 0.2 - happy SHP'ing.

Using JSMGR to Compress a Single Script

JSMGR as currently designed handles the compression and combination of several scripts very well. But what if you just want to access a single script?

Currently in order to access a single script you'll have to use the following URL:

http://<host>/js?s=path/to/js-file.js

It is just a tag bit more inconvenient than

http://<host>/js/path/to/js-file.js

Fortunately it is simple for us to handle pathinfos:

;; the javascript! loader will depend on a having a paticular setting
(define (js/css! (scripts ($query* "s")) #:base (base (current-directory)) #:compress! (compress! yui-compress!))
 (define (scripts-helper)
   (open-js/css-files/base
    (if (not (null? ($pathinfo)))
        (cons (string-join (filter (lambda (segment)
                                     (not (string=? segment "")))
                                   ($pathinfo))
                           "/")
              scripts)
        scripts)
    base
    compress!))
 (raise
  (http-client-response->response
   (make-http-client-response "1.1"
                              200
                              "OK"
                              '(("Content-Type" . "text/javascript"))
                              (scripts-helper))
   (lambda (x) x))))

Now if we want to pass a single script "normally" - we can do that.

This single script support is currently only available in the shp adapter, not the servlet adapter.

Manage CSS through SHP and JSMGR

Besides javascript, CSS is another type of text resource that suffer same problems that javascript does:

• lack of modularity
  
• lack of compression
  

The problem is almost exactly the same that we should be able to use JSMGR to handle CSS as well!

Luckily - YUI compressor works with both javascript and CSS, so by using YUI compressor our work is limited (you can of course implement your own compress! routine). What we need is to add the signature specific to CSS, we have a couple of approaches:

1. have css-* functions that matches the javascript-* functions
   
2. change the signature of javascript-* functions to it does not have the type of the script in its name, and instead, takes it in a #:type parameter
   

It seems that at the compression level, approach #2 is more logical, especially since YUI compressor works with either simply by the file extension type. So instead of open-javascript-files and open-javascript-files/base, we should have open-js/css-files and open-js/css-files/base. And javascript! will also be renamed as js/css!.

The only trouble we will encounter at this time with this approach is that when we need to implement another compressor we'll have to follow YUI's interface, but since that's out of scope, that's worry for another day.

At the xexpr-generator level, though, it makes sense to have the two types separated, since by default they generate different signatures:

• js-url/xexpr generates <script> block, but we'll call it script* to match it more with xexpr
  
• css-url/xexpr generates <link rel=styleet> block, and we'll call this css* to match it more with xexpr
  

So, we'll adapt the signature accordingly:

(define css-base-path (make-parameter "/css"))

(define (css-url path . paths)
 (let ((url (string->url (css-base-path))))
   (set-url-query! url (map (lambda (path)
                              (cons 's path))
                            (cons path paths)))
   (regexp-replace* #px"(\\&)([^a])" (url->string url) ";\\2")))

(define (css* path . paths)
 `(link ((rel "stylesheet") (type "text/css") (href ,(apply css-url path paths)))))

Now we can handle CSS with JSMGR as well!

Managing your Javascripts through SHP (4) - Shorthand for Generating the JSMGR querystring

To really simplify the development it would be nice to be able to call something like the following to issue the javascript reference:

;; in any SHP script 
`(script ((src ,(js-link "jquery.js" "jquery.ui.js" ...))) "")

and it will translate into the appropriate link. Let's see how it can be done.

Assuming the path is hardcoded (let's say "/js/"), then it's pretty straight forward:

(define js-base-path (make-parameter "/js"))

(define (js-url path . paths) 
  (let ((url (string->url (js-base-path))))
    (set-url-query! url (map (lambda (path)
                               (cons 's path))
                             (cons path paths)))
    (url->string url)))

Given the base path is set in a parameter, you can parameterize it to a different value. It would be nice to "automatically" set the js-base-path based on where you call the value, but that might be more difficult than it's worth so it's out of scope for now.

Managing your Javascripts through SHP (3) - Integrate with Servlets

If you do not want to use SHP but would like to use jsmgr, it provides a servlet adapter that you can use - just do the following in your servlet:

;; your servlet... 
(require (planet bzlib/jsmgr/servlet-util)) 
(define start
  (make-start "the-path-to-javascript-here..."))

And the servlet will have the exact same functionality. The following is the servlet-util.ss that enabled the above:

(require web-server/http/request-structs
         web-server/http/response-structs
         net/url
         mzlib/etc
         scheme/contract
         (planet bzlib/shp:1:1/request)         
         (planet bzlib/shp:1:1/proxy)         
         (planet bzlib/http/client)
         "loader.ss"
         )

(define (request-helper request)
  (parameterize (($request request))
    ($query* "s")))

;; we want to take the request's query object (which are quite available through our definitions...) 
;; and then map it to 
(define (make-start base) 
  (lambda (request)
    (let ((scripts (request-helper request)))
      (http-client-response->response 
       (make-http-client-response "1.1" 
                                  200 
                                  "OK" 
                                  '(("Content-Type" . "text/javascript; charset=utf-8"))
                                  (open-javascript-files/base scripts base))
       (lambda (x) x)))))

(provide/contract 
 (make-start (-> path-string? (-> request? response/c)))
 )

Managing your Javascripts through SHP (2) - Integrate with SHP

Since we now can retrieve the combined scripts, the integration basically will focus on mapping the request query values to the scripts, and return the combined, mini-fied scripts as the response. Let's try to do that within SHP.

We'll use s as the query key name. And it can be repeated multiple times. What we want is to write an SHP script that looks like the following:

;; js
;; loading javascripts
;; default to using s key, but can be customized
(javascript!) ;; or (javascript! ($query* "s"))

The definition of javascript! will roughly look like the following:

(define (javascript! (scripts ($query "s")) #:base (base (current-directory)))
(http-client-response->response
(make-http-client-response "1.1"
                        200
                        "OK"
                        '(("Content-Type" . "text/javascript; charset=utf-8"))
                        (apply javascript-loader #:base base
                               scripts)) ;; this is a mismatch
(lambda (x) x)))

And we encounter our first mismatch - our original loader was a closure over the base path. While we can still create the closure as previously designed, the closure is wasted since we are not keeping it around. It's simpler to have a new interface that takes in the base path - and we'll take the opportunity to refactor the code a bit:

(define (yui-compress! path min-path)
(system (string-join
       (list "java"
             "-jar"
             (path->string (build-path (this-expression-source-directory)
                                       "yuicompressor.jar"))
             "-o" (path->string min-path) (path->string path))
       " ")))

(define (open-javascript-files paths (compress! yui-compress!))
(define (min-path-helper path)
(string->path (string-append (path->string path) ".min")))
(define (helper path)
(let ((min-path (min-path-helper path)))
  (when (or (not (file-exists? min-path))
            (> (file-or-directory-modify-seconds path)
               (file-or-directory-modify-seconds min-path)))
    (compress! path min-path))
  (open-input-file min-path)))
(apply input-port-append #t (map helper paths)))

(define (open-javascript-files/base base paths (compress! yui-compress!))
(define (helper path)
(build-path base path))
(open-javascript-files (map helper paths) compress!))

Now yui-compress! is extracted from the loader, and two versions of the loader are exposed: open-javascript-files and open-javascript-files/base, which takes in an additional base argument. And they both take in an optional compress! argument that you can use to switch out yui-compress! if you want to use another javascript compressor.

Now the javascript! handler would look like:

;; the javascript! loader will depend on a having a paticular setting
(define (javascript! (scripts ($query* "s")) (base (current-directory)))
(raise
(http-client-response->response
 (make-http-client-response "1.1"
                            200
                            "OK"
                            '(("Content-Type" . "text/javascript; charset=utf-8"))
                            (open-javascript-files/base scripts base)
                            (lambda (x) x)))))

And we now can add the module (we are calling this bzlib/jsmgr/shp) to our SHP instance:

;; required.shp
(require (planet bzlib/jsmgr/shp))

And the script.shp should contain the following:

;; script.shp
(javascript! #:base "some-path-here...")

Testing it out give us a cryptic error:

Exception

The application raised an exception with the message:

open-input-file: cannot open input file: "../collects/planet/main.ss" (The system cannot find the file specified.; errno=2)

Although the error is cryptic, we know it arises from our SHP handler's require-modules! procedure. It turns out that because the procedure calls flatten, we have flattened (planet bzlib/jsmgr) into two separate collections and hence caused the error. Below fixes the error:

(define (require-modules! terms)
  (define (require! module)
    (namespace-require module))
  (define (helper listof-modules)
    (parameterize ((current-namespace handler-namespace))
      (for-each (lambda (modules)
                  (for-each require! modules))
                listof-modules)))
  (helper (map cdr (filter require-exp? terms))))

Now the Javascript Manager is available through SHP.

Managing your Javascripts through SHP

If you are doing extensive AJAX development (which many people are these days), you probably are writing quite a bit of OOP Javascript code, along with using libraries such as Prototype, jQuery, Dojo, etc.

The trouble with javascript is that it lacks modern module management facilities that we have come to expect from programming languages, especially given its importance in the browser world. You either develop all of your code in one large script, or you risk the browser needing to download numerous small scripts, which is inefficient. Furthermore, if you plan on using javascript compressors such as the YUI compressor, if you make updates to the source script you would have to recopmress the code manually.

It would be nice to have a tool that can help with the above issues:

• allows you to code in multiple small scripts, but automatically combine the smaller scripts into a single script as you specify
  
• automatically helps you recompress the code if you make changes to them, so you just need to worry about the source (which will not be served directly unless specified)
  

Let's look at what we need to accomplish the goal:

• ability to load and combine multiple files into a single response
  
• ability to detect timestamp of the source and the compressed scripts, and re-compress if the source script's timestamp is newer than the compressed scripts

You might have come across "script builders" such as jQuery UI, and Mootools - you can use them as a mental model for the goal (for your own scripts!).

Compressor Choice

Currently the best choice of javascript compressor appears to be YUI compressor (if you are not trying to obfuscate the scripts too much) as it provides the best combination of compression ratio as well as the decompression speed (obfuscators such as Packer loses speed during the unpack phase), so it is our choice for compressor. It is licensed under BSD license, so we can redistribute the binary without issues. In the future we might add support for other compressors, but for now it's out of scope.

Prerequisite

YUI compressor requires Java (version >= 1.4), which you'll have to install separately and make it available in the system path.

First Attempt - a Single Javascript File
Let's see if we can get a simple version of this up and running - let's first just serve out a javascript file sitting somewhere in our system. No compressions. It is quite straight forward.

(define (make-loader root-path)
(lambda (path)
  (open-input-file (build-path root-path path))))

The basic signature will be make-loader, which will take a path and returns a procedure we can then use to load the file. If the file does not exist an appropriate error will be thrown (which we'll have to handle once integrating into servlets). We'll also have to handle setting the content-type during integration.

Second Attempt - Multiple Javascript Files

Now let's see if we can serve multiple javascript files. PLT Scheme makes this easy with input-port-append:

(require scheme/port)
(define (make-loader root-path)
(lambda (path . paths)
  (define (helper path)
    (open-input-file (build-path root-path path)))
  (apply input-port-append #t (map helper (cons path paths)))))

All of the input-port are now appended together, so when we exhausted the data from the first port, we'll then retrieve the data from the second port, so on.

Adding Compression

So far, so good. Let's now try to see if we can add compression in here.

Let's just say that the YUI compressor live in the same directory as the module, and Java is in the path. And all of the compressed script's path is the same as the source path, with the addition of ".min":

(require scheme/system scheme/string)
(define (make-loader root-path) 
  (lambda (path . paths) 
    (define (min-path-helper full-path)
      (string->path (string-append (path->string full-path) ".min")))
    (define (path-helper path)
      (build-path root-path path))
    (define (helper path)
      (let* ((path (path-helper path))
             (min-path (min-path-helper path)))
        (when (or (not (file-exists? min-path))
                  (< (file-or-directory-modify-seconds min-path)
                     (file-or-directory-modify-seconds path)))
          (system (string-join (list "java" 
                                     "-jar" 
                                     (path->string (build-path (this-expression-source-directory)
                                                               "yuicompressor.jar"))
                                     "-o" (path->string min-path) (path->string path))
                               " ")))
        (open-input-file min-path)))
    (apply input-port-append #t (map helper (cons path paths)))))

The only thing we need to make sure to handle is the error that could result from the system - it returns #f when the call failed, so we need something a bit more verbose (including error messages) in order to throw error about the failed compilation (it is best to have such error messages show up during your development, rather than during production).

Allright - the next step would be for us to integrate this into SHP. Stay tuned.