Managing your Javascripts through SHP (4) - Shorthand for Generating the JSMGR querystring

To really simplify the development it would be nice to be able to call something like the following to issue the javascript reference:

;; in any SHP script 
`(script ((src ,(js-link "jquery.js" "jquery.ui.js" ...))) "")

and it will translate into the appropriate link. Let's see how it can be done.

Assuming the path is hardcoded (let's say "/js/"), then it's pretty straight forward:

(define js-base-path (make-parameter "/js"))

(define (js-url path . paths) 
  (let ((url (string->url (js-base-path))))
    (set-url-query! url (map (lambda (path)
                               (cons 's path))
                             (cons path paths)))
    (url->string url)))

Given the base path is set in a parameter, you can parameterize it to a different value. It would be nice to "automatically" set the js-base-path based on where you call the value, but that might be more difficult than it's worth so it's out of scope for now.

Managing your Javascripts through SHP (3) - Integrate with Servlets

If you do not want to use SHP but would like to use jsmgr, it provides a servlet adapter that you can use - just do the following in your servlet:

;; your servlet... 
(require (planet bzlib/jsmgr/servlet-util)) 
(define start
  (make-start "the-path-to-javascript-here..."))

And the servlet will have the exact same functionality. The following is the servlet-util.ss that enabled the above:

(require web-server/http/request-structs
         web-server/http/response-structs
         net/url
         mzlib/etc
         scheme/contract
         (planet bzlib/shp:1:1/request)         
         (planet bzlib/shp:1:1/proxy)         
         (planet bzlib/http/client)
         "loader.ss"
         )

(define (request-helper request)
  (parameterize (($request request))
    ($query* "s")))

;; we want to take the request's query object (which are quite available through our definitions...) 
;; and then map it to 
(define (make-start base) 
  (lambda (request)
    (let ((scripts (request-helper request)))
      (http-client-response->response 
       (make-http-client-response "1.1" 
                                  200 
                                  "OK" 
                                  '(("Content-Type" . "text/javascript; charset=utf-8"))
                                  (open-javascript-files/base scripts base))
       (lambda (x) x)))))

(provide/contract 
 (make-start (-> path-string? (-> request? response/c)))
 )

Managing your Javascripts through SHP (2) - Integrate with SHP

Since we now can retrieve the combined scripts, the integration basically will focus on mapping the request query values to the scripts, and return the combined, mini-fied scripts as the response. Let's try to do that within SHP.

We'll use s as the query key name. And it can be repeated multiple times. What we want is to write an SHP script that looks like the following:

;; js
;; loading javascripts
;; default to using s key, but can be customized
(javascript!) ;; or (javascript! ($query* "s"))

The definition of javascript! will roughly look like the following:

(define (javascript! (scripts ($query "s")) #:base (base (current-directory)))
(http-client-response->response
(make-http-client-response "1.1"
                        200
                        "OK"
                        '(("Content-Type" . "text/javascript; charset=utf-8"))
                        (apply javascript-loader #:base base
                               scripts)) ;; this is a mismatch
(lambda (x) x)))

And we encounter our first mismatch - our original loader was a closure over the base path. While we can still create the closure as previously designed, the closure is wasted since we are not keeping it around. It's simpler to have a new interface that takes in the base path - and we'll take the opportunity to refactor the code a bit:

(define (yui-compress! path min-path)
(system (string-join
       (list "java"
             "-jar"
             (path->string (build-path (this-expression-source-directory)
                                       "yuicompressor.jar"))
             "-o" (path->string min-path) (path->string path))
       " ")))

(define (open-javascript-files paths (compress! yui-compress!))
(define (min-path-helper path)
(string->path (string-append (path->string path) ".min")))
(define (helper path)
(let ((min-path (min-path-helper path)))
  (when (or (not (file-exists? min-path))
            (> (file-or-directory-modify-seconds path)
               (file-or-directory-modify-seconds min-path)))
    (compress! path min-path))
  (open-input-file min-path)))
(apply input-port-append #t (map helper paths)))

(define (open-javascript-files/base base paths (compress! yui-compress!))
(define (helper path)
(build-path base path))
(open-javascript-files (map helper paths) compress!))

Now yui-compress! is extracted from the loader, and two versions of the loader are exposed: open-javascript-files and open-javascript-files/base, which takes in an additional base argument. And they both take in an optional compress! argument that you can use to switch out yui-compress! if you want to use another javascript compressor.

Now the javascript! handler would look like:

;; the javascript! loader will depend on a having a paticular setting
(define (javascript! (scripts ($query* "s")) (base (current-directory)))
(raise
(http-client-response->response
 (make-http-client-response "1.1"
                            200
                            "OK"
                            '(("Content-Type" . "text/javascript; charset=utf-8"))
                            (open-javascript-files/base scripts base)
                            (lambda (x) x)))))

And we now can add the module (we are calling this bzlib/jsmgr/shp) to our SHP instance:

;; required.shp
(require (planet bzlib/jsmgr/shp))

And the script.shp should contain the following:

;; script.shp
(javascript! #:base "some-path-here...")

Testing it out give us a cryptic error:

Exception

The application raised an exception with the message:

open-input-file: cannot open input file: "../collects/planet/main.ss" (The system cannot find the file specified.; errno=2)

Although the error is cryptic, we know it arises from our SHP handler's require-modules! procedure. It turns out that because the procedure calls flatten, we have flattened (planet bzlib/jsmgr) into two separate collections and hence caused the error. Below fixes the error:

(define (require-modules! terms)
  (define (require! module)
    (namespace-require module))
  (define (helper listof-modules)
    (parameterize ((current-namespace handler-namespace))
      (for-each (lambda (modules)
                  (for-each require! modules))
                listof-modules)))
  (helper (map cdr (filter require-exp? terms))))

Now the Javascript Manager is available through SHP.

Managing your Javascripts through SHP

If you are doing extensive AJAX development (which many people are these days), you probably are writing quite a bit of OOP Javascript code, along with using libraries such as Prototype, jQuery, Dojo, etc.

The trouble with javascript is that it lacks modern module management facilities that we have come to expect from programming languages, especially given its importance in the browser world. You either develop all of your code in one large script, or you risk the browser needing to download numerous small scripts, which is inefficient. Furthermore, if you plan on using javascript compressors such as the YUI compressor, if you make updates to the source script you would have to recopmress the code manually.

It would be nice to have a tool that can help with the above issues:

• allows you to code in multiple small scripts, but automatically combine the smaller scripts into a single script as you specify
  
• automatically helps you recompress the code if you make changes to them, so you just need to worry about the source (which will not be served directly unless specified)
  

Let's look at what we need to accomplish the goal:

• ability to load and combine multiple files into a single response
  
• ability to detect timestamp of the source and the compressed scripts, and re-compress if the source script's timestamp is newer than the compressed scripts

You might have come across "script builders" such as jQuery UI, and Mootools - you can use them as a mental model for the goal (for your own scripts!).

Compressor Choice

Currently the best choice of javascript compressor appears to be YUI compressor (if you are not trying to obfuscate the scripts too much) as it provides the best combination of compression ratio as well as the decompression speed (obfuscators such as Packer loses speed during the unpack phase), so it is our choice for compressor. It is licensed under BSD license, so we can redistribute the binary without issues. In the future we might add support for other compressors, but for now it's out of scope.

Prerequisite

YUI compressor requires Java (version >= 1.4), which you'll have to install separately and make it available in the system path.

First Attempt - a Single Javascript File
Let's see if we can get a simple version of this up and running - let's first just serve out a javascript file sitting somewhere in our system. No compressions. It is quite straight forward.

(define (make-loader root-path)
(lambda (path)
  (open-input-file (build-path root-path path))))

The basic signature will be make-loader, which will take a path and returns a procedure we can then use to load the file. If the file does not exist an appropriate error will be thrown (which we'll have to handle once integrating into servlets). We'll also have to handle setting the content-type during integration.

Second Attempt - Multiple Javascript Files

Now let's see if we can serve multiple javascript files. PLT Scheme makes this easy with input-port-append:

(require scheme/port)
(define (make-loader root-path)
(lambda (path . paths)
  (define (helper path)
    (open-input-file (build-path root-path path)))
  (apply input-port-append #t (map helper (cons path paths)))))

All of the input-port are now appended together, so when we exhausted the data from the first port, we'll then retrieve the data from the second port, so on.

Adding Compression

So far, so good. Let's now try to see if we can add compression in here.

Let's just say that the YUI compressor live in the same directory as the module, and Java is in the path. And all of the compressed script's path is the same as the source path, with the addition of ".min":

(require scheme/system scheme/string)
(define (make-loader root-path) 
  (lambda (path . paths) 
    (define (min-path-helper full-path)
      (string->path (string-append (path->string full-path) ".min")))
    (define (path-helper path)
      (build-path root-path path))
    (define (helper path)
      (let* ((path (path-helper path))
             (min-path (min-path-helper path)))
        (when (or (not (file-exists? min-path))
                  (< (file-or-directory-modify-seconds min-path)
                     (file-or-directory-modify-seconds path)))
          (system (string-join (list "java" 
                                     "-jar" 
                                     (path->string (build-path (this-expression-source-directory)
                                                               "yuicompressor.jar"))
                                     "-o" (path->string min-path) (path->string path))
                               " ")))
        (open-input-file min-path)))
    (apply input-port-append #t (map helper (cons path paths)))))

The only thing we need to make sure to handle is the error that could result from the system - it returns #f when the call failed, so we need something a bit more verbose (including error messages) in order to throw error about the failed compilation (it is best to have such error messages show up during your development, rather than during production).

Allright - the next step would be for us to integrate this into SHP. Stay tuned.