BZLIB/SHP Web API & Converters

The previous post describes the basics on how to use the web API.  This post will focus on integrating your module with the web API.

As shown before, you can create a web API by creating an SHP script as follows:

;; /api/add2 
(:api-args (a number?) (b number?)) 
(+ a b) 

Where both a and b are validated as number?.  It would be nice if we can validate any type of scheme values, as long as the value can be created via the request input. 

For example - let's say that you have a struct with the following definition:

(define-struct foo (bar baz)) 

We want to do the following:

(:api-args (foo foo?)) ;; takes in the foo struct 

And let web API handle the rest.   This is achieved via converters.

Converters

The mappings between the request and the api args are done via converters, which maps the parameter key against the type's test function (such as number?).   And when you want to use the converter in the api-args expression, you specify the <test?> function in the parameter position in one of the following forms:

BZLIB/SHP.plt 0.4 now available - Web API

A new version of SHP.plt is now available via planet.  This is a major rewrite of SHP and provides two main upgrades:

• a "web API" interface - your web script can now be exposed as an "API" (think XMLRPC/JSON), and it automatically works with either XMLRPC or JSON (details below)
  
• general performance enhancement - the scripts are now compiled and cached to reduce disk IO.  If the scripts are updated then they are automatically recompiled
  

As usual, the code is released under LGPL.

Installation 

(require (planet bzlib/shp:1:3)) 

SHP requires some newer dependencies (bzlib/base:1:6, bzlib/date:1:3, bzlib/parseq:1:3, bzlib/xml:1:3, bzlib/mime:1:0), and the current versions of PLT Scheme and Racket have issues with version dependencies (the link: module mismatch bug), so you might have to clear out the planet cache and recompile them again.

As usual, SHP comes with a small example site that you can play with under the example sub directory - cd to the example directory and run (require "web.ss") will start the example site.  The example site is still just trivial code right now - it will eventually be enhanced and separated into its own package.

Cached Compiled Script

All of the scripts are now compiled and cached.  This has some potential performance benefit, since we will only access the file content when the file timestamp changes (meaning the file has been touched and/or modified).  As this is a non-visible feature, we won't spend much time discussing it, except to note that the change is not just done for performance reasons - it is also done to enable and simplify the design of web api, which is discussed below.

Web API

Under the example site you can find the script shp/api/add2, which contains the following:

;; -*- scheme -*- -p 
(:api-args (a number?) (b number?)) 
(+ a b)

This is the new *web api* - it takes in 2 numbers, a and b, and return the added result. To write an api script, you must use the :api-args expression, and then supply the arguments inside. The arguments can be specified in the following forms:

(:api-args a b) ;; both a & b are non-validating and you get what's passed in

(:api-args (a number?) (b string?)) ;; a expects a number, and b expects a string 

(:api-args (a number? 3) (b number? 5)) ;; a & b both expect numbers, and both have default values if they are not passed in (a defaults to 3, and b defaults to 5). 

When you run the example site you can access the api via the following http call:

GET /api/add2 HTTP/1.0 

When running the above in browser you should get back an XMLRPC response:

Content-Type: text/xml; charset=utf-8 

<methodResponse>
<fault>
<value>
<string>required: a</string>
</value>
</fault>
</methodResponse>

XMLRPC is the default response mode for web APIs.  What it returns by default as shown above is an error message, because neither a or b is passed in.

To pass in the values - you just need to specify them in the query string as following:

BZLIB/FLEXER 0.1 - FLEX Integration with SHP - Now Available

bzlib/flexer is now available through planet.

As described in the FLEX/SHP posts, bzlib/flexer provides integration between FLEX and SHP, allowing you to do the following:

1. directly embed flash video files by calling (flash <url;> ...)
2. directly code in MXML within SHP and compile into flash video  with (mxml ...)
   
3. optimizing the compilation of MXML

The package is released under LGPL. 

Prerequisite

You'll need the Adobe Flex SDK 3 downloaded and installed separately, and make sure that the flex compiler mxmlc is on the system path.

You'll also need bzlib/shp, with the planet version >= 1.2. 

Installation & Configuration


Install via planet with

(require (planet bzlib/shp:1:2))
(require (planet bzlib/flexer))

Make sure to require bzlib/flexer in your SHP required script as well:

Flex Compiler Integration Optimization

This is a continuation of the FLEX Integration Series - please refer to the previous posts for more details:

1. Building FLEX Integration with SHP
2. FLEX Compiler Integration with SHP

Now we'll cover how to optimize the integration for performance.

The easiest way to determine whether a new compilation is necessary is by comparing the timestamp of the source code vs the timestamp of the object code - even the venerable make utilize this simple check.

So what we need to do is to determine the timestamp of the shp script that we call mxml, and then compare it against the flash video file.  Since mxmlc compilation time is long, if the timestamp of the shp source script has a later timestamp than the flash video, it is very safe to assume that the source script have been changed.

Determining the Timestamp 

PLT Scheme offers (this-expression-source-directory) and (this-expression-source-file-name) to help determine the location.  But as those are macros and applies to the location of their presence, they are not exactly helpful to our cause unless we want to write them out everytime we have to use mxml like this:

(mxml (this-expression-source-directory) path ...)  

We want the value to be automatically available rather than manually available, so we'll have to improve our SHP handler.

(define __PATH__ (make-parameter (shp-handler-path ($server)))) ;; looks like C macro... 
 (define (evaluate-script path)

  (evaluate-terms (file->values path) path))


(define (evaluate-terms terms path)
  (require-modules! terms) ;; first register the required modules 
  ;; then we filter out the required statement and evaluate the rest of the terms as a proc. 
  (eval `(lambda ,(terms->args terms) 
           (parameterize ((__PATH__ path))
             . ,(terms->exps terms)))
        handler-namespace))

Now we have the path available within the SHP script scope, and we can use it to check the timestamp.

Flex Compiler Integration with SHP

This is a continuation of Building FLEX Integration with SHP - please refer to it for refresher.

The next step of the integration is to do FLEX development directly within SHP. If you are used to develop FLEX apps in IDE such as Flex Builder, you might not necessary need this capability.  But consider the following:

• potential for modularizing the flex scripts
• potential for graceful degradations into ajax or pure html 
• possibility for automating the generation of simple flex-based solutions 

 All of which are powerful building blocks for web development, so we'll give it a shot.  Let's get started.

Goal

We want to be able to write in MXML and ActionScripts in SHP, and have the script being compiled in real-time into the flash movie and served to browser transparently.

Example:

Here's a mxml shp script for hello world:

(mx:app (mx:script "private function clickHandler(evt:Event):void {
    messageDisplay.text = \"I am Glad, it does.\";
}")
     (mx:label "Flex without Flex Builder")
     (mx:button #:label "Yes, It Works!" #:click "clickHandler(event)")
     (mx:label #:id "messageDisplay"))

Which should be compiled into the following mxml:

<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml">
<mx:Script>
<![CDATA[
private function clickHanlder(evt:Event):void {
    messageDisplay.text = "I am Glad, it does.";
}
 ]]>
</mx:Script>
<mx:Label text="Flex without Flex Builder" />
<mx:Button label="Yes, It Works!" click="clickHanlder(event)" />
<mx:Label id="messageDisplay" />
</mx:Application> 

And then compiled into a flash video file, and finally served to the client via the (flash) inclusion.  And we should check the timestamp of the SHP script against the compile flash video so we won't waste CPU cycle by compiling the same script over and over again.

The first thing to do is to ensure we can generate the correct MXML.  It is mostly straight forward:

(define (mx:app . widgets) 
  `(mx:Application ((xmlns:mx "http://www.adobe.com/2006/mxml"))
                   . ,widgets))

(define (mx:label (text "") #:id (id #f))
  `(mx:Label ((text ,text)
              ,@(if (not id) '() `((id ,id))))))

(define (mx:button #:label (label "") #:click (click #f))
  `(mx:Button ((label ,label) 
               ,@(if (not click) '() `((click ,click))))))




;; more mxml widget definitions  

With the above we now can generate the MXML, albeit in an incomplete form.  The next step would then be to get the generated mxml *saved* to a predefined location, instead of serving them.

Building Flex Integration with SHP

Besides AJAX, FLEX is the other major player in the RIA world (Silverlight might be a future contender, but it currently is not).  We already provided basic integration with Javascripts, now we want to provide the same level of integration with FLEX.


The lightest layer of integration would be to manage the flex files as static resources, and just make sure they work well within SHP.  But we can also opt for a closer integration by having SHP compile the flash video files, and make the compilation seamless.  Let's develop the light integration first.

What we want are the following:

• keep the flex video files in a static directory similar to other resources, i.e. in the $htdocs/flex/ directory
• simplify the generation of the <object> tag to load the appropriate video file

The tag should be as simple as the following SHP script:

;;  foo.shp ;; loads the foo.swf flash video
(flash "foo" #:width 550 #:height 400) 

Which should then be translated into the following xexpr:

<object width="550" height="400" id="foo" > 

<param name="allowscriptaccess" value="always" /> 

<param name="movie" value="/flex/foo.swf" />
<embed src="/flex/foo.swf" width="550" height="400" name="foo" 

      type="application/x-shockwave-flash" allowscriptaccess="always" >
</embed></object>

This is quite straight forward:

(define flash-base-path (make-parameter "/flash")) 

(define (path->id path)
  (string-join (path-helper path) "_"))

(define (path-helper path)
  (cond ((pair? path) path)
        ((string? path) (regexp-split #px"\\/" path ))
        ((path? path) (path-helper (path->string path)))))

(define (flash-url path)
  (string-join (cons (flash-base-path) (path-helper path)) "/"))

(define (flash (path ($pathinfo)) #:id (id #f) #:base (base (flash-base-path)) #:height (height 400) #:width (width 400))
  (define (helper path id width height)
  `(object ((width ,width) 
            (height ,height) 
            (id ,id))
           (param ((name "movie") (value ,path)) "")
           (param ((name "allowscriptaccess") (value "always")) "")
           (embed ((src ,path)
                   (width ,width)
                   (height ,height)
                   (name ,id)
                   (type "application/x-shockwave-flash")
                   (allowscriptaccess "always")
                   ) "")))
  (helper (flash-url path)
          (if (not id) (path->id path) id)
          (number->string width)
          (number->string height)))

With the ability to generate the flash embedding xexpr, we now just need to create the static flash directory, require the module (we'll call it bzlib/flexer), and then use the above shp script in our code.

We'll talk about the tighter integration next time.  Cheers.

SHP 0.2 and JSMGR 0.1, HTTP 0.1 Are Now Available

SHP 0.2 is now available through planet, along with JSMGR 0.1 and HTTP 0.1. SHP 0.2 adds a number of enhancements on top of SHP 0.1 toward making SHP simple to use:

• simpler startup
  
• default and configurable chrome - so you can switch the UI chrome depending on the page
  
• proxy integration - a built-in http/s proxy so you can use AJAX with ease
  
• JSMGR, which allows you to develop javascript and css in modular fashion but yet serve them combined and compressed; it is optionally available (you'll have to require it) in SHP
  
• HTTP client abstraction, which include https protocol - this is the basis of proxy
  

All code are released under LGPL.

Simple Startup

Now to start a new site - you just need to do:

;; startup.ss or startup.scm
(require (planet bzlib/shp/start))
(start-shp-server! <path> #:htdocs <htdocs-path> ...)

The following parameters are available:

• path: this is the root directory for your SHP scripts
  
• #:port: the port of the server - default to 8080
  
• #:default: this is the path of the default script for each directory - defaults to "index"
• #:not-found: the not-found script for the site (within the SHP root directory) - default to #f as it is optional
  
• #:required: the required script for the site - default to /include/required - the site will function if the script does not exist within the SHP directory
  
• #:topfilter: the topfilter script for the site - default to #f. If it is specified it must exist or an error will occur
  
• #:chrome: the chrome script for the site - default to #f. If specified it must exist.
  

Default and Configurable Chrome

Instead of using topfilter to factor out the common chrome, you can now specify a default chrome (through the startup) and dynamically change chrome within the scripts (NOTE - there is only one chrome for a given request).

To specify the default chrome - do it when you startup the shp server via the #:chrome parameter. Make sure the chrome script does exist.

To disable chrome within the script, issue the following:

;; inside your shp script
($chrome #f)

To change chrome to another chrome script, issue the following:

;; inside your shp script
($chrome "/new/chrome/path")

A chrome script takes in one parameter, which is the actual results evaluated of all of the inner scripts (so it differs from the topfilter's parameter, which is a procedure). Remember chrome is evaluated last of all scripts, so please ensure there aren't evaluation order dependencies between chrome and other scripts.

Proxy Integration
A HTTP proxy is built-in by default in SHP to help you integrate with other web services (your own or a third party's). All you need to do is use the following script:

;; /proxy script
(proxy!)

And it'll automatically convert the pathinfo into the target URL, and pass the rest of the data in query string or post to pass it to the URL.

Headers are currently filtered, and only headers with the prefix of "bzl-" are passed through (with the "bzl-" prefix stripped).

This is designed for using with AJAX, where there is issue of the same-origin policy, so you can query web services residing in different domains without issues. It is not a full web browser proxy (yet).

Proxy is built on top of bzlib/http, which will be described later.

JSMGR Integration

An optional integration is available with JSMGR, which helps you to develop javascript and css scripts in modular fashion but allows you to serve them in combined and compressed mode.

To use JSMGR, add the following to your required script:

(require (planet bzlib/jsmgr/shp)) 

If you have not previously downloaded JSMGR - the above will do it for you the first time you run the code in SHP script, but since JSMGR is quite large as it includes YUI compressor, it's probably best to do so in DrScheme or mzscheme REPL.

Then you can serve js and css with the following code:

;; either /js or /css script
(js/css! #:base <your-javascript-or-css-path-here>)

And you can use helpers to generate the URL for you:

;; in your shp scripts
(script* <js1> <js2> ...) ;; generates a <script> xexpr
(css* <css1> <css2> ...) ;; generates a <link> xexpr

They generate the block by utilizing a known url path (i.e., /js for javascript and /css for css) - if you decide to use a different url you'll have to reset the following in your topfilter script:

;; topfilter script
(parameterize ((js-base-path "/new/js/url")
                        (css-base-path "/new/css/url"))
...)

JSMGR is distributed with YUI compressor, which is licensed by Yahoo! under BSD. I do not believe there are license compatibility issues, but let me know if that is not the case.

You can switch to a different compressor of your choice, as long as you implement a compress! procedure that has the following signature:
(-> path-string? path-string? any). The first is the path to the actual script (need to exist), and the second is the path for the compressed script (need not exist).

JSMGR also comes with a servlet-util module that allow you to integrate JSMGR into a regular servlet rather than SHP. This is not a well tested feature, however, so let me know if you run into issues.

;; your servlet
(require (planet bzlib/jsmgr/servlet-util))
(define start (make-start <base-path-to-the-javascript-or-css-directory>))
(provide start)

HTTP Client Abstraction

SHP proxy depends on bzlib/http, which will be automatically installed if you install SHP. You can use it as a replacement for the net/url's get-impure-port and post-impure-port. Besides working with the http and file protocol, http-get and http-post also work with https protocol.

(require (planet bzlib/http/client))
(http-get <url> <list-of-headers>)
(http-post <url> <data> <list-of-headers>)

The list of headers is a list of string pairs so the headers are easy to construct. The data for http-post is a list of bytes, so you'll have to manually construct the values for now.

bzlib/http also depends on bzlib/net, which provides additional utilities to deal with network code. This module will currently not be supported so I won't document how it can be used.

That's it for SHP 0.2 - happy SHP'ing.

Using JSMGR to Compress a Single Script

JSMGR as currently designed handles the compression and combination of several scripts very well. But what if you just want to access a single script?

Currently in order to access a single script you'll have to use the following URL:

http://<host>/js?s=path/to/js-file.js

It is just a tag bit more inconvenient than

http://<host>/js/path/to/js-file.js

Fortunately it is simple for us to handle pathinfos:

;; the javascript! loader will depend on a having a paticular setting
(define (js/css! (scripts ($query* "s")) #:base (base (current-directory)) #:compress! (compress! yui-compress!))
 (define (scripts-helper)
   (open-js/css-files/base
    (if (not (null? ($pathinfo)))
        (cons (string-join (filter (lambda (segment)
                                     (not (string=? segment "")))
                                   ($pathinfo))
                           "/")
              scripts)
        scripts)
    base
    compress!))
 (raise
  (http-client-response->response
   (make-http-client-response "1.1"
                              200
                              "OK"
                              '(("Content-Type" . "text/javascript"))
                              (scripts-helper))
   (lambda (x) x))))

Now if we want to pass a single script "normally" - we can do that.

This single script support is currently only available in the shp adapter, not the servlet adapter.

A Unified Startup Script

Up to now we have kept the startup script in two separate places: one is the servlet.ss so it can be readily used as a servlet, and the other is web.ss that will serve the single servlet. Satisfying our adherence to the Pareto principle, we will support out of the box only the single servlet method, so we'll combine the two files into one.

By combining the two into one we'll also solve an interesting issue: up to now shp-handler does not really have access to the htdocs path, since that value is consumed by the serve/servlet but is not really passed to make-shp-handler in the separated approach. Now we'll remedy this problem:

(define (start-shp-server! path
                       #:htdocs (htdocs #f)
                       #:port (port 8080)
                       #:default (default "index")
                       #:not-found (not-found #f)
                       #:required (required "/include/required")
                       #:topfilter (topfilter #f)
                       #:chrome (chrome #f))
(define start
(make-shp-handler path
                  #:default default
                  #:not-found not-found
                  #:required required
                  #:topfilter topfilter
                  #:chrome chrome
                  #:htdocs htdocs))
(thread (lambda ()
        (serve/servlet start #:port port
                       #:servlet-path "/"
                       #:launch-browser? #f
                       #:listen-ip #f
                       #:servlet-namespace (list (path->string (this-expression-file-name)))
                       #:servlets-root path
                       #:servlet-regexp #px".*"
                       #:server-root-path path
                       #:extra-files-paths (list (if (path? htdocs)
                                                     htdocs
                                                     (string->path htdocs)))
                       ))))

The single function now creates the shp-handler and start web-server at the same time. All that's left to do is to pass the value of the htdocs into shp-handler. We want to be able to access the value in shp script as follows:

($htdocs)

So to accomplish the following, we will first add an additional attribute to shp-handler struct:

(define-struct shp-handler (... htdocs)
...)

Then we will modify the constructor function to take in an htdocs parameter:

(define (*make-shp-handler path
                       ...
                       #:htdocs (htdocs #f))
(make-shp-handler path
                ...
                (if (not htdocs)
                    (build-path path 'up "file")
                    htdocs)
                ))

Which defaults to the the "file" directory in the same directory that "shp" directory live in.

Finally - we have the accessor function to be used in the SHP scripts:

(define ($htdocs)
(shp-handler-htdocs ($server)))

Lastly - let's make sure we have error checking to test for both the shp and htdocs path's existence.

(define (*make-shp-handler path
                        ...
                        #:htdocs (htdocs #f))
  (unless (directory-exists? path)
 (error 'make-shp-handler "path ~a does not exist." path))
(let ((htdocs
      (let ((htdocs (if (not htdocs)
                        (build-path path 'up "file")
                        htdocs)))
        (if (directory-exists? htdocs)
            htdocs
            (error 'make-shp-handler "htdocs path ~a does not exist." htdocs)))))
   ....))

Now we can simple do the following to start the site with any path.

(require (planet bzlib/shp/start))
(start-shp-server! <path> #:htdocs <htdocs-path>)

Make the Not Found Path Optional

Currently you'll have to supply the not-found path for the app to work, this slows down the creation of the site, so we'll try to make it optional.

The idea is simple - since the package includes a not-found script, we should use that by default when it is not specified.

(define (*make-shp-handler path
                         #:default (default "index")
                         #:not-found (not-found #f)
                         #:required (required "required")
                         #:topfilter (topfilter #f)
                         #:chrome (chrome #f))
(make-shp-handler path
                  default
                  (if (not not-found)
                      (make-script (build-path (this-expression-source-directory) "example" "shp" "notfound") 0)
                      not-found)
                  (init-script required path) topfilter chrome))

we'll also have to change the segment->path and segment->partial-path function to ensure make them play well with having a script struct as the value of not-found:

(define (not-found-path base)
(let ((not-found (shp-handler-not-found ($server))))
  (if (script? not-found)
      (script-path not-found)
      (build-path base not-found))))

;; partially matching the path from the beginning of the path
(define (segments->partial-path segments)
(define (helper rest path default)
  ;; if we did not find any match return not-found
  (cond ((null? rest)
         (not-found-path (shp-handler-path ($server)))) ...)
...)

(define (segments->path segments (partial? #t))
....
        (else
         (not-found-path script)))))

So now if you decide not to specify the not-found script path, the code will still work.

Manage CSS through SHP and JSMGR

Besides javascript, CSS is another type of text resource that suffer same problems that javascript does:

• lack of modularity
  
• lack of compression
  

The problem is almost exactly the same that we should be able to use JSMGR to handle CSS as well!

Luckily - YUI compressor works with both javascript and CSS, so by using YUI compressor our work is limited (you can of course implement your own compress! routine). What we need is to add the signature specific to CSS, we have a couple of approaches:

1. have css-* functions that matches the javascript-* functions
   
2. change the signature of javascript-* functions to it does not have the type of the script in its name, and instead, takes it in a #:type parameter
   

It seems that at the compression level, approach #2 is more logical, especially since YUI compressor works with either simply by the file extension type. So instead of open-javascript-files and open-javascript-files/base, we should have open-js/css-files and open-js/css-files/base. And javascript! will also be renamed as js/css!.

The only trouble we will encounter at this time with this approach is that when we need to implement another compressor we'll have to follow YUI's interface, but since that's out of scope, that's worry for another day.

At the xexpr-generator level, though, it makes sense to have the two types separated, since by default they generate different signatures:

• js-url/xexpr generates <script> block, but we'll call it script* to match it more with xexpr
  
• css-url/xexpr generates <link rel=styleet> block, and we'll call this css* to match it more with xexpr
  

So, we'll adapt the signature accordingly:

(define css-base-path (make-parameter "/css"))

(define (css-url path . paths)
 (let ((url (string->url (css-base-path))))
   (set-url-query! url (map (lambda (path)
                              (cons 's path))
                            (cons path paths)))
   (regexp-replace* #px"(\\&)([^a])" (url->string url) ";\\2")))

(define (css* path . paths)
 `(link ((rel "stylesheet") (type "text/css") (href ,(apply css-url path paths)))))

Now we can handle CSS with JSMGR as well!

Handling UI Chromes - Make it Switchable

You might have noticed that since topfilter gets executed with every single request, it is a nice place to specify the UI chrome so we do not need to separately specify it for every page.

The drawback with such approach is that once you have placed the chrome into topfilter, it'll accompany all of the responses (unless you raise an error or a response object to escape the continuation), regardless whether you intend it or not.

In majority of the cases, you'll be just fine, since most of the web pages would want to have such a chrome. But what if you are generating non-html pages? For example, if you want to dynamically generate a javascript file, or if you want to return an XML data (i.e. setting up a web-service). In these situation, a default chrome that runs 100% of the time defeats the purpose. Hence we want to have a "switchable" chrome that is defaulted to some value most of the time, but can be turned off as necessary.

The first thing we need again is a parameter:

(define $chrome (make-parameter #f))

What should go into the parameter is the path of the chrome file.

Then in order to execute the chrome, we'll wrap around the inner handler with:

(define (make-chrome-based-handler inner) 
  (lambda () 
    (let ((result (inner))) 
      (if ($chrome) 
          (include! ($chrome) result)
          result))))

(define (include! path 
                  #:topfilter (topfilter #f)
                  #:partial? (partial? #f)
                  . args) 
  (define (make-script path partial?)
    (evaluate-script (if (script? path)
                         (script-path path)
                         (resolve-path path partial?))))
  (define (helper topfilter)
    (let ((proc (make-script path partial?)))
      (let ((chrome-handler (make-chrome-based-handler (lambda () (apply proc args)))))
        (if topfilter 
            (topfilter chrome-handler)
            (chrome-handler)))))
  (helper (if topfilter (make-script topfilter #f) topfilter)))
;; adding the chrome parameter
(define-struct shp-handler (path default not-found required topfilter chrome)
  #:property prop:procedure
  (lambda ($struct request)
    (handle-request $struct request)))
;; parameterize the chrome parameter
(define (handle-request server request)
  (parameterize (($server server) ...)
    (parameterize (... 
                   ($chrome (shp-handler-chrome server)))
       ...)))

With the above, we can then put chrome into its own file:

;; a default chrome 
(:args inner) 
`(html (body ,inner)) ;; NOTE inner is not a procedure 

You'll notice that chrome looks a lot like the topfilter. And yes, in a way, chrome is a specialized filter, but it differ from the topfilter in that the inner is not a procedure call. Instead, the values are first evaluated and finally pass to chrome. This way we can modify the $chrome parameter to disable it half way in the code.

So - when you use chrome you will need to remember that it is evaluated last. This generally should not matter, but it does when your code have order dependencies. Make sure your chrome will work as the last thing being executed in your code.

Productionize the Chrome Code

The above chrome code has a serious bug - it causes infinite recursion. The reason is that make-chrome-base-handler calls include! and include! calls make-chrome-base-handler, so we cannot use include! inside make-chrome-base-handler:

(define (make-chrome-based-handler inner) 
  (lambda () 
    (let ((result (inner))) 
      (if ($chrome) 
          ;; I see - chrome itself now will cause a recursion!! 
          ((evaluate-script (resolve-path ($chrome) #f)) result)
          result))))

With the above, your handler will return, but if you use include! in your script you'll notice they also have the chrome applied to them - our include! does not know when chrome should be applied (only at the outermost layer), so we'll also need to fix that:

(define (include! path 
                  #:topfilter (topfilter #f)
                  #:partial? (partial? #f)
                  #:chrome? (chrome? #f)  ;; disable by default.
                  . args) 
  (define (make-script path partial?)
    (evaluate-script (if (script? path)
                         (script-path path)
                         (resolve-path path partial?))))
  (define (helper topfilter)
    (let ((proc (make-script path partial?)))
      (let ((handler
             (let ((handler (lambda () (apply proc args))))
               (if chrome? (make-chrome-based-handler handler) handler))))
        (if topfilter 
            (topfilter handler)
            (handler)))))
  (helper (if topfilter (make-script topfilter #f) topfilter)))

And we need to ensure the outermost layer is called with chrome enabled:

(define (handle-request server request)
   ... 
        (make-response (include! (request-uri request) 
                                 #:topfilter (shp-handler-topfilter server)
                                 #:chrome? #t
                                 #:partial? #t))))))

Now we have a default and switchable chrome, and to turn the chrome off, just call ($chrome #f).

Managing your Javascripts through SHP (4) - Shorthand for Generating the JSMGR querystring

To really simplify the development it would be nice to be able to call something like the following to issue the javascript reference:

;; in any SHP script 
`(script ((src ,(js-link "jquery.js" "jquery.ui.js" ...))) "")

and it will translate into the appropriate link. Let's see how it can be done.

Assuming the path is hardcoded (let's say "/js/"), then it's pretty straight forward:

(define js-base-path (make-parameter "/js"))

(define (js-url path . paths) 
  (let ((url (string->url (js-base-path))))
    (set-url-query! url (map (lambda (path)
                               (cons 's path))
                             (cons path paths)))
    (url->string url)))

Given the base path is set in a parameter, you can parameterize it to a different value. It would be nice to "automatically" set the js-base-path based on where you call the value, but that might be more difficult than it's worth so it's out of scope for now.

Managing your Javascripts through SHP (3) - Integrate with Servlets

If you do not want to use SHP but would like to use jsmgr, it provides a servlet adapter that you can use - just do the following in your servlet:

;; your servlet... 
(require (planet bzlib/jsmgr/servlet-util)) 
(define start
  (make-start "the-path-to-javascript-here..."))

And the servlet will have the exact same functionality. The following is the servlet-util.ss that enabled the above:

(require web-server/http/request-structs
         web-server/http/response-structs
         net/url
         mzlib/etc
         scheme/contract
         (planet bzlib/shp:1:1/request)         
         (planet bzlib/shp:1:1/proxy)         
         (planet bzlib/http/client)
         "loader.ss"
         )

(define (request-helper request)
  (parameterize (($request request))
    ($query* "s")))

;; we want to take the request's query object (which are quite available through our definitions...) 
;; and then map it to 
(define (make-start base) 
  (lambda (request)
    (let ((scripts (request-helper request)))
      (http-client-response->response 
       (make-http-client-response "1.1" 
                                  200 
                                  "OK" 
                                  '(("Content-Type" . "text/javascript; charset=utf-8"))
                                  (open-javascript-files/base scripts base))
       (lambda (x) x)))))

(provide/contract 
 (make-start (-> path-string? (-> request? response/c)))
 )

Managing your Javascripts through SHP (2) - Integrate with SHP

Since we now can retrieve the combined scripts, the integration basically will focus on mapping the request query values to the scripts, and return the combined, mini-fied scripts as the response. Let's try to do that within SHP.

We'll use s as the query key name. And it can be repeated multiple times. What we want is to write an SHP script that looks like the following:

;; js
;; loading javascripts
;; default to using s key, but can be customized
(javascript!) ;; or (javascript! ($query* "s"))

The definition of javascript! will roughly look like the following:

(define (javascript! (scripts ($query "s")) #:base (base (current-directory)))
(http-client-response->response
(make-http-client-response "1.1"
                        200
                        "OK"
                        '(("Content-Type" . "text/javascript; charset=utf-8"))
                        (apply javascript-loader #:base base
                               scripts)) ;; this is a mismatch
(lambda (x) x)))

And we encounter our first mismatch - our original loader was a closure over the base path. While we can still create the closure as previously designed, the closure is wasted since we are not keeping it around. It's simpler to have a new interface that takes in the base path - and we'll take the opportunity to refactor the code a bit:

(define (yui-compress! path min-path)
(system (string-join
       (list "java"
             "-jar"
             (path->string (build-path (this-expression-source-directory)
                                       "yuicompressor.jar"))
             "-o" (path->string min-path) (path->string path))
       " ")))

(define (open-javascript-files paths (compress! yui-compress!))
(define (min-path-helper path)
(string->path (string-append (path->string path) ".min")))
(define (helper path)
(let ((min-path (min-path-helper path)))
  (when (or (not (file-exists? min-path))
            (> (file-or-directory-modify-seconds path)
               (file-or-directory-modify-seconds min-path)))
    (compress! path min-path))
  (open-input-file min-path)))
(apply input-port-append #t (map helper paths)))

(define (open-javascript-files/base base paths (compress! yui-compress!))
(define (helper path)
(build-path base path))
(open-javascript-files (map helper paths) compress!))

Now yui-compress! is extracted from the loader, and two versions of the loader are exposed: open-javascript-files and open-javascript-files/base, which takes in an additional base argument. And they both take in an optional compress! argument that you can use to switch out yui-compress! if you want to use another javascript compressor.

Now the javascript! handler would look like:

;; the javascript! loader will depend on a having a paticular setting
(define (javascript! (scripts ($query* "s")) (base (current-directory)))
(raise
(http-client-response->response
 (make-http-client-response "1.1"
                            200
                            "OK"
                            '(("Content-Type" . "text/javascript; charset=utf-8"))
                            (open-javascript-files/base scripts base)
                            (lambda (x) x)))))

And we now can add the module (we are calling this bzlib/jsmgr/shp) to our SHP instance:

;; required.shp
(require (planet bzlib/jsmgr/shp))

And the script.shp should contain the following:

;; script.shp
(javascript! #:base "some-path-here...")

Testing it out give us a cryptic error:

Exception

The application raised an exception with the message:

open-input-file: cannot open input file: "../collects/planet/main.ss" (The system cannot find the file specified.; errno=2)

Although the error is cryptic, we know it arises from our SHP handler's require-modules! procedure. It turns out that because the procedure calls flatten, we have flattened (planet bzlib/jsmgr) into two separate collections and hence caused the error. Below fixes the error:

(define (require-modules! terms)
  (define (require! module)
    (namespace-require module))
  (define (helper listof-modules)
    (parameterize ((current-namespace handler-namespace))
      (for-each (lambda (modules)
                  (for-each require! modules))
                listof-modules)))
  (helper (map cdr (filter require-exp? terms))))

Now the Javascript Manager is available through SHP.

Managing your Javascripts through SHP

If you are doing extensive AJAX development (which many people are these days), you probably are writing quite a bit of OOP Javascript code, along with using libraries such as Prototype, jQuery, Dojo, etc.

The trouble with javascript is that it lacks modern module management facilities that we have come to expect from programming languages, especially given its importance in the browser world. You either develop all of your code in one large script, or you risk the browser needing to download numerous small scripts, which is inefficient. Furthermore, if you plan on using javascript compressors such as the YUI compressor, if you make updates to the source script you would have to recopmress the code manually.

It would be nice to have a tool that can help with the above issues:

• allows you to code in multiple small scripts, but automatically combine the smaller scripts into a single script as you specify
  
• automatically helps you recompress the code if you make changes to them, so you just need to worry about the source (which will not be served directly unless specified)
  

Let's look at what we need to accomplish the goal:

• ability to load and combine multiple files into a single response
  
• ability to detect timestamp of the source and the compressed scripts, and re-compress if the source script's timestamp is newer than the compressed scripts

You might have come across "script builders" such as jQuery UI, and Mootools - you can use them as a mental model for the goal (for your own scripts!).

Compressor Choice

Currently the best choice of javascript compressor appears to be YUI compressor (if you are not trying to obfuscate the scripts too much) as it provides the best combination of compression ratio as well as the decompression speed (obfuscators such as Packer loses speed during the unpack phase), so it is our choice for compressor. It is licensed under BSD license, so we can redistribute the binary without issues. In the future we might add support for other compressors, but for now it's out of scope.

Prerequisite

YUI compressor requires Java (version >= 1.4), which you'll have to install separately and make it available in the system path.

First Attempt - a Single Javascript File
Let's see if we can get a simple version of this up and running - let's first just serve out a javascript file sitting somewhere in our system. No compressions. It is quite straight forward.

(define (make-loader root-path)
(lambda (path)
  (open-input-file (build-path root-path path))))

The basic signature will be make-loader, which will take a path and returns a procedure we can then use to load the file. If the file does not exist an appropriate error will be thrown (which we'll have to handle once integrating into servlets). We'll also have to handle setting the content-type during integration.

Second Attempt - Multiple Javascript Files

Now let's see if we can serve multiple javascript files. PLT Scheme makes this easy with input-port-append:

(require scheme/port)
(define (make-loader root-path)
(lambda (path . paths)
  (define (helper path)
    (open-input-file (build-path root-path path)))
  (apply input-port-append #t (map helper (cons path paths)))))

All of the input-port are now appended together, so when we exhausted the data from the first port, we'll then retrieve the data from the second port, so on.

Adding Compression

So far, so good. Let's now try to see if we can add compression in here.

Let's just say that the YUI compressor live in the same directory as the module, and Java is in the path. And all of the compressed script's path is the same as the source path, with the addition of ".min":

(require scheme/system scheme/string)
(define (make-loader root-path) 
  (lambda (path . paths) 
    (define (min-path-helper full-path)
      (string->path (string-append (path->string full-path) ".min")))
    (define (path-helper path)
      (build-path root-path path))
    (define (helper path)
      (let* ((path (path-helper path))
             (min-path (min-path-helper path)))
        (when (or (not (file-exists? min-path))
                  (< (file-or-directory-modify-seconds min-path)
                     (file-or-directory-modify-seconds path)))
          (system (string-join (list "java" 
                                     "-jar" 
                                     (path->string (build-path (this-expression-source-directory)
                                                               "yuicompressor.jar"))
                                     "-o" (path->string min-path) (path->string path))
                               " ")))
        (open-input-file min-path)))
    (apply input-port-append #t (map helper (cons path paths)))))

The only thing we need to make sure to handle is the error that could result from the system - it returns #f when the call failed, so we need something a bit more verbose (including error messages) in order to throw error about the failed compilation (it is best to have such error messages show up during your development, rather than during production).

Allright - the next step would be for us to integrate this into SHP. Stay tuned.

Continuing of HTTP Call & Proxy Integration (4) - Customize Responses

If you are doing AJAX you know that the XMLHttpRequest object is quite brittle - different browsers offer different behaviors and bugs. One such bug is that IE does not properly the more complicated cousins of text/xml, such as application/atom+xml. In such situations it would be nice to have the proxy customizing the response before it is passed back to the browser.

The simplest way of accomplishing conversion of content-type is to take an keyword parameter:

(define (proxy! (url ($pathinfo)) (headers ($headers))
                #:content-type (content-type (lambda (x) x)))
  (define (helper url headers)
    (raise 
     (http-client-response->response
      (case ($method)
        ((post) (http-post url (request-post-data/raw ($request)) headers))
        ((get) (http-get url headers))
        (else (error 'proxy "proxy method ~a not supported" ($method))))
      content-type)))
  (call-with-values 
   (lambda () 
     (normalize-url+headers url headers))
   helper))

This means we should also modify http-client-response->response:

(define (http-client-response->response r content-type)
  (define (get-content-type r)
    (define (helper header)
      (string->bytes/utf-8 
       (content-type (if (not header) 
                         "text/html; charset=utf-8"
                         (cdr header)))))
     (helper (assf (lambda (key)
                     (string-ci=? key "content-type"))
                   (http-client-response-headers r))))
  (define (normalize-headers r)
    (map (lambda (kv)
           (make-header (string->bytes/utf-8 (car kv))
                        (string->bytes/utf-8 (cdr kv))))
         (http-client-response-headers r)))
  (define (make-generator)
    (lambda (output) 
      (let loop ((b (read-bytes 4096 r)))
        (cond ((eof-object? b)
               (void))
              (else
               (output b)
               (loop (read-bytes 4095 r)))))))
  (make-response/incremental (http-client-response-code r)
                             (string->bytes/utf-8 (http-client-response-reason r))
                             (current-seconds)
                             (get-content-type r)
                             (normalize-headers r)
                             (make-generator)))

And now our proxy can be customized if we want to filter out extended XML headers:

;; proxy.shp
(proxy! #:content-type 
 (lambda (ct) 
  (if (regexp-match #px"^application/.+xml.*$" ct)
      "text/xml"
      ct)))

Continuing of HTTP Call & Proxy Integration (3)

To satisfy the requirement of proxy, the following serves as the basic skeletion:

(define (proxy! (url ($pathinfo)) (headers ($headers)))
  (case ($method)
    (("post") (http-post (url-helper url) (request-post-data/raw ($request)) headers))
    (("get") (http-get (url-helper url) headers))
    (else (error 'proxy "proxy method ~a not supported" ($method)))))

As we can see, we already have most of the HTTP connection code defined, except we need to ensure all of the custom headers, as well as the user/password credentials are properly retrieved. For that we need to massage the url and the headers.

Filter Out Non-Custom Headers

The first step is to filter out the non-custom headers:

(define (custom-header? header)
  (string-ci=? "bzl-" (substring (car header) 0 (max (string-length (car header)) 4))))

(define (convert-header header)
    (cons (regexp-replace #px"^bzl-(.+)$" (car header) "\\1")
          (cdr header)))

(define (headers->custom-headers headers) 
  (map convert-header (filter custom-header? header)))

We also want to keep some regular headers such as Content-Type and Content-Length:

(define (headers->custom-headers headers) 
  (append (filter (lambda (header)
                    (or (string-ci=? (car header) "content-type")
                        (string-ci=? (car header) "content-length")))
                  headers)
          (map convert-header (filter custom-header? headers))))

Extract User Credential from URL

In case the user credentials is supplied in user:password form in the path, we want to extract it and push it onto the headers:

(define (url-helper url)
  (cond ((url? url) url)
        ((string? url) (string->url url))
        (else ;; this is based on pathinfo... 
         (let ((url (string-join url "/")))
           ;; keep the url query 
           (set-url-query! url (url-query ($uri)))
           url))))

(define (url->auth-header url) 
  ;; helper removes the additional \r\n appended by base64-encode
  (define (remove-extra-crlf auth) 
    (substring auth 0 (- (string-length auth) 2)))
  (if (not (url-user url))
      #f 
      (cons "Authorization"
            (string-append "Basic "
                           (remove-extra-crlf 
                            (bytes->string/utf-8
                             (base64-encode
                              (string->bytes/utf-8 (url-user url)))))))))

And finally we modify proxy! to use the newly generated url & headers:

(define (normalize-url+headers url headers) 
  (let ((url (url-helper url))
        (headers (headers->custom-headers headers)))
    (let ((auth (url->auth-header url))) ;; in case the auth info is passed in via url. 
      (let ((headers (if (not auth) headers
                         (cons auth headers))))
        (values url headers)))))

(define (proxy! (url ($pathinfo)) (headers ($headers)))
  (define (helper url headers)
    (case ($method)
      (("post") (http-post url (request-post-data/raw ($request)) headers))
      (("get") (http-get url headers))
      (else (error 'proxy "proxy method ~a not supported" ($method)))))
  (call-with-values 
   (lambda () 
     (normalize-url+headers url headers))
   helper))

Convert the Responses

Since the http-get and http-post returns http-client-response instead of response/basic, we'll have to have an adapter to convert from one type to another, and once it's converted, we can handle it the same way as redirect! to raise the response/basic.

(define (http-client-response->response r)
  (define (get-content-type r)
    (define (helper header)
      (if (not header) 
          #"text/html; charset=utf-8"
          (string->bytes/utf-8 (cdr header))))
    (helper (assf (lambda (key)
                    (string-ci=? key "content-type"))
                  (http-client-response-headers r))))
  (define (normalize-headers r)
    (map (lambda (kv)
           (make-header (string->bytes/utf-8 (car kv))
                        (string->bytes/utf-8 (cdr kv))))
         (http-client-response-headers r)))
  (define (make-generator)
    (lambda (output) 
      (let loop ((b (read-bytes 4096 r)))
        (cond ((eof-object? b)
               (void))
              (else
               (output b)
               (loop (read-bytes 4095 r)))))))
  (make-response/incremental (http-client-response-code r)
                             (string->bytes/utf-8 (http-client-response-reason r))
                             (current-seconds)
                             (get-content-type r)
                             (normalize-headers r)
                             (make-generator)))
 
(define (proxy! (url ($pathinfo)) (headers ($headers)))
  (define (helper url headers)
    (raise 
     (http-client-response->response
      (case ($method)
        ((post) (http-post url (request-post-data/raw ($request)) headers))
        ((get) (http-get url headers))
        (else (error 'proxy "proxy method ~a not supported" ($method)))))))
  (call-with-values 
   (lambda () 
     (normalize-url+headers url headers))
   helper))

Fixing the URL

The code so far almost works, except that since empty strings are stripped from pathinfo, we'll get http:/www.google.com instead of http://www.google.com when we reconstruct the pathinfo into url, so we'll fix that by ensuring an additional empty path is reintroduced:

(define (join-url segments) 
  (define (helper segments)
    (string-join segments "/"))
  (cond ((null? segments) (error 'join-url "invalid segments: ~a" segments))
        ((string-ci=? (car segments) "http:")
         (helper (list* (car segments) "" (cdr segments))))
        ((string-ci=? (car segments) "https:")
         (helper (list* (car segments) "" (cdr segments))))
        (else 
         (helper (list* "http:" "" (cdr segments))))))
(define (url-helper url)
  (cond ((url? url) url)
        ((string? url) (string->url url))
        (else ;; this is based on pathinfo... 
         (let ((url (string->url (join-url url))))
           ;; keep the url query 
           (set-url-query! url (url-query ($uri)))
           (display (format "~a\n" (url->string url)) (current-error-port))
           url))))

With these it is now easy to add proxy capability into SHP:

;; proxy
(proxy!) 

Voila - how you can use proxy to wrap around internal or 3rd party web services!

Continuing of HTTP Call & Proxy Integration (2)

In the last post we basically have a working rudimentary HTTP client. It should not be a lot of work to create a rudimentary proxy out of the HTTP client.

Basically - we want to more or less pass the request to the destination web service, with as minimal fuss as possible.

By default, the proxy call should look something like the following:

http://host:port/http://target/path
http://host:port/https://target2/path2

But of course we also want to allow for hardcoded path to possibly map directly to another url, for example:

http://host:port/google => http://www.google.com/

In order for the above to work, we would want to have scripts called http: and https: located at the SHP root path with the following syntax:

;; http: or https:
(proxy! (pathinfo->url))

But unfortunately http: and https: are not valid file names in Windows, so the above technique cannot be used with Windows (but should be usable with Linux). and we'll settle for the following:

http://host:port/proxy/http://target/path
http://host:port/proxy/https://target2/path2

Then the code for proxy should be:

;; proxy
(proxy! (path->info->url (cdr ($pathinfo))))

Custom Headers

Many web services (such as Twitter, Del.icio.us, and Blogger) do not take all of the client headers with their APIs, since they are not expecting the browser making direct connections and hence do not need all of the headers issued from browsers.

What this means is that we should filter out most of the headers except needed ones, a simple way to determine what's needed is via custom headers. We'll choose to create our custom headers via prefix of bzl-.

So, any headers started with bzl- shall be passed onto the target, and other ones (except mandatory headers such as Content-Type, Content-Length) should be filtered out. This way it would allow fine grained control by us.

Since all of the headers are already available in $headers parameter, the conversion should be automatic.

Authentication & Passing User Credentials Securely

A special header is the Authorization header, which contains the user's login and password. HTTP Basic is basically insecure, and HTTP Digest will require a challenge and a response - for now we'll only handle passing HTTP Basic.

There are a couple of ways of passing user credentials - one is via the above mentioned custom header mechanism (i.e. bzl-Authorization), and the other is via the user field of the URL value, i.e.

http://user:password@host/path...

Then within our proxy it looks like:

http://host:port/proxy/http://user:password@host/path...

Since either way is insecure without a SSL, it does not really matter which way is preferred. But what we need to make sure is not to use the regular Authorization header, since that should be used for potentially authenticating against the proxy itself.

Review of Requirements

Let's reiterate the requirements:

• a pass through of request to the proxied target, and a pass through of responses
• take the URL from the pathinfo of the request
  
• very simple usage within SHP (i.e. no fancy setup of anything)
• a clean way of passing through headers
  
• a clean (& mostly secure) way of passing credentials for the 3rd party web service
  

Let's see how we can develop such a proxy to match the requirements. To be continued...

Continuing of Integration Features - HTTP Call & Proxy

In order to fulfill the role of an integrator, we want to add the ability to make HTTP calls to other services (either within your own network or 3rd party web services). The reason such capability is needed is simple: unless we plan on developing everything under the sun, we'll need to interface with other applications that provides crucial and non-trivial services to speed up your development.

PLT Scheme already provides basic URL fetching capability through the net/url module, which can serve as a basis for our development. Let's get started.

HTTPS Calls

The first thing to note is that net/url does not support SSL connections, so we'll need to add the support ourselves.

(require openssl
        scheme/tcp
        net/url
        mzlib/trace
        scheme/contract
        )

;; https->impure-port
;; base function to handle the connection over https
(define (https->impure-port method url (headers '()) (data #f))
 (let-values (((s->c c->s) (ssl-connect (url-host url)
                                        (if (url-port url) (url-port url) 443)))
              ((path) (make-url #f #f #f #f
                                (url-path-absolute? url)
                                (url-path url)
                                (url-query url)
                                (url-fragment url))))
   (define (to-server fmt . args)
     (display (apply format (string-append fmt "\r\n") args) c->s))
   ;; (trace to-server)
   (to-server "~a ~a HTTP/1.0" method (url->string path))
   (to-server "Host: ~a:~a" (url-host url)
              (if (url-port url) (url-port url) 443))
   (when data
     (to-server "Content-Length: ~a" (bytes-length data)))
   (for-each (lambda (header)
               (to-server "~a" header)) headers)
   (to-server "")
   (when data
     (display data c->s))
   (flush-output c->s)
   (close-output-port c->s)
   s->c))

;; get-impure-port/https
;; a GET version of https call
(define (get-impure-port/https url (headers '()))
 (https->impure-port "GET" url headers))

;; post-impure-port/https
;; a POST version of https call
(define (post-impure-port/https url data (headers '()))
 (https->impure-port "POST" url headers data))

The above provided get-impure-port/https and post-impure-port/https, which mimics net/url's get-impure-port and post-impure-port. We are only interested in impure port as we want to be able to manipulate the headers, since it's trivial to add pure port on top of it.

With the above we can then create an abstraction over both https & http url fetching:

;; http-client-response holds all of the metadata (code, status, headers)
;; as well as the data stream
(define-struct http-client-response (version code reason headers input)
  #:property prop:input-port 4)

;; read-http-status
;; parse the http-status of the response.
(define (read-http-status in) 
  (define (helper match)
    (if match
        (list (cadr match) (caddr match) (cadddr match))
        match))
  (define (reader in)
    (read-folded-line in))
  (trace reader)
  (helper (regexp-match #px"^HTTP/(\\d\\.\\d)\\s+(\\d+)\\s+(.+)$" (reader in))))

;; a helper over the make-http-client-response
(define (*make-http-client-response in) 
  (define (helper version code reason)
    (make-http-client-response version (string->number code) reason (read-headers in) in))
  (let ((status (read-http-status in))) 
    (if (not status)
        (error 'make-http-client-response "invalid http response")
        (apply helper status)))) 
;; helper over url conversion 
(define (url-helper url)
  (if (string? url) (string->url url)
      url)) 

;; converting headers over to headers that can be used by get/post-impure-port 
(define (headers-helper headers)
  (map (lambda (kv) 
         (format "~a: ~a" (car kv) (cdr kv))) 
       headers))

;; http-get 
;; abstraction over http GET 
(define (http-get url (headers '())) 
  (define (helper url) 
    (*make-http-client-response 
     ((if (string-ci=? (url-scheme url) "https")
          get-impure-port/https
          get-impure-port) 
      url (headers-helper headers))))
  (helper (url-helper url)))

;; http-post 
;; abstarction over http POST 
(define (http-post url data (headers '())) 
  (define (helper url)
    (*make-http-client-response 
     ((if (string-ci=? (url-scheme url) "https")
          post-impure-port/https
          post-impure-port)
      url data (headers-helper headers))))
  (helper (url-helper url)))

The abstraction more of less follows the net/url's approach, except that it wraps around both http & https procedures, adding convenient header handlings, as well as providing an abstraction over the http response to parse through all of the metdata, but yet still retain their values (unlike get/post-pure-port which gets rid of all of the status and headers).

Reading and Parsing RFC822 Headers

RFC822 compliant headers requires non-trivial treatment. While the concept of headers that's made of key/values appear simple, in fact they are not for many historical reasons that are well captured in all of the RFC's. Below are a list of things that we need to be aware of:

• RFC822 headers might span multiple lines in the style of "folded line" (the line continues if the following line starts with non-terminating whitespace, which includes #\space and #\tab), and it might keep going indefinitely
  
• The header values might contain comments (enclosed in parentheses), which are nestable, and generally the comments should be ignored but might not be (for example - many server generate date fields with comment to denote the timezone)
  
• Because of the traditional SMTP line width limitations, generating headers might require breaking the line into the folded line along line width limitation (which generally is around 70)
• Also due to the traditional ASCII oriented nature of network protocols, there are two additional encodings defined (called Q and B) that parsers should be able to handle appropriately in order to correctly parse headers
  

It would be cool to support all of the above capabilities, but given we are only using headers in HTTP situation (which generally are not subjected to the SMTP line & encoding limits) we'll limit ourselves to the following for the immediate purpose:

• generating a single line per header
• parse a folded line per header, but do not handle encodings
• assume the character set to be UTF-8 (otherwise throw errors)
  

The following handles the folded line:

;; read-folded-line
;; read folded line according to RFC822.
(define (read-folded-line in) 
  (define (folding? c)
    (or (equal? c #\space)
        (equal? c #\tab)))
  (define (return lines) 
    (apply string-append "" (reverse lines)))
  (define (convert-folding lines)
    (let ((c (peek-char in)))
      (cond ((folding? c) 
             (read-char in)
             (convert-folding lines))
            (else
             (helper (cons " " lines))))))
  (define (helper lines)
    (let ((l (read-line in 'return-linefeed)))
      (if (eof-object? l) 
          (return lines)
          (let ((c (peek-char in)))
            (if (folding? c) ;; we should keep going but first let's convert all folding whitespaces... 
                (convert-folding (cons l lines))
                ;; otherwise we are done... 
                (return (cons l lines)))))))
  (helper '()))

Then to read all of the headers is to read in all of the folded lines until we encounter an empty line (which would either be EOF or the separator between headers and the data).

;; a header is simply a pair of strings... 
(define (header? h)
  (and (pair? h)
       (string? (car h))
       (string? (cdr h))))

;; header->string: does not generate terminator 
(define (header->string h) 
  (format "~a: ~a" (car h) (cdr h)))

;; string->header
;; convert a string into a header?
(define (string->header line)
  (define (helper match)
    (if match
        (cons (cadr match) (caddr match))
        #f))
  (helper (regexp-match #px"^([^:]+)\\s*:\\s*(.+)$" line)))
;; reading header.
;; RFC822 headers are actually non-trivial for parsing purposes.  
;; first it requires the handling of "folded line", which means that any line 
;; that does not end directly in 
(define (read-headers in) 
  (define (return lines)
    (map line->header lines))
  (define (helper lines) 
    (let ((l (read-folded-line in))) 
      (if (string=? l "") ;; we are done... 
          (return lines)
          (helper (cons l lines)))))
  (helper '()))

The above should cover the majority of cases that we'll encounter during interactions with web services, and we'll fix issues as we encounter them.

To be continued...