Parsing "Encoded Word" in RFC Headers (3) - Parsing & Decoding

In the previous two posts we have built capabilities to generate encoded words, now it's time to decode them. We'll start with detecting whether a string is an encoded word.

Testing for Encoded Word

Remember that an encoded word has the following format:

=?<charset>?<Q or B>?<encoded data>?=

The above format can be succintly specified via regular expression:

(define encoded-word-regexp #px"=\\?([^\\?]+)\\?(?i:(b|q))\\?([^\\?]+)\\?=")

(define (encoded-word? str)
  (regexp-match encoded-word-regexp str))

Since the format is not recursive, regular expression is good enough, even though it can be considered as ugly. You can certainly try using other approaches, such as a hand written lexer, or using parser-tools to do the job. For now we keep things simple.

Decoding

Once we can test whether a string is an encoded word, we can then use it to handle the decoding:

(define (encoded-word->string str)
  (if-it (encoded-word? str)
         (apply decode-encoded-word (cdr it))
         str))

If the string is an encoded word, we decode it, otherwise we return it verbatim. This way it allows regular string to be passed into this function.

The decode-encoded-word function looks like the following:

Parsing "Encoded Word" in RFC Headers (2) - Charset Handling & Multiple Encoded Words

In the previous post we discussed the Q and B encodings, and ended with a bug on mismatching charset if the charset is not utf-8, let's try to fix the bug here.

It would be nice if we can use local charsets such as iso-8559-1 or big5 if we know for sure that the charset contains all of the characters that appears in the string (of course, it is the developer's responsibility to choose the right charset; the code will error out if the charset does not match the data).

PLT Scheme provides a convert-stream to help handle converting bytes from one charset to another. We can build helpers that takes strings or bytes and return string or bytes on top of this function. What we want are something like:

(bytes/charset->string #"this is a string" "ascii") ;; => returns a string
(bytes/charset->bytes/utf-8 <bytes> <charset>) ;; => returns a bytes

The idea is that we'll convert the input data to input-port, and then retrieve the data from the output-port, which will be a bytes port.

So let's start with a helper function that'll take in an input-port, and the charsets and then return a bytes:

(define (port->bytes/charset in charset-in charset-out)
  (call-with-output-bytes 
   (lambda (out)
     (convert-stream charset-in in charset-out out))))

Then we can have the following:

(define (bytes->bytes/charset bytes charset-in charset-out)
  (port->bytes/charset (open-input-bytes bytes) charset-in charset-out))

And we can define converting bytes to and from utf-8:

(define (bytes/charset->bytes/utf-8 bytes charset)
  (bytes->bytes/charset bytes charset "utf-8")) 

(define (bytes/utf-8->bytes/charset bytes charset)
  (bytes->bytes/charset bytes "utf-8" charset))

And finally we can then return strings on top of these two functions:

;; there are more to handle (specifically charsets).
(define (bytes/charset->string bytes charset)
  (bytes->string/utf-8 (bytes/charset->bytes/utf-8 bytes charset)))

(define (string->bytes/charset string charset)
  (bytes/utf-8->bytes/charset (string->bytes/utf-8 string) charset))

With the above functions, we can now ensure to convert the encoded word into the correct charset:

Parsing "Encoded Word" in RFC Headers

If you want to correctly handle internet message headers as defined in RFC822 or as improved by RFC2822, you'll find that you currently have no way of handling encoded words, which is defined separately in RFC1342.

Below is the example of encoded words in message headers from RFC1342:

From: =?US-ASCII?Q?Keith_Moore?= <moore@cs.utk.edu>
To: =?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= <keld@dkuug.dk>
CC: =?ISO-8859-1?Q?Andr=E9_?= Pirard <PIRARD@vm1.ulg.ac.be>
Subject: =?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=
 =?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?=

which should be decoded into

From: Keith Moore <moore@cs.utk.edu>
To: Keld Jørn Simonsen <keld@dkuug.dk>
CC: André Pirard <PIRARD@vm1.ulg.ac.be>
Subject: If you can read this you understand the example.

But currently, net/head cannot handle the encode words and they are not parsed:

(extract-all-fields <the-above-string>)
;; => 
'(("From" . "=?US-ASCII?Q?Keith_Moore?= ")
 ("To" . "=?ISO-8859-1?Q?Keld_J=F8rn_Simonsen?= ")
 ("CC" . "=?ISO-8859-1?Q?Andr=E9_?= Pirard ")
 ("Subject"
  .
  "=?ISO-8859-1?B?SWYgeW91IGNhbiByZWFkIHRoaXMgeW8=?=\r\n =?ISO-8859-2?B?dSB1bmRlcnN0YW5kIHRoZSBleGFtcGxlLg==?="))

So we'll need to handle it ourselves. Let's get started.

The Format of an Encoded Word

An encoded word has the following format:

=?<charset>?<Q or B>?<encoded data>?=

And an encoded word should not exceed 75 bytes (including all the delimiters). If the string being encoded cannot fit in the length, then multiple encoded words should be separated by space or linear folding whitespace (\r\n\s*). Encoded words can coexist with plain text in the same header (shown above in the Cc header).

Developing a Memcached Client (1) - Storage

[This is a continuation of the network development in PLT post]

Memcached is all the rage these days with the high scalability guys.  Legends have it that as soon as you slap a couple of memcached in between your web app and your database server, all of your scalability issues go away.  Even if that's glossing over of details, memcached certainly offers advantages over accessing the discs all the time, and we want it in PLT.

Let's try to build a memcached client.

Steps to Build a Client (or a Server)

The steps to build either a client or a server is pretty straight forward (independent of the complexity of the protocol):

1. study the protocol 
2. write the "output functions" that sends the info to the other party 
3. write the "input functions" that parses the input from the other party 
4. combine the input & output functions, and manage the states of the connection (if applicable)

You are likely to mix the steps up instead of doing them in the order prescribed above and that's okay.  The steps are numbered to help us to think of them as an orderly process, but the process is likely more iterative.

Allright - let's get started.

Study the Protocol 

The key to build a network interface is to understand the protocols.  It is best if the protocol is open and published, so you have something written that you can study (hopefully in a language you understand).  The next best option is to reverse engineer from an open source implementation, and the worst is to reverse engineer from sniffing the network interactions between the clients and the server.  I am glad such memcached's protocol is already published so we do not have to go down the other routes.

Please refer to the memcached protocols throughout this process as the official reference.  We'll sprinkle information from the official doc in the posts as necessary.

First a quick overview of the memcached protocol.

• request/response protocol - there are no unexpected responses that the client has to handle 
• multiple requests and responses within a single session 
• *mostly* line-based protocol - the commands are single lines that are terminated by CRLF; the data is also terminated by CRLF, but it is actually determined by length parameter
• can store, retrieve, and delete the content (as well as flush) 
• there are also statistics that can be retrieved, but the protocol appears in flux for the stats (so it'll be out of scope for now) 
• the protocol itself is *stateless* as each of the requests and responses are independent from each other (each request is atomic) 

Overall the protocol is simple yet non-trivial, which gives us a great example for network development.  We'll start with the storage part of the protocol.

Storage API

Memcached has multiple storage commands for different situations:

SHP 0.2 and JSMGR 0.1, HTTP 0.1 Are Now Available

SHP 0.2 is now available through planet, along with JSMGR 0.1 and HTTP 0.1. SHP 0.2 adds a number of enhancements on top of SHP 0.1 toward making SHP simple to use:

• simpler startup
  
• default and configurable chrome - so you can switch the UI chrome depending on the page
  
• proxy integration - a built-in http/s proxy so you can use AJAX with ease
  
• JSMGR, which allows you to develop javascript and css in modular fashion but yet serve them combined and compressed; it is optionally available (you'll have to require it) in SHP
  
• HTTP client abstraction, which include https protocol - this is the basis of proxy
  

All code are released under LGPL.

Simple Startup

Now to start a new site - you just need to do:

;; startup.ss or startup.scm
(require (planet bzlib/shp/start))
(start-shp-server! <path> #:htdocs <htdocs-path> ...)

The following parameters are available:

• path: this is the root directory for your SHP scripts
  
• #:port: the port of the server - default to 8080
  
• #:default: this is the path of the default script for each directory - defaults to "index"
• #:not-found: the not-found script for the site (within the SHP root directory) - default to #f as it is optional
  
• #:required: the required script for the site - default to /include/required - the site will function if the script does not exist within the SHP directory
  
• #:topfilter: the topfilter script for the site - default to #f. If it is specified it must exist or an error will occur
  
• #:chrome: the chrome script for the site - default to #f. If specified it must exist.
  

Default and Configurable Chrome

Instead of using topfilter to factor out the common chrome, you can now specify a default chrome (through the startup) and dynamically change chrome within the scripts (NOTE - there is only one chrome for a given request).

To specify the default chrome - do it when you startup the shp server via the #:chrome parameter. Make sure the chrome script does exist.

To disable chrome within the script, issue the following:

;; inside your shp script
($chrome #f)

To change chrome to another chrome script, issue the following:

;; inside your shp script
($chrome "/new/chrome/path")

A chrome script takes in one parameter, which is the actual results evaluated of all of the inner scripts (so it differs from the topfilter's parameter, which is a procedure). Remember chrome is evaluated last of all scripts, so please ensure there aren't evaluation order dependencies between chrome and other scripts.

Proxy Integration
A HTTP proxy is built-in by default in SHP to help you integrate with other web services (your own or a third party's). All you need to do is use the following script:

;; /proxy script
(proxy!)

And it'll automatically convert the pathinfo into the target URL, and pass the rest of the data in query string or post to pass it to the URL.

Headers are currently filtered, and only headers with the prefix of "bzl-" are passed through (with the "bzl-" prefix stripped).

This is designed for using with AJAX, where there is issue of the same-origin policy, so you can query web services residing in different domains without issues. It is not a full web browser proxy (yet).

Proxy is built on top of bzlib/http, which will be described later.

JSMGR Integration

An optional integration is available with JSMGR, which helps you to develop javascript and css scripts in modular fashion but allows you to serve them in combined and compressed mode.

To use JSMGR, add the following to your required script:

(require (planet bzlib/jsmgr/shp)) 

If you have not previously downloaded JSMGR - the above will do it for you the first time you run the code in SHP script, but since JSMGR is quite large as it includes YUI compressor, it's probably best to do so in DrScheme or mzscheme REPL.

Then you can serve js and css with the following code:

;; either /js or /css script
(js/css! #:base <your-javascript-or-css-path-here>)

And you can use helpers to generate the URL for you:

;; in your shp scripts
(script* <js1> <js2> ...) ;; generates a <script> xexpr
(css* <css1> <css2> ...) ;; generates a <link> xexpr

They generate the block by utilizing a known url path (i.e., /js for javascript and /css for css) - if you decide to use a different url you'll have to reset the following in your topfilter script:

;; topfilter script
(parameterize ((js-base-path "/new/js/url")
                        (css-base-path "/new/css/url"))
...)

JSMGR is distributed with YUI compressor, which is licensed by Yahoo! under BSD. I do not believe there are license compatibility issues, but let me know if that is not the case.

You can switch to a different compressor of your choice, as long as you implement a compress! procedure that has the following signature:
(-> path-string? path-string? any). The first is the path to the actual script (need to exist), and the second is the path for the compressed script (need not exist).

JSMGR also comes with a servlet-util module that allow you to integrate JSMGR into a regular servlet rather than SHP. This is not a well tested feature, however, so let me know if you run into issues.

;; your servlet
(require (planet bzlib/jsmgr/servlet-util))
(define start (make-start <base-path-to-the-javascript-or-css-directory>))
(provide start)

HTTP Client Abstraction

SHP proxy depends on bzlib/http, which will be automatically installed if you install SHP. You can use it as a replacement for the net/url's get-impure-port and post-impure-port. Besides working with the http and file protocol, http-get and http-post also work with https protocol.

(require (planet bzlib/http/client))
(http-get <url> <list-of-headers>)
(http-post <url> <data> <list-of-headers>)

The list of headers is a list of string pairs so the headers are easy to construct. The data for http-post is a list of bytes, so you'll have to manually construct the values for now.

bzlib/http also depends on bzlib/net, which provides additional utilities to deal with network code. This module will currently not be supported so I won't document how it can be used.

That's it for SHP 0.2 - happy SHP'ing.

Managing your Javascripts through SHP (4) - Shorthand for Generating the JSMGR querystring

To really simplify the development it would be nice to be able to call something like the following to issue the javascript reference:

;; in any SHP script 
`(script ((src ,(js-link "jquery.js" "jquery.ui.js" ...))) "")

and it will translate into the appropriate link. Let's see how it can be done.

Assuming the path is hardcoded (let's say "/js/"), then it's pretty straight forward:

(define js-base-path (make-parameter "/js"))

(define (js-url path . paths) 
  (let ((url (string->url (js-base-path))))
    (set-url-query! url (map (lambda (path)
                               (cons 's path))
                             (cons path paths)))
    (url->string url)))

Given the base path is set in a parameter, you can parameterize it to a different value. It would be nice to "automatically" set the js-base-path based on where you call the value, but that might be more difficult than it's worth so it's out of scope for now.

Continuing of HTTP Call & Proxy Integration (4) - Customize Responses

If you are doing AJAX you know that the XMLHttpRequest object is quite brittle - different browsers offer different behaviors and bugs. One such bug is that IE does not properly the more complicated cousins of text/xml, such as application/atom+xml. In such situations it would be nice to have the proxy customizing the response before it is passed back to the browser.

The simplest way of accomplishing conversion of content-type is to take an keyword parameter:

(define (proxy! (url ($pathinfo)) (headers ($headers))
                #:content-type (content-type (lambda (x) x)))
  (define (helper url headers)
    (raise 
     (http-client-response->response
      (case ($method)
        ((post) (http-post url (request-post-data/raw ($request)) headers))
        ((get) (http-get url headers))
        (else (error 'proxy "proxy method ~a not supported" ($method))))
      content-type)))
  (call-with-values 
   (lambda () 
     (normalize-url+headers url headers))
   helper))

This means we should also modify http-client-response->response:

(define (http-client-response->response r content-type)
  (define (get-content-type r)
    (define (helper header)
      (string->bytes/utf-8 
       (content-type (if (not header) 
                         "text/html; charset=utf-8"
                         (cdr header)))))
     (helper (assf (lambda (key)
                     (string-ci=? key "content-type"))
                   (http-client-response-headers r))))
  (define (normalize-headers r)
    (map (lambda (kv)
           (make-header (string->bytes/utf-8 (car kv))
                        (string->bytes/utf-8 (cdr kv))))
         (http-client-response-headers r)))
  (define (make-generator)
    (lambda (output) 
      (let loop ((b (read-bytes 4096 r)))
        (cond ((eof-object? b)
               (void))
              (else
               (output b)
               (loop (read-bytes 4095 r)))))))
  (make-response/incremental (http-client-response-code r)
                             (string->bytes/utf-8 (http-client-response-reason r))
                             (current-seconds)
                             (get-content-type r)
                             (normalize-headers r)
                             (make-generator)))

And now our proxy can be customized if we want to filter out extended XML headers:

;; proxy.shp
(proxy! #:content-type 
 (lambda (ct) 
  (if (regexp-match #px"^application/.+xml.*$" ct)
      "text/xml"
      ct)))

Continuing of HTTP Call & Proxy Integration (3)

To satisfy the requirement of proxy, the following serves as the basic skeletion:

(define (proxy! (url ($pathinfo)) (headers ($headers)))
  (case ($method)
    (("post") (http-post (url-helper url) (request-post-data/raw ($request)) headers))
    (("get") (http-get (url-helper url) headers))
    (else (error 'proxy "proxy method ~a not supported" ($method)))))

As we can see, we already have most of the HTTP connection code defined, except we need to ensure all of the custom headers, as well as the user/password credentials are properly retrieved. For that we need to massage the url and the headers.

Filter Out Non-Custom Headers

The first step is to filter out the non-custom headers:

(define (custom-header? header)
  (string-ci=? "bzl-" (substring (car header) 0 (max (string-length (car header)) 4))))

(define (convert-header header)
    (cons (regexp-replace #px"^bzl-(.+)$" (car header) "\\1")
          (cdr header)))

(define (headers->custom-headers headers) 
  (map convert-header (filter custom-header? header)))

We also want to keep some regular headers such as Content-Type and Content-Length:

(define (headers->custom-headers headers) 
  (append (filter (lambda (header)
                    (or (string-ci=? (car header) "content-type")
                        (string-ci=? (car header) "content-length")))
                  headers)
          (map convert-header (filter custom-header? headers))))

Extract User Credential from URL

In case the user credentials is supplied in user:password form in the path, we want to extract it and push it onto the headers:

(define (url-helper url)
  (cond ((url? url) url)
        ((string? url) (string->url url))
        (else ;; this is based on pathinfo... 
         (let ((url (string-join url "/")))
           ;; keep the url query 
           (set-url-query! url (url-query ($uri)))
           url))))

(define (url->auth-header url) 
  ;; helper removes the additional \r\n appended by base64-encode
  (define (remove-extra-crlf auth) 
    (substring auth 0 (- (string-length auth) 2)))
  (if (not (url-user url))
      #f 
      (cons "Authorization"
            (string-append "Basic "
                           (remove-extra-crlf 
                            (bytes->string/utf-8
                             (base64-encode
                              (string->bytes/utf-8 (url-user url)))))))))

And finally we modify proxy! to use the newly generated url & headers:

(define (normalize-url+headers url headers) 
  (let ((url (url-helper url))
        (headers (headers->custom-headers headers)))
    (let ((auth (url->auth-header url))) ;; in case the auth info is passed in via url. 
      (let ((headers (if (not auth) headers
                         (cons auth headers))))
        (values url headers)))))

(define (proxy! (url ($pathinfo)) (headers ($headers)))
  (define (helper url headers)
    (case ($method)
      (("post") (http-post url (request-post-data/raw ($request)) headers))
      (("get") (http-get url headers))
      (else (error 'proxy "proxy method ~a not supported" ($method)))))
  (call-with-values 
   (lambda () 
     (normalize-url+headers url headers))
   helper))

Convert the Responses

Since the http-get and http-post returns http-client-response instead of response/basic, we'll have to have an adapter to convert from one type to another, and once it's converted, we can handle it the same way as redirect! to raise the response/basic.

(define (http-client-response->response r)
  (define (get-content-type r)
    (define (helper header)
      (if (not header) 
          #"text/html; charset=utf-8"
          (string->bytes/utf-8 (cdr header))))
    (helper (assf (lambda (key)
                    (string-ci=? key "content-type"))
                  (http-client-response-headers r))))
  (define (normalize-headers r)
    (map (lambda (kv)
           (make-header (string->bytes/utf-8 (car kv))
                        (string->bytes/utf-8 (cdr kv))))
         (http-client-response-headers r)))
  (define (make-generator)
    (lambda (output) 
      (let loop ((b (read-bytes 4096 r)))
        (cond ((eof-object? b)
               (void))
              (else
               (output b)
               (loop (read-bytes 4095 r)))))))
  (make-response/incremental (http-client-response-code r)
                             (string->bytes/utf-8 (http-client-response-reason r))
                             (current-seconds)
                             (get-content-type r)
                             (normalize-headers r)
                             (make-generator)))
 
(define (proxy! (url ($pathinfo)) (headers ($headers)))
  (define (helper url headers)
    (raise 
     (http-client-response->response
      (case ($method)
        ((post) (http-post url (request-post-data/raw ($request)) headers))
        ((get) (http-get url headers))
        (else (error 'proxy "proxy method ~a not supported" ($method)))))))
  (call-with-values 
   (lambda () 
     (normalize-url+headers url headers))
   helper))

Fixing the URL

The code so far almost works, except that since empty strings are stripped from pathinfo, we'll get http:/www.google.com instead of http://www.google.com when we reconstruct the pathinfo into url, so we'll fix that by ensuring an additional empty path is reintroduced:

(define (join-url segments) 
  (define (helper segments)
    (string-join segments "/"))
  (cond ((null? segments) (error 'join-url "invalid segments: ~a" segments))
        ((string-ci=? (car segments) "http:")
         (helper (list* (car segments) "" (cdr segments))))
        ((string-ci=? (car segments) "https:")
         (helper (list* (car segments) "" (cdr segments))))
        (else 
         (helper (list* "http:" "" (cdr segments))))))
(define (url-helper url)
  (cond ((url? url) url)
        ((string? url) (string->url url))
        (else ;; this is based on pathinfo... 
         (let ((url (string->url (join-url url))))
           ;; keep the url query 
           (set-url-query! url (url-query ($uri)))
           (display (format "~a\n" (url->string url)) (current-error-port))
           url))))

With these it is now easy to add proxy capability into SHP:

;; proxy
(proxy!) 

Voila - how you can use proxy to wrap around internal or 3rd party web services!

Continuing of HTTP Call & Proxy Integration (2)

In the last post we basically have a working rudimentary HTTP client. It should not be a lot of work to create a rudimentary proxy out of the HTTP client.

Basically - we want to more or less pass the request to the destination web service, with as minimal fuss as possible.

By default, the proxy call should look something like the following:

http://host:port/http://target/path
http://host:port/https://target2/path2

But of course we also want to allow for hardcoded path to possibly map directly to another url, for example:

http://host:port/google => http://www.google.com/

In order for the above to work, we would want to have scripts called http: and https: located at the SHP root path with the following syntax:

;; http: or https:
(proxy! (pathinfo->url))

But unfortunately http: and https: are not valid file names in Windows, so the above technique cannot be used with Windows (but should be usable with Linux). and we'll settle for the following:

http://host:port/proxy/http://target/path
http://host:port/proxy/https://target2/path2

Then the code for proxy should be:

;; proxy
(proxy! (path->info->url (cdr ($pathinfo))))

Custom Headers

Many web services (such as Twitter, Del.icio.us, and Blogger) do not take all of the client headers with their APIs, since they are not expecting the browser making direct connections and hence do not need all of the headers issued from browsers.

What this means is that we should filter out most of the headers except needed ones, a simple way to determine what's needed is via custom headers. We'll choose to create our custom headers via prefix of bzl-.

So, any headers started with bzl- shall be passed onto the target, and other ones (except mandatory headers such as Content-Type, Content-Length) should be filtered out. This way it would allow fine grained control by us.

Since all of the headers are already available in $headers parameter, the conversion should be automatic.

Authentication & Passing User Credentials Securely

A special header is the Authorization header, which contains the user's login and password. HTTP Basic is basically insecure, and HTTP Digest will require a challenge and a response - for now we'll only handle passing HTTP Basic.

There are a couple of ways of passing user credentials - one is via the above mentioned custom header mechanism (i.e. bzl-Authorization), and the other is via the user field of the URL value, i.e.

http://user:password@host/path...

Then within our proxy it looks like:

http://host:port/proxy/http://user:password@host/path...

Since either way is insecure without a SSL, it does not really matter which way is preferred. But what we need to make sure is not to use the regular Authorization header, since that should be used for potentially authenticating against the proxy itself.

Review of Requirements

Let's reiterate the requirements:

• a pass through of request to the proxied target, and a pass through of responses
• take the URL from the pathinfo of the request
  
• very simple usage within SHP (i.e. no fancy setup of anything)
• a clean way of passing through headers
  
• a clean (& mostly secure) way of passing credentials for the 3rd party web service
  

Let's see how we can develop such a proxy to match the requirements. To be continued...

Continuing of Integration Features - HTTP Call & Proxy

In order to fulfill the role of an integrator, we want to add the ability to make HTTP calls to other services (either within your own network or 3rd party web services). The reason such capability is needed is simple: unless we plan on developing everything under the sun, we'll need to interface with other applications that provides crucial and non-trivial services to speed up your development.

PLT Scheme already provides basic URL fetching capability through the net/url module, which can serve as a basis for our development. Let's get started.

HTTPS Calls

The first thing to note is that net/url does not support SSL connections, so we'll need to add the support ourselves.

(require openssl
        scheme/tcp
        net/url
        mzlib/trace
        scheme/contract
        )

;; https->impure-port
;; base function to handle the connection over https
(define (https->impure-port method url (headers '()) (data #f))
 (let-values (((s->c c->s) (ssl-connect (url-host url)
                                        (if (url-port url) (url-port url) 443)))
              ((path) (make-url #f #f #f #f
                                (url-path-absolute? url)
                                (url-path url)
                                (url-query url)
                                (url-fragment url))))
   (define (to-server fmt . args)
     (display (apply format (string-append fmt "\r\n") args) c->s))
   ;; (trace to-server)
   (to-server "~a ~a HTTP/1.0" method (url->string path))
   (to-server "Host: ~a:~a" (url-host url)
              (if (url-port url) (url-port url) 443))
   (when data
     (to-server "Content-Length: ~a" (bytes-length data)))
   (for-each (lambda (header)
               (to-server "~a" header)) headers)
   (to-server "")
   (when data
     (display data c->s))
   (flush-output c->s)
   (close-output-port c->s)
   s->c))

;; get-impure-port/https
;; a GET version of https call
(define (get-impure-port/https url (headers '()))
 (https->impure-port "GET" url headers))

;; post-impure-port/https
;; a POST version of https call
(define (post-impure-port/https url data (headers '()))
 (https->impure-port "POST" url headers data))

The above provided get-impure-port/https and post-impure-port/https, which mimics net/url's get-impure-port and post-impure-port. We are only interested in impure port as we want to be able to manipulate the headers, since it's trivial to add pure port on top of it.

With the above we can then create an abstraction over both https & http url fetching:

;; http-client-response holds all of the metadata (code, status, headers)
;; as well as the data stream
(define-struct http-client-response (version code reason headers input)
  #:property prop:input-port 4)

;; read-http-status
;; parse the http-status of the response.
(define (read-http-status in) 
  (define (helper match)
    (if match
        (list (cadr match) (caddr match) (cadddr match))
        match))
  (define (reader in)
    (read-folded-line in))
  (trace reader)
  (helper (regexp-match #px"^HTTP/(\\d\\.\\d)\\s+(\\d+)\\s+(.+)$" (reader in))))

;; a helper over the make-http-client-response
(define (*make-http-client-response in) 
  (define (helper version code reason)
    (make-http-client-response version (string->number code) reason (read-headers in) in))
  (let ((status (read-http-status in))) 
    (if (not status)
        (error 'make-http-client-response "invalid http response")
        (apply helper status)))) 
;; helper over url conversion 
(define (url-helper url)
  (if (string? url) (string->url url)
      url)) 

;; converting headers over to headers that can be used by get/post-impure-port 
(define (headers-helper headers)
  (map (lambda (kv) 
         (format "~a: ~a" (car kv) (cdr kv))) 
       headers))

;; http-get 
;; abstraction over http GET 
(define (http-get url (headers '())) 
  (define (helper url) 
    (*make-http-client-response 
     ((if (string-ci=? (url-scheme url) "https")
          get-impure-port/https
          get-impure-port) 
      url (headers-helper headers))))
  (helper (url-helper url)))

;; http-post 
;; abstarction over http POST 
(define (http-post url data (headers '())) 
  (define (helper url)
    (*make-http-client-response 
     ((if (string-ci=? (url-scheme url) "https")
          post-impure-port/https
          post-impure-port)
      url data (headers-helper headers))))
  (helper (url-helper url)))

The abstraction more of less follows the net/url's approach, except that it wraps around both http & https procedures, adding convenient header handlings, as well as providing an abstraction over the http response to parse through all of the metdata, but yet still retain their values (unlike get/post-pure-port which gets rid of all of the status and headers).

Reading and Parsing RFC822 Headers

RFC822 compliant headers requires non-trivial treatment. While the concept of headers that's made of key/values appear simple, in fact they are not for many historical reasons that are well captured in all of the RFC's. Below are a list of things that we need to be aware of:

• RFC822 headers might span multiple lines in the style of "folded line" (the line continues if the following line starts with non-terminating whitespace, which includes #\space and #\tab), and it might keep going indefinitely
  
• The header values might contain comments (enclosed in parentheses), which are nestable, and generally the comments should be ignored but might not be (for example - many server generate date fields with comment to denote the timezone)
  
• Because of the traditional SMTP line width limitations, generating headers might require breaking the line into the folded line along line width limitation (which generally is around 70)
• Also due to the traditional ASCII oriented nature of network protocols, there are two additional encodings defined (called Q and B) that parsers should be able to handle appropriately in order to correctly parse headers
  

It would be cool to support all of the above capabilities, but given we are only using headers in HTTP situation (which generally are not subjected to the SMTP line & encoding limits) we'll limit ourselves to the following for the immediate purpose:

• generating a single line per header
• parse a folded line per header, but do not handle encodings
• assume the character set to be UTF-8 (otherwise throw errors)
  

The following handles the folded line:

;; read-folded-line
;; read folded line according to RFC822.
(define (read-folded-line in) 
  (define (folding? c)
    (or (equal? c #\space)
        (equal? c #\tab)))
  (define (return lines) 
    (apply string-append "" (reverse lines)))
  (define (convert-folding lines)
    (let ((c (peek-char in)))
      (cond ((folding? c) 
             (read-char in)
             (convert-folding lines))
            (else
             (helper (cons " " lines))))))
  (define (helper lines)
    (let ((l (read-line in 'return-linefeed)))
      (if (eof-object? l) 
          (return lines)
          (let ((c (peek-char in)))
            (if (folding? c) ;; we should keep going but first let's convert all folding whitespaces... 
                (convert-folding (cons l lines))
                ;; otherwise we are done... 
                (return (cons l lines)))))))
  (helper '()))

Then to read all of the headers is to read in all of the folded lines until we encounter an empty line (which would either be EOF or the separator between headers and the data).

;; a header is simply a pair of strings... 
(define (header? h)
  (and (pair? h)
       (string? (car h))
       (string? (cdr h))))

;; header->string: does not generate terminator 
(define (header->string h) 
  (format "~a: ~a" (car h) (cdr h)))

;; string->header
;; convert a string into a header?
(define (string->header line)
  (define (helper match)
    (if match
        (cons (cadr match) (caddr match))
        #f))
  (helper (regexp-match #px"^([^:]+)\\s*:\\s*(.+)$" line)))
;; reading header.
;; RFC822 headers are actually non-trivial for parsing purposes.  
;; first it requires the handling of "folded line", which means that any line 
;; that does not end directly in 
(define (read-headers in) 
  (define (return lines)
    (map line->header lines))
  (define (helper lines) 
    (let ((l (read-folded-line in))) 
      (if (string=? l "") ;; we are done... 
          (return lines)
          (helper (cons l lines)))))
  (helper '()))

The above should cover the majority of cases that we'll encounter during interactions with web services, and we'll fix issues as we encounter them.

To be continued...