BZLIB/SHP.plt 0.4 now available - Web API

A new version of SHP.plt is now available via planet.  This is a major rewrite of SHP and provides two main upgrades:

• a "web API" interface - your web script can now be exposed as an "API" (think XMLRPC/JSON), and it automatically works with either XMLRPC or JSON (details below)
  
• general performance enhancement - the scripts are now compiled and cached to reduce disk IO.  If the scripts are updated then they are automatically recompiled
  

As usual, the code is released under LGPL.

Installation 

(require (planet bzlib/shp:1:3)) 

SHP requires some newer dependencies (bzlib/base:1:6, bzlib/date:1:3, bzlib/parseq:1:3, bzlib/xml:1:3, bzlib/mime:1:0), and the current versions of PLT Scheme and Racket have issues with version dependencies (the link: module mismatch bug), so you might have to clear out the planet cache and recompile them again.

As usual, SHP comes with a small example site that you can play with under the example sub directory - cd to the example directory and run (require "web.ss") will start the example site.  The example site is still just trivial code right now - it will eventually be enhanced and separated into its own package.

Cached Compiled Script

All of the scripts are now compiled and cached.  This has some potential performance benefit, since we will only access the file content when the file timestamp changes (meaning the file has been touched and/or modified).  As this is a non-visible feature, we won't spend much time discussing it, except to note that the change is not just done for performance reasons - it is also done to enable and simplify the design of web api, which is discussed below.

Web API

Under the example site you can find the script shp/api/add2, which contains the following:

;; -*- scheme -*- -p 
(:api-args (a number?) (b number?)) 
(+ a b)

This is the new *web api* - it takes in 2 numbers, a and b, and return the added result. To write an api script, you must use the :api-args expression, and then supply the arguments inside. The arguments can be specified in the following forms:

(:api-args a b) ;; both a & b are non-validating and you get what's passed in

(:api-args (a number?) (b string?)) ;; a expects a number, and b expects a string 

(:api-args (a number? 3) (b number? 5)) ;; a & b both expect numbers, and both have default values if they are not passed in (a defaults to 3, and b defaults to 5). 

When you run the example site you can access the api via the following http call:

GET /api/add2 HTTP/1.0 

When running the above in browser you should get back an XMLRPC response:

Content-Type: text/xml; charset=utf-8 

<methodResponse>
<fault>
<value>
<string>required: a</string>
</value>
</fault>
</methodResponse>

XMLRPC is the default response mode for web APIs.  What it returns by default as shown above is an error message, because neither a or b is passed in.

To pass in the values - you just need to specify them in the query string as following:

GET /api/add2?&a=50&b=90 HTTP/1.0 

which will return the following:

Content-Type: text/xml; charset=utf-8

<methodResponse>
<params>
<param>
<value>
<int>140</int>
</value>
</param>
</params>
</methodResponse>

The add2 script contains args of (a number?) and (b number?), which mean that both a & b expects a number input.  So if we pass a non-number to the api

GET /api/add2?&a=not-a-number&b=also-not-a-number HTTP/1.0 

we get the following:

<methodResponse>
<fault>
<value>
<string>invalid-conversion: "not-a-number"</string>
</value>
</fault>
</methodResponse>

Which shows that the validation takes place for each of the arguments.  We will talk about how the underlying validation magic works later.

XMLRPC Request Payload

Now the API doesn't just work for query string parameters - you can pass in an XMLRPC payload as well.  To do so you need to use POST instead of GET, and put the following XMLRPC request into the payload:

POST /api/add2 HTTP/1.0 
Content-Type: text/xml; charset=utf-8 

<methodCall>
<methodName>add2</methodName>
<params>
<param><name>a</name><value><int>50</int></value></param>
<param><name>b</name><value><int>90</int></value></param>
</params>
</methodCall>

Which will return the same result:

Content-Type: text/xml; charset=utf-8

<methodResponse>
<params>
<param>
<value>
<int>140</int>
</value>
</param>
</params>
</methodResponse>

Partial Path Dispatch with XMLRPC methodName

You might have noticed in the above that the token add2 is specified twice in the request - once in the path, and once in the methodName parameter in the payload.  And if you were to use a bogus method name such such as add3 in the methodName parameter, you will see that it is being ignored - the path has precedence.

So - the dispatch rule is - if the path matches the script exactly, the methodName parameter will be ignored.

This is designed to follow the same dispatching rule of regular SHP scripts, and to ensure that one script compiles into one api. 

But if you like to use the regular XMLRPC dispatch, it also works - you just need to remove the add2 from the path, as follows:

POST /api HTTP/1.0 # notice - no add2 here 
Content-Type: text/xml; charset=utf-8 

<methodCall>
<methodName>add2</methodName>
<params>
<param><name>a</name><value><int>50</int></value></param>
<param><name>b</name><value><int>90</int></value></param>
</params>
</methodCall>

So instead of posting to /api/add2, just post to /api, and specify add2 in the methodName parameter.  This is called partial path dispatch, for the lack of a better term for now.

NOTE - in order for partial dispatch to work, the /api directory must not contain an index script, because otherwise the index script will be matched first prior to the partial dispatch kicking in.

Partial Path Dispatch With Query String

There is another way of doing partial path dispatch, and that involve the using of query string.  Yes, you can specify query string with POST requests, and the query string values will be parsed properly in SHP.

The way to do it is to specify a query key that starts with **.  So for example, if we want to dispatch to /api/add2, we can dispatch as follows:

POST /api?**add2 HTTP/1.0 # note the **add2 key in the query string. 
Content-Type: text/xml; charset=utf-8 

<methodCall>
<methodName>add3</methodName><!-- note the wrong method name --> 
<params>
<param><name>a</name><value><int>50</int></value></param>
<param><name>b</name><value><int>90</int></value></param>
</params>
</methodCall>

The above has exactly the same effect as if you have done a direct dispatch - the ** prefix will be stripped, and then appended to the path.  And this rule, along with the direct dispatch, supersedes the method name in the xmlrpc payload.

This design exists for a reason - to deal with web forms that have multiple submit buttons. 

Web forms that have multiple submit buttons usually have each button mapped to different actions, and instead of requiring you to do manual dispatch on the server side based on which button is clicked (the name & value of the submit button clicked gets submitted to the server side along with the data), you can partition the API calls based on the name of the button (partition on name instead of the value is better, because that way it allows you to localize the value without worrying about breaking the script). 

For example, if you have a server-side-based calculator that has the +, -, *, /, = submit buttons, you can theoretically have the following scripts:

/api/add 
/api/subtract
/api/multiply
/api/divide
/api/equal 

And then have the buttons mapped to the name of **add, **subtract, **multiply, **divide, and **equal, and map the form's action to /api.  SHP will do the dispatching for you correctly.

JSON Request & Responses

Besides working with query strings and XMLRPC request and responses, the same api script also can handle JSON requset & responses - you just need to POST the payload with the appropriate content-type, which is text/json.

POST /api/add2 HTTP/1.0 
Content-Type: text/json; charset=utf-8 

{ a : 50 , b : 90 } 

And the response will automatically be a JSON as well:

Content-Type: text/json; charset=utf-8 

140 

Since JSON does not have a method name parameter, it does not have a corresponding partial path dispatch rule, but the query string's partial path dispatch rule remains in effect. 

And of course if you are using JSON you will want to use JSONP, which you can by specifying an additional ~jsonp query string parameter as follows:

POST /api/add2?~jsonp=myCallback HTTP/1.0 
Content-Type: text/json; charset=utf-8 

{ a : 50 , b : 90 } 

Which will generate the following response as a JSONP result:

Content-Type: text/json; charset=utf-8

return myCallback( 140 ); 

That explains the basic rules with regards to using the web api.  The next post will discuss the syntax of the web API, as well as how to extend it so you can use other types.