BZLIB/SHP Web API & Converters

The previous post describes the basics on how to use the web API.  This post will focus on integrating your module with the web API.

As shown before, you can create a web API by creating an SHP script as follows:

;; /api/add2 
(:api-args (a number?) (b number?)) 
(+ a b) 

Where both a and b are validated as number?.  It would be nice if we can validate any type of scheme values, as long as the value can be created via the request input. 

For example - let's say that you have a struct with the following definition:

(define-struct foo (bar baz)) 

We want to do the following:

(:api-args (foo foo?)) ;; takes in the foo struct 

And let web API handle the rest.   This is achieved via converters.

Converters

The mappings between the request and the api args are done via converters, which maps the parameter key against the type's test function (such as number?).   And when you want to use the converter in the api-args expression, you specify the <test?> function in the parameter position in one of the following forms:

(<name> <test?> <default>) ;; this form means this parameter is an optional parameter, and if no values are passed in it will return the default value.

(<name> <test?>) ;; this form means the parameter is a required parameter – if no value passed in it will error.  Any value passed in will be validated and converted (if it fails the conversion it will error)

<name> ;; this form means the parameter does not have any validation – any value will be passed verbatim. 

Because the <test?> function is used as the mapping to the actual underlying converter object, we will call the underlying converter object as the "<test?> converter". i.e., the actual converter mapped by number? is called the "number? converter" (the quotes are dropped going forward).

You can define your own converters to use in the api-args expression to validate your own objects.  To do so, you can define either a scalar or a struct converter as of bzlib/shp:1:3. 

Scalar vs. Struct Converters

There are two different types of converters - scalar and struct. They differ in the type of input required. This is best explained with a JSON request.

Scalar maps to the JSON number and strings, while struct maps to JSON objects (JSON arrays are automatically mapped to lists).

As an example - the following JSON object has a scalar value for both foo and bar key, but a struct value for the baz key:

{ foo : 1 , bar : "test me" , baz : { abc : 1 , def : 2 } } 

So we will use a scalar converter for both foo and bar field (number? and string? respectively), and we will use a struct converter for the baz field that takes in an abc and def fields as number?.

The mapping is pretty straightforward for XMLRPC request as well, except that XMLRPC request method call does not handle named parameters, so we will map the arguments by positions. Otherwise, XMLRPC's string and integers maps to scalars, and struct maps to struct converters (the XMLRPC array maps to lists automatically).

The mapping in query-based request is a bit more complicated, since the query string is only consisted of key/value pairs, it does not handle nested hierarchy of objects by default. We solve this problem by using the dot notation (familiar to most OOP developers) to simulate the hierarchy in the key name themselves. As an example, the query string below maps to the JSON object above:

&foo=1&bar=test%20me&baz.abc=1&baz.def=2

The baz JSON object is flattened into baz.abc and baz.def key/value pairs. This flattening can be done for arbitrary levels.

Hence - you need to ensure that the request are constructed correctly based on the above rules. The rules for JSON & XMLRPC are specified in their corresponding specs, and the query-string rules are specified here.

Define a Scalar Converter

To define a scalar converter, we use the define-scalar-converter! syntax.

(define-scalar-converter! <test?> 
  (<type?> <transform-from-type?>) ...) 

For example – the following is the definition of the number? converter:

(define-scalar-converter! number? (string? string->number)) 

It roughly means the following:

(lambda (x) 
  (cond ((number? x) x) 
        ((string? x) 
         (let ((x (string->number x))) 
           (if x
               x 
               (error 'invalid-conversion "~s" x))))
        (else (error 'invalid-type "~s" x))))

If the passed in value is already the desired type, we just let it pass through.  Otherwise we test to see if it is one of the known types (or error out), and try to convert the value.  If the converter succeeds, return the value, otherwise, throw an exception.

Although the other type of converters is called struct converter, we can define a scalar converter for a struct as well, as long as the struct can be mapped from a scalar value. NOTE - you cannot define both a scalar and struct converter for a struct, since all converters shares the same namespace.

Here's an example of a scalar converter for a struct – we will create a scalar converter for url? from net/url.

(define-scalar-converter! url? 
  (string? string->url) 
  (bytes? (compose string->url bytes->string/utf-8))

Define Struct Converters

To define a struct converter, we use the define-struct-converter! syntax.

(define-struct-converter! name ((field converter?) ...)) 

For example – the following defines a struct converter for the foo struct above:

(define-struct-converter! foo ((bar number?) (baz string?))) 

The struct converter looks like the struct form in provide/contract, but instead of contract expressions, each field is accompanied by a converter instead.

The field converter must already have been defined, or else the definition will fail and throw an exception.

The field converter can of course be either a scalar converter or a struct converter as well.

Converter Definition Orders and Other Design Decisions

Converters should be defined in the order from the most general types to the most specific types, similarly to how object hierarchies are defined.  For example, if you have a sub struct, you should define the converter for the sub struct after you have defined the converter for the parent struct.  And if you want to have converter for both number? and integer?, you should first define the number? converter and then define the integer? converter.

One of the design decisions for the converters is that once a converter is defined it is immutable – future definitions are simply ignored.

Furthermore, both scalar converter and struct converter shares the same namespace.

The above two points means that you can only define either a scalar converter or a struct converter for any given type, and once it is defined it cannot be changed.

Where to Define Converters

Since the design philosophy behind bzlib/shp is that shp scripts should be used for presentations and the application logics should reside in regular PLT Scheme/Racket module/packages, converters are best defined in the respective modules.

However, for ad hoc purposes, the required script (the single shp script that is responsible for loading the required modules) has been extended with bzlib/shp 0.4 so it can now handle ad hoc defintions as well. So if you want to define converters in the required script you can do so, but remember that if it seems like these definition should be shared they might need to eventually migrate into their respective modules.

That's it for now.  Feel free to leave comment if there are any questions.  Enjoy.