Thursday, August 13, 2009

Adding "Global" (sortof) Variables...

Let's say we want to create a database connection and we want the database connection available during the whole script execution phase, but we do not want to have to touch all of the scripts to either initialize or pass the variable, what do we do?

Because the variable will be available to the top level scripts as well, it means that the variable needs to be passed through parameters rather than passing through procedure arguments, and because the way parameterize works, we'll need a wrapper function outside of the script. In short, we need a filter.

A filter is a procedure the form of:

(-> handler? response/c)
(define (filter inner) ...)

And basically, we'll pass the handler's function into the filter, and then let the filter do its job, then call the handler, possibly do some post processing, and finally return the result.

In the case of setting the global parameters, the filter will call parameterize and then call inner within the parameterize block.

It is simple to develop such a filter, since our scripts already takes in arguments:

;; somefilter.shp
(:args inner)
(parameterize ((foo (some-value)))
(inner))

And since the script takes an argument - it cannot be invoked directly via the browser. Now we just need to modify the handler to support such a filter. Let's tackle the support for this specific case (of having a top level filter for all scripts).
The first thing we do is the same as adding required module: add an attribute for shp-handler:

(define-struct shp-handler
(path default not-found required topfilter) ...)

Then we want to call the toplevel filterif it is available with something like the following:

(define (envoke-topfilter-if-available inner)
(if (shp-handler-topfilter ($server))
(include! (shp-handler-topfilter ($server)) inner)
(inner)))

(define-struct shp-handler (path default not-found required topfilter)
#:property prop:procedure
(lambda ($struct request)
...
(make-response (if (thunk? proc)
(envoke-topfilter-if-available proc)
(include! (shp-handler-not-found ($server)))))))))

We then also need to update the wrapper *make-shp-handler procedure with the additional parameter. Given we now have four optional parameters and it can get cumbersome to specify them, we'll change them to keyword parameters:

(define (*make-shp-handler path
#:default (default "index.shp")
#:not-found (not-found "notfound.shp")
#:required (required "required.shp")
#:topfilter (topfilter #f))
(make-shp-handler path default not-found (init-script required path) topfilter))


Now you can parameterize at topfilter and know that you'll get the correct value in all of the scripts! Since the topfilter gets run after the required script, make sure you have the parmeter defined and exported from your required modules, and things will work accordingly.

Outer vs Inner Scripts

Since now some scripts will compile into functions that takes parameters, they cannot be called directly by the dispatcher anymore. In a way this is a nice side feature: it automatically distinguishes the outer vs inner scripts.

In PHP it is always a bother to "hide" the inner scripts so they are not accessible directly by the browser, because there are no effective way of telling PHP that these particular scripts should only be run by including other scripts, except via some url rewrite-fu. So we'll be one-upping PHP ;)

The idea is simple - if the script comes into a procedure that requires parameters at the top level of the request handler, then it should run notfound.shp instead.

Hence we need to be able to tell the two types of procedures apart:

(define (thunk? p)
(and (procedure? p)
(let ((a (procedure-arity p)))
(cond ((arity-at-least? a)
(= (arity-at-least-value a) 0))
((number? a) (= a 0))
((list? a) (member 0 a))))))

Now we can fix shp-handler to check for the procedure's arity:

(define-struct shp-handler (path default not-found required)
#:property prop:procedure
(lambda ($struct request)
;; evaluate if
(parameterize (($pathinfo ($pathinfo))
($request request)
($server $struct))
(eval-script-if-changed! (shp-handler-required ($server)))
(let ((proc (evaluate-terms
(file->values
(url->shp-path (request-uri request))))))
(make-response (if (thunk? proc)
(proc)
(include! (shp-handler-not-found ($server)))))))))

Now the shared scripts are automatically hidden for you ;)

Shared Scripts vs Required Modules


It is worth noting that while both shared scripts and modules are "shared" resources, they play different roles. Each shared script represents only a single procedure, while the module can comprise many different procedures and other values. Most of the scripts should in fact be focused on generating representations, rather than doing any sort of substantial calculation. It is not that the shp script cannot handle complex functions, but PLT Scheme already provided one of the most advanced module systems available and there is no reason not to take advantage of it. SHP script's primary focus is on quick development cycle similar to PHP, so use them accordingly.

Wednesday, August 12, 2009

Passing Variables Between the Scripts

As you already know, each SHP scripts are compiled into its own procedures, and hence the scopes of procedures apply - there are no shared variables across scripts. Unless you want to pass everything through parameters (which is painful as you will need a toplevel script to parameterize the value of the parameter to ensure different threads not overriding each other).

Since each scripts are procedures, the best way to pass variables around is to pass them as arguments to each procedure call. Something like the following should work:

(include! "/path/to/script" var1 var2 ...)

This means we need a way to specify the arguments of the script! Something like:

;; inner script
(:args x y z)
;; ... the rest of the script

What we need then is to lift out the args clause (there should only be one!) during the compilation phase and compile it into part of the procedure signature. Let's see how this would work!

First we need to determine which particular expression is the argument expression, and then we need to filter out the arg expression from each of the script:

;; determining the args
(define (args-exp? term)
(and (pair? term) (equal? (car term) ':args)))

;; get the args expressions
(define (terms->args terms)
(define (helper args)
(cond ((null? args) '()) ;; if none just return null
((not (null? (cdr args))) ;; cannot have more than one
(error 'filter-args "multiple args statement: ~a" args))
(else (cdr (car args)))))
(helper (filter args-exp? terms)))


Then we should push the args expression into the eval statement:

;; terms->exps
(define (terms->exps terms)
(let ((exps (filter (lambda (exp)
(and (not (require-exp? exp))
(not (args-exp? exp))))
terms)))
(if (null? exps)
'("") ;; ensure there is at least one exp in the lambda.
exps)))

;; abstract the eval process
(define (evaluate-terms terms)
(require-modules! terms) ;; first register the required modules
;; then we filter out the required statement and evaluate the rest of the terms as a proc.
(eval `(lambda ,(terms->args terms)
. ,(terms->exps terms))
handler-namespace))

Note that previous we actually have request as the first argument, but since we already passed request via parameter, we no longer need it to take up the spot in the procedure, so we should fix all of the places that calls evaluate-terms:

(define (eval-script-if-changed! script)
(unless (not (file-exists? (script-path script)))
(let ((timestamp (file-or-directory-modify-seconds (script-path script))))
(when (> timestamp (script-timestamp script))
(set-script-timestamp! script timestamp)
(let ((proc
(evaluate-terms
(file->values (script-path script)))))
(proc))))))

(define-struct shp-handler (path default not-found required)
#:property prop:procedure
(lambda ($struct request)
;; evaluate if
(parameterize (($pathinfo ($pathinfo))
($request request)
($server $struct))
(eval-script-if-changed! (shp-handler-required ($server)))
(let ((proc (evaluate-terms
(file->values
(url->shp-path (request-uri request))))))
(make-response (proc))))))

(define (include! path . args)
(let ((proc (evaluate-terms
(file->values
(segments->path (path->segments path) #f)))))
(apply proc args)))

Notice that in both shp-handler and eval-script-if-changed, the calling of proc takes zero arguments. It means that the required script and other top level scripts should not require any arguments. include! takes a list of parameters, and it works for regular, optional, and rest parameters.

Allright - with the declaration of formal arguments we can now pass variables around! We might enable the parameters to be bound by contracts in the future, but we'll tackle other problems first.

Include Other Files - Solve the Evaluation Order Bug

If we want the evaluation order to be done correctly, what we need is to compile the inner functions before we compile the outer functions, so we can make sure that the compilation is done correctly.

If we are solving this in the "general" sense, it can get quite hairy, because we'll need to invert the evaluation process by doing the following:
  1. walk through the expressions to pull out included scripts
  2. parse the included scripts to look for their required statements
  3. evaluate those first, and throw away the rest of the statements
  4. repeat 2 & 3 recursively
  5. then finally go back and eval the rest of the expressions, which will then run through the included scripts and this time create the procedures and return the results
So we would need to create a separate compilation phase for evaluating the require statements in the included scripts, and then finally redo the eval phase for generating the output from the them.

While the above solves the problem in a general sense, it is neither elegant nor the only approach to solve the problem, not to mention the problem's impact is actually "small" (the exception is only thrown on the very first time the scripts are eval'd before the require modules are loaded), and hence we would get very little return solving it that way.

Sometimes the best way to solve the problem is to redefine the problem, and this is one of such case.

20% Effort for 80% Gain

The reason we want to have require statements inside includes is to keep them in one place (since the require statements have global scope anyways) so we can save on typings. So if we look at it from that perspective, even having to include them in all of the outermost scripts is too much work. It's probably easier if we can simply have a specialized scripts to contain all of the required statements.

This is very similar in concept to the "notfound.shp", which gets invoked when we cannot find the right script to dispatch. With such a specialized script, we simply need to make sure that it is evaluated whenever it changes.

The first step is to add the required path to the shp-handler struct

(define-struct shp-handler (path default not-found required) ..)


The next step is to check to see if the script exists and if it has changed, if so, evaluate it. In order to do so we'll need to keep track of the timestamp of the script. In order to do so we either add another field to shp-handler, or we will have to something else besides the path to the required field.

(define-struct script (path (timestamp #:mutable))) ;; for comparing to new timestamp

Then we can create the struct by check to

(define (normalize-path path (base (shp-handler-path ($server))))
(apply build-path base (path->segments path)))

(define (init-script path (base (shp-handler-path ($server))))
(let ((path (normalize-path path base)))
(make-script path 0))) ;; 0 is epoch time.

We do not want to call init-script every time, so we will need a wrapper for make-shp-handler:

(define (*make-shp-handler path ;; will be renamed when provided out
(default "index.shp")
(not-found "notfound.shp")
(required "required.shp"))
(make-shp-handler path default not-found (init-script required path)))

Then we just need to make sure this file is tested to see if it's changed, and evaluated if necessary.

(define (eval-script-if-changed! script)
(unless (not (file-exists? (script-path script)))
(let ((timestamp (file-or-directory-modify-seconds (script-path script))))
(when (> timestamp (script-timestamp script))
(set-script-timestamp! script timestamp)
(let ((proc
(evaluate-terms
(file->values (script-path script)))))
(proc ($request)))))))

and finally ensure that it is called before dispatching the request:

(define-struct shp-handler (path default not-found required)
#:property prop:procedure
(lambda ($struct request)
;; evaluate if
(parameterize (($pathinfo ($pathinfo))
($request request)
($server $struct))
(eval-script-if-changed! (shp-handler-required ($server)))
(let ((proc (evaluate-terms
(file->values
(url->shp-path (request-uri request))))))
(make-response (proc request))))))


Allright - with this change we now have a common require script in place. If it does not exist, nothing will happen, and we only evaluate it if the file actually has changed. And if you like to add the require statement to the individual scripts - that still works! Of course - this means that evaluating the inner include require statements before the outer function gets compiled will not be supported.

Problem solved.

Including other files like PHP

If we want to develop complex scripts we will need to be able to modularize the pages so we are not completely repeating ourselves, so we want to be able to include other scripts!

We should have something like the following:

(include! "/path/to/script/file")

The include would then be embeddable within the output xexprs, or just for the side effects (for example, the include file contains the common required modules).

Although relative path seem interesting - we want include! to work as similar to the script dispatcher as possible, so we will only support the full path.

Getting the Root Path

Currently the root path is held in the closure of make-shp-handler so is inaccessible outside its scope. We can pass store the value in a parameter, but as previously noted, if we started to increase the number of configuration values it's probably best to store the whole value into a configuration object. It's time to create that configuration object.

In a way - the configuration object basically contains the settings of the server, so we'll call this the $server object. And since we are in schemeland, there is no reason this object should just hold configuration values - we can have it hold the server (well - the shp handler) itself, so it's full invokable.

(define $server (make-parameter #f))

The simplest way to do so is to convert shp-handler into a struct, the swiss army knife of PLT Scheme:

(define-struct shp-handler (path default not-found)
#:property prop:procedure
(lambda ($struct request)
(parameterize (($pathinfo ($pathinfo))
($request request)
($server $struct))
(let ((proc (evaluate-terms
(file->values
(url->shp-path (request-uri request))))))
(make-response (proc request))))))


With the above definition we now can make a shp-handler by calling make-shp-handler and use it as a procedure (we also need to remove the previous make-shp-handler to avoid name conflict). And since we parameterize $server to $struct itself, we now have access to the shp-handler structure within the scope of the request.

Since we now can access the config values through accessors such as shp-handler-path, shp-handler-default, and shp-handler-notfound within the scope, we no longer have to pass them around through function parameters, so the following functions have their signatures changed.

;; partially matching the path from the beginning of the path
(define (normalize-partial-path segments)
(define (helper rest path default not-found)
;; if we did not find any match return not-found
(cond ((null? path) (build-path path not-found))
;; then we test to see if this is a directory path & whether the default file exists for the directory
((and (no-extension? (car rest))
(file-exists? (build-path path (car rest) default)))
($pathinfo (cdr rest)) ;; updating the pathinfo so it can be accessed.
(build-path path (car rest) default))
((file-exists? (build-path path (car rest))) ;; otherwise the segment is a file and see if it exists...
($pathinfo (cdr rest)) ;; udpdating the pathinfo so it can be accessed.
(build-path path (car rest)))
(else
(helper (cdr rest) (build-path path (car rest)) default not-found))))
(helper segments (shp-handler-path ($server)) (shp-handler-default ($server)) (shp-handler-not-found ($server))))

(define (segments->path segments (partial? #t))
;; test for full path first and then partial path.
(let ((script (apply build-path (shp-handler-path ($server)) segments)))
(cond ((file-exists? script) script)
((and (directory-exists? script)
(file-exists? (build-path script default)))
(build-path script default))
(partial?
(normalize-partial-path segments))
(else
(build-path script (shp-handler-not-found ($server)))))))

(define (url->shp-path url)
(segments->path (url->path-segments url) #t))

For those who love function purity it looks bad - but scheme is a practical language, and SHP will not enforce the concept of pure functional programming.

And include! now looks like this:

(define (path->segments path)
(filter (lambda (path)
(not (equal? path "")))
(regexp-split #px"\\/" path)))

(define (include! path)
(let ((proc (evaluate-terms
(file->values
(segments->path (path->segments path) #f)))))
(proc ($request))))

Pure Side Effect Scripts


For scripts that exists primarily for side effects (such as a common require module list), because we need to generate a procedure (even if it's a dummy procedure), we'll insert a dummy return value (i.e. an empty string) to satisfy the procedure creation rules if it does not exists after the filtering.

;; terms->exps
(define (terms->exps terms)
(let ((exps (filter (compose not require-exp?) terms)))
(if (null? exps)
'("") ;; ensure there is at least one exp in the lambda.
exps)))

;; abstract the eval process
(define (evaluate-terms terms)
(require-modules! terms) ;; first register the required modules
;; then we filter out the required statement and evaluate the rest of the terms as a proc.
(eval `(lambda (request)
. ,(terms->exps terms))
handler-namespace))


With this we can have a script that only contains side effects without having to worry about it returning any value, because the platform takes care of it for us.

Bug - Evaluation Order


The code as written above has one subtle bug, and that is the evaluation order of the include file. Basically - if you include a common required module script, the first time the script gets evaluated is when the containing script gets compiled, but because the definition resides inside the inner include, it does not get fully evaluated until the compilation failed. So the first time around when you are executing the script you will get:

reference to an identifier before its definition: ...

We'll talk about how to solve this problem in the following post, stay tuned.

Basic Environment Access - Response Helpers

Now we have simple way of accessing the request object, we also want to do the same with the response object.

Basically - we should be able to modify the outgoing status, headers, cookies, etc.

Below are the basic features we should have:

($status 'ok) ;; change the status to 200 OK
($content-type "text/html; charset=utf-8")
(header! "Content-Length" 120)

To do so, we'll have to create our own response object, rather than relying on web-server to convert xexpr into response object (we will need to do this anyways if we are ever going to support other type of output). And then we'll use parameters to hold the output value until we are ready to generate the response object.

The following handles status:

;; the list of http-status, code & text. this is internal and might change
(define http-status '((continue 100 "Continue")
(switching-protocols 101 "Switching Protocols")
(ok 200 "OK")
(created 201 "Created")
(accepted 202 "Accepted")
(non-authoritative-information 203 "Non-Authoritative Information")
(no-content 204 "No Content")
(reset-content 205 "Reset Content")
(partial-content 206 "Partial Content")
(multiple-choices 300 "Multiple Choices")
(moved-permanently 201 "Moved Permanently")
(found 302 "Found")
(see-other 303 "See Other")
(not-modified 304 "Not Modified")
(use-proxy 305 "Use Proxy")
(temporary-redirect 307 "Temporary Redirect")
(bad-request 400 "Bad Request")
(unauthorized 401 "Unauthorized")
(payment-required 402 "Payment Required")
(forbidden 403 "Forbidden")
(not-found 404 "Not Found")
(method-not-allowed 405 "Method Not Allowed")
(not-acceptable 406 "Not Acceptable")
(proxy-authentication-required 407 "Proxy Authentication Required")
(request-timeout 408 "Request Timeout")
(conflict 409 "Conflict")
(gone 410 "Gone")
(length-required 411 "Length Required")
(precondition-failed 412 "Precondition Failed")
(request-entity-too-large 413 "Request Entity Too Large")
(request-uri-too-long 414 "Request-URI Too Long")
(unsupported-media-type 415 "Unsupported Media Type")
(request-range-not-satisfied 416 "Requested Range Not Satisfiable")
(expectation-failed 417 "Expectation Failed")
(internal-server-error 500 "Internal Server Error")
(not-implemented 501 "Not Implemented")
(bad-gateway 502 "Bad Gateway")
(service-unavailable 503 "Service Unavailable")
(gateway-timeout 504 "Gateway Timeout")
(version-not-supported 505 "HTTP Version Not Supported")))

(define $status (make-parameter 'ok))

;; for testing to see whether the stauts is valid.
(define (valid-status? status)
(assoc status http-status))

;; convert the status into code
(define ($status->code)
(cadr (valid-status? ($status))))

;; convert the status into text
(define ($status->text)
(string->bytes/utf-8 (caddr (valid-status? ($status)))))


The following handles headers & special header such as content-type:

(define $content-type (make-parameter "text/html; charset=utf-8"))

(define $headers (make-parameter '()))

(define (header! key val (replace? #t))
;; add the keyval to the headers!
($headers (cons (cons key val) ($headers))))

;; this function might require changes because response objects have kept changing
(define ($headers->headers)
(map (lambda (kv)
(make-header (string->bytes/utf-8 (car kv))
(string->bytes/utf-8 (cdr kv))))
($headers)))



And finally we convert the output and the parameters into a response object:

(define (make-response output)
(cond ((xexpr? output)
(make-response/full ($status->code)
($status->text)
(current-seconds)
(string->bytes/utf-8 ($content-type))
($headers->headers)
(list
(string->bytes/utf-8
(xexpr->string output)))))))

(define (make-shp-handler path
#:default (default "index.shp")
#:not-found (not-found "notfound.shp"))
(lambda (request)
(parameterize (($pathinfo ($pathinfo))
($request request))
(let ((proc (evaluate-terms
(file->values
(url->shp-path (request-uri request) path default not-found)))))
(make-response (proc request))))))


There might be other special headers such as last-modified, etag, and possibily doc-type that we'll add, but for now we have our basic response helpers.

Basic Environment Access - Request Helpers

Although we have access to the request object, its current form is not yet as abstracted as what PHP has - let's bring it closer. What we want to do is to write shp scripts like:

`(p ,(url->string ($uri)) ,($query "key"))

The first thing we should do is to create a $request parameter, which we'll fill with

(define $request (make-parameter #f))

The simplest function then is $uri:

(define ($uri)
(request-uri ($request)))

We then add function to handle the key/value pairs of $query, which includes both the url query as well as the post data, and $header for looking up header values.

(define ($header key)
(headers-assq* (string->bytes/utf-8 key) (request-headers/raw ($request))))

(define ($query* key)
(define (helper bindings)
(map (lambda (b)
(if (binding:form? b)
(bytes->string/utf-8 (binding:form-value b))
b))
bindings))
(helper (filter (lambda (binding)
(string-ci=? (bytes->string/utf-8 (binding-id binding))
key))
(request-bindings/raw ($request)))))

(define ($query key)
(let ((vals ($query* key)))
(if (null? vals) #f
(car vals))))

For query - since we can have duplicate values, two forms are provided - $query* to return multiple values, and $query to return singular values (or #f if not found).

To make it all work - we just need to parameterize $request during the calling:

(define (make-shp-handler path
#:default (default "index.shp")
#:not-found (not-found "notfound.shp"))
(lambda (request)
(parameterize (($pathinfo ($pathinfo))
($request request))
(let ((proc (evaluate-terms
(file->values
(url->shp-path (request-uri request) path default not-found)))))
(proc request)))))


There are more accessors to provide, but for now this provide enough $request abstraction.