Thursday, September 17, 2009

Create a Driver for bzlib/dbi (1) - DBI Internals

Since we claimed bzlib/dbi as an extensible abstract database interface for PLT Scheme, let's take a look on how to extend it.  As an exercise we'll create a database driver that uses the filesystem as the database. 

Motivation 

What we want is to use the filesystem as a simple form of database, where the path of the file is the key, and the content is the value.  We should be able to select the data, update, delete, and insert new data.  Let's see what we can come up with.

Overview of Creating an Database Driver

The first step is to require bzlib/base (contains bzlib/base/registry) and bzlib/dbi (contains the interface that we need to extend):

#lang scheme/base
(require (planet bzlib/base) (planet bzlib/dbi)) 

Then we need to create four functions for the driver that matches the following signature:

(connect driver . args) ;; a connect function that takes in the database driver definition, and a variable list of arguments
(disconnect handle) ;; a disconnect function that takes in the database handle
(prepare handle statement) ;; a prepare function that takes in the database handle and a statement
(query handle statement args) ;; a query function that takes in the database handle, the statement (or the key to the prepared statement, and the args as a key/value plist

The above four functions are sufficient for creating a database driver, and if we need to provide custom transaction functions, we will customize three additional functions:

(begin-trans handle) ;; to provide the begin transaction clause for the database handle
(commit handle) ;; to provide the commit transaction clause for the database handle
(rollback handle) ;; to provide the rollback transaction clause for the database handle 

Overriding the transaction functions is an optional step as you can use the default version, which are named default-begin, default-commit, and default-rollback.  All of the default version basically issues the corresponding SQL statements to the underlying database connection.

Then we just need to register the functions as a database driver:

Tuesday, September 15, 2009

Flex Compiler Integration Optimization

This is a continuation of the FLEX Integration Series - please refer to the previous posts for more details:

  1. Building FLEX Integration with SHP
  2. FLEX Compiler Integration with SHP

Now we'll cover how to optimize the integration for performance.

The easiest way to determine whether a new compilation is necessary is by comparing the timestamp of the source code vs the timestamp of the object code - even the venerable make utilize this simple check.

So what we need to do is to determine the timestamp of the shp script that we call mxml, and then compare it against the flash video file.  Since mxmlc compilation time is long, if the timestamp of the shp source script has a later timestamp than the flash video, it is very safe to assume that the source script have been changed.

Determining the Timestamp 

PLT Scheme offers (this-expression-source-directory) and (this-expression-source-file-name) to help determine the location.  But as those are macros and applies to the location of their presence, they are not exactly helpful to our cause unless we want to write them out everytime we have to use mxml like this:


(mxml (this-expression-source-directory) path ...)  
We want the value to be automatically available rather than manually available, so we'll have to improve our SHP handler.


(define __PATH__ (make-parameter (shp-handler-path ($server)))) ;; looks like C macro... 
 (define (evaluate-script path)

  (evaluate-terms (file->values path) path))


(define (evaluate-terms terms path)
  (require-modules! terms) ;; first register the required modules 
  ;; then we filter out the required statement and evaluate the rest of the terms as a proc. 
  (eval `(lambda ,(terms->args terms) 
           (parameterize ((__PATH__ path))
             . ,(terms->exps terms)))
        handler-namespace))
Now we have the path available within the SHP script scope, and we can use it to check the timestamp.

Flex Compiler Integration with SHP

This is a continuation of Building FLEX Integration with SHP - please refer to it for refresher.

The next step of the integration is to do FLEX development directly within SHP. If you are used to develop FLEX apps in IDE such as Flex Builder, you might not necessary need this capability.  But consider the following:
  • potential for modularizing the flex scripts
  • potential for graceful degradations into ajax or pure html 
  • possibility for automating the generation of simple flex-based solutions 
 All of which are powerful building blocks for web development, so we'll give it a shot.  Let's get started.

Goal

We want to be able to write in MXML and ActionScripts in SHP, and have the script being compiled in real-time into the flash movie and served to browser transparently.

Example:

Here's a mxml shp script for hello world:

(mx:app (mx:script "private function clickHandler(evt:Event):void {
    messageDisplay.text = \"I am Glad, it does.\";
}")
     (mx:label "Flex without Flex Builder")
     (mx:button #:label "Yes, It Works!" #:click "clickHandler(event)")
     (mx:label #:id "messageDisplay"))
Which should be compiled into the following mxml:

<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml">
<mx:Script>
<![CDATA[
private function clickHanlder(evt:Event):void {
    messageDisplay.text = "I am Glad, it does.";
}
 ]]>
</mx:Script>
<mx:Label text="Flex without Flex Builder" />
<mx:Button label="Yes, It Works!" click="clickHanlder(event)" />
<mx:Label id="messageDisplay" />
</mx:Application> 
And then compiled into a flash video file, and finally served to the client via the (flash) inclusion.  And we should check the timestamp of the SHP script against the compile flash video so we won't waste CPU cycle by compiling the same script over and over again.

The first thing to do is to ensure we can generate the correct MXML.  It is mostly straight forward:


(define (mx:app . widgets) 
  `(mx:Application ((xmlns:mx "http://www.adobe.com/2006/mxml"))
                   . ,widgets))

(define (mx:label (text "") #:id (id #f))
  `(mx:Label ((text ,text)
              ,@(if (not id) '() `((id ,id))))))

(define (mx:button #:label (label "") #:click (click #f))
  `(mx:Button ((label ,label) 
               ,@(if (not click) '() `((click ,click))))))




;; more mxml widget definitions  
With the above we now can generate the MXML, albeit in an incomplete form.  The next step would then be to get the generated mxml *saved* to a predefined location, instead of serving them.

Monday, September 14, 2009

Building Flex Integration with SHP

Besides AJAX, FLEX is the other major player in the RIA world (Silverlight might be a future contender, but it currently is not).  We already provided basic integration with Javascripts, now we want to provide the same level of integration with FLEX.


The lightest layer of integration would be to manage the flex files as static resources, and just make sure they work well within SHP.  But we can also opt for a closer integration by having SHP compile the flash video files, and make the compilation seamless.  Let's develop the light integration first.

What we want are the following:
  • keep the flex video files in a static directory similar to other resources, i.e. in the $htdocs/flex/ directory
  • simplify the generation of the <object> tag to load the appropriate video file
The tag should be as simple as the following SHP script:

;;  foo.shp ;; loads the foo.swf flash video
(flash "foo" #:width 550 #:height 400) 

Which should then be translated into the following xexpr:


<object width="550" height="400" id="foo" > 
<param name="allowscriptaccess" value="always" /> 
<param name="movie" value="/flex/foo.swf" />
<embed src="/flex/foo.swf" width="550" height="400" name="foo" 
      type="application/x-shockwave-flash" allowscriptaccess="always" >
</embed></object>

This is quite straight forward:

(define flash-base-path (make-parameter "/flash")) 

(define (path->id path)
  (string-join (path-helper path) "_"))

(define (path-helper path)
  (cond ((pair? path) path)
        ((string? path) (regexp-split #px"\\/" path ))
        ((path? path) (path-helper (path->string path)))))

(define (flash-url path)
  (string-join (cons (flash-base-path) (path-helper path)) "/"))

(define (flash (path ($pathinfo)) #:id (id #f) #:base (base (flash-base-path)) #:height (height 400) #:width (width 400))
  (define (helper path id width height)
  `(object ((width ,width) 
            (height ,height) 
            (id ,id))
           (param ((name "movie") (value ,path)) "")
           (param ((name "allowscriptaccess") (value "always")) "")
           (embed ((src ,path)
                   (width ,width)
                   (height ,height)
                   (name ,id)
                   (type "application/x-shockwave-flash")
                   (allowscriptaccess "always")
                   ) "")))
  (helper (flash-url path)
          (if (not id) (path->id path) id)
          (number->string width)
          (number->string height)))
With the ability to generate the flash embedding xexpr, we now just need to create the static flash directory, require the module (we'll call it bzlib/flexer), and then use the above shp script in our code.

We'll talk about the tighter integration next time.  Cheers.

Saturday, September 12, 2009

Extensible Abstract Database Interface for PLT Scheme

There are not a lot of choices of relational database interfaces for PLT - part of the reason is just that the community is not yet the size of Perl or Python, so not as many infrastructure packages yet.

However, PLT does have the interfaces for three of the most important open source databases - postgresql, mysql, and sqlite, so your open source database needs are likely addressed. But the challenge is that each sports a different interface, so you couldn't really abstract the database away to the degree that you can in other languages.

sqlid was probably the closest thing for PLT to have a single interface and multiple drivers, but it hasn't appeared to be active for a while, so I decided to take the plunge to create an abstracte database interface so all three packages can function uniformly. Hopefully it can attract collaborations from driver writers.

Intoducing bzlib/dbi - extensible abstract database interface for PLT

Inspired by Perl's DBI, the package bzlib/dbi follows the separation between the interface and the driver, so if you want to use a particular driver, you just need to require:

(require (planet bzlib/dbi)
(planet bzlib/dbd-<driver>))
Currently there are three such drivers available:
All of the four packages are released under LGPL.

To connect to the database, you just need to issue connect and pass it the symbol that identifies the driver, along with the arguments expected by the underlying driver. Once you have required bzlib/dbi and the appropriate driver, for example, to connect against schematics/spgsql:

(define handle (connect 'spgsql '#:server "localhost" '#:port 5432 ...))

And to connect against jaymccarthy/sqlite:

(define handle (connect 'jsqlite ':temp:)) ;; creates a temp database

To connect against jaz/mysql:

(define handle (connect 'jazmysql "localhost" ...))

Note that if the underlying driver expects keyword arguments, pass in the keyword with a quote, so PLT will treat it as an argument rather than a keyword for connect (which will result in an exception, since connect has no keyword arguments).

To disconnect, just do:

(disconnect handle)

Query and Named Parameters

The goal of bzlib/dbi is to keep the interface as simple as possible, so there is currently only one function that handles the queries:

(query handle "select * from ..." args)


The arguments is a list of pairs of symbol and values. It is currently not optional (so if you have no arguments you would still have to pass in '()).

The query statements takes in named parameteres instead of ordinal parameters, so instead of writing:

select * from table1 where c1 = ? and c2 = ?

You write

select * from table1 where c1 = ?c1 and c2 = ?c2 -- ?c1 & ?c2 are named parameters.

Then you pass in '((c1 . val1) (c2 . val2)) as the argument.

The nice thing about named parameters is that you do not have to worry about matching the order of the parameters, and you only need to specify the value once if you have to pass in the same value to multiple positions. The name can consist of regular alphanumeric characters, underscores, dashes, and exclamation mark.

bzlib/dbi currently standardizes on using question mark as the placeholder and translates the query and the parameters to their corresponding version for each of the drivers. This can be customized.

Prepared Statements

bzlib/dbi also provides the ability to create prepared statements:

;; (prepare handle name-symbol query)
;; example
(prepare handle 'select-table1 "select * from table1 where c1 = ?c1 and c2 = ?c2")
(query handle 'select-table1 `((c1 . ,val1) (c2 . ,val2)))
Basically - the prepared statement is created and kept along with the handle, and identified via the name symbol. And it is run via the same query interface by passing in the name symbol (instead of the query) along with the arguments.

Unlike other database interfaces where the prepared object is returned to the caller for passing around, bzlib/dbi hides them to 1) simplify the interface, and 2) minimize the chance of error when the same underlying handle is used by multiple threads at once - I settle on this design choice when running into issues with a prepared statement run by a separate thread where the handle was terminated by its originating thread so you wouldn't have to manage the same object with multiple threads.

Transactions

You can trigger transactions manually with the SQL statement.

(begin-trans handle)
(commit handle)
(rollback handle)

Or you can use the with-trans syntax, which wraps the transaction management with exception handling - so if there are errors triggered it would automatically cause rollback of the transaction.

(with-trans (handle1 ...)
exp ...)

You can wrap multiple database handles and if one errors all will be rolled back - this is a simple distributed transaction.

Data Type Mapping

The data type mapping depends on the underlying driver, but the following are the mapping for the general types:
  • integer <-> sql integer
  • number <-> sql float
  • srfi/19 date time <-> sql date/time
  • string <-> sql string/text
  • bytes <-> sql binary/blobs
  • null <-> sql null
For bzlib/dbd-spgsql, there is also a mapping for the array type as a list.

The recordset is returned as a list of lists. The first row holds the column names (so there will always be one row in a select query even if there are no results).

Additional Query Helpers

The following provides additional helpers that wraps around query to simplify the developer's tasks in certain situations:
  • exec - used for non select queries
  • rows - strips the column names and just return the rows
  • row - returns the first row (and raise error if no rows)
  • row/false - returns the first row or #f if there are no rows
  • exists? - a shorthand for row/false (looks more natural for people used to SQL)
  • cell - returns the first cell of the first row or throw an error if there are no rows
  • cell/null - returns the first cell of the first row or returns null
  • cell/false - returns the first cell of the first row or returns #f
All of the above helpers also takes in an optional converter parameter right after the args parameter, so you can directly process the result before it returns. The converter should be a function that can take in as many arguments as there are columns in the query. The full signature of these helpers look like:

(query-helper handle symbol-or-query key/val-args converter)


Extensible Database Interface

An important principle of the design is to make the database interface extensible, so code can be reused and new database driver can be developed more easily (of course it would still require intimate knowledge with the particular database's protocol). Hence the interface is extensible with the chaining of drivers.

The following drivers are currently available for extending the capability of the three database drivers.
  • app - provides a kill-safe interface over the raw database driver
  • pool - provides a database connection pool interface over the raw database driver
Kill-Safe App Driver

To make use of the kill-safe app driver, you need to pass in the 'app driver identifier to connect:

(define kill-safe-handle (connect 'app #f <inner-driver-args> ...))

So for example, if you want to make schematics/spgsql kill-safe, use the following:

(define handle (connect 'app #f 'spgsql '#:server "localhost" '#:port 5432 ...))

And if you want to do so with jaymccarthy/sqlite (which doesn't do much since jaymccarthy/sqlite is FFI-based and already serializes all threads):

(define handle (connect 'app #f 'jsqlite "database.db"))

What happens is that bzlib/dbi will initiate the app driver, which then initiates the underlying database driver. The #f arg is the argument that tells the app driver whether there is a pool driver that helps handle the response. You only need to explicitly pass this value in if you are initiating app driver directly.

Database Connection Pool

The database connection pool driver further builds on top of the app driver and manages the database connections. It works in conjunction with the app driver. For example - to initiate a database pool of maximum 10 database connections:

(define pool-handle (connect 'pool 10 'spgsql '#:server "localhost" '#:port 5432 ...))

You do not need to specify the app driver in this case, since the pool driver does that for you internally.

The pool driver is a bit special in that it does not directly handle the queries, but offers up one of its connections to the caller and let the caller use the connection until the caller thread dies. To retrieve a connection from the pool driver, do:

(query pool-handle 'connect '())

which would return the underlying app driver handle.

The app and pool driver demonstrates how bzlib/dbi can be extended. You can certainly extend it further to fulfill your needs (an example of possible future extension would be to create a load balancer handle that sits in front of a database cluster).

TODO

There are a lot more the design of bzlib/dbi can accomodate, but one of the important thing to do is to collaborate with the driver writers so there can be a joint effort. Since bzlib/dbi is the new kid on the block, the effort will have to come later after people have chance to test out the interface. But now you can switch between the three major database with relative ease.

Erlang-Style Programming in PLT

Similar to Erlang, PLT Scheme also offers microthreads. While the PLT's concurrency implementation might not yet be as capable as Erlang's (currently PLT is not yet multicore enabled), we can still employ similar development style.

The base of PLT's thread primitives are documented in the PLT reference documentation. Specifically, to spawn a thread, we just need to call (thread <procedure>). To send the thread a message, use (thread-send <thd> <msg>). The target thread should call (thread-receive) to retrieve the message. This basic pattern works well with two threads communicating with each other. Once more than two threads are involved, we run into issues with just (thread-receive) since we have no way to verify which thread sends which message, let along sending back the right response to the right receipient. We'll need something more.

Introducing bzlib/thread - implementing the erlang selective receive pattern in PLT Scheme. The code is released under LGPL.

In Erlang, the receive is pattern match enabled, and we want something similar in PLT scheme. bzlib/thread provides the pattern match enabled receive for PLT Scheme, with the syntax receive/match:

(require (planet blib/thread)) ;; load the package.
;; default syntax - with pattern match
(receive/match
(match-pattern exp ...) ...)

;; timer syntax - just the timer.
(receive/match
(after time exp ...))

;; extended syntax - combine the pattern match with timer.
(receive/match
(match-pattern exp ...) ...
(after time exp ...))

The match-pattern has the exact same syntax as scheme/match, since scheme/match provides the underlying matching capability. You can also specify an after clause so you can specify the number of seconds (does not have to be integer) elapsed for a timer-based event to trigger even without thread messages.

Beyond Erlang's Receive Capability

While PLT Scheme's concurrency capability is not yet as capable as Erlang, its language facility has more to offer than Erlang's. By default PLT Scheme provides a powerful synchronization framework that includes many different types of events (the after clause is built on top of the alarm event), and it would be a shame if we cannot take advantage of all those event capabilities within our receive/match, hence receive/match is enhanced with a sync clause:

;; with the sync clause - it is also pattern-match enabled...
(receive/match
(pattern-match exp ...) ...
(after time exp ...)
(sync (pattern exp ...) ...))

Hence you can pass in a list of custom events so the syntax will handle all of the synchronizations within one clause. The sync clause also have pattern matching, so you can use it to match against the specific events that was triggered and dispatch to the correct clause branch.

The receive/match form the basis of multi-threads communications.

Communication Bewteen Multiple Threads

As stated earlier, the challenge with the bare thread-send and thread-receive is that you have to handle your own message dispatching if there are multiple threads trying to communicate with each other. While receive/match provides the additional pattern matching capability, it only simplifies your effort to coordinate all of the threads, but you still have to devise the scheme in which you can identify the sending thread (so you can send back an appropriate response). To do so, we need to codify the structure of the message so it includes information about the sender. The simplest way is to add the sending thread into the message as follows:

(thread-send thd (list (current-thread) args))

Then you just need to ensure your receive/match matches the signature:

(receive/match ((list (? thread? thd) args) (do-whatever) ...) ...)
bzlib/thread codifies the above signature in thread-call, so all you have to do is:

(thread-call thd args)
;; or with a timeout
(thread-call thd args timeout)

and the above message signature will be sent.

Once you've received the message via receive/match and extracted your args for processing, you can then use thread-reply to send a message back to the sender:

(thread-reply sender result)
;; or you can reply on behalf of another thread
(thread-reply sender result thread-on-behalf-of)


Handling Exceptions and Respond to Sender

What if the processing raised an exception and you need to send the exception back to the sender? To avoid confusion between a regular reply (which holds legit values) and an exception, you should use send-exn-to to return the exception, which does the following:

(thread-send thd (cons exn thread) #f) ;; #f means no error is raised if the receiving thread is dead.

Then you just match for the exception pattern to check whether an exception was raised:

(receive/match ((cons (? exn? e) (? thread? sender)) (do-something...)) ...)


Casting Messages

The above thread-call, thread-reply, and send-exn-to codifies the communication signatures between threads so the correct reply can be sent back to the originator, but what if you do not need to respond back to the originator? In that case you have a "cast" pattern, and bzlib/thread provides thread-cast and thread-cast* for you.

(thread-cast thd arg)
(thread-cast* thd arg arg1 ...) ;; equals (thread-cast thd (list arg arg1 ...))

They are very similar to the bare thread-send, except they are written with the kill-safe pattern, so they'll first attempt to wake the receiving thread if it was suspended, and then send over the arguments. Use thread-cast or thread-cast* if you do not need to get a response back.

Applications and Getting Closer to OTP

Erlang is famous for its nine-nines uptime, and their OTP modules have a lot to do with it. It takes a lot of effort to implement OTP and get all of the bugs out, so OTP is not coming to PLT soon. But app which is part of bzlib/thread is the first step toward constructing OTP for PLT.

App basically provides a simple structure over the receiving thread, so you have an structure that you can pass around and manipulate. The function make-application tries to simplify the creation of an application:

(make-application call cast init-state)

You just need to pass the call function (run when triggered via thread-call) and the cast function (run when triggered via thread-cast), as well as the init-state, which are the values that the application will hold internally between the thread calls or casts.

The call function should have the following signature:

(sender-thread passed-in-args app-state . -> . (cons/c result app-state))

The returned app-state do not have to be the same as the passed in app-state, but it needs to be compatible to the function for the next call.

The cast function should have the following signature:

(passed-in-args app-state . -> . (cons/c result app-state))

The result will be disgarded since there isn't a response back to the sender.

app-call and app-cast are provided as wrappers over thread-call and thread-cast. Unlike thread-call and thread-cast, app-call and app-cast takes variable parameter lists.

(app-call app cmd #:timeout (number? +inf.0) arg1 ...)
(app-cast app cmd arg1 ...)

cmd is a symbol that you can use to dispatch to the correct function within your app, assuming your app provides multiple function as its APIs.

That's it for now - have fun programming in erlang style in PLT Scheme.

Wednesday, September 2, 2009

Conditional Module Inclusion and Compilation

Inspired by the thread of "conditional module inclusion and compilation" in PLT mailing list, bzlib/os provides a way to conditionally run code depending on the system type.

To start, require bzlib/os:
(require (planet bzlib/os))


Conditional Require

To require modules only within windows, use require/windows.

(require/windows <require-spec> ...)

The <require-spec> takes the same form as regular require statements. In Mac OS or Unix, the above statement evalutes to (void).

To do the same for Mac OS, use require/macosx. And use require/unix for Unix.

To specify multiple platform in one statement - use require/os.

(require/os (:windows <required-spec> ...)
(:macosx <required-spec> ...)
(:unix <required-spec> ...)
(else <required-spec> ...))

Use :windows to specify the window's branch inclusion. Use :macosx for Macs, and :unix for Unix. All are optional.

Use else to specify non-platform specific includes. If the else branch exists, it must be the last branch.

Conditional Provide

There are also os-dependent provide statements, provided to mirror the require statements (even though it's probably less likely to be used).
  • provide/windows evals to provide on Windows
  • provide/macosx evals to provide on Macs
  • provide/unix evals to provide on Unix
  • provide/os has the same structure has require/os - use :windows, :macosx, and :unix to write OS-dependent provide branch, and use else to write OS independent branch. If the else branch exists it must be the last branch


Both the require/* and provide/* behaves the same as regular require and provide in that they can only be called at top level or module level (provide only works at module level).

Conditional Expressions

The most general form is conditional expressions, which can be called at any position.

For windows, use +:windows:

(+:windows <exp> <exp2>) evaluates to <exp>, and <exp2> otherwise.

(+:windows <exp>) evaluates to <exp>, and (void) otherwise. It is equivalent to (+:windows <exp> (void)).

For Macs, use +:macosx in the same way as above. For Unix, use +:unix.

If you need to write more than 2 OS branch, use +:os, which is the foundation of all above macros:

(+:os (:windows exp)
(:macosx exp)
(:unix exp)
(else exp))

Similar to require/os and provide/os, one of the branches with labels of :windows, :macosx and :unix get evaluated for the respective OS. The else branch is evaluated for platform independent expression; and if it exists it must be the last expression.

It is possible to write (+:os), which gets evaluated to (void).

Caveats

Eli has cautioned that one must make sure no .zo files are copied across platforms by using this pattern.

In general one should write platform independent code, but of course that would not be possible in all situations. It is always better to isolate the OS-specific code to their own modules, and make sure all those modules all expose the same signatures; Unit can aid with this pattern.

Example:
  • bar.ss exposes bar and baz
  • bar-windows.ss exposes bar and baz that work specifically for windows
  • foo.ss includes bar.ss & bar-windows.ss via (require/os (:windows "bar-windows.ss") (else "bar.ss"))