Wednesday, September 23, 2009

Developing a Memcached Client (1) - Storage

[This is a continuation of the network development in PLT post]

Memcached is all the rage these days with the high scalability guys.  Legends have it that as soon as you slap a couple of memcached in between your web app and your database server, all of your scalability issues go away.  Even if that's glossing over of details, memcached certainly offers advantages over accessing the discs all the time, and we want it in PLT.

Let's try to build a memcached client.

Steps to Build a Client (or a Server)

The steps to build either a client or a server is pretty straight forward (independent of the complexity of the protocol):
  1. study the protocol 
  2. write the "output functions" that sends the info to the other party 
  3. write the "input functions" that parses the input from the other party 
  4. combine the input & output functions, and manage the states of the connection (if applicable)
You are likely to mix the steps up instead of doing them in the order prescribed above and that's okay.  The steps are numbered to help us to think of them as an orderly process, but the process is likely more iterative.

Allright - let's get started.

Study the Protocol 

The key to build a network interface is to understand the protocols.  It is best if the protocol is open and published, so you have something written that you can study (hopefully in a language you understand).  The next best option is to reverse engineer from an open source implementation, and the worst is to reverse engineer from sniffing the network interactions between the clients and the server.  I am glad such memcached's protocol is already published so we do not have to go down the other routes.

Please refer to the memcached protocols throughout this process as the official reference.  We'll sprinkle information from the official doc in the posts as necessary.

First a quick overview of the memcached protocol.
  • request/response protocol - there are no unexpected responses that the client has to handle 
  • multiple requests and responses within a single session 
  • *mostly* line-based protocol - the commands are single lines that are terminated by CRLF; the data is also terminated by CRLF, but it is actually determined by length parameter
  • can store, retrieve, and delete the content (as well as flush) 
  • there are also statistics that can be retrieved, but the protocol appears in flux for the stats (so it'll be out of scope for now) 
  • the protocol itself is *stateless* as each of the requests and responses are independent from each other (each request is atomic) 
Overall the protocol is simple yet non-trivial, which gives us a great example for network development.  We'll start with the storage part of the protocol.

Storage API

Memcached has multiple storage commands for different situations:

Tuesday, September 22, 2009

Network Development in PLT: Overview

One of the nice thing about PLT Scheme is that you can write native scheme network apps, rather than having to use FFI to interface with C libraries for the network apps. The nice thing about using native scheme approach is that it works with PLT's threads, unlike FFI, which blocks all PLT threads (and effectively synchronizes the network access). And according to Geoffrey, FFI also have the disadvantage of having to worry about whether it expects 32-bit or 64-bit pointers.

The challenge with network development though is that we seldom have to do it, so it's hard to know all of the ins and outs of network development. This series of posts will provide a tutorial of doing networking development. The goal is to keep the principles as general as possible so it applies toward other programming languages, but obviously this is a scheme blog.

General Concept

We'll quickly go over the basic network architecture concept for the sake of completeness.

In general, networking involves clients and servers interacting with each other by sending information to each other and (if necessary) wait for responses from the other party.

Clients are programs that sends out the requests, and servers are programs that "fulfill" the requests (and possibly send back a response).  A program can be both a client and a server at the same time.

The information they send each other are serialized to bytes, which will be reconstructed by the receivers into meaningful representations that they can interpret.

Depending on the nature of the work, clients and servers might only need to send information to each other once and be done (HTTP is one such protocol), but sometimes they need to communicate back and forth to accomplish the task (SMTP is one such protocol), and such situation it might be necessary for the client and the server to keep track of the state in order to manage the work.

So, at a high level architecture, we need to focus on the following in order to do network development:

  • manage the connection (initiation, termination, etc) 
  • serialize and send information to the other party 
  • receive information from the other party and interpret the meaning (and act accordingly)
  • track and manage the state if the protocol requires it 
The above applies to all network developments.  The specific details and the associated complexity comes down to the specific protocols you are developing.

Let's take a look at what PLT offers to help us handle each of the needs.

Network Connection Management 

By default PLT offers network programming in TCP, UDP, and SSL.  You can require them into your module:




(require scheme/tcp scheme/udp openssl)

We'll focus on just TCP network development in this tutorial since most network protocols will be TCP-based.

Initiate Connections 

If you are developing a client, you can initiate a client connection with:

Monday, September 21, 2009

bzlib/dbd-file - filesystem-based database driver - now available

As previously promised, bzlib/dbd-file, a demonstration of how to extend bzlib/dbi around filesystem, is now available via planet.

You can revisit the posts that chronicled its developments:
  1. Overview of the DBI internals 
  2. a draft driver - list directories and open files
  3. enhance the driver - save files atomically
  4. enhance the driver - delete files atomically 
Prerequisite 


There isn't a prerequisite for this module, but there are a few caveats:
  • No transaction support (only atomicity)
  • Atomicity not guaranteed on Windows (it'll almost work most of the time)
  • No prepared queries 
  • No SQL - this is not a SQL database driver
Installation 

The installation over planet is straight forward:

;; in REPL  
(require (planet bzlib/dbi))
(require (planet bzlib/dbd-file))


Usage 

You should set aside a particular directory as the root for the database.  Let's call that <root-path<.  As with any database - you should not manually access the files within that directory to minimize the chance of destroying the database.

To connect to the database:

;; use the 'file driver  
(define handle (connect 'file <root-path>))  
;; or use the 'file/rs driver 
(define handle (connect 'file/rs <root-path>))

The difference between the two drivers is that 'file/rs returns values that are compatible with the DBI query helper functions such as rows, rows/false, cell/false, since it converts the underlying return value into recordset structure, so it is just a bit less efficient.

To disconnect from the database:

(disconnect handle) ;; NOOP  

The call itself is unnecessary since there are no external resources to clean up.  The prepare is also a NOOP.

To "select" the data from the files by paths:

(query handle 'open  `((path . "/path1") (path . "/path/to/file2") ...)) 
;; => a list of bytes, each bytes is the total content of the file 

 The paths used with the driver (either returned values or passed arguments) are always a jailed absolute path, which will be merged with the root directory to form the actual underlying path.

To "select" the paths a particular base path:

(query handle 'list `()) ;; list the files in the root directory
(query handle 'list `((path . "/some/path"))) ;; list the files in some path 
;; => returns a list of the jailed paths  

To "insert" or "update" a particular path:

(query handle 'save! `((path . "/some/path") (content . #"some bytes")))  

Both the path and content parameters are required in this particular case.  The file will be saved atomically.

To "delete" files or directories:

;; delete files (not directories)
(query handle 'delete! `((path . "/some/path") (path . "/some/other/path") ...))
;; delete empty directories

(query handle 'rmdir! `((path . "/some/path") (path . "/some/other/path") ...))


;; delete either files or directories 
(query handle 'rm-rf! `((path . "/some/path") (path . "/some/other/path") ...))



'delete! will only delete files, and will error if trying to delete directories. 'rmdir! will only delete empty directories, and will error trying to delete either files or non-empty directories.  rm-rf! will delete either files or directories (whether empty or not).  The deletion are atomic on non-Windows platform.

Notes About Windows

Since Windows locks opened filehandles, and generally have background processes such as antivirus softwares that randomly open files for inspections, you might find the save and deletion operations error out intermittently.  You can of course handle the errors and then retry the operations to get around the issues, but basically supporting atomic file-based save/deletion is out of scope.

That's it for now - enjoy.

BZLIB/FLEXER 0.1 - FLEX Integration with SHP - Now Available

bzlib/flexer is now available through planet.

As described in the FLEX/SHP posts, bzlib/flexer provides integration between FLEX and SHP, allowing you to do the following:
  1. directly embed flash video files by calling (flash <url;> ...)
  2. directly code in MXML within SHP and compile into flash video  with (mxml ...)
  3. optimizing the compilation of MXML
The package is released under LGPL. 

Prerequisite

You'll need the Adobe Flex SDK 3 downloaded and installed separately, and make sure that the flex compiler mxmlc is on the system path.

You'll also need bzlib/shp, with the planet version >= 1.2. 

Installation & Configuration


Install via planet with

(require (planet bzlib/shp:1:2))
(require (planet bzlib/flexer))

Make sure to require bzlib/flexer in your SHP required script as well:

Saturday, September 19, 2009

Create a Driver for bzlib/dbi (4) - Concluding Filesystem Driver

This is the fourth installment of the extending bzlib/dbi series - for refreshers, see the following previous posts:
  1. Overview of the DBI internals 
  2. a draft driver - list directories and open files
  3. enhance the driver - save files atomically
Basically we now have the SQL equivalent of select, insert, and update.  The next stop is to provide the ability to delete files and directories.

The Deletion Capability 

PLT Scheme offers the following for deleting files and directories:
  • (delete-file file-path) - delete the path if it is a file; otherwise raise an error
  • (delete-directory dir-path) - delete the path if it is an empty directory; otherwise raises exceptions 
  • (delete-directory/files file-or-dir-path) - delete the path whether it is a file or directory, and if it is an non-empty directory, the sub files and directories will first be deleted recursively 
It seems that delete-directory/files is the most convenient to use.  However, there might be situations you want to ensure that you are deleting either a directory or a file, and if you are deleting a directory you only want to delete an empty directory, and we want to make sure our API reflects that, so we will have 3 separate calls:

;; delete files use delete! 
(query handle 'delete! `((path . path1) ...))
;; delete empty-only directories uses rmdir!
(query handle 'rmdir! `((path . path1) ...))
;; delete either file or directories use rm-rf! 
(query handle 'rm-rf! `((path . path1) ...)) 

Similar to SQL's delete statement, the above query can delete multiple paths at once.  However, given transactions is not supported - the deletes are done independently, and if any of the delete fails the rest will stay undeleted.

The following addition to file-query accomplishes the above:

Friday, September 18, 2009

Create a Driver for bzlib/dbi (3) - Continuing Filesystem Driver

Previously we have discussed the internals of DBI and how to extend it, as well as crafted the first draft of the filesystem database driver.  We are now onto #3 of the DBI extension series.  We'll continue by adding the ability to save data to the driver.

Save The Data

Saving the data is equivalent to both insert and update in SQL venacular.  The simplest version looks like below:

(define (file-query handle stmt (args '())) 
  ... 
  (case stmt
    ... 
    ((save!)
     (call-with-output-file (path-helper (assoc/cdr 'path args))
       (lambda (out) 
         (write-bytes (assoc/cdr 'content args) out))
       #:exists 'replace))
    (else 
     (error 'file-query "unknown statement: ~a" stmt))))

With the above we now can save data into a particular file with the following usage:

(query handle 'save! `((path . "/foo/bar/baz.txt") (content . #"this is the content")))  
But unfortunately, there are a few hiccups:
  • there is no verification that path and content key/value pairs are passed in (for 'list and 'open queries the path key/value pairs are optional) 
  • there is no guarantee the directory of the path exists (and if not it will result in an error) 
  • the above saving is not an atomic operation and can corrupt the data
So we'll address each of the issue to ensure we have a solid implementation for saving data.

Argument Verification 

To verify the arguments, we can use let/assert! from the next version of bzlib/base (which will be released together with this driver) as follows:

(define (file-query handle stmt (args '())) 
  ... 
  (case stmt
    ... 
    ((save!)
     (let/assert! ((path (assoc/cdr 'path args))
                   (content (assoc/cdr 'content args)))
                  (call-with-output-file (path-helper path)
                    (lambda (out) 
                      (write-bytes content out))
                    #:exists 'replace)))
    (else 
     (error 'file-query "unknown statement: ~a" stmt))))

let/assert! checks to see if the values returned are false, and if so raises the error, else binds the variable call the inner expression.  It also behaves like let* instead of let, as the subsequent variable can see the previous variable.

Gaurantee of Directory Path

To ensure the directory for the path already exists - we can utilize make-directory* to create the parent directory for the path, but we need to make sure the parent path of the path is not a file (so we can create a directory):

Thursday, September 17, 2009

Create a Driver for bzlib/dbi (2) - Filesystem Driver

In our previous post - Create a Driver for bzlib/dbi (1) - DBI Internals - we discussed the motivation and DBI internals so we can get started to implement our filesystem-based driver.  If you find some of the following a bit opaque - read the previous post for some explanations.


Caveats about Filesystems and the Driver

Since each filesystem has different behaviors it is difficult to guarantee particular performance characteristics about the drivers. For now the following are the caveats:
  • No transactional support - it takes more effort to build in transactional support for bare filesystems - something we will not tackle for now 
  • No atomic writes for Windows filesystem - as Windows filesystem does not fully support atomic rename (Windows program holds lock on the file if it is opened during the rename and will cause an error), we also cannot make guarantee that writes will be successful in Windows.  Under Unix variants we can guarantee atomic writes 
Furthermore, since our goal is to have a simple driver, the following are also out of scope:
  • SQL statement mappings - for now we will not support SQL statements
  • Prepared statements - since we are not supporting complex SQL mappings, there is no reason to have prepared statement capabilities; i.e. the prepare call will be a no op.
With caveats out of the way - let's determine what we should be able to do at a minimum:
  • manage all files within a single directory as the database data 
  • open files by path and return their contents 
  • listing the files in a particular directory (within the base directory) 
  • save data against a particular path (this can either be an "insert" or "update" operation) atomically on unix platform (might cause errors on Windows given the limitation of Windows platform) 
  • delete a particular file 
  • delete a particular directory 
  • create a particular directory (even without the intermediate directories) 
 Allright - let's get started.

Connect, Disconnect, Prepare, and Transaction Handling 

Since we want to have a directory representing the root of the database, our database connection is really the root directory:

#lang scheme/base 
(require (planet bzlib/base)
         (planet bzlib/dbi)
         )

(define (file-connect driver path) 
  (assert! (directory-exists? path)) ;; assert! comes from bzlib/base 
  (make-handle driver path (make-immutable-hash-registry) 0)) 

Disconnect is even more straight forward, since there isn't any external resources that have to be released:

(define (file-disconnect handle)
  (void)) 

And since prepare is out of scope, it is also a NOOP:

(define (file-prepare handle stmt)
  (void)) 

Furthermore, transaction support is also out of scope - we have more NOOPs:

(define (file-begin handle)
  (void))
(define (file-commit handle)
  (void))
(define (file-rollback handle)
  (void)) 

The default transaction functions will not suffice here since they issue the corresponding SQL statements against the handle.

Assuming we have the corresponding file-query defined we now have a complete driver with:

(registry-set! drivers 'file
               (make-driver file-connect
                            file-disconnect
                            file-query
                            file-prepare
                            file-begin
                            file-commit
                            file-rollback))
    
Now we just need to flesh out file-query, which is the meat of the driver: