BZLIB/DATE & BZLIB/DATE-TZ 0.2 Now Available

bzlib/date and bzlib/date-tz are now available via planet. They are released under LGPL. You can find the previous documentation for previous usage.

The changes included are:

• re-exports for SRFI-19 functions
• Wrapper functions for PLT date objects (you can use the functions with PLT date objects instead of with SRFI date objects)
• day comparison functions
• RFC822 date parsers and generators
• additional date manipulation functions

SRFI19 Re-export
Previous you have to explicitly include srfi/19 to use the functions within SRFI19 as follows:

(require srfi/19 (planet bzlib/date)) 

Now you just have to do the following:

(require (planet bzlib/date/srfi)) 

And almost all srfi/19 functions will be re-exported along with the functions within bzlib/date.

The exceptions are date->string and string->date, neither of which are exported from srfi/19. This is because we may want to use those names for our own date parsers and generator functions. I'll examine the details before deciding whether to re-export those or create our own.

PLT Date Wrappers

You now can use PLT date objects instead of srfi/19 date objects (I do not really know why they are different date objects in the first place...). You can just do the following:

(require (planet bzlib/date/plt)) 

Introducing BZLIB/DATE & BZLIB/DATE-TZ - Date Time Manipulation Libraries

BZLIB/DATE & BZLIB/DATE-TZ are now available on planet. They provide additional date manipulation capability on top of SRFI/19, including timezone manipulation.

They are released under LGPL.

Usage and Installation

(require (planet bzlib/date)) 
(require (planet bzlib/date-tz)) 

bzlib/date provides date manipulations. bzlib/date provides timezone manipulations. Their usages are separately discussed below.

bzlib/date

To create a date object, you can use bulid-date, which provides a more natural year/month/day/hour/minute/second order of the parameters:

(build-date <year> <month> <day> <hour> <minute> <second> #:tz <offset>) 

And it would do the right thing if you enter February 31st:

(build-date 2009 2 31) ;; => #(struct:tm:date 0 0 0 0 3 3 2009 0) 

By default, the tz offset is 0, which equates to GMT (see below for timezone support). Only year, month, and day are required.

Date Comparisons
The following function compares two dates to determine their orders:

• (date>? <d1> <g2>)
• (date<? <d1> <g2>)
• (date>=? <d1> <g2>)
• (date<=? <d1> <g2>)
• (day=? <d1> <g2>)
• (date!=? <d1> <g2>)
• (date===? <d1> <g2>)

day=? only compares the year/month/day values, and date===? means they are the same date with the same tz offset.

Conversions

You can convert between date and seconds with (date->seconds <date>) and (seconds->date <seconds>).

You can add to a date with (date+ <date> <number-of-days>). The number of day can be a non-integer.

You can find out the gaps between two dates with (date- <date1> <date2>).

You can create an alarm event with date via (date->alarm <date>) or (date->future-alarm <date>). The difference between the two is that date->future-alarm will return false if the date is in the past.

Dealing with Weekdays

To find out the weekday of a particular date, you can use (week-day <date>).

To find out the date of the nth-weekday (e.g., first sunday, 3rd wednesday, last Friday) of a particular month, use nth-week-day:

(nth-week-day <year> <month> <week-day> <nth> <hour> <minute> <second> #:tz <offset>) 

For the week-day argument, use 0 for Sunday, and 6 for Saturday. For the nth argument, use 1, 2, 3, 4, 5, or 'last. hour, minute, second, and offset are optional (same as build-date, and the other functions below that have them).

To find out the date of a particular weekday relative to another date, use one of the following:

• week-day>=mday
• week-day<=mday
• week-day>mday
• week-day<mday

They all share the same arguments, which are year, month, week-day, month-day, hour, minute, second, and offset.
The usage is something like:

;; the sunday after May 15th, 2009
(week-day>mday 2009 5 0 15) ;; 5/17/2009 
;; the friday before September 22nd, 2008 
(week-day<mday 2009 9 5 22) ;; 9/19/2009 

The hour, minute, second, and offset parameters are there for you to customize the return values:

;; the sunday after May 15th, 2009
(week-day>mday 2009 5 0 15 15 0 0 #:tz -28800) ;; 5/17/2009 15:00:00-28800
;; the friday before September 22nd, 2008 
(week-day<mday 2009 9 5 22 8 30 25 #:tz 14400) ;; 9/19/2009 08:00:00+14400

bzlib/date-tz

By default, you need to parameterize the current-tz parameter, which defaults to America/Los_Angeles. The timezone names are the available names from the olson database.

To determine the offset of any date for a particular timezone, use tz-offset:

(parameterize ((current-tz "America/New_York")) 
  (tz-offset (build-date 2008 3 9))) ;; => -18800 
(parameterize ((current-tz "America/New_York")) 
  (tz-offset (build-date 2008 3 10))) ;; => -14400 

If you want to separate between the standard offset and the daylight saving offset, you can use tz-standard-offset or tz-daylight-saving-offset:

(let ((d1 (build-date 2008 3 9))
        (d2 (build-date 2008 3 10)))
    (parameterize ((current-tz "America/New_York"))
      (values (tz-standard-offset d1)
              (tz-daylight-saving-offset d1)
              (tz-daylight-saving-offset d2))))
;; => -18800 (std)
;; => 0 (dst on 3/9/2008)
;; => 3600 (dst on 3/10/2008) 

Conversion

To reset a date's tz offset, you can use the helper function date->tz, which will reset the offset for you:

(let ((date (build-date 2008 3 10 #:tz -18800))) 
  (parameterize ((current-tz "America/New_York")) 
    (date->tz date))) 
;; => #(struct:tm:date 0 0 0 0 10 3 2008 -14400)

This function is meant for you to fix the offsets for dates that belong to a particular timezone but did not correctly account for the offset - it does not switch the timezone for you.

Couple other functions makes it even easier to work with timezone.

(build-date/tz <year> <month> ...)
(date+/tz <date> <number-of-days>)

They basically creates the date object and calls date->tz so the offset is properly adjusted based on the timezone.

Besides using current-tz, you can also pass it explicitly to tz-offset, tz-standard-offset, tz-daylight-saving-offset, date->tz, build-date/tz, and date+/tz. You pass it in in the following forms:

(tz-offset <date> "America/Los_Angeles")
(tz-daylight-saving-offset <date> "Asia/Kolkata")
(tz-standard-offset <date> "Europe/London")
(date->tz <date> "Europe/London")
(build-date/tz <year> <month> <day> #:tz "America/New_York")
(date+/tz <date> <number-of-days> "America/Los_Angeles")

Convert from One Timezone to Another

To covert the timezone of a date so you get the same date in a different timezone, use tz-convert:

(tz-convert <date> >from-timezone> <to-timezone>)

All parameters are required.

(tz-convert (build-date 2008 3 10 15) "America/New_York" "America/Los_Angeles")
;; => #(struct:tm:date 0 0 0 12 10 3 2008 -25200) ;; 2008/3/10 12:00:00-25200
(tz-convert (build-date 2008 3 10 15) "America/New_York" "GMT")
;; ==> #(struct:tm:date 0 0 0 19 10 3 2008 0) ;; 2008/10/10 19:00:00+00:00

That's it for now - enjoy.

Handling Time Zone in Scheme (5): Pre-Convert the Rules

This is part five of the timezone series - you can find the previous posts on this subject to get up to speed on the details:

1. motivation and overview
2. parsing the zoneinfo database
3. compute the offsets
4. convert the rules into dates

We previously have finished the first draft of tz-offset, and it works correctly in majority of the situations. But unfortunately, there is one issue with it:

The code will not work correctly when there are gaps between the years where the rules are applicable.

Fortunately, under regular use of the code, we will not encounter this issue. But the issue definitely exists. Below is the US rule:

  ("US"
   (1918 1919 - 3 (last 0) (2 0 0 w) 3600 "D")
   (1918 1919 - 10 (last 0) (2 0 0 w) 0 "S")
   (1942 1942 - 2 9 (2 0 0 w) 3600 "W")
   (1945 1945 - 8 14 (23 0 0 u) 3600 "P")
   (1945 1945 - 9 30 (2 0 0 w) 0 "S")
   (1967 2006 - 10 (last 0) (2 0 0 w) 0 "S")
   (1967 1973 - 4 (last 0) (2 0 0 w) 3600 "D")
   (1974 1974 - 1 6 (2 0 0 w) 3600 "D")
   (1975 1975 - 2 23 (2 0 0 w) 3600 "D")
   (1976 1986 - 4 (last 0) (2 0 0 w) 3600 "D")
   (1987 2006 - 4 (match 0 >= 1) (2 0 0 w) 3600 "D")
   (2007 +inf.0 - 3 (match 0 >= 8) (2 0 0 w) 3600 "D")
   (2007 +inf.0 - 11 (match 0 >= 1) (2 0 0 w) 0 "S"))

You can see that there is a gap between 1945 and 1967 for applicable rules. You probably are not going to calculate the offsets for 1948 most of the time, but it would be nice if we do not have to face the potential issues.

What we want is to have the last rule of 1945 continue to be applicable until 1967 in this case. And that means we need to be able to skip over multiple years, which our current design does not account for. The nice thing is that the situation is not as dire as it sounds, since most of the timezones that I inspected visually do not make use of the "US" rule during this gap. But it would be nice to know such potential bug will not exist.

As we have found out in the previous posts, inferring the "previous" rules with the data format is difficult, and it's easier to compute the applicable rules for a given year, the easiest solution is to pre-compute the dates for every year that we need to worry about. In this case, any of the gaps will automatically be filled with the previously applicable rules, with the applicable years.

The Applicable Years

It should be obvious that the timezone concept has definite bound toward the past, as GMT was not established until 1675, and US does not adopt the timezone until 1918. The minimum year from the zoneinfo database is 1916:

Handling Time Zone in Scheme (4): Rule Conversion

Previously we have discussed timezone handling in a series of posts:

1. motivation and overview
2. parsing zoneinfo database
3. calculating offsets

To continue from the third post where we are in the midst of calculating daylight saving offsets, we have figured out the applicable rules, and we now need to convert them into date structs so we can determine the exact boundary.

Going back to our two rule examples for America/Los_Angeles:

  (2007 +inf.0 - 3 (match 0 >= 8) (2 0 0 w) 3600 "D")
  (2007 +inf.0 - 11 (match 0 >= 1) (2 0 0 w) 0 "S")

We want to convert them into the applicable values (for both 2009 and the previous year - 2008):

2009/3/8 02:00:00-08:00 
2009/11/1 02:00:00-07:00
2008/3/9 02:00:00-08:00 
2008/11/2 02:00:00-07:00 

In order to do so, we'll first have to be able to convert the ON (day of the month) into the correct date value, and then we'll have to convert the AT (time of the date) into the correct time value. Let's get started.

Day of the Month

The simplest ON format is a day number (ranging from 1-31), and for that we do not have to do too much. But there are also two other formats that are based on weekdays:

'(last 0) ;; => last sunday (sunday = 0, monday = 1 ..., saturday = 6) 
'(match 0 >= 5) ;; => sunday on or after 5th 
'(match 2 <= 10) ;; => tuesday on or before 10th 

That means we need to be able to convert them to the appropriate day of the month based on the year and the month.

Doomsday Algorithm and Weekday Calculation

To be able to calculate the weekday-based date values, we first need to be able to calculate the weekday of a particular date. For that we can make use of the doomsday algorithm, which are based on the concept that there is a doomsday every month, and they are easy to remember based on a moniker (4/4, 6/6, 8/8, 10/10, 12/12, ...). The linked explanation makes it sounds more complicated than it actually is - below is the oneline doomsday algorithm in scheme:

(define (doomsday y)
  (modulo (+ 2 (floor (+ y (/ y 4) (- (/ y 100)) (/ y 400)))) 7))

Then with doomsday we can calculate the weekday of a date:

Handling Time Zone in Scheme (3): Computing Offsets

This is the third post of the timezone handling series. Please see the previous posts for additional details:

1. motivation and overview
2. parsing zoneinfo database

At this point we are able to parse the zoneinfo database files into a giant structure that contains all of the applicable zones and rules. Let's see how to utilize the data.

There are two main purposes for using the time zone information, the first is to figure out the actual offset for a given zone for a particular date:

(tz-offset (build-date 2009 7 1 ...) "America/Los_Angeles") ;; => -25200 
(tz-offset (build-date 2009 12 1 ...) "America/Los_Angeles") ;; => -28800 

And the second use is to convert between different timezones of the same date:

(tz-convert (build-date 2009 7 1 0 0 0) "America/Los_Angeles" "GMT") 
;; => (date 2009 7 1 7 0 0 0 0) 
(tz-convert (build-date 2009 7 1 0 0 0) "America/Los_Angeles" "Asia/Taipei") 
;; => (date 2009 7 1 15 0 0 0 28800)

It should be clear that tz-convert can be built on top of tz-offset, so we first should figure out how tz-offset to compute the offsets.

Basic Idea
The zoneinfo database structure more or less denotes how we should be making use of the library:

1. look up the zone - find the zone that you are looking for
2. based on the zone, map to the correct set of rules by testing which rules is applicable (based on the UNTIL field of the zone value)
3. apply the set of rules against the date to find the proper daylight saving offset

With that in mind, let's first load the parsed structure into something more useable. Let's convert it into a hashtable loading the zones by name, which in turn holds the references to the applicable rules, off of which we can then use to compute actual offsets. Assuming the zoneinfo database has been serialized to a file, then the following would load the data back in:

(define (make-zoneinfo zi)
  (make-links (make-zones (caddr zi) (make-rules (cadr zi)))
              (cadddr zi)))

(make-zoneinfo (call-with-input-file path read))

Basically - we'll first load the rules, and then pass the rules into make-zones, which will then be passed to make-links to complete building the hashtable.

With the following structure definitions for zone and rule, we have:

(define-struct zone (offset rule format until) #:transparent)

(define-struct rule (from to type month date time offset format) #:transparent)

(define (make-rules rules)
  (make-immutable-hash (map (lambda (kv)
                              (cons (car kv)
                                    (map (lambda (rule)
                                           (apply make-rule rule)) 
                                         (cdr kv))))
                            (cdr rules))))

(define (make-zones zones rules)
  (make-immutable-hash (map (lambda (kv)
                              (cons (car kv)
                                    (map (lambda (zone)
                                           (define (helper offset rule format until)
                                             (make-zone offset (hash-ref rules rule #f) format until))
                                           (apply helper zone))
                                         (cdr kv))))
                            (cdr zones))))

And make-link left-folds over the zones and the links list:

Handling Time Zone in Scheme (2): Parsing The Zoneinfo Database

Continuing from the motivation post on the difficulty of handling timezone, we'll now focus on solving the first problem - parsing the zoneinfo database.

As previously discussed, the zoneinfo database format is pretty straight forward:

• comments starts from # until the end of the line
• records are line oriented, and there are no line continuation
• there are 3 main record types:
  
  • rule - describes daylight saving rules
  • zone - describes different time zones and how the offset (and the applicable daylight saving rules) change over the years
  • link - describes aliases for the different zones

What we want to accomplish is to parse the zoneinfo database and convert into scheme values that are more consumable. Let's try to map them out.

Overall Structure

It would be nice if we can parse the files into structure that looks like:

(zoneinfo (rule ("US" rule1 ...) ...) 
          (zone ("America/New_York" zone-rule1 ...) ...) 
          (link (Link1 ...) ...)) 

The above value would be easy to load back into memory (just a singe read operation). And all of the related rules are all grouped together.

Since the records are line oriented - when we parse them it would look like the following:

(rule "US" ...)
(rule "US" ...) 
...
(zone "America/New_York" ...) 
(zone "America/New_York" ...) 
...
(link ...) 
...

Assuming we are able to generate the above records, we can group them via the group we introduced during the development of the memcached client, by first grouping on rule, zone, and link, and within each group we then further group based on the sub key, and then cons 'zoneinfo in front.

;; 2-layer grouping 
(cons 'zoneinfo 
      (map (lambda (kv)
             (cons (car kv)
                   (group (cdr kv))))
           (group zis)))

Once the transformation is done - we just need to serialize the data out to a file.

(call-with-output-file <target>
  (lambda (out)
     (print zoneinfo out)) 
  #:exists 'replace) 

Then we just need to make sure that we can parse the files (yes - the zoneinfo database consists of multiple files) so we can generate the records to feed into the 2-layer grouping. Since append the results from parsing multiple files is trivial, we'll just discuss the parsing of a single file.

Parsing The Zoneinfo File

PLT Scheme offers a lot of fantastic support for parsing files. We will not go through them all here, but at the high level, there is the parser tools that are basically lex and yacc in scheme, and there is also the parser combinators that you can use to construct your monadic combinator parsers. At a low level, you can use all of the read-* functions to read just about anything you want from an input port.

Handling Time Zone in Scheme: Motivation & Overview

Timezone is a difficult problem, more difficult than it has to be. Probably the biggest challenge is the daylight saving time. It changes with the whims of politicians and governments. Once a while, a city/state will just arbitrarily decide to change their timezone. How should timezone software handle such changes?

In such case, we might naturally assume the answer is to keep all of the same hours but for a different timezone. This answer would have been sufficient when things are not global: what if some of the appointments are with people from other timezone?

Luckily such situation does not arise very often, and neither does the changing of the daylight saving switching dates. But they illustrate the difficulty of timezone handlings.

Insufficient Solution

Most software stores an offset along with a time to denote the timezone offset from GMT, and sometimes even a check to determine whether it is a daylight saving time. In PLT Scheme - the PLT date object has both.

(define-struct date (second minute hour day month year week-day year-day dst? time-zone-offset))

But such solution is brittle in the case of date manipulations, even without the drastic circumstances above. What if you want to calculate the date four month ahead? How do you know whether or not the time should have the same daylight saving offset applied?

A possible solution is to call the C date functions, which can handle the date calculation correctly, pending the TZ environment variable. The drawback of the approach is that if your problem needs to be timezone aware, then you'll be constantly swapping your environment. Besides the fact that such approach will serialize all of your threads, it is also less desirable.

Let's see if we can bring timezone handling into scheme.

Zoneinfo Database

How does C date functions knows how to calculate dates? It consults all of the timezone information with a database called zoneinfo. This database contains all of the past and current timezones and their corresponding offsets. This database is the best authoritative source if you need to handle timezones.

On linux/mac - type man zic, and you will get the details on the format of the zoneinfo files. The zoneinfo files are line oriented, with comment lines starting with #. There are two main constructs we are interested from the files are the zones and the rules: