ananelson.com

PLEASE UPDATE YOUR FEED

2009-09-23T15:06:36+01:00

This feed has moved. If you'd like to keep reading, please move to the new feed

A DIY HTML R Transcript Viewer

2009-09-23T13:06:36+01:00

Please visit the website for source code downloads and syntax highlighting.

This is a fun and very useful little script which I wrote today that I want to share. It generates a HTML file with embedded images of the results of running an R script. So, you have a little R script:

# http://addictedtor.free.fr/graphiques/RGraphGallery.php?graph=63

palette(rainbow(12, s = 0.6, v = 0.75))
quartz()
stars(mtcars[, 1:7], len = 0.8, key.loc = c(12, 1.5),
           main = "Motor Trend Cars", draw.segments = TRUE)

And you want to run this and share the output as HTML. Well, running this command:

rout2html tiny-example.R

will evaluate the script in R, build a very pretty HTML file and include all the images. Check it out!

I’ll get to the fun stuff of how this is implemented soon, but first a little about why this is useful. Have you heard the song Baby Got Stats?

Well, if you’re a born-again convert to the command line like me, the first word that’s going to jump out at you from this song is “logfile”. Logfile? A logfile means you are working interactively in a GUI and then later you are going to go back through the logfile to extract the relevant results. This is not a good idea. It means your results are not easily reproducible. This is bad for you if you need to change something or want to double-check that you’ve done your calculations correctly. This is bad for anyone else if they want to check or review your results. A better approach is to write a script and revise that script until it gives you the output you are looking for. That’s reproducible, testable, documentable, and easily sharable.

The problem with GUI apps is that they encourage you to take a logfile approach, rather than a batchfile approach. When you fire up the R GUI, the most natural thing you can do is to start typing. Resist! GUIs should be used as a scratchpad or for interactive help. Instead open your favourite text editor, type up a script (your text editor’s syntax highlighting is going to be MUCH nicer than the R GUI’s), run it using rout2html and view the results in your web browser. Make changes to the script, run the command again, refresh your browser. Now iterate. This approach works in Windows too! (Well, in principle this approach works in Windows. For now rout2html is a bash script. I’ll look into writing a Windows .bat file or making it system-independent.)

There are other applications for rout2html. You can use this to automate a remote R script and view the results in HTML format, for instance if you wanted to analyze performance of a web application. (Check out splunk if you need to parse server logs.) If for some reason you were trying to use R but couldn’t run a GUI, like on a remote server, then you could use rout2html to generate HTML transcripts and view them via a web browser.

So, how does this work? Well, rout2html is a bash script which basically does 3 things:

Runs your R script in R via R CMD BATCH.
Highlights the resulting Rout file using Pygments
Looks for special image codes in the HTML file and replaces them with HTML image tags.

Unfortunately the R Console transcript highlighter has just been submitted to Pygments, so it’s not available in the latest released version. I’ll update this post if my patch gets accepted. In the mean time you’ll need to download the source to Pygments, apply my patch and run setup.py install or setup.py develop.

In order to get images to show up, you’ll need to have a call to quartz() or some other supported graphics device in your R script before each plot. If you’ve been working with the GUI a lot, you probably have that in there already anyway. rout2html runs a script called redefine-graphics-devices.R before it calls your script:

quartz <- function(...) {
    png(file = random_filename_generator('png'), ...)
}

X11 <- function(...) {
    quartz(...)
}

pdf <- function(file = f, ...) {
    quartz(...)
}

jpeg <- function(file = f, ...) {
    quartz(...)
}

random_filename_generator <- function(file_extension) {
    random_filename <- paste(runif(1, 100000, 1000000000), file_extension, sep=".")
    cat(file_extension, "_filename:", random_filename, "\n", sep="")
    return(random_filename)
}

This basically hijacks the graphics devices so they all generate a PNG file with a randomly generated filename, and so the R transcript contains a reference to this PNG filename. The 3rd step in rout2html looks for these filename references and converts them to HTML img tags so the final HTML actually contains the images. The nice thing about this is that, assuming you have explicitly called a graphics device before you start plotting to it, you don’t have to change a thing. You can go back and forth between running this script in the GUI via source(), or via R CMD BATCH yourself, or via rout2html, and it will work in all 3 cases. If you have specified a filename for a PDF or JPEG device, this will be ignored when run via rout2html. The only issue will arise with the PNG device. Because this device is not overwritten (for now) if you have called the png() device, then you’ll need to cat the filename manually in the same format as the random_filename_generator() function, e.g. png_filename:my-custom-filename.png, followed by a newline. Then an img tag should be created for you.

png("i-want-this-filename.png")
plot(c(1,2,3))
cat("png_filename:i-want-this-filename.png\n")

Take a look at the output.

So, in order to run this you need a modified Pygments, the rout2html script somewhere on your PATH, and the redefine-graphics-devices.R script in the same directory as your R script. My plan is to eventually have this implemented as an R package + an executable script, so that R can look after making redefine-graphics-devices available. For now, if you are skilled enough to get this working then you are probably skilled enough to modify it for your needs. Also, all HTML and PNG images will be created in the directory with your R script and redefine-graphics-devices.R. You are welcome to move the HTML and PNG after they are generated, but keep them together since the image tags just reference the filename. CSS is bundled into the header of the HTML. If you wanted to rewrite this script to use a syntax highlighting library other than Pygments, then in addition to changing the pygmentize command, you would need to change the Ruby regexp based on what the output line would look like.

#!/bin/bash

# Save any old workspace.
if [ -e ".RData.xyztmp" ]
then
    echo "A file named .RData.xyztmp already exists. Please remove this before proceeding."
    exit 1
fi

if [ -e ".RData" ]
then
    mv .RData .RData.xyztmp
fi

# Execute redefine-graphics-devices.R and save results in fresh workspace.
# Don't want an outfile here.
R CMD BATCH --no-restore --save redefine-graphics-devices.R /dev/null

# Now run the R script passed in as $1, 
# which will have access to anything defined in our new workspace.
# $2 is passed to the R script as an argument, 
# this will just be ignored unless you look for it,
# but it gives you the option to pass args to your script.

R CMD BATCH --restore --no-save "--args $2" $1 $1out
rm .RData 

# Syntax highlight and create HTML file using Pygments
pygmentize -o $1.html -O full $1out

# Now look for png_filename: or jpg_filename: and convert to HTML img tags.
RUBY_CMD="gsub(/<span class=\"go\">(png|jpg)_filename:([0-9\.A-Za-z-]+\.(png|jpg))<\/span>/, '<img src=\"\2\" \/>')" 
ruby -pe "$RUBY_CMD" < $1.html > $1-with-images.html

# Restore any old workspace.
if [ -e ".RData.xyztmp" ]
then
    mv .RData.xyztmp .RData
fi

OSSBarCamp Talk on Automated Documentation

2009-09-20T18:16:19+01:00

Please visit the website for source code downloads and syntax highlighting.

You can download PDFs of the handout from my talk at the September 2009 OSSBarCamp in the sidebar. Unfortunately the hyperlinks aren’t live in the PDFs, so here are the software projects I mentioned in my talk. The outline was done using FreeMind which I’ve just started playing around with.

Webby

Webby is primarily a tool for building static websites, but its combination of Rake and a generic filter/helper structure makes it phenominally powerful for managing any kind of collection of documents. I use Webby as the glue for my Automated Documentation “stack”.

Gorgyrella

Gorgyrella is my Ruby port of the Python library Idiopidae. Gorgyrella lets you add special comments to your source code to define sections, which you can then pull out to include in documenation or to execute with Dobby.

Dobby

Dobby is still in its infancy, but its goal is to allow you to run snippets of code in an interpreter and display the output in a document. What makes it especially convenient for writing tutorials is that you can keep a Dobby session open, run a setup script invisibly, and only show the output of running a few lines of script at a time but with the memory of any previous commands still intact. Dobby currently supports R and Python, but in principle can be adapted to support any software with a command line interface.

For syntax highlighting, I prefer Pygments but Gorgyrella also has support for UltraViolet and CodeRay. It is very easy to add support for other syntax highlighting packages.

The projects which use Webby+Gorgyrella for their documentation are:

Surpass

Write native Excel files using just pure Ruby. This one has the screenshot automation.

RBloomberg

Interface between R + Bloomberg financial data.

Specifically for Arrrrrrrrrr users¹, the rcom server allows integration with R and various Windows packages which can be scripted using COM/OLE. In particular, R2PPT integrates R with PowerPoint.

I currently use Bazaar for version control. It’s a distributed version control system and very easy to install and get started with. Once you have it installed if you want to check out the source code for Surpass to see how the documentation is compiled, then you just need to type:


bzr branch http://ananelson.com/code/surpass

and this will give you a bzr repo named “surpass” in whatever directory you typed the command. I’ll try to set something up with zipped copies of the latest version for those who don’t want to install Bazaar. Hey! I can automate that with Webby! ;-)

¹ Laura picked a great day for OSSBarCamp.

A Comments Engine for Webby

2009-07-05T16:08:33+01:00

Please visit the website for source code downloads and syntax highlighting.

The most common complaint about using Webby for a blogging platform is probably “with static HTML files how can you handle comments?”. The standard answers are:

Don’t bother with comments.
Use Disqus or some other 3rd party commenting plugin.
Build some sort of commenting engine yourself.

Today I am going to describe my contribution to this third option. I have extracted the comments engine I use for this blog. For now, you need to install it from source since it’s not ready to be pushed out as a gem.

bzr branch http://ananelson.com/code/webby_email_comments

A gem and a nicer name will be forthcoming when I think they are deserved. For now, the code works but it’s very hard-coded to the way I do things, so if you aren’t comfortable modifying this to the way you work, you’re better off waiting until this is a little more polished and configurable.

Here’s basically how this works:

Someone fills in a comment form on one of my blog posts.
The form is posted to a PHP script which generates an email which goes to a dedicated email account.
Every time I run a “webby build”, a script checks for new emails and pulls the contents into a local DataMapper database. The emails are marked as read once they have been successfully downloaded.
When a blog post page is rendered, it checks in this DataMapper database and pulls in any comments for that page.

This does mean that you have to render every page freshly in order to include a new comment. There are other possible approaches, such as using JavaScript calls to incorporate comments dynamically. I took this one first since it was the easiest to implement, and it means that comments are available for indexing to search engines.

If I want to refresh my local cache of comments, I can simply delete the sqlite3 database, mark all the emails as unread, and the next time I build they will all be fetched again. If I receive a spam comment or an unwanted comment, I simply delete the email (or just mark it as read) before running “webby build”. I can also interact with comments, delete unwanted ones etc., via DataMapper. If you have legacy comments you want to include, you can just import them into the database.

Installing WebbyEmailComments

WebbyEmailComments depends on DataMapper and JSON. To get the source, you’ll need Bazaar and to build the gem, you’ll need bones.

sudo easy_install bzr # Assuming you have Python's SetupTools installed, or see bazaar-vcs.org for install directions
sudo gem install bones
sudo gem install json
sudo gem install dm-core dm-more

bzr branch http://ananelson.com/code/webby_email_comments
cd webby_email_comments
sudo rake gem:install

In your webby project Sitefile, you need to require "webby_email_comments".

Configuration

Webby uses the Loquacious config engine. This makes it very easy to add additional config settings to Webby, which is done in lib/config.rb in webby_email_comments:

Loquacious.configuration_for(:webby) {
  email nil, :desc => "Webby author's email address (your own personal email address)"
  
  comments {
    email nil, :desc => "Your dedicated email address for comments"
    login nil, :desc => "The login id to log in to the email server, probably the same as your email address"
    password nil, :desc => "The password to log in to the email server"
    reply_template nil, :desc => "A message automatically sent to everyone who posts a comment to let them know their comment was received. No message sent if this is nil."
    host 'imap.gmail.com', :desc => "IMAP server, defaults to gmail's IMAP"
    port 993, :desc => "IMAP port"
    ssl true, :desc => "Whether to use SSL for connecting to IMAP"
    db "email-comments-db.sqlite3", :desc => "Filename for database to store comments."
    php_bin "emailbin", :desc => "Location to put PHP handler files."
    inbox "Inbox", :desc => "The name of the Inbox IMAP folder. This folder is checked for new comment messages."
  }
}

You can set all of these new configuration options in your Sitefile, just as you do your other Webby options, as shown below.

You can view all the Webby configuration options and their current values by calling:

<%= Loquacious.help_for(:webby).show :values => true %>

It gives you quite a lot of detail, here is a short excerpt:


    Location to put PHP handler files.
     - comments.php_bin          => "emailbin"

    IMAP port
     - comments.port             => 993

That’s a very convenient way of seeing what settings are available, and checking that values are being set correctly.

Using WebbyEmailComments

Create a Comment Form

There is a comment form helper built in. In your layout template, just call <%= comment_form %> whereever you want the comment form to appear. I actually call the comment form like this:


    <%= comments_for_page %>
    <br />
    <% if @page.dirty -%>
    <%= comment_form %>
    <% end -%>

comments_for_page is the helper which displays the actual comments.

This means that in order to enable comments on a blog post, I must add dirty: true to the Webby metadata for that page. This is a good idea anyway since it will mean that new comments will get added to that page every time you build your site.

There are a few spam prevention attempts built in to WebbyEmailComments (I have no idea how effective they are as I haven’t studied my logs yet). One of them is to have a secret code built in to the form and the receiving PHP script. In order for this to work, the PHP script and the comment forms must be generated at the same time so they share the same key.

Hence, I recommend you follow the same procedure:

To enable comments on a blog post, set dirty=true.
In your blog layout, show the comment form only if dirty=true.
Ensure that dirty=true is kept in the PHP form (it’s there by default).
When you are ready to close comments for a post, simply remove dirty=true from the metadata.

Also, in your CSS you should have something like this:

.emphasise {
    visibility: hidden;
    display: none;
}

to hide the math question which is just there to trick bots.

PHP Processing

There is a template/ directory in webby_email_comments which contains the PHP script to process completed forms, along with a copy of XPertMailer, an open source PHP library to create emails. I have minimal PHP experience so if anyone wants to suggest a better library, I’m happy to add that in, although XPertMailer seems to work very well and is very easy to use.

This PHP script is dynamic, it gets processed via ERb each time to incorporate the secret key and to fill in user preferences like what your email address for comments is.

If you run the webby_email_comments executable (which should be available after you install the gem), from the root directory of your webby installation, it will create a directory called “emailbin” in your webby content directory and install your PHP handling and the XPertMailer library. You can change the name of this directory in the config.

You can then edit the PHP code yourself if you want to make any changes, and you should just upload this directory along with the rest of your site, taking care to check that the permissions on the PHP files are correct (e.g. not world-writable).

Moderating Comments

The easiest way to get started is to create a gmail account like “blog-comments-ananelson@gmail.com” and specify this in your Sitefile as follows:

require "webby_email_comments" # must require this before setting configs

SITE.email = 'ana@ananelson.com'
SITE.comments.email = 'my.comments.email.address@gmail.com'
SITE.comments.reply_template = "Hey, thanks for leaving a comment. :-) ..."
SITE.comments.login = 'my.comments.email.address@gmail.com'
SITE.comments.password = 'passw0rd' # yeah, right

The SITE.comments.email is your dedicated email for comments, SITE.email is your real email address and the reply_template is an acknowledgement message which gets sent back to people who leave a comment. (No acknowledgement is sent if reply_template is nil.) A copy of each comment gets sent to your personal email as well as the comments email so you know when you have received a comment. Just delete emails of unwanted comments in your comments.email, or mark them as spam if you want to try to train gmail (or whatever spam filter you use) to recognize spam.

I have a rake task to check for new comments.

task :comments do
  puts "initializing comments database..."
  WebbyEmailComments.db_init

  puts "fetch latest comments..."
  if WebbyEmailComments.process_inbox
    puts "ok"
  else
    puts "unable to connect to comments email, skipping..."
  end
end

process_inbox does all the work. It returns false if there has been a SocketError, such as when you don’t have internet access, and true if all is well.

By specifying task :build => :comments, you can ensure that the latest comments are fetched every time you rebuild. If you want to check comments more methodically, then don’t add this dependency and run webby comments manually.

Currently only IMAP is supported. The default host, port and ssl settings will work if you use gmail. Otherwise set the variables in your Sitefile to the appropriate values.

The database defaults to email-commends-db.sqlite3 in your webby root directory, you can change this (such as to put it in a comments/ directory or somewhere else out of the way) by specifying SITE.comments.db in your Sitefile.

Displaying Comments

As discussed above, place <%= comments_for_page %> in your template wherever you wish comments to be displayed.

Getting Help, Asking Questions

If you have questions, feel free to post a comment here or on the Webby mailing list. Once again, this is early days and for now this is intended for people who are pretty comfortable with using Webby already.

Using Bazaar With Capistrano on Rails Machine

2009-03-23T22:05:37+00:00

Please visit the website for source code downloads and syntax highlighting.

How to convert a Subversion repository to Bazaar using Tailor.
Modifying your deploy.rb to use Bazaar for Capistrano deployment.

Why?

I have been using the standard Rails Machine Capistrano setup (as described in the original Five-Minute Rails Application Deployment page). The only alteration I have made to this is the use of SVK as a local wrapper for Subversion. (I started writing this blog post some months ago, and the Rails Machine instructions now suggest using Git rather than Subversion for version control.)

My setup has served me well overall, but apart from not getting to use Bazaar (which I really enjoy), I have 2 gripes with it:

If I am starting a new project which I plan to deploy on Rails Machine, I have to either start over with a new repository when I am ready to deploy (losing my repository history up to that point), or I have to manually upload and import my repository contents after creating a blank repository as per the Five Minute guide.

If I remove a project from Rails Machine, I have the same problem in reverse. I need to manually export and recreate the repository elsewhere if I want to keep it.

These aren’t everyday issues, I probably have to deal with this sort of situation only a few times every year. But, when I was starting a new project recently I began working with Bazaar and, when the time came to deploy, I decided to try to find a way around switching to Subversion. After that, I became anxious to convert all my deployed projects to the same setup.

So, my new approach is:

A local Bazaar repository for development, pushed to some other remote location for backup and remote access.
A Capistrano deployment recipe based on the Copy strategy. The Copy strategy is very straightforward, and it avoids the need to install Bazaar (and all the extra bits needed for SSH) on your deployment server.

Get Out of Subversion

The first step in the process is to convert my existing Subversion repositories to Bazaar. You can, of course, choose to ignore your version history and just import the current version of your project into a blank Bazaar repository. You lose your version history this way, but for some projects the simplicity of this approach might outweigh the posterity value of having the full code history.

There is, however, a tool which will let you export your whole repository history from Subversion into Bazaar. There are, in fact, several such tools, but I am going to use Tailor which supports several version control systems (including Git).

Unless you happen to have darcs already installed, I don’t suggest you try to get the latest development version of Tailor. Much easier to just download a released version of Tailor. If you do decide to install darcs, read this page first, especially the part about how the GHC can take 3 hours to compile.

Using Tailor

Preliminaries

Tailor needs a directory in which it can do its dirty work. To ensure we always start with a clean setup, I prefer to write a script to prepare the environment:

mkdir ~/tailor-tmp
rm ~/tailor-tmp/tailor.state

rm -rf ~/tailor-tmp/work/
mkdir ~/tailor-tmp/work/

This can and should be run prior to each time you actually run tailor. (The workflow I am describing here is for a once-off use of Tailor to convert a repository. Tailor can also be used to keep 2 repositories in sync, in which case of course you would not delete the working directory and state file.)

Creating a Config File

The tailor documentation describes the options available to you in the config file, but it’s not written in a way which makes it easy to write a file from scratch. I eventually figured out a recipe which worked for me by getting Tailor to create a blank config file and then tweaking it.

You can get Tailor to generate a blank config file as easily as:

tailor --verbose > blank-tailor-config.tailor

However, moving into ~/tailor-tmp first and passing a few simple tailor switches makes the config file much more complete:

cd ~/tailor-tmp
tailor --verbose -s svn -t bzr -R /path/to/repo > blank-tailor-config.tailor

This gives you:

[DEFAULT]

verbose = True
[project]

target = bzr:target

start-revision = INITIAL

root-directory = /Users/ana/tailor-temp

state-file = tailor.state

source = svn:source

subdir = .
[bzr:target]
[svn:source]

repository = /path/to/repo

You can read more about all these options in the projects section of the Tailor documentation.

The only item we need to change in [DEFAULT] or [project] is to set

subdir = work

The subdir is where Tailor does its magic, and it’s crucial to start with an empty subdirectory each time you run Tailor, hence this directory (along with the tailor.state file) is deleted and recreated in the prepare.sh script.

Using start-revision = INITIAL is not the same as start-revision = 1. start-revision = INITIAL uses --stop-on-copy to help calculate when the current repository was created and, hence, when it should start its export. If you are working off a repository which was forked midway through its life, but you nonetheless want the entire history, then set start-revision = 2 (or whenever the trunk/ directory was added to the repository, probably revision 2 if Capistrano created your repository for you). You might also decide to look at your log and skip over the first few irrelevant commits.

Next, we have to tell Tailor where our source repository is. The source repository is the remote Subversion repository on your Rails Machine server. You should replace /path/to/repo with the actual location of your remote repository, and then to indicate that we only want the trunk, we add module = trunk.

[svn:source]
repository = svn+ssh://deploy@subdomain.railsmachina.com/var/www/apps/appname/repos
module = trunk

If you need branches or tags, then you can run Tailor again and export these separately, or if you want to keep a Subversion-style setup (i.e. trunk/ branches/ tags/) you can set module = /. Or, take a look at the svn-tags and svn-branches options in the Tailor docs.

We don’t need to specify a target repository, since Tailor will automatically initialize a blank Bazaar repository when it runs.

Once the config file is all ready, actually running tailor is a one-liner:

tailor --configfile config.tailor

If you have set up SSH keys, then you can sit back and relax, watching the verbose output. If you don’t have SSH keys, you will need to enter your password several times during the process, at least once for every revision which is exported.

If all goes smoothly, then the ~/tailor-tmp/work directory will be a bazaar repository. You can copy this cleanly by branching it:

bzr branch ~/tailor-tmp/work ~/bzr-repositories/projectname

You should bzr push this to a remote location for safe-keeping.

If you have multiple repositories to export, all you need to change is the source repository location and you can run the script again.

Troubleshooting Tailor

If you get strange error messages, make sure you have deleted all temporary files. If in doubt, backup your config file and remove the entire ~/tailor-tmp/ directory and start over. (Note: there is no need to have the config file stored in ~/tailor-tmp/, it can be anywhere.)

It’s possible that Bazaar will choke on importing some portion of your Subversion output. I have stumbled across bizarre combinations of adding and deleting directories which confuse Bazaar. Hopefully these minor bugs have all been fixed in the latest version. If this happens to you, do file a bug. If you can’t get around it, then best option I can suggest is to start the import at a higher revision. If Bazaar doesn’t like revision 11, then set start-revision = 12 in your config file and live with the loss.

Using Capistrano with Bazaar

There are just a few settings to change in deploy.rb to use your new Bazaar repository for deployment.

Change the :repository switch from something like this:

set :repository, "svn+ssh://#{user}@#{domain}#{deploy_to}/repos/trunk"

to something like this:

set :repository, "."
set :deploy_via, :copy
set :copy_strategy, :export
set :copy_cache, true

And you’ll need to specify the form of version control being used. Since Subversion is the default, there may be a commented-out line of code in your deploy.rb like this:

#set :scm,"subversion"

Uncomment this line and use the string “bzr” to indicate Bazaar:

set :scm, "bzr"

There is one final tweak necessary, and this is a slight alteration to the railsmachine gem itself.

You’ll need to add bzr to the list of allowed version control systems, and you’ll need to create an empty file named bzr.rb in the lib/railsmachiine/recipes/scm/ directory.

With these changes made, you should now be able to cap deploy from your new Bazaar repository.

Surpass Talk to Python Ireland

2009-03-23T20:05:37+00:00

Please visit the website for source code downloads and syntax highlighting.

I gave a talk about Surpass and XLWT recently to Python Ireland, as part of an evening themed as joint Ruby/Python talks. Surpass is a native Ruby library for creating Excel spreadsheets, which I will be writing a lot more about in the near future. It's a port to Ruby of the Python library XLWT. Vicky Lee did a fantastic job of recording and editing my talk, and it's up on Vimeo if you'd like to hear me talk about why you might want to create Excel spreadsheets, and how to do it with Surpass (or XLWT).

You can also download the slides in PDF format.

Python/Ruby Ireland Talks - March 2009 (1 of 3) from whykay on Vimeo.

Inspect an Array of Variables

2008-10-23T20:05:37+01:00

Please visit the website for source code downloads and syntax highlighting.

Most likely, everyone’s been doing this forever but I just figured it out:

i = 1
want = "want"
to = 2
see = :inspect
lots = 1..100
of, variables, at, once = "of variables at once".split

puts "Generally, I prefer this:"
puts [i, want, to, see, lots, of, variables, at, once].inspect

puts "\ninstead of this:"
puts i, want, to, see, lots, of, variables, at, once

Generally, I prefer this:
[1, "want", 2, :inspect, 1..100, "of", "variables", "at", "once"]

instead of this:
1
want
2
inspect
1..100
of
variables
at
once

It’s a handy shortcut which I discovered while porting a Python project to Ruby recently, which involved lots of manual debugging.

it, works, for_python, too = range(0,4)

print(it, works, for_python, too)

A Rails Application Factory

2008-08-28T21:14:19+01:00

Please visit the website for source code downloads and syntax highlighting.

This is a post about creating fresh Rails applications quickly and maintaining them easily over time. The approach I take here could easily be applied outside of Rails to any software project.

My Philosophy of Friction

I like to think about productivity in terms of “friction”. My inner economist wants me to call this “marginal cost” but my inner consultant thinks “friction” will sell better. The principle I try to organize by is that things which you want yourself to do on a regular basis: exercise, eat well, churn out a shiny prototype in next to no time; should be quick and painless. You should try to reduce the friction between you and your desired outcomes. So, make it easy to eat well by stocking the kitchen with plenty of healthy, easy-to-prepare food. If you never go to the gym because there’s never any parking, or you have to make a tricky right turn to get into the car park (I live in Ireland, remember, where right turns are the tricky ones), this is friction and the more friction there is, the less likely you are to do something. Join a different gym, or make yourself consciously aware of the friction and make the decision to go to the gym anyway. Friction is nefarious since it works on our subconscious, sometimes just being aware of it can remove its influence, but staying aware of it is difficult.

Reducing friction for good things, and increasing friction for bad things (put those cookies on the TOP shelf), can be a simple and effective guideline for orienting your practices towards your goals. (But, please do remember that marginal thinking does not show you the entire picture. Somewhere you need to think about the sum of what you are doing, not just marginal increases or decreases. Friction is very useful as a short-term heuristic, though, and it’s also usually the right tool for predicting behaviour in other people, especially users of software. Check out Why We Buy by Paco Underhill, one of my all-time favourite books for its content and basically all about friction.)

So, this week I decided to reduce a few forms of friction in my programming life. To begin with, I wanted to improve my strategy for creating a fresh Rails application.

Creating A Fresh App

Of course, actually building a generic Rails application is very straightforward, you simply type:

rails example

and you will have a new directory called example which contains a skeleton Rails application. Very quickly, though, a skeleton Rails application isn’t going to be enough. Most developers will have a collection of plugins, rake tasks and other goodies which they want to install on every application. I head straight for rspec and active_scaffold. Also, I don’t have any use for public/index.html or the test/ directory so I will always delete these. The details don’t really matter, the point is that there is a series of repetitive tasks which we do whenever we set out to create a new Rails application.

My original approach to automating this was to write a shell script which went through the various steps for me:

# Usage: bash setup_rails projectname
rails $1
cd $1
ruby script/plugin install svn://rubyforge.org/var/svn/rspec/tags/CURRENT/rspec
ruby script/plugin install svn://rubyforge.org/var/svn/rspec/tags/CURRENT/rspec_on_rails
ruby script/generate rspec
ruby script/plugin install http://svn.caldersphere.net/svn/main/plugins/rspec_autotest
svn export http://activescaffold.googlecode.com/svn/trunk vendor/plugins/active_scaffold
svn export http://activescaffoldexport.googlecode.com/svn/trunk vendor/plugins/activescaffoldexport
svn export http://dev.agent.ie/svn/rails/generators/ajax_rspec_resource/ lib/generators/ajax_rspec_resource
ruby script/plugin install exception_notification

echo "About to drop and recreate development and test databases. Next password prompt is for mysql user password."
echo "drop database if exists $1_development;" | mysql -u root -p
echo "drop database if exists $1_test;" | mysql -u root -p
echo "create database $1_development;" | mysql -u root -p
echo "create database $1_test;" | mysql -u root -p

Wow, mysql and svn, that takes me back. ;-)

With this script, I could create a new Rails application in a few seconds and my favourite plugins would be installed and configured, databases would be set up and a custom generator installed and ready to go.

This was definitely better than nothing, but it had some problems. I would tend to have a spurt of starting a few new projects, and then there would be a lull of a few months where I was working on existing projects. By the time I went back to use this script again, things would have changed. Plugins would have moved or I would be using different ones. Or, I would forget I had written the script and find myself halfway through setting up an application manually before I remembered that, yes, I invented an easier way to do this once upon a time. But, the crucial flaw in the setup_rails.sh concept is that it does nothing to help you automate the maintenance of your Rails applications. I had automated the birth process, but not the much more onerous ongoing maintenance which is necessary for any piece of software with external dependencies.

So, finally, this morning I worked out a strategy which is going to solve all of these problems. There’s no script involved, just some really easy branching and merging using, in my case, the Bazaar version control system.

The strategy is:

create a skeleton Rails application configured just the way I like it named template-rails
each time I want to create a new Rails application, I bzr branch template-rails my_new_app. This creates a copy of the contents of template-rails in a new directory called my_new_app, and the my_new_app branch “remembers” that it originated from template-rails.
each time I upgrade a plugin, tweak a rake task, write a new spec helper or anything else that I want to share amongst more than one application, I make this change to template-rails and then bring the changes into each application by just typing bzr merge and, later, committing the changes.

So, this covers both creating and maintaining applications in one easy step. Because I interact with version control on a daily basis, I’m not going to forget what my setup is, whereas I might easily forget about a six-month-old shell script tucked away in some bin/ directory.

Bazaar Tricks

I’m sure you could do this with any version control system that has good (low-friction!) support for branching and merging. I happen to be partial to bazaar because, coming from subversion, it took me 5 minutes to learn the basics but I’m still discovering wonderful little features that make my life so much easier. Just this morning, for example, I learned that if you bzr rm --new --keep, bazaar will remove whichever files you have just accidentally added to your repository without deleting them, they just go back to being “unknown” files. This is GREAT when you add several directories without adding, say, “.git” to your .bzrignore file. You can just undo the add, tell .bzr to ignore the pattern, then safely add just the files you want. When I use a plugin from github I git clone it and leave the git branch information intact. It doesn’t conflict with bzr and I can easily pull down changes.

A trick which is particularly useful for working with a template like I have been describing is described in Section 7.2.3 of the Bazaar user guide. While I would probably go into template-rails/ to pull down the latest version of a plugin, if I am editing something I have written myself like a rake task or a spec helper, chances are I’m going to be working on this and testing this in a live application. But, once I have upgraded or fixed the code, I probably want to bring this change back into my template so it can be shared with my other apps and any apps I create by branching this template in the future. Here is a nice little workflow assuming I start out in my example/ application directory:

cd ../template-rails
bzr merge --uncommitted example/
bzr diff
bzr commit . -m "Refactor matchers to derive description automatically from class name."
cd ../example
bzr merge
bzr commit . -m "Merged changes from template."

The merge --uncommitted example/ is just a fantastic little command which gets whatever changes you have made in the example/ branch but not yet committed (this does not know about “unknown” files, so if you create a new file you have to add it to the repository for it to show up here). So, you can happily develop and test in any of your branches and pulling the changes back to the template is a couple of quick commands away.

Sharing Code

A side effect of adopting a system like this is, because you have reduced the pain (or friction, or marginal cost) associated with sharing code between your applications, you are more likely to share code. You are more likely to write test helpers, custom rake tasks and your own plugins. Sharing code between your applications is a great way to improve it and the lessons you learn from testing your plugin on its 2nd Rails application will improve its performance and stability in the Rails application you originally developed it for. You are more likely to keep all your Rails applications running on an up-to-date version of Rails too.

I have mentioned Rails and some Rails-specific terminology quite a lot, but of course the idea of setting out a standard structure in a template and branching from this could be applied to very many situations. In fact, because Rails has a pretty good directory structure by default, I’m guessing I will feel the benefit of this approach much more in non-Rails project. For example, I am doing a lot of data analysis work using just DataMapper and a console, no web framework of any kind required, and I have started to work with a certain self-imposed project structure but now I am going to set up a template so that I don’t succumb to the temptation to re-invent this structure on every project.

One of the reasons I am pretty excited about sharing code between applications easily is that I have been developing some very fun tools over the past few weeks for rapid model prototyping in Rails using rspec and railroad. Stay tuned.

JRuby Wrapping For ANTLR

2008-06-27T18:33:55+01:00

Please visit the website for source code downloads and syntax highlighting.

I use ANTLR, a fantastic parser generator, for various parsing tasks and experimenting with DSLs¹. ANTLR is one of the tools that makes it possible to “write code that writes code”. Previously I wrote about capturing Java’s output and error streams in JRuby, and I used this trick to write a wrapping for ANTLR which makes it very easy to unit test grammars as well as to integrate ANTLR parsers into a Ruby class. I call it AntlrVelvet.

AntlrVelvet is just a Ruby module, so you need to require the file and then include the module in a class:

require 'antlr_velvet'

class Expr
  include AntlrVelvet  
end

And… that’s it. We now have a parser! By default, AntlrVelvet assumes you have given your class the same name as you gave your parser, in this case Expr. (You can easily override this.) ExprLexer.class and ExprParser.class need to be on the Java CLASSPATH somewhere, but if they’re in the same directory as antlr_velvet.rb this will be taken care of automatically.

Let’s take a quick look at Expr.g, the ANTLR grammar file, and see what super-powers we have given our Ruby class with these 4 lines of boilerplate.

// http://www.antlr.org/wiki/display/ANTLR3/Expression+evaluator
// [The "BSD licence"] Copyright (c) 2005-2008 Terence Parr All rights reserved.

grammar Expr;

@header {
import java.util.HashMap;
}

@members {
/** Map variable name to Integer object holding value */
HashMap memory = new HashMap();
}

prog:   stat+ ;
                
stat:   expr NEWLINE {System.out.println($expr.value);}
    |   ID '=' expr NEWLINE
        {memory.put($ID.text, new Integer($expr.value));}
    |   NEWLINE
    ;

expr returns [int value]
    :   e=multExpr {$value = $e.value;}
        (   '+' e=multExpr {$value += $e.value;}
        |   '-' e=multExpr {$value -= $e.value;}
        )*
    ;

multExpr returns [int value]
    :   e=atom {$value = $e.value;} ('*' e=atom {$value *= $e.value;})*
    ; 

atom returns [int value]
    :   INT {$value = Integer.parseInt($INT.text);}
    |   ID
        {
        Integer v = (Integer)memory.get($ID.text);
        if ( v!=null ) $value = v.intValue();
        else System.err.println("undefined variable "+$ID.text);
        }
    |   '(' expr ')' {$value = $expr.value;}
    ;

ID  :   ('a'..'z'|'A'..'Z')+ ;
INT :   '0'..'9'+ ;
NEWLINE:'\r'? '\n' ;
WS  :   (' '|'\t')+ {skip();} ;

Expr.g is taken directly from the ANTLR wiki, it is an expression evaluator. We can assign integer values to variables and perform simple arithmetic operations.

Let’s start feeding in some strings. If you have JRuby and ANTLR installed, you can download antlr_velvet_demo.rb etc. and play along.

AntlrVelvet has convenience methods for parsing files as well as strings. We’ll just parse strings. To initialize a new string parser, you do:

parser = Expr.string_parser("1 + 1")

parser is now all ready to parse, but the parsing hasn’t happened yet. Because AntlrVelvet is designed for unit testing, it doesn’t make any assumptions about how you want to parse. If you call parser.prog, it will parse according to the Expr grammar’s main prog rule. You might only want to test a part of your grammar, however, such as an atom or an expr. You can call any rule defined in your grammar, e.g. parser.atom, and the string you passed at initialization will be evaluated against that rule.

So, let’s try this now:

puts parser.prog

we get:

line 0:-1 missing NEWLINE at '<EOF>'

Oops. Okay, we tried comparing "1 + 1" to the prog rule and we got an error, because the prog rule expects to process one or more stat rules, each of which are newline-terminated. If we add a newline at the end:

parser = Expr.string_parser("1 + 1\n")
puts parser.prog

we get:

nil

nil? Honestly, how hard is 1 + 1 = 2? Okay, looking back at Expr.g I see that prog doesn’t actually return a value. Let’s try an expr since according to the grammar that should return an int.

parser = Expr.string_parser("1 + 1")
puts parser.expr

2! Excellent. After all this we can add 1 and 1.

Now, let’s take another look at stat, since that really does seem to be the main action. We don’t return anything, but we do write to System.out.println(). Remember AntlrVelvet captures Java’s output and error streams so it can look for parsing errors and raise an exception if it sees one. If you want to get at the contents of the output stream, you do this via the output method.

parser = Expr.string_parser("1 + 1\n")
parser.stat
puts parser.output.inspect

["2"]

So, we have an array with a 2 in it. This is because stat prints out expr.value every time it gets called. Now, let’s try calling prog again with some additional input.

parser = Expr.string_parser("
1 + 1
x = 1
y = 2
3*(x+y)
")
parser.prog
puts parser.output.inspect

["2", "9"]

The first “2” comes from 1 + 1. The lines x = 1 and y = 2 are of the form ID = expr NEWLINE, they are variable assignments rather than expr evaluations, so nothing is added to the output buffer, instead the parser executes the memory.put ... action. The final line is an expr which evaluates to 9 and so this is added to the output buffer. Note the extra newline at the end of our input, if this wasn’t there we would end up with an EOF error. (Of course you can easily write a grammar which doesn’t need a newline at the end if you want to.)

Let’s do one more example.

puts Expr.string_parser("1 + 2 * (3 + 4 * (5 + 6 * (7 + 8 * 9)))").expr

There’s some nice mutual recursion in Expr.g that lets it handle nested parentheses like this, go check it out if you haven’t seen it before.

Now, Expr.g doesn’t do anything mathematically that we couldn’t do in Ruby. What it does do is let us accept a string and only perform this calculation if the string conforms to our grammar. So, if, say, you were writing a web application and wanted to allow your users to perform arithmetic on your website, but for some reason you didn’t feel like doing this via² <%= eval(params[:arithmetic_expression]) %>, something like an ANTLR grammar might let you nicely sandbox and sanitize your user’s inputs. And, Expr.g might not be able to do anything all that fancy, but a more complex grammar might give you (and your untrustworthy users) some really interesting functionality.

So, to recap, Expr.string_parser returns an object of class Expr, a Ruby class. We can call any of the grammar’s rules as a method on that object and they will parse the string according to that rule, and the method will return a value if the rule returns a value.

After we have done the parsing, we can call parser.output to see what’s in the output buffer. This will either be an array of strings or nil. We can also call parser.parser or parser.lexer to return the ExprParser or ExprLexer, the actual ANTLR objects which do the parsing. Remember Expr is just our wrapper class. We can even call parser.token_stream. And, since this is JRuby, we can have some introspection fun with these objects:

puts java_methods(parser.parser.methods)

already_parsed_rule
atom
backtracking_level
begin_resync
consume_until
display_recognition_error
emit_error_message
end_resync
equals
expr
grammar_file_name
hash_code
input
java_class
java_object
match
match_any
memoize
mismatch_is_missing_token
mismatch_is_unwanted_token
mult_expr
notify
notify_all
number_of_syntax_errors
prog
recover
recover_from_mismatched_set
report_error
reset
rule_invocation_stack
rule_memoization_cache_size
source_name
stat
synchronized
to_java_object
to_string
to_strings
token_names
token_stream
trace_in
trace_out
wait

puts java_methods(parser.token_stream.methods)

consume
discard_off_channel_tokens
discard_token_type
equals
get
hash_code
index
java_class
java_object
la
lt
mark
notify
notify_all
release
reset
rewind
seek
size
source_name
synchronized
to_java_object
to_string
token_source
tokens
wait

puts java_methods(parser.lexer.methods)

already_parsed_rule
backtracking_level
begin_resync
char_error_display
char_index
char_position_in_line
char_stream
consume_until
display_recognition_error
emit
emit_error_message
end_resync
equals
grammar_file_name
hash_code
java_class
java_object
line
m_id
m_int
m_newline
m_t__10
m_t__11
m_t__12
m_t__13
m_t__8
m_t__9
m_tokens
m_ws
match
match_any
match_range
memoize
mismatch_is_missing_token
mismatch_is_unwanted_token
next_token
notify
notify_all
number_of_syntax_errors
recover
recover_from_mismatched_set
report_error
reset
rule_invocation_stack
rule_memoization_cache_size
skip
source_name
synchronized
text
to_java_object
to_string
to_strings
token_names
trace_in
trace_out
wait

So, here we have the internals of ANTLR laid out. You can see the values of any of these methods. Play with them. Use them for testing. Go into an interactive jirb session and follow the lexing and parsing step-by-step until you know exactly how the grammar does its thing.

By the way, AntlrVelvet uses method_missing to pass method calls like prog and stat on to the embedded ExprParser object, but you could also explicitly define any of these methods and call the ExprParser yourself. This can be useful if you want to do some Ruby post-processing or perhaps debugging. If you do this, you will need to invoke @AntlrVelvet@’s capture_java_streams yourself if you don’t want Java writing to your console. And remember, in JRuby Java’s System.out and Ruby’s $stdout are totally independent of each other. So if you have Ruby’s $stdout redirected to a logfile, Java’s System.out will still write to the console unless it, too, has been redirected.

class Expr
  def atom
    puts "Your atom is " + @parser.atom.to_s
  end
end

Expr.string_parser("99").atom

Your atom is 99

On to testing! I’m just using Ruby’s Test::Unit for now. To begin with, I define some custom assertions and convenience methods:

  def assert_nothing_left(parser)
    assert(parser.token_stream.index == parser.token_stream.size)
  end

  def assert_something_left(parser)
    assert(parser.token_stream.index < parser.token_stream.size)
  end
  
  def assert_valid_input(method, input_string)
    assert_nothing_raised do
      parser = Expr.string_parser(input_string)
      parser.send(method)
      assert_nothing_left(parser)
    end
  end
  
  def assert_too_much_input(method, input_string)
    parser = Expr.string_parser(input_string)
    parser.send(method)
    assert_something_left(parser)
  end
  
  def assert_invalid_input(method, input_string)
    assert_raise(AntlrError) { Expr.string_parser(input_string).send(method) }
  end
  
  def parsed_value(method, input_string)
    Expr.string_parser(input_string).send(method)
  end
  
  def parser_output(method, input_string)
    parser = Expr.string_parser(input_string)
    parser.send(method)
    parser.output
  end

To recap the examples from earlier in test form:

  
  def test_prog_without_newline
    assert_invalid_input(:prog, "1 + 1")
  end

  def test_prog_with_newline
    assert_valid_input(:prog, "1 + 1\n")
  end
  
  def test_expr
    assert_equal 2, parsed_value(:expr, "1 + 1")
  end

  def test_stat_output
    assert_equal ["2"], parser_output(:stat, "1 + 1\n")
  end
  
  def test_prog_output
    str = "
     1 + 1
     x = 1
     y = 2
     3*(x+y)
     "

     assert_equal ["2", "9"], parser_output(:prog, str)
   end
   
   def test_nasty_expr
    assert_equal 3839, parsed_value(:expr, "1 + 2 * (3 + 4 * (5 + 6 * (7 + 8 * 9)))")
   end

The assert_valid_input, assert_invalid_input and assert_too_much_input just take a symbol and an input string for the parser. assert_invalid_input expects the parsing to throw an AntlrError, meaning that ANTLR wrote something to Java’s System.err. This is used when you want to make sure that the input string is illegal for the rule in question, either too short or containing illegal characters or sequences. assert_too_much_input means that the input is acceptable, but after processing there’s something left over. In order for assert_valid_input to pass, ANTLR must not find any errors and the input string must be entirely consumed.

parsed_value and parser_output also take a symbol and an input string as arguments. They parse the string and return the grammar rule’s return value or the text written to System.out respectively. Then you can write assertions for what the value or output should be, as shown. assert_nothing_left and assert_something_left are mostly for internal use, you have to initialize and run a parser yourself before passing the parser in to these functions, but of course you can call these directly if you want to.

I use ANTLR’s Java target because, as of this writing, the Ruby target for ANTLRv3 isn’t complete enough to be practical. Even though I’m not a fan of Java, I have to say that I really don’t mind the little bit of it I have to write in a ANTLR grammar. If you are a Ruby programmer, don’t let a bit of Java scare you away from the possibilities that ANTLR opens up. And, if you are a Java programmer, I hope this has demonstrated that JRuby can be a useful complement to Java, giving you a fantastic scripting language and testing environment for development, even if it doesn’t play any role in your final product.

As usual, all the source code is available for download from the sidebar. If you have any difficulty running any of the files, please let me know in the comments.

¹ However, if you want to write a syndicated advice column protocol then you might want a more state-machine-oriented tool like Ragel.

² Joke!! I’m KIDDING people!! You really think I would put something so unsafe in my code? Of course I meant <%=h eval(params[:arithmetic_expression]) %>.