Skip to content

Building Go Test It: Fun with Scala and REST APIs

Go Test It

Go Test It logo

Go Test It, the awesome new web testing product I am currently working on, was announced on Monday. I had hinted at it back in December, but now that it is taking shape it was time for me to start spreading the word.

Please see the Go Test It website for details about the product; in summary, it allows web developers and QA teams to very quickly and easily record functional tests on their web application, and plays them back in our own cloud-hosted cross-browser infrastructure. It is going to be the fastest way ever to get started with cross-browser testing, and also provide great tools for running your existing test suites. With Go Test It, there will be no more excuses for not having great coverage of automated functional tests.

On this blog I would like to share from time to time some of the technical challenges we have overcome in the process of building this product.

Today: Simple REST APIs in Scala

The Go Test It system consists of a number of different components using different programming languages, different frameworks, different infrastructures. The main user front-end is a Rails app, but not all tasks are so well suited to Ruby. In particular, the multi-threaded parts of the system which do a lot of the internal coordination are being written in Scala, a very elegant JVM-based language which lends itself well to this sort of thing.

The various components of the system need to communicate, and depending on the type of interaction needed, we’re using a combination of a message queue and REST APIs. There is good Java library support for both, and since Scala can seamlessly use Java libraries, it all fits together very nicely.

Jersey is a really neat Java (and, by extension, Scala) library for writing REST APIs. (It falls into a similar category as Sinatra in the Ruby world.) For example, a resource which returns a HTML document can be written in just a few lines of Scala code:

1
2
3
4
5
6
7
8

package com.example.restapi
import javax.ws.rs._
 
@Path("/hello")
class Hello {
  @GET @Produces(Array("text/html"))
  def doGet = "<html><body><h1>hello jersey/scala world!</h1></body></html>"
}

A resource is defined by a class, annotated with the path(s) under which it can be accessed, and has one or more methods to handle different HTTP verbs (GET, POST, PUT, DELETE), and if you want, also different methods for different content types.

Coda Hale has written up some really nice examples of using Jersey and doing nice clean error handling of input. His examples are in Java, but translating them into Scala is a very simple exercise — more or less a matter of removing the type declarations and the semicolons.

But here comes the tricky bit. Given the beautiful code snippet above, how do you actually build and run the thing? That is something I wrestled with for a while, and here I would like to reveal my solution. You can build it with Maven (for the Ruby guys, it is basically RubyGems and Rake rolled into one, intended for building Java projects with automatic dependency management). The process is a bit scary, but once it works, it is magic — it automatically downloads all the right packages, sets the right classpaths and packages up the results of your build into a war file (web archive).

But the scary bit first. You need to declare your dependencies and build properties to Maven, and the prescribed way of doing that is using the most ugly XML configuration file on the planet. I’m really sorry for the ugliness, but I think it’s worth it. However, the good news is that you only need to set it up once and never touch it again; once your build is working, you can create resources and handle them in Scala to your heart’s content. And for a bonus, see below for a Capistrano recipe which allows you to easily deploy the resulting war file to a Tomcat server, again and again.

First, put the following in a file called pom.xml in the base directory of your project:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153

< ?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
  <modelversion>4.0.0</modelversion>
  <groupid>com.example</groupid>
  <artifactid>restapi</artifactid>
  <packaging>war</packaging>
  <version>1.0-SNAPSHOT</version>
  <name>restapi</name>
 
  <dependencies>
    <!-- Scala standard library -->
    <dependency>
      <groupid>org.scala-lang</groupid>
      <artifactid>scala-library</artifactid>
      <version>2.7.2</version>
    </dependency>
 
    <!-- JUnit for testing -->
    <dependency>
      <groupid>junit</groupid>
      <artifactid>junit</artifactid>
      <version>3.8.2</version>
      <scope>test</scope>
    </dependency>
 
    <!-- Jersey and Servlet (for the REST API) -->
    <dependency>
      <groupid>com.sun.jersey</groupid>
      <artifactid>jersey-server</artifactid>
      <version>1.0.3</version>
    </dependency>
 
    <dependency>
      <groupid>javax.servlet</groupid>
      <artifactid>servlet-api</artifactid>
      <version>2.4</version>
      <scope>provided</scope>
    </dependency>
 
    <dependency>
      <groupid>com.sun.jersey.test.framework</groupid>
      <artifactid>jersey-test-framework</artifactid>
      <version>1.0.3</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
 
  <build>
    <plugins>
      <!-- Compile to Java 1.6 -->
      <plugin>
        <groupid>org.apache.maven.plugins</groupid>
        <artifactid>maven-compiler-plugin</artifactid>
        <configuration>
          <source>1.6</source>
          <target>1.6</target>
        </configuration>
        <executions>
          <execution>
            <phase>compile</phase>
            <goals>
              <goal>compile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
 
      <!-- Build Scala sources -->
      <plugin>
        <groupid>org.scala-tools</groupid>
        <artifactid>maven-scala-plugin</artifactid>
        <version>2.10</version>
        <executions>
          <execution>
            <id>scala-compile-first</id>
            <phase>process-resources</phase>
            <goals>
              <goal>add-source</goal>
              <goal>compile</goal>
            </goals>
          </execution>
          <execution>
            <id>scala-test-compile</id>
            <phase>process-test-resources</phase>
            <goals>
              <goal>testCompile</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
 
      <!-- Run the application using "mvn glassfish:run" -->
      <plugin>
        <groupid>org.glassfish</groupid>
        <artifactid>maven-glassfish-plugin</artifactid>
      </plugin>
    </plugins>
 
    <pluginmanagement>
      <plugins>
        <plugin>
          <groupid>org.scala-tools</groupid>
          <artifactid>maven-scala-plugin</artifactid>
          <version>2.9.1</version>
        </plugin>
 
        <plugin>
          <groupid>org.apache.maven.plugins</groupid>
          <artifactid>maven-compiler-plugin</artifactid>
          <version>2.0.2</version>
        </plugin>
      </plugins>
    </pluginmanagement>
  </build>
 
  <!-- List of repositories where the various dependencies and plugins can be found -->
  <repositories>
    <repository>
      <id>scala-tools.org</id>
      <name>Scala-tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </repository>
 
    <repository>
      <id>maven2-repository.dev.java.net</id>
      <name>Java.net Repository for Maven</name>
      <url>http://download.java.net/maven/2/</url>
      <layout>default</layout>
    </repository>
 
    <repository>
      <id>glassfish-repository</id>
      <name>Java.net Repository for Glassfish</name>
      <url>http://download.java.net/maven/glassfish</url>
    </repository>
  </repositories>
 
  <pluginrepositories>
    <pluginrepository>
      <id>scala-tools.org</id>
      <name>Scala-tools Maven2 Repository</name>
      <url>http://scala-tools.org/repo-releases</url>
    </pluginrepository>
 
    <pluginrepository>
      <id>maven2-repository.dev.java.net</id>
      <name>Java.net Repository for Maven</name>
      <url>http://download.java.net/maven/2/</url>
      <layout>default</layout>
    </pluginrepository>
  </pluginrepositories>
</project>

That was a terrible mouthful. Urgh. But worry not, the worst is over. Just one other XML file, and this one goes in src/main/webapp/WEB-INF/web.xml (sorry, no syntax highlighting on this one due to a bug in my syntax highlighting plugin):

<?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.4" xmlns="http://java.sun.com/xml/ns/j2ee"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
  <servlet>
    <servlet-name>Example.com REST API</servlet-name>
    <servlet-class>com.sun.jersey.spi.container.servlet.ServletContainer</servlet-class>
    <init-param>
      <param-name>com.sun.jersey.config.property.packages</param-name>
      <param-value>com.example.restapi</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  <servlet-mapping>
    <servlet-name>Example.com REST API</servlet-name>
    <url-pattern>/*</url-pattern>
  </servlet-mapping>
  <session-config>
    <session-timeout>30</session-timeout>
  </session-config>
</web-app>

And that’s all the boilerplate you need. You can now write the actual code, with Scala source files in src/main/scala and Java source files in src/main/java (if you want to use Java alongside Scala). For example, place the source for the Hello resource above in src/main/scala/com/example/restapi/Hello.scala. Tests go in src/test/scala and src/test/java respectively.

Now go and build it! (You need to have Maven 2.0.9 or newer installed.) The two most useful commands are mvn package, which compiles and packages your project, and places the result in target/restapi-1.0-SNAPSHOT.war, and mvn glassfish:run, which launches an embedded Glassfish application server and serves your new REST API on http://localhost:8080/ for local testing. The example resource which we defined above would then be accessible at http://localhost:8080/hello in your web browser.

If you don’t have the dependencies yet, Maven will go away and download them. This will take a while, but they won’t be re-downloaded once you have them. It’s a bit frightening at first to watch Maven doing its magic, but in my limited experience so far it has always done a good job.

Now for the bonus points. Just building your app locally is not enough; you also need to deploy it to a server and make it accessible to the world. In our case this is Tomcat, but any standard Java web container should do. I wanted to use Capistrano for deployment, because it gets a lot of things right and also fits well into the rest of our system which does have a lot of Ruby in it. There is an existing Capistrano recipe for deploying war files to Tomcat, but it needed a bit of improvement, so my own version is included below.

These deployment instructions assume you’re running a Debian Lenny system, with a restapi user account; for other systems, you’ll need to customise it to suit.

Create the following shell script in /etc/tomcat5.5/symlink-webapp and make it executable:

1
2
3
4
5
6
7

#!/bin/sh
# Allows an unprivileged user to deploy an application to Tomcat. Add a rule for this
# script and /etc/init.d/tomcat to /etc/sudoers. Then to deploy an application, call:
#     sudo /etc/tomcat5.5/symlink-webapp webapp-name /location/of/war/file
NAME=`echo "$1" | sed -e 's/[^a-zA-Z0-9\.-]//g'`
echo "ln -f -s \"$2\" /var/lib/tomcat5.5/webapps/$NAME"
ln -f -s "$2" /var/lib/tomcat5.5/webapps/$NAME

Next, add the following line to /etc/sudoers:

restapi ALL = NOPASSWD: /etc/init.d/tomcat5.5, /etc/tomcat5.5/symlink-webapp

With that set up, here’s the Capfile for you to put in the base directory of your project:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106

# This recipe is based on:
# http://github.com/andynu/capistrano-recipes/blob/master/capfile_tomcat
 
load 'deploy'
set :application, "restapi"
 
# DEPLOYMENT SCHEME
set :scm, :none
set :deploy_via, :copy
set :repository do 
  fetch(:deploy_from)
end
 
# LOCAL
set :war_path, "#{File.dirname(__FILE__)}/target/*.war"
 
# TOMCAT SERVERS
role :webserver, "server.example.com"
set :tomcat_home, "/var/lib/tomcat5.5"
set :tomcat_ctrl, "/etc/init.d/tomcat5.5"
set :deploy_to, "/home/restapi/deploy"
 
# USER LOGIN
set :user, "restapi"
ssh_options[:keys] = File.expand_path('~/.ssh/my_private_ssh_key')
set :use_sudo, false
default_run_options[:pty] = true
 
set :deploy_from do 
  dir = "/tmp/prep_#{release_name}"
  system("mkdir -p #{dir}")
  dir
end
 
# simple interactions with the tomcat server
namespace :tomcat do
  desc "start tomcat"
  task :start do
    sudo "#{tomcat_ctrl} start"
  end
 
  desc "stop tomcat"
  task :stop do
    sudo "#{tomcat_ctrl} stop"
  end
 
  desc "stop and start tomcat"
  task :restart do
    tomcat.stop
    tomcat.start
  end
 
  desc "tail :tomcat_home/logs/*.log and logs/catalina.out"
  task :tail do
    stream "tail -f #{tomcat_home}/logs/*.log #{tomcat_home}/logs/catalina.out"
  end
end
 
# Before everything else, build the application with maven and copy
# the war file to a temporary directory.
before 'deploy:update_code' do
  cmd = "mvn clean package"
  puts cmd; system cmd or raise "\"#{cmd}\" failed"
  set :war, Dir[war_path].first
  cmd = "cp #{war} #{deploy_from}"
  puts cmd; system cmd or raise "\"#{cmd}\" failed"
end
 
# Restart tomcat at the end of deployment
namespace :deploy do
  task :restart do
    cmd = "/etc/tomcat5.5/symlink-webapp #{application}.war #{deploy_to}/current/`basename #{war}`"
    puts cmd
    sudo cmd
    tomcat.restart
  end
end
 
# Disable all the default tasks that
# either don't apply, or I haven't made work.
namespace :deploy do
  [ :upload, :cold, :start, :stop, :migrate, :migrations ].each do |default_task|
    desc "[internal] disabled"
    task default_task do
      # disabled
    end
  end
 
  namespace :web do
    [ :disable, :enable ].each do |default_task|
      desc "[internal] disabled"
      task default_task do
        # disabled
      end
    end
  end
 
  namespace :pending do
    [ :default, :diff ].each do |default_task|
      desc "[internal] disabled"
      task default_task do
        # disabled
      end
    end
  end
end

So there we go. For the first time use cap deploy:setup to create the directories on the server, and thereafter always type cap deploy. Those 11 keystrokes are all you need to fetch all the dependencies (no more troubles replicating your build environment on different development machines), build a clean copy of the entire app, ship it off to the server and run it. And your app can be all in beautiful, concise and type-safe Scala. I have survived the XML deluge and I am happy :-)

Posted in business, software, techie notes. Tagged with , , , , , , .

By Martin Kleppmann
Wednesday, 13 May 2009

View Comments

  • Martin,

    I really enjoy reading your site and especially appreciate the posts on Scala and Jersey. I took a crack at implementing this post in SBT and was pleasantly surprised how easy it was, I wrote up the result here: http://jeffdevine.com/2009/09/27/simple-build-tool-or-how-i-rtfm-and-stopped-hating-on-maven/

    Keep up the great work.
  • martinkl
    @Alan - thanks for the pointer, SBT looks very useful. I will give it a try.
  • Have you considered SBT (http://code.google.com/p/simple-build-tool/) for your scala building and dep tracking? I was using an ugly pom.xml also before I converted.
  • @Martin

    I'm not jumping into the Scala/Ruby fray from a month or so ago, I just wanted to hear your thoughts.

    Thank you!
  • martinkl
    Yes, I've certainly considered JRuby but not actually tried it out yet. I have several reasons for wanting to use Scala -- type safety is a bigger benefit than Ruby's flexibility for this particular application, it's functional approach and Actor model fits in nicely here, and Scala's XML handling capabilities are miles ahead of Ruby's. However, for the front-end web app, I still think that Ruby is hard to beat. Choose the most suitable tool for the job, I think.
  • Just wanted to know if you had considered JRuby?
blog comments powered by Disqus