First day

This conference started with a meta-talk (well, obviously a keynote) about systems and complexity by Zach Tellman. He shared quite interesting ideas with us. The keynote was finished with some recommendations for further readings:

Two talks about side projects followed. One of them was quite funny: Bodil Stokke talked about building your own Lisp. Her Lisp is called Bodol. You can find the pon… eh, slides here. After this Philip Meier talked about liberator, a promising library for building RESTful APIs in Clojure. Then Joseph Wilk took us on the journey to creative machines. He raised the question, if machines can be creative. And he delivered his answer: musical creativity. In this project he gives the machine a set of tones and an association network. With this setup new music is created based on a probabilistic approach. Underneath the awesome Clojure library Overtone is used. It’s on my list to play around with it. It should be on yours, too.

Then the highlight of this day was presented: an ARDrone that is controlled by a LeapMotion pad. It is like a self made Kinect for ARDrones. The code and workspace setup was also quite nice. Here is a live-recording of the demo:

The day was finished by a talk about code smells in Clojure. An interesting topic because the typical smells of object-orientated languages are not applicable to a Lisp derivate. You can find a detailed summary of this talk here.

Second day

The last day I had a blast. I could attend the best keynote in my life so far, given by Stuart Halloway. It was a sarcastic keynote about narcissistic design: what you can do to keep your job forever. How you can make yourself irreplaceable. It’s pity that this talk wasn’t recorded. But I found a recording of this talk from another conference (it’s not as good as on the EuroClojure but better than nothing):

Again, I recommend to read the detailed notes here by Tero Parviainen. And here are two quotes from the keynote:

Two migration talks followed. One was about the Daily Mail. They are probably serving the longest website on this planet (8,5 meters long when printed on paper). Their old system is described on this slide:

So, they rewrote the whole system in Clojure. Now the codebase is only 13k lines big. Quite impressive!

Sam Aaron followed with his talk about Meta-eX: the first live coding band on earth. He showed us how he makes music with Clojure. Here is a recording of his live demo:

Afterwards two talks about enterprise integration were given. The libraries Lamina, EEP and Storm were shown.

Impressions

The crowd at the EuroClojure conference consisted of hard Lisp’ers, Clojurians who came from languages like Ruby and Java to Clojure and me: a Clojure beginner with Scala background.

An interesting observation (for an apple fanboy): less macbooks as on other conferences. More interesting for you: emacs seems to be the editor of choice for most Clojure developers - instead of IDEs like Eclipse or IntelliJ.

When you would ask me, what could characterize the Clojure community I would answer: diversity and pragmatism. The combination is quite interesting. They don’t want to live in an academic world. They want to solve real-world problems. And it seems they have an expressive and concise language to accomplish this.

Summary

Was it worth it? Definitely yes! It were two great days with lots of different subjects, from nerdvana to enterprise. Beside interesting libraries such as liberator, I saw lots of cool projects.

The first day felt like being on Chaos Communication Congress but without being afraid someone hacks your smartphone or macbook. After this the second day broadened my view on the Clojure community and also showed more real-world usages.

I will continue my travel into the deep rabbit hole called Clojure and its ecosystem, for sure.

• be aware of pre-allocation done by MongoDB
• memory fragmentation is important to keep an eye on when dealing with updates on collections
• predict how your schema (although you have none in Mongo but your data still have some kind of structure) may change in future and how this will affect your sizing requirements
• collection.drop() can be significantly faster than removing a document from one collection (drop also reduces memory fragmentation because then you typically create a new collection from scratch)
• put different collections on different drives to scale write operations
• put the journal on an extra drive
• one index per query: you need to use compound indices if you need indices for several fields in a query
• try covered indices for performance improvements on queries you are doing quite often
• monitor with mongostat and mongotop
• when using mongostat interesting columns for inspect correct sizing, index setup and configuration are faults, locked, idx miss, qr, qw, ar, aw
• also nice web frontends for monitoring: Server Density, the MongoDB Monitoring Service (free) and of course you can use tools like Nagios, Graphite or Ganglia
• have a look at the aggregation framework of MongoDB, looks quite promising
• Spring Data is the way to go for using MongoDB with Java due Morphia seems to be dead (or not much interaction in community) and Hibernate OGM is not worth to be mentioned at all

I can only recommend anyone of you who is interested in NoSql stores to try MongoDB. On education.10gen.com you can also sign up for several free online trainings.

Stay tuned.

The goal is to execute all JUnit tests that are suffixed with IntegrationTest, start a Tomcat before (I want to test a deployed rest api) and afterwards shutdown the running Tomcat.

For that you have to exclude all integration tests in the normal test phase of Maven.

1
2
3
4
5
6
7
8
9
10

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <version>2.12.1</version>
  <configuration>
    <excludes>
      <exclude>**/*IntegrationTest*</exclude>
    </excludes>
  </configuration>
</plugin>

Now you should add all integration tests to the Maven Failsafe Plugin and activate the execution of the plugin in your build.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-failsafe-plugin</artifactId>
  <version>2.12.4</version>
  <configuration>
    <includes>
      <include>**/*IntegrationTest*</include>
    </includes>
  </configuration>
  <executions>
    <execution>
      <goals>
        <goal>integration-test</goal>
        <goal>verify</goal>
      </goals>
    </execution>
  </executions>
</plugin>

After this the only missing step is a working Tomcat integration in your build. In this example I’m using the Tomcat 7 Maven Plugin for this (the Tomcat 6 plugin should also work). With the start and stop goals you can control a Tomcat instance. Important is that you fork the process, otherwise your build would be blocked by the running Tomcat. You can do similiar things with the Jetty Maven Plugin.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

<plugin>
  <groupId>org.apache.tomcat.maven</groupId>
  <artifactId>tomcat7-maven-plugin</artifactId>
  <version>2.0</version>
  <configuration>
    <path>/</path>
  </configuration>
  <executions>
    <execution>
      <id>start-tomcat</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>run</goal>
      </goals>
      <configuration>
        <fork>true</fork>
      </configuration>
    </execution>
    <execution>
      <id>stop-tomcat</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>shutdown</goal>
      </goals>
    </execution>
  </executions>
</plugin>

With this little setup you get full support for integration tests with Maven and Tomcat in your web applications, especially if you building rest apis. I hope that this Maven snippet helps you to improve your builds.

Update [11.02.2013] Changed suffix of include and exclude configurations. Thanks to a hint by Erich Eichinger in the comment section.

If you cannot hold your best ones you are facing many other problems:

• hiring other great developers becomes more difficult because the more smart one developer is the more he wants to work with other even smarter developers to learn, to improve himself and last but not least to have fun at work
• to improve your technical staff by techniques like pair programming and code reviews becomes more difficult due the smart developers are away
• your internal code quality may decrease or you may become slower at delivering valuable software
• if your code quality decreases, it makes it even less attractive for smart developers to join your company (none of them want to deal with a shitty codebase although everyone of them may read “Working Effectively with Legacy Code” by Michael Feathers)

On the other hand we are actually facing a shift from “one developer hero rescues the world/project” to “we develop a strong team, where super-programmers are not needed anymore”. Yeah, that’s the theory and it sounds great. But medicore developers do not learn new technologies, methods, frameworks and programming languages in there spare time. It’s not their passion. And that is OK. I think that great developers make the difference between “just another IT company” and “company that leads the change in IT”.

So, I think you need those great and smart developers if you want to adapt to new technologies and markets.

As mentioned in this blog posting, the Enforcer Maven Plugin helps. It offers a dependency convergence rule that checks the following:

If a project has two dependencies, A and B, both depending on the same artifact, C, this rule will fail the build if A depends on a different version of C then the version of C depended on by B.

That is exactly what I want in all of my projects managed by Maven! Therefore, I add the enforcer plugin to the pom file and enable the DependencyConvergence rule. I find it useful when the enforcer plugin is executed during the validate phase. This causes the build to break before compiling if the dependencies do not converge. Then you have to exclude the unwanted transitive dependencies or change the versions. Very handy!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-enforcer-plugin</artifactId>
  <version>1.0.1</version>
  <configuration>
    <rules>
      <DependencyConvergence/>
    </rules>
  </configuration>
  <executions>
    <execution>
      <id>enforce</id>
      <goals>
        <goal>enforce</goal>
      </goals>
      <phase>validate</phase>
    </execution>
  </executions>
</plugin>

Just try this in one of your projects. You will be quite surprised what for a mess your actual dependency definitions are.

Achievements

• gained a lot experience in working in teams and how to manage critical situations
• improved knowledge about continuous delivery
• improved skills in TDD, BDD, ATDD
• gained more experience in managing code quality
• improved skills in scripting builds
• improved usage of Sonar
• held many talks on conferences and events about testing, Sonar and Scala
• finally started this blog :-)

Technologies

• Play Framework 1 & 2 (Java, Scala)
• RabbitMQ, only for a spike but I enjoyed it a lot
• Elasticsearch, bad documentation but nevertheless a great technology
• MongoDB, the classic NoSql store
• Redis
• Spring MVC mostly for building REST apis
• JavaScript, oh geez a lot of JavaScript
• AngularJS, only a bit
• PhoneGap, the I/O related part for iOS
• Vagrant for setting up dev VMs
• Chef, only changing some configuration in existing scripts
• Ruby for web development
• Sinatra for a tiny web project, great framework

Events

• visited many Java Usergroup Berlin events
• attended some conferences: BedCon, gearconf, Java Forum Stuttgart
• organized tech talks at my current employer, adesso AG

In the Play Framework documentation they show you that it is quite easy to set up a test with a FakeApplication by the following code snippet:

1
2
3
4
5
6
7
8
9
10

@Test
public void findById() {
  running(fakeApplication(), new Runnable() {
    public void run() {
      Computer macintosh = Computer.find.byId(21l);
      assertThat(macintosh.name).isEqualTo("Macintosh");
      assertThat(formatted(macintosh.introduced)).isEqualTo("1984-01-24");
    }
  });
}

So, you might think when you have many test methods you just copy this pattern and everything is fine. But that is not the case!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

@Test
public void testOne() {
  running(fakeApplication(), new Runnable() {
    public void run() {
      // test logic
    }
  });
}

@Test
public void testTwo() {
  running(fakeApplication(), new Runnable() {
    public void run() {
      // test logic
    }
  });
}

Here are two examples that FakeApplication can be difficult to use:

• (Play-Scala 2.0.4) FakeApplication onStop not being called after Test
• [play 2.0.3] Two fakeApplication tests only one will succeed but the other throws error 500

I ran into the same problems that are mentioned in the first mailinglist posting. In the project where I’m using MongoDB together with Play 2 the tests with a FakeApplication started to become unreliable. There were issues with cleaning up test data after and before each test method. Also the shutdown and restart of a FakeApplication was sometimes error prone.

The best solution for this problems is to start one FakeApplication per test class, not per test method. Here is the test helper class that I use for this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import play.test.FakeApplication;

import static play.test.Helpers.*;

public abstract class AbstractOneFakeApplicationIntegrationTest {

    private static FakeApplication fakeApplication;

    @BeforeClass
    public static void startFakeApplication() {
        fakeApplication = fakeApplication();
        start(fakeApplication);
    }

    @AfterClass
    public static void shutdownFakeApplication() {
        stop(fakeApplication);
    }
}

You only need to inherit from it and every test has access to a running FakeApplication. One disadvantage is that you have more work to clean up any data that is written for your tests into databases, filesystem or any other persistent system. But you get reliable integration tests and the tests are much faster because you only start one FakeApplication per class and not dozens of them as in the per method approach

Beside this little issues that FakeApplication can cause Play 2 offers great testing support.

Therefore, you have to take care of two things: disable the http module because you do not want to have network communication in your tests and configure a writable data directory. You could also use elasticsearch with full in-memory mode but this is not as stable as you may think. Just let Lucene write on disk in a temporary directory.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

public class EmbeddedElasticsearchServer {

    private static final String DEFAULT_DATA_DIRECTORY = "target/elasticsearch-data";

    private final Node node;
    private final String dataDirectory;

    public EmbeddedElasticsearchServer() {
        this(DEFAULT_DATA_DIRECTORY);
    }

    public EmbeddedElasticsearchServer(String dataDirectory) {
        this.dataDirectory = dataDirectory;

        ImmutableSettings.Builder elasticsearchSettings = ImmutableSettings.settingsBuilder()
                .put("http.enabled", "false")
                .put("path.data", dataDirectory);

        node = nodeBuilder()
                .local(true)
                .settings(elasticsearchSettings.build())
                .node();
    }

    public Client getClient() {
        return node.client();
    }

    public void shutdown() {
        node.close();
        deleteDataDirectory();
    }

    private void deleteDataDirectory() {
        try {
            FileUtils.deleteDirectory(new File(dataDirectory));
        } catch (IOException e) {
            throw new RuntimeException("Could not delete data directory of embedded elasticsearch server", e);
        }
    }
}

By using getClient you can get access to the embedded server. And as you can see the data directory is deleted when the embedded elasticsearch server will be stopped. With this class available you can write an abstract test helper class for your test infrastructure.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

public abstract class AbstractElasticsearchIntegrationTest {

    private EmbeddedElasticsearchServer embeddedElasticsearchServer;

    @Before
    public void startEmbeddedElasticsearchServer() {
        embeddedElasticsearchServer = new EmbeddedElasticsearchServer();
    }

    @After
    public void shutdownEmbeddedElasticsearchServer() {
        embeddedElasticsearchServer.shutdown();
    }

    protected Client getClient() {
        return embeddedElasticsearchServer.getClient();
    }
}

You can find a full example there: elasticsearch-integration-test

First of all, don’t get me wrong: I like the Play Framework a lot. I just think there are chances for optimisation if it comes to testing and modularisation.

After searching for Spring integration in Play Framework 2 I discovered this demo by Guillaume Bort on how to combine Spring and Play 2. The name of this repository suggests that is does work with Play 2.0 but it only works with Play 2.1 because the so called managed controller classes instantiation is only available on Play 2.1 branch. And in fact with Play 2.1 you can easily integrate every dependency injection framework you want by implementing a getControllerInstance method on a Global object.

I want to show you how to integrate Spring into your Play 2.0.x application.

There is already a Spring module for Play 2. But I don’t like the fact that you have to use at least two lines of XML to get annotation-based configuration and component-scanning in place. As Chris Beams states in his slides about modern enterpise app config: “this is kind of ironic”. If you like XML files you can also try this module: play-2.0-spring-module. I prefer annotation-based configuration. But this module inspired my solution a lot.

Ok, the goal is to autowire our controllers. Therefore, you have to intercept the controller creation by Play in some decent way. The trick is to add a stupid ControllerFactory that delegates static controller calls to non-static controller classes and let all routes go through this factory.

1
2
3
4
5
6

public class ControllerFactory {

    public static Application application() {
        return new Application();
    }
}

1
2

GET     /                           controllers.ControllerFactory.application.index()
GET     /personalized/:name         controllers.ControllerFactory.application.helloTo(name: String)

The controllers can be non-static and therefore also instantiated by an application context. In the next step you simply bootstrap an application context in your Global object.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

public class Global extends GlobalSettings {

    private static ApplicationContext applicationContext;

    @Override
    public void onStart(Application application) {
        applicationContext = new AnnotationConfigApplicationContext(SpringConfiguration.class);
    }

    public static <T> T getBean(Class<T> beanClass) {
        if (applicationContext == null) {
            throw new IllegalStateException("application context is not initialized");
        }
        return applicationContext.getBean(beanClass);
    }
}

With this you get an application context everytime Play starts. In the last step you autowire every controller in the static delegating methods of ControllerFactory.

1
2
3
4
5
6
7

public class ControllerFactory {

    public static Application application() {
        // magic happens: Spring autowires your controller :-)
        return Global.getBean(Application.class);
    }
}

Now you have fully fledged Spring integration in your Play application and the Application controller can be autowired.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

@Component
public class Application extends Controller {

    @Autowired
    private HelloWorldService helloWorldService;

    @Autowired
    private PersonalizedHelloWorldService personalizedHelloWorldService;

    public Result index() {
        return ok(helloWorldService.sayHello());
    }

    public Result helloTo(String name) {
        return ok(personalizedHelloWorldService.sayHelloTo(name));
    }
}

You can find the full example project with more explanations there: play20-spring-integration

I looked for projects on github that supports my requirements and came up with these three possibilities:

• jmockmongo This is a mock implementation of the MongoDB protocol and works purely in-memory.
• nosqlunit This library offers more features than jmockmongo. You can use it to test several nosql stores and configure data sets for your tests. For testing MongoDB it uses jmockmongo under the hood. You can also use a local or remote running MongoDB instance for your tests if the in-memory approach does not suit your needs.
• embedmongo With embedmongo you get a platform independent way of running local MongoDB instances. It provides helper classes to bootstrap MongoDB during tests. The most important feature imho: You do not need to install MongoDB. embedmongo downloads MongoDB automatically for windows, linux and mac when bootstraping your tests.

Because I wanted a running MongoDB instance and no in-memory solution I finally chose embedmongo. There is a nice helper class which lets you easily setup a MongoDB on a free port of your system. You can also choose the version.

1

new MongodForTestsFactory().with(Version.V2_2_0).newMongo();

Furthermore I had the needs to integrate the MongoDB test instance into the lifecycle of Play. In Play one typically uses a fake instance for proper integration tests.

1
2
3
4
5
6

running(fakeApplication(), new Runnable() {
    @Override
    public void run() {
        // here you insert your test code
    }
});

One can intercept the bootstrap of a Play 2 application with a so called Global object that is placed in the default package (or another package that you have configured in the application.conf). There you can override methods that are called from Play during start and shutdown of your application. The perfect place to bootstrap your own MongoDB instance.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

import com.mongodb.Mongo;
import de.flapdoodle.embed.mongo.distribution.Version;
import de.flapdoodle.embed.mongo.tests.MongodForTestsFactory;
import play.Application;
import play.GlobalSettings;
import play.Logger;

import java.io.IOException;

public class Global extends GlobalSettings {

    private MongodForTestsFactory mongodForTestsFactory;

    @Override
    public void onStart(Application application) {
        if (application.isProd()) {
            /*
             * here you can setup mongodb for production,
             * e.g. connect to remote mongodb instance
             */
        } else {
            startLocalMongoInstance();
        }
    }

    @Override
    public void onStop(Application application) {
        if (mongodForTestsFactory != null) {
            Logger.info("Shutdown local mongo instance...");
            mongodForTestsFactory.shutdown();
            Logger.info("Local mongo instance was stopped");
        }
    }

    private void startLocalMongoInstance() {
        Logger.info("Start local mongo instance...");
        try {
            mongodForTestsFactory = new MongodForTestsFactory();
            Mongo mongo = mongodForTestsFactory.with(Version.V2_2_0).newMongo();
            // that's it: now you've got the local mongo instance
            Logger.info("Local mongo instance was successfully started");
        } catch (IOException e) {
            Logger.error("Unable to start local mongo instance", e);
            throw new RuntimeException(e);
        }
    }
}

This solution starts a MongoDB instance each time you start Play in development mode and for each test where you start a fake application. In production you typically connect to a remote MongoDB instance or cluster.

And sorry for the oh so funny title of this first blog post. I did not find a better one. ;-)

So far so good. I hope you will enjoy the content.