17 things I learnt at QCon London

1: I used to be 4th order ignorant, but now I’m always 3rd or better

Putting the “re” into Architecture, Kevlin Henney

According to the five orders or ignorance, 4th order ‘meta-ignorance’ is to lack knowledge of the five orders of ignorance (broadly equivalent to the Rumsfeldian “[un]known [un]knowns”).

2: Elegant and simple EIP implementations are possible with Apache Camel

Take a ride on Camel, James Strachan

I was particularly impressed with the Scala DSL, where, for example, to read messages from a queue, and post them to three load-balanced RESTful services:

"apachemq:events" ==> {
    loadbalance roundrobin {
        to ("http://eventHandlerA")
        to ("http://eventHandlerB")
        to ("http://eventHandlerC")
    }
}

3: I am both a software engineer, and a craftsman

Craft and Software Engineering, Glenn Vanderburg

This is because, unlike other engineers (structural, chemical etc..), I both design and build the product (working software). Although, this is increasingly true of other types of engineering (electrical, mechanical) because of increasingly advanced software and production techniques using CAD tools. So (my conclusion), according to this metaphor, either I’m still just an engineer, or all engineers are now craftsmen. Whether or not it affects this argument, I did like the quote:

Programs should be written for people to read, and only incidentally for machines to execute. – “Structure and Interpretation of Computer Programs” by Abelson and Sussman

4: Until recently, Twitter received, on average, more updates to the social graph than tweets

Data Architecture at Twitter Scale, Nick Kallen

That is, counter-intuitively, there were more ‘follow’, ‘unfollow’, ‘block’ etc.. events in total than tweets. As these social graph updates represent CRUD-like behaviour with strict consistency constraints, whereas tweets are mostly append-only with only eventual consistency required, handling these events makes a significant contribution to the complexity required to scale Twitter.

5: New Zealand is Facebook’s Guinea Pig

Scaling the Social Graph: Infrastructure at Facebook, Jason Sobel

Often when Facebook release a new product or service, they will release it to a restricted geographical area to verify it’s scalability, popularity, etc… using a system called GateKeeper. Presumably New Zealand serves as a good test bench as it is an affluent area with a relatively small population, and more accurate GeoIP because it’s an island. Interestingly, I once heard that chocolate bar manufacturers do the same thing (NZ only) to trial new bars on the international market.

6: Encapsulating your domain model in a API encourages simplicity

Why I Chose Mongo DB for guardian.co.uk, Matthew Wall

Using MongoDB encouraged the Guardian team to reduce the complexity of the domain model, such that it could be simply expressed in an extensible API (an API that can mutate over time without breaking compatibility). This was achieved by modelling the API at two levels:

  1. Property groups, which change infrequently and adhere to a contract
  2. Properties, which are not prescribed, allowing the simple addition of properties over time

e.g.

"user": {
    "publicFields": {
        "username": "2346247"
    },
    "privateFields": {
        "postCode" : "N11GU",
        "country": "United Kingdom"
    }
}

7: Using more memory for Java applications increases performance, even if they don’t need the extra memory

Performance Tuning for Java Applications, George Barnett

George had compelling data showing up to 28% reduction in response time for average load to Jira from http://jira.atlassian.com when attempting to ‘drown’ the JVM with larger and larger heap sizes. The basic rules of thumb were:

  • Larger heap = better performance
  • Diminishing returns as more memory is provided. Most improvement comes from first doubling of the heap size above the required size

The explanation was that the GC uses different memory allocation strategies if more heap space is available. If there is plenty of space, these strategies are more efficient.

8: Functional languages are ideal for building elegant web frameworks

Clojure & the Web, Glenn Vanderburg

The functional paradigm complements the fundamental function of the web: read text in, write text out. Also, lazy evaluation allows for elegant solutions to building web frameworks in Clojure, as demonstrated by Compojure and Ring, where the flow of the application does not have to be controlled to ensure efficient operation.

9: You can wrap up a web-app so it looks like an app, with PhoneGap

HTML5 and the dawn of Rich Mobile Web Applications, James Pearce

James said we’re currently at 1995 on the mobile web (e.g. app stores = yahoo website index). We’re in the slightly bizarre position that in order to create a ‘app’ that’ll sit in the AppStore, Market etc.. it has to be wrapped up in a customised container so it can pretend to be an ‘app’. This is because we haven’t got sufficiently user-friendly solutions to discoverability on the web.

10: One-way package dependencies can be enforced with Maven and SonarJ

Where Did My Architecture Go? Preserving software architecture in its implementation, Eoin Woods

Enforcing dependency rules on a Java project such as ‘package A’ can access ‘package B’ but not vice versa is possible with a number of tools such as JDepend, Macker and SonarJ. SonarJ allows integration with Maven to enforce rules and indicate allowed exceptions to rules.

11: Fail fast: Always build pretotypes before prototypes

Innovation at Google, Patrick Copeland

Where ‘pretotype’ is a displeasingly poor shortening of another imaginary word: ‘pretend-o-type’. It’s a word for a prototype that is as simplistic as possible, such as a piece of cardboard instead of an iPhone, or a person-in-a-box responding to voice commands, instead of text-to-speech software.

12: Handle what you can, but let someone else handle the rest (in Erlang?)

Let It Crash… Except when you shouldn’t, Steve Vinoski

As a refinement of Joe Armstrong’s ‘Let it crash’ philosophy, as applied to Erlang, this should not be taken as an excuse to remove all guards or defensive programming techniques from your code. You should not be afraid to let the system crash when the error is outside of your control, but foreseeable problems should be accounted for (e.g. legitimate network failures). It seems to me that the presence of Erlang’s ‘heart’ process makes this approach more appropriate than with other languages or architectures.

13: Being in an environment where everything ‘just works’ liberates developers to write great software

Better is Better, Steve Freeman

In a environment where administrators lighten the work load, operations provide robust platforms and developers have ready access to the necessary technologies, developers can focus on producing quality software. However, to achieve this takes a constant effort; always working towards that which is ‘better’.

14: Not everything on Mars that looks like a winking bunny is always a winking bunny

Innovations and Integrations: Applying trendy technologies to NASA Mission Operations Planning, Mark Powell

Sometimes it’s a tiny fragment of air-bag material ripped from a lander module that, when highly pixelated and cycled through the three colour layers, looks like a winking rabbit.

15: NoXML Spring is ready (except JPA)

Spring 3.1 and Beyond – Themes & Trends, Juergen Hoeller

Following on from @Configuration in 3.0, with the addition of @Profile (environment-sensitive property profiles) and abstracted configurable caching support, NoXML Spring is increasingly viable.

16: The Earth is an ellipsoid

Unifying the Search Engine and NoSQL DBMS with a Universal Index, Jason Hunter

Unlike most things in MarkLogic, geo-location queries cannot be solved entirely through the use of indexed term-lists. XML structure, free-text search and range queries can.

17: An inverse relationship exists between use of IBM productivity tools and productivity

Scaling Lean & Agile, Craig Larman

Advertisements

Aggregate! Aggregate! Aggregate! Using linked data to make websites more interesting.

The way content is modelled is pivotal to the overall, technical design of a website, in particular, the choice of content management system.

Many bespoke CMSs are built using a relational database as the content repository. Most generic, off-the-shelf CMSs now use NoSQL technology, specifically document stores, to provide the necessary flexibility of schema design. Aside from these two, key content storage technologies, a small number of CMSs, usually hosted services, make direct use of triple stores: content repositories that use linked data technology.

Each type of content repository has advantages and disadvantages:

  • Relational database: Supported by a wealth of mature technology and offers relational integrity. However, schema change can be slow and complex.
  • Document store: Provides schema flexibility, horizontal scalability and content ‘atomicity’ (the ability to handle content in discreet chunks, for versioning or workflow). However, referential integrity is hard to maintain and queries across different sets of data are harder to achieve.
  • Triple store: Provides logical inference and a fundamental ability to link concepts across domains. However, the technology is less mature.

I think there are two diametric content modelling approaches; most approaches will fit somewhere, on a sliding scale, between the two extremes. Each exhibits properties that make some things simple, and some things complex:

  • Atomic content: Content is modelled as a disconnected set of content ‘atoms’ which have no inter-relationships. Within each atom is a nugget of rich, hierarchically structured content. The following content management features are easier to implement if content is modelled atomically:
    • Versioning
    • Publishing
    • Workflow
    • Access control
    • Schema flexibility
  • Relational content: Content is modelled as a graph of small content fragments with a rich and varied set of relationships between the fragments. The following content management features are easier to implement if content is modelled relationally:
    • Logical inference
    • Relationship validation/integrity
    • Schema standardisation

Atomic vs. Relational

The following diagram indicates where the different content repositories fit on this scale:

Atomic to Relational

For the scenario outlined below, two ‘advanced’ features have been used to highlight tension between atomic and related content models. The versioning feature causes problems for more related models, and the tag relationships feature causes problems for more atomic models.

The online film shop

A use-case to explain why linked data technology can help to build rich, content-driven websites

An online film shop needs to be built. It will provide compellingly presented information about films, and allow users to purchase films and watch them in the browser.

Here are some (very) high-level user-stories:

“As a user I want to shop for films at, http://www.onlinecinemashop.com, and watch them online after purchase”

“As an editor I need to manage the list of available films, providing enough information to support a compelling user experience.

An analysis indicates that the following content is needed:

  • Films
    • The digital movie asset
    • Description (rich-text with screen-grabs & clips)
    • People involved (actors, directors)
    • Tags (horror, comedy, prize-winning, etc…)

Some basic functionality has been developed already…

  • A page for each film
    • A list of participants, ordered by role
  • A page for each person
    • A list of the films they have participated in
  • An A-Z film list
  • A shopping basket and check-out form
  • A login-protected viewing page for each film

But now, the following advanced functionality is needed…

  • Feature 1, Versioned film information: This supports several features including:
    • Rolling-back content if mistakes are made
    • Allowing editors to set up new versions for publishing at a later date, whilst keeping the current version live
    • Providing a workflow, allowing approval of a film’s information, whilst continuing to edit a draft version
  • Feature 2, Tagged content: Tags are used to express domain concepts, these concepts have different types of relationships which will support:
    • Rich navigation, e.g. “This director also worked with…”
    • Automated, intelligent aggregation pages, e.g. An aggregation page for ‘comedy’ films, including all sub-genres like ‘slapstick’

Using a relational database

Using a relational database, a good place to start is by modelling the entities and relationships…

Entities: Film, Person, Role, Term
Relationships: Participant [Person, Role, Film], Tag [Term, Film]

Assuming that the basic feature set has been implemented, the advanced features are now required:

Advanced feature 1: Versioned films

Trying to version data in a relational database is not easy: often a ‘parent’ entity is needed for all foreign keys to reference, with ‘child’ entities for the different versions, each annotated with version information. Handling versioning for more than one entity and the schema starts getting complicated.

Advanced feature 2: Tag relationships

When tagging content, one starts by trying to come up with a single, simple list of terms that collectively describe the concepts in the domain. Quickly, that starts to break down. Firstly a 2-level hierarchy is identified; for example, putting each tag into a category: ‘Horror: Genre’, ‘Paris: Location’, and so on. Next, a multi-level hierarchy is discovered: ‘Montmatre: Paris: France: Europe’. Finally, this breaks down as concepts start to overlap and repeat. Here are a few examples: Basque Country (France & Spain), Arnold Schwarzenegger (Actor, Politician, Businessman), things get even trickier with Bruce Dickinson. Basically, real-world concepts do not fall easily into a taxonomy, and instead form a graph.

Getting the relationships right is important for the film shop. If an editor tags a film as ‘slapstick’, they would expect it to appear on the ‘comedy’ aggregation page. Likewise, if an editor tagged a film as ‘Cannes Palme d’Or 2010 Winner’ they would expect it to be shown on the ‘2010 Cannes Nominees’ aggregation page. These two ‘inferences’ use different relationships, but achieve the correct result for the user.

With a relational database 2-level hierarchies introduce complexity, multi-level hierarchies more so, and once the schema is used to model a graph, the benefits of a relational schema have been lost.

Relational database: Advanced features summary

  • Feature 1 is complex
  • Feature 2 is complex

Using a document store

NoSQL storage solutions are currently very popular, offering massive scalability and a chance to break-free from strict schema and normal-forms. From a document store perspective. the original entities look very different, compiled into a single structure:

<film>
<name>Pulp Fiction</name>
<description>The lives of two mob hit men...</description>
<participants>
<participant role="director">Quentin Tarantino</participant>
<participant role="actor">Bruce Willis</participant>
...
</participants>
<tags>
<tag>Palme d'Or Winner 1994 Cannes Film Festival</tag>
<tag>Black comedy</tag>
<tag>Neo-noir</tag>
...
</tags>
</film>

The lists of available tags or participants may be handled using separate documents, or possibly ‘dictionaries’ of allowed terms.

Assuming again that the basic feature set has been implemented, the advanced features are now required:

Advanced feature 1: Versioned films

Keep multiple copies the film document, each marked to indicate the different versions. All the references are inside the document, so even the tags and roles are versioned too. Many off-the-shelf CMS products offer built-in document versioning based on this process.

Advanced feature 2: Tag relationships

This is even worse now. Before, with a relational database, we could keep adding more structures, using SQL queries to extract the hierarchical relationships. It was only when moving to graphs that relationships became too complex to manage. Without SQL to query a structure of tags, bespoke plug-in code is required to extract meaning from a bespoke hierarchy of terms. This is now too complicated to build anything simple and reusable.

Document store: Advanced features summary

  • Feature 1 is simple
  • Feature 2 is very complex

Using a triple store

In triple stores, all the content is reduced to triples. So all our content would look something like this…

http://www.imdb.com/title/tt0450345/ -> dc:title -> “The Wicker Man”

Assuming again that the basic feature set has been implemented, the advanced features are now required:

Advanced feature 1: Versioned films

Versioning triples would not be straightforward. A complex overlaid system of ‘named graphs’ could be used to separate out content items into different versions, but this would be more complex than for relational databases.

Advanced feature 2: Tag relationships

This is simple! SPARQL queries will offer us all the inferred or aggregated content that we require, as long as the ontology has been created, and curated, with care.

Triple store: Advanced features summary

  • Feature 1 is very complex
  • Feature 2 is simple

Why a single content model can’t solve all your problems

In the introduction I explained how each of the different content repositories has advantages and disadvantages. It is possible to model most domains, entirely, in either relational databases, document stores or triple stores. However, as the examples above show, some features are too complicated to implement for some content repositories. The solution, in my opinion, is to use different content repositories, depending on what you need to do, even with the same domain.

As an analogy, serving a search engine from a relational database is not a good idea, as the content model is not well tuned to free-text searching. Instead a separate search engine would be created to serve the specific functionality, where relational content is flattened to produce a number of search indices, allowing fast, free-text search capabilities. As this example shows, for a particular problem, content must be modelled in an appropriate way.

Using a document store + triple store

For this solution, document content is converted into triples by means of a set of rules. These rules will be processed by means of a ‘bridge’ which will convert the change stream provided by the document store, such as Atom, into RDF or SPARQL. In this way, a triple store will contain a copy of the document data in the form of triples (with a small degree of latency due to the conversion process). The system now provides a document store for fast, scalable access to the full content, and a SPARQL endpoint providing all the inference and aggregation that is needed.

The following diagram represents the architecture used in this solution. It combines a document store, and a triple store, to provide both atomic content management and relational structures and inferences:

Architecture

Continuing with the advanced feature implementation…

Advanced feature 1: Versioned films

See ‘document stores’.

Advanced feature 2: Tag relationships

See ‘triple stores’

Document store + triple store: Advanced features summary

  • Feature 1 is simple
  • Feature 2 is simple

By the leveraging the innate advantages of two different content models, all the advanced features are simple to achieve!

Cross-domain aggregation…

What seems interesting about this proposed solution, is that it involves the use of semantic web technologies without publishing any data interchange formats on the web. It would, however, seem a shame to stop short of publishing linked data. I would advocate that this approach is additionally used as a platform to support the publication of linked data, where the overhead of doing so, through the use of a triple store, is significantly reduced.

Combining this approach with ‘open linked data’ and SPARQL endpoints would demonstrate the power of modelling content as triples: expressing relationships in a reusable, standardised way, allows content to be aggregated within, or between, systems, ecosystems, or organisations.

Q&A

If a relational database is not used for content, how can data integrity be maintained?

In short, it is not. Integrity is applied at the point of entry through form validation and referential look-ups. Integrity then becomes the responsibility of systems higher up the chain (for example, the code that renders the page: if an actor doesn’t exist for a film, do not show them). If data integrity really is needed (for example, financial or user data), maybe an SQL database is a better choice; it is still possible to bridge content into the triple store if it is needed.

How is the ontology data managed?

This is a tricky one, and it really depends….I think there are three choices:

  1. Curate the domain ontology manually: Editing and importing RDF/turtle. Needs people with the relevant skills. Could be ideal for small, stable ontologies
  2. Use a linked data wiki: Some tools and frameworks exist for building semantic wikis (e.g. Ontowiki). This could be too generic an approach for some, more complex, ontologies.
  3. Use a relational database CMS + data bridge: Despite the drawbacks of relational databases for content, it may still be a practical solution for ontological data. By building a CMS on a relational database we get all the mature database technology and data integrity, only leaving the tricky process of bridging the ontology data into a triple store.

And finally…

Some of the ideas in this post are based on projects happening at the BBC, many thanks in particular to the BBC News & Knowledge team for sharing their recent endeavours into semantic web technologies.

The Future of the Java SE

Here’s a few interesting points I picked out from JAVAWUG BOF 54 The Future of the Java SE Platform and Beyond….

  • JDK 7 is packed full of goodies, but they might not be in Java SE so don’t get your hopes up
  • Loads of little changes coming with SE 7 e.g. String switches, Multi-type exception catching
  • JDK will be much more modular (project Jigsaw): this will lead to smaller downloads, different subsets of the JDK
  • Closures are already here! Anonymous inner types have provided this functionality since ’97
  • Closures are coming! Real closures without all the boiler plate, function types etc.. may/may not be in SE 7, but they’re coming. There will be a common lower level representation of closures to allow exchange of closures between JVM languages.
  • Easy multi-core development features will come around eventually, even though they’re already 10 years late.
  • JCP transparency will improve to enhance participation in the community and improve decision making.
  • No appropriate forum for JVM language implementers; currently there is too much disparity in approach
  • Don’t scare off the ‘cool crowd’ (Ruby, Python etc..) with too much conservatism vs. don’t break what is already a success by blindly adding features.
  • Don’t experiment with Java instead harness what is being learnt by other JVM languages and pull in the best bits.

Testing Recipes, Part 2 – Utility methods

From part 1…

Should the testing frameworks or methodologies used, influence the design of the implementation code? I’m sure most would respond in the negative, but sometimes this is very hard to avoid. I have been attempting to overcome some of this influence by finding ways to test code that I had previously refactored to make it more ‘testable’. To achieve this, I have been building on use of the standard Mockito syntax, to apply the extensions provided by PowerMock (with Mockito).

Next up, private static utility methods.

When writing a service, as the complexity increases, it is useful to extract more services to provide sub-functionality. This approach is commonly the best way to extract inner functionality, but sometimes this functionality is very contained and local and does warrant a new service. In these scenarios, one will often start to refactor the code to produce utility methods.

Take the follow example: I wish to make a service to provide string padding (I have tried to simplify the context of this example, so excuse the unrealistic service). This service is not supposed to be a multi-purpose padding utility, but instead needs to provide a restricted functionality to its users. Here’s the interface:

public interface Padder {

	public String padWithSpaces(String text, int size);

	public String padWithHashes(String text, int size);

	public String padWithDashes(String text, int size);

}

A padder will pad the supplied text to the indicated size using spaces or hashes. So how do we go about building up the implementation using tests? Notice that the methods provide similar functionality but with varying initial configuration. We could start with the first method and explore all the edge cases with tests:

anEmptyStringInputWillReturnAStringOfSpacesWithALengthOfSize()
aSizeLongerThanTheLengthOfTheStringReturnsASpacePaddedStringUpToTheSize() 
aSizeShorterThanTheLengthOfTheStringReturnsAStringTruncatedToALengthOfSize() 
aSizeEqualToTheLengthOfTheStringReturnsTheSameString()  

An then we test the next method, where the first two tests can be ‘copied’.

anEmptyStringInputWillReturnAStringOfHashesWithALengthOfSize()
aSizeLongerThanTheLengthOfTheStringReturnsAHashesPaddedStringUpToTheSize() 

and again for the final method

anEmptyStringInputWillReturnAStringOfDashesWithALengthOfSize()
aSizeLongerThanTheLengthOfTheStringReturnsADashesPaddedStringUpToTheSize() 

Often this creeping addition of virtually identical tests happens as a service gets extended over time and often the duplication is missed.

My suggestion is that when the common functionality is identified through refactoring of the implementation, we address this common functionality at the test level (whilst considering if a separate service might be the cleaner route).

Testing private static utility methods using PowerMock

Let us assume that we intend to refactor the service to provide an internal private static utility method to manage the core functionality, as follows:

public class PadderImpl implements Padder {
	
	public String padWithSpaces(String text, int size){
		return pad(text, " ", size);
	}

	public String padWithHashes(String text, int size){
		return pad(text, "#", size);
	}

	private static final String pad(String text, String padding, int size) {
		//pad the string with a supplied padding
	}

}

Note that we want to keep the utility method private because we do not intend to expose this service to our users as it would complicate the interface.

We can now directly test the utility method, which is purporting to provide a generic padding service. We can use PowerMock’s WhiteboxImpl.invokeMethod() method to invoke the private utility method in the tests:

import static org.junit.Assert.assertEquals;
import static org.powermock.reflect.internal.WhiteboxImpl.invokeMethod;

import org.junit.Before;
import org.junit.Test;

public class PadderImplTest {
	
	private Padder padder;
	
	@Before
	public void setUp(){
		padder = new PadderImpl();	
	}
	
	@Test
	public void anEmptyStringInputWillReturnAStringOfPaddingWithALengthOfSize() throws Exception {
		String result = invokeMethod(padder, "pad", "", " ", 1);
		
		assertEquals(" ", result);
	}

	@Test
	public void aSizeLongerThanTheLengthOfTheStringReturnsAPaddedStringUpToTheSize() throws Exception {
		String result = invokeMethod(padder, "pad", "text", "*", 8);
		
		assertEquals("text****", result);
	}

	@Test
	public void aSizeShorterThanTheLengthOfTheStringReturnsAStringTruncatedToALengthOfSize() throws Exception {
		String result = invokeMethod(padder, "pad", "sandwich", "*", 4);
		
		assertEquals("sand", result);
	}
	
	@Test
	public void aSizeEqualToTheLengthOfTheStringReturnsTheSameString() throws Exception {
		String result = invokeMethod(padder, "pad", "sandwich", "*", 8);
		
		assertEquals("sandwich", result);
	}
	
	@Test
	public void paddingWithALongStringCausesThePaddingToBeTruncatedIfNecessary() throws Exception {
		String result = invokeMethod(padder, "pad", "$", "abc", 5);
		
		assertEquals("$abca", result);
	}

}

We now have a padding utility service that we can trust as tested within our service. Next we need to verify the behaviour of each of the service methods. This can now be done in terms of interaction with the utility method. Testing our static method requires the test to be altered to be run as a PowerMock test and to prepare the class with the static method using PrepareForTest:

import static org.junit.Assert.assertEquals;
import static org.powermock.api.mockito.PowerMockito.mockStatic;
import static org.powermock.api.mockito.PowerMockito.verifyPrivate;
import static org.powermock.reflect.internal.WhiteboxImpl.invokeMethod;

import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.powermock.core.classloader.annotations.PrepareForTest;
import org.powermock.modules.junit4.PowerMockRunner;

@RunWith(PowerMockRunner.class)
@PrepareForTest(PadderImpl.class)
public class PadderImplTest {

We can now add the final tests:

	@Test
	public void padWithSpacesCallsPadWithASpace() throws Exception {
		mockStatic(PadderImpl.class);
		
		padder.padWithSpaces("text", 10);
		
		verifyPrivate(PadderImpl.class).invoke("pad", "text", " ", 10);
	}
	
	@Test
	public void padWithHashesCallsPadWithAHash() throws Exception {
		mockStatic(PadderImpl.class);
		
		padder.padWithHashes("text", 10);
		
		verifyPrivate(PadderImpl.class).invoke("pad", "text", "#", 10);
	}

mockStatic() sets up our class for static verification, and using the verifyPrivate method, we can verify that the private static utility method is called correctly.

Adding an additional public method is as simple as describing only its interaction with the utility method, rather than copying the same set of more complex padding tests:

	@Test
	public void padWithDashesCallsPadWithADash() throws Exception {
		mockStatic(PadderImpl.class);
		
		padder.padWithDashes("text", 10);
		
		verifyPrivate(PadderImpl.class).invoke("pad", "text", "-", 10);
	}

One of the advantages with this approach is that refactoring to extract a service (where previously we had used a utility method) is much easier. This is because the tests reflect the service-like behaviour of the utility method rather than the black-box behaviour of the public methods.

There are problems with this approach however. If one is not careful to treat the utility method as ‘service-like’ then the coverage could be compromised and bugs introduced. It is not the way to test any internal method and it could be particularly dangerous if used to test entirely at a method-level. Generally speaking, the service should be treated as a black box, but occasionally it pays off to compromise and test internal method calls.

Testing Recipes, Part 1 – Self-service

Should the testing frameworks or methodologies used, influence the design of the implementation code? I’m sure most would respond in the negative, but sometimes this is very hard to avoid. I have been attempting to overcome some of this influence by finding ways to test code that I had previously refactored to make it more ‘testable’. To achieve this, I have been building on use of the standard Mockito syntax, to apply the extensions provided by PowerMock (with Mockito). This first example uses just Mockito.

The first problem I will address is how to test services that use their own services (‘self-service’). This scenario is best explained by the code below showing a ‘client’ service that allows the service-user to call methods on the client.

public class Client {

	public void create(){
		//perform action
	}

	public void delete(){
		//perform action
	}

	public void recreate(){
		delete();
		create();
	}

}

Three methods are available: create(), delete() and recreate(). Each method performs an action on the client, but recreate() is different because the action performed makes use of other service actions. More complex scenarios could be presented where a method would call other service methods multiple times, or vary calls according to business logic, but this simple example is suitable to explain a typical use-case.

The testing problem comes when wanting to test the recreate() method. Let’s assume that our client makes calls to a ‘databaseService’ to perform the underlying actions.

Approach 1 – Test the underlying interactions

A simple approach would be as follows:

	@Test
	public void recreateDeletesThenCreates() {
		client.recreate();

		//verify underlying actions
		verify(databaseService).runSQL('DELETE...');
		verify(databaseService).runSQL('INSERT...');
	}

The problem, I feel, with this test, is that we are not testing the functionality of the recreate() method, but instead we are treating the Client as a black box and testing how the client interacts with the underlying service. This would be the correct approach for the simpler create() and delete() methods but we are now open to duplication in our test suite. If the lower level methods become more complex, then tests for the higher level methods will increasingly duplicate already tested functionality. This problem begins immediately because the line

verify(databaseService).runSQL('DELETE...');

will presumably already be present in the test for delete(). If we want to refactor delete() then we have to change two or more tests.

Approach 2 – Extract an ‘extended’ service

Alternatively we can extract an ‘extended’ service which provides only the functionality that is defined in terms of the basic service methods:

public class BasicClient {

	public void create(){
		//perform action
	}

	public void delete(){
		//perform action
	}

}

public class ExtendedClient {

	private BasicClient client;

	public void recreate(){
		client.delete();
		client.create();
	}

}

We can then test the recreate() in terms of interactions with the BasicClient (the setInternalState() method is part of the Mockito Whitebox class):

public class ExtendedClientTest {

	@Mock BasicClient basicClient;

	ExtendedClient extendedClient;

	@Before
	public void setUp(){
		MockitoAnnotations.initMocks(this);

		extendedClient = new ExtendedClient();
		setInternalState(extendedClient, "client", basicClient);
	}

	@Test
	public void recreateCallsDeleteAndCreateOnTheBasicClient() {
		extendedClient.recreate();

		verify(basicClient).delete();
		verify(basicClient).create();
	}

}

The problem I see here is that we have let our testing needs influence our implementation. The approach of extracting services to reduce complexity is often the correct approach to writing more testable code. However, if no concept is reflected in the domain and the extended service is extracted purely for the purposes of testing the higher level methods, then extracting the service only adds complexity, and should therefore be avoided.

Approach 3 – Verify interaction between methods using spy

My suggested approach is to take advantage of Mockito’s spy() method. Creating a spy of the Client creates an object that can be treated like a mock (stubbing and method verification is possible), but maintains the original implemented code until stubbed.

public class ClientTest {

	Client client;
	
	@Before
	public void setUp(){
		client = spy(new Client());		
	}
	
	@Test
	public void recreateCallsDeleteThenCreate() {
		client.recreate();
		
		verify(client).delete();
		verify(client).create();
	}
	
}

We now have a Client that cleanly represents the service we wish to offer, but at the same time we can test method in terms of their interactions – testing methods in this way is generally best avoided but our Client presents a counter-example. Using this approach we must be careful to avoid the common pitfall of testing the implementation rather than the behaviour. I feel however that this approach allows us to build up interactions between methods inside a class without resulting in an explosion of test complexity.

Even the comments for the spy() method itself warn against it’s usage:

Real spies should be used carefully and occasionally, for example when dealing with legacy code.

and in general I concur, but the example presented above represents a scenario where attempting to avoid using this feature will comprise the design of the system.

These techniques are entirely compatible with a test-driven approach and provide a much cleaner way of adding higher level methods to services that already provide a set of lower level methods. I intend to explore testing internal private utility methods later in this series which is also closely related to this problem.

Thanks to Johan for recommending improvements to the code examples.

Why Android isn’t ready for TDD, and how I tried anyway

Pick a station to find out more...

Over the last few weeks I’ve been attempting to develop an small application for the Android development platform. In particular, I’ve been applying the techniques of test-driven development (TDD) to this project in an attempt to understand whether they are compatible with the platform. But before launching into more a more detailed explanation of the problems I’ve faced I will provide a brief description of the app I have written…

As an avid listener of BBC radio, particularly 6music and Radio 4, I always want to find out about what I’m listening to. I find the text provided by my DAB radio is both limited and slow. Sometimes it says something like “Why not tell us what you’d like to listen to blah blah” (or similar generic message, and not the currently playing track) which takes forever to scroll across my 30 character LCD display. The solution? It’s called ‘BBC Radio What’s On’: on your Android phone you select from a list of available radio stations (see above).

What's on 6music?

Then you are shown a page with information about:

1) What programme is on now
2) What’s on next
3) What were the last 10 tracks played on this station*

*This data is only as accurate as the data provided by the BBC radio teams, which vary from show to show

So, as you can see, a very simple application, but (arguably) very useful. As far as I am aware, no BBC web page provides this data in one place. The nearest to this is the new configurable BBC mobile homepage. This allows the user to select radio stations to appear on the homepage indicating what show is on now, but no information about current tracks.

Development

Most people I have spoken to about Android development have been broadly positive. The speed and ease with which you can create and deploy a basic application on the emulator or a real phone is impressive. The tight integration with Eclipse makes development simple (performance issues aside, see below). Where I really found development problematic was when attempting to continue developing in the way I do when building web applications (i.e. using test-driven development and dependency injection).

Why testing is a problem

A basic unit testing scenario is that I want to test some code that makes use of a third-party utility library. In my application I wanted to make a service that reads a JSON file from a URL (using my HttpService) and returns a object representing the JSON in way that reflects the domain. This object is a RadioStation object which contains meta-data about a particular radio station. The method updateRadioStation(radioStation) takes a radio station and populates the fields based on JSON. My test, shown below, is trying to capture this functionality:

@Mock HttpService httpService;

@Test
public void loadsDataCorrectlyFromJSON() throws IllegalStateException, IOException, JSONException {
    RadioStation radioStation = new RadioStation("x", "service", "radio x");

    when(httpService.callUrlAndReturnString(anyString()))
        .thenReturn(FileResources.BBC_RADIO_STATION_JSON);

    radioStationLoader.updateRadioStation(radioStation);

    assertEquals(radioStation.getCurrentShow().getType(), "On Now:");
    assertEquals(radioStation.getCurrentShow().getTitle(), "Cerys on 6");
    assertEquals(radioStation.getCurrentShow().getSubtitle(), "16/10/2009");
    assertEquals(radioStation.getCurrentShow().getPid(), "b00n9nfv");
    assertEquals(radioStation.getCurrentShow().getSynopsis(), 
                 "Music, chat and more archive sessions with Cerys.");
}

Unfortunately, if we run this test as a junit test, we get the following error:

java.lang.RuntimeException: Stub!
    at org.json.JSONObject.(JSONObject.java:35)
    at daverog.bbcradio.service.RadioStationLoader.updateRadioStation(...:29)
    at daverog.bbcradio.service.RadioStationLoaderTest.loadsDataCorrectlyFromJSON(...:36)
    etc...

The problem here is that the android.jar supplied with the SDK is stubbed out with no implementation code. The solution that seems to be expected is that you should run your unit tests on the emulator or a real phone. Google have provided all the tools to do this; in fact it is an intrinsic part of the platform. To run a test on the emulator, the Eclipse plugin supplies a new run target ‘Android junit test’ (or simply install the application as normal and all unit tests will be run). This will load up the emulator, load the whole application onto the device, and run the test inside the Dalvik virtual machine*.

Dalvik virtual machine
From Wikipedia, the free encyclopedia
Dalvik is the virtual machine which runs the Java platform on Android mobile devices.
It runs applications which have been converted into a compact Dalvik Executable (.dex) format suitable for systems that are constrained in terms of memory and processor speed.
Dalvik was written by Dan Bornstein, who named it after the fishing village of Dalvík in Eyjafjörður, Iceland, where some of his ancestors lived.

By running on the emulator and against the fully working Dalvik-compiled platform the unit test will succeed. Well, it would succeed, if you’d done the following:

  • Made all tests extend some kind of Android test case class (e.g. AndroidTestCase)
  • Stopped using a mocking framework
  • Downgraded to junit 3
  • Tweaked the imports to match Android’s package changes (e.g. framework.junit from org.junit)

and worst of all?

  • Wait at least 10 seconds for even a single simple test to run

The five points above make testing on the emulator a largely impractical process. Testing needs to be simple and responsive, and developers need to be able to leverage powerful matching and mocking frameworks. The rest of this blogpost assumes that the above five points hold true, so I should briefly explain some of the alternatives I tried or thought of that did not work:

1) Import a mocking framework (e.g. mockito) into the project as an additional dependency.

Any imported jars containing class files not compiled to Dalvik bytecode (most) will not work. Attempting to compile the source along with your project will not work either because most libraries will make extensive use of parts of the Java language not compatible with Dalvik: it uses its own library built on a subset of the Apache Harmony Java implementation.

2) Obtain a version of the android.jar where the utility libraries are not stubbed.

Tried and failed, but please tell me if you find one.

That the deployment environment (Dalvik) can place restrictions on the way in which code is tested is very frustrating. For the same reason, we also have a restriction on the libraries we can use as part of the application. The Android platform provides a very extensive array of libraries that make it easier to write most applications, particularly those with a web-focus. However, I feel one is missing that is central to creating applications using test-driven development and dependency injection: Spring. The inclusion of a lightweight IOC container would be an excellent addition to the Android platform.

I have one final bugbear that I need to address. It is not important because it will be possible to fix, but it caused me more frustration than any other part of the development process. I am referring to the performance of Eclipse when using the Android SDK plugin. When Eclipse first loads and you are working with an Android project, the responsiveness is quite acceptable, but, for whatever reason, Eclipse increasingly slows. Use of the emulator is particularly detrimental and after around half an hour of development it can take 10-20seconds to switch between two file tabs. This is unacceptably slow and I find myself rebooting Eclipse and the emulator every half hour to alleviate the problem. Moan over.

An approach to TDD on Android

Based on the assumption that running tests on the emulator is too restrictive and slow, I wanted to come up with an approach that at least partially facilitates test-driven development. First, in Eclipse, we create a new Java (not Android) project call [ProjectName]Test which will be dependent on the main project and hold all the tests and test resources. By having a separate non-Android project we can add resources and code that do not conform to the Android project specification and satisfy the automatic validation. This project will be setup to use a standard JRE system library. Now we can add tests to this project that test the classes in the main project.

The first general suggestion I’d make is: keep things simple by trying to make as many services as possible not dependent on the parts of the Android platform that are not compatible with a conventional JVM. This avoids the “Stub!” error I mentioned above. The LastFmRssReader service does not use any Android-specific libraries so it can be tested using whatever testing or mocking frameworks you wish:

public class LastFmRssReader {

    ...

    private RssFeedParser parser;

    public List getPlayedTracksForLastFmUser(String lastFmUser) {
        parser.setUrl(HTTP_WS_AUDIOSCROBBLER_COM_1_0_USER+lastFmUser+RECENTTRACKS_RSS);

        List playedTracks = new ArrayList();
        List messages = parser.parse();

        for(Message message:messages){
            String title = message.getTitle();
            String artist = "";
            ...
            playedTracks.add(new PlayedTrack(artist, title, message.getDate()));
        }

        return playedTracks;
    }

}

an example test:

@Test
public void readerCallsCorrectURLAndExtractsArtistAndTitleFromMessageTitle(){
    LastFmRssReader lastFmRssReader = new LastFmRssReader(rssFeedParser);

    ArrayList messages = new ArrayList();
    Message message= new Message();
    message.setTitle("artist - title");
    messages.add(message);        

    when(rssFeedParser.parse()).thenReturn(messages);

    List tracks = lastFmRssReader.getPlayedTracksForLastFmUser("testuser");

    assertEquals(tracks.get(0).getArtist(), "artist");
    assertEquals(tracks.get(0).getTitle(), "title");

    verify(rssFeedParser).setUrl(
        "http://ws.audioscrobbler.com/1.0/user/testuser/recenttracks.rss");
    verify(rssFeedParser).parse();
}

This approach should work well, particularly for larger applications where more and more services perform an internal role where external libraries are not required. It’s also the best way to test domain objects which usually don’t need mocked services. Where this approach fails is where much of the code written is either using libraries or the parts of Android API, which would be common in small applications (such as ‘BBC Radio What’s On’).

Using the approach above, two problems remain:

  1. Testing parts of the system that interact with the Android API
  2. Testing parts of the system that use common libraries embedded in the Android platform

I am stumped regarding the best way to test code that uses the Android API. I have been reverting to testing on the emulator using Junit 3 and no mocking framework. The android.test.mock package may help with this, but seems to be just a bunch of stubbed out implementations for you to override as you require, so does not offer nearly as much power as a mocking framework.

I have found an unconventional way of testing with the common libraries embedded in the Android platform. This approach is the best example of how awkward testing can be on Android.

If we go back to the test I first used as an example:

@Mock HttpService httpService;

@Test
public void loadsDataCorrectlyFromJSON() throws 
        IllegalStateException, IOException, JSONException {
    RadioStation radioStation = new RadioStation("x", "service", "radio x");

    when(httpService.callUrlAndReturnString(anyString()))
        .thenReturn(FileResources.BBC_RADIO_STATION_JSON);

    radioStationLoader.updateRadioStation(radioStation);

    assertEquals(radioStation.getCurrentShow().getType(), "On Now:");
    assertEquals(radioStation.getCurrentShow().getTitle(), "Cerys on 6");
    assertEquals(radioStation.getCurrentShow().getSubtitle(), "16/10/2009");
    assertEquals(radioStation.getCurrentShow().getPid(), "b00n9nfv");
    assertEquals(radioStation.getCurrentShow().getSynopsis(), 
        "Music, chat and more archive sessions with Cerys.");
}

This test cause a “Stub!” exception when run as a unit test.

Here’s the implementation I am testing (it’s loading json from a URL and populating a domain object):

import org.json.JSONArray;
import org.json.JSONException;
import org.json.JSONObject;

public class RadioStationLoader {
    ...

    public void updateRadioStation(RadioStation radioStation) throws 
            IllegalStateException, IOException, JSONException{
        String url = HOST + radioStation.getBrandId()
                + "/programmes/schedules" + radioStation.getServiceTypeURLExtension()
                + "/upcoming.json";

        String body = httpService.callUrlAndReturnString(url);

        JSONObject json = new JSONObject(body); <--- This is where the exception is thrown

        JSONObject schedule = ((JSONObject) json.get("schedule"));

        JSONObject service = ((JSONObject) schedule.get("service"));
        String serviceTitle = service.getString("title");

        radioStation.setName(serviceTitle);

        JSONObject now = ((JSONObject) schedule.get("now"));
        JSONObject currentBroadcast = ((JSONObject) now.get("broadcast"));
        JSONObject next = ((JSONObject) schedule.get("next"));
        JSONArray broadcasts = ((JSONArray) next.get("broadcasts"));
        JSONObject nextBroadcast = ((JSONObject) broadcasts.getJSONObject(0));

        radioStation.setCurrentShow(extractRadioShowFromBroadcastJson(
            "On Now:", currentBroadcast));
        radioStation.setNextShow(extractRadioShowFromBroadcastJson(
            "On Next:", nextBroadcast));
    }

    private RadioShow extractRadioShowFromBroadcastJson(
            String type, JSONObject broadcastJson) throws JSONException{
        String start = broadcastJson.getString("start");
        String end = broadcastJson.getString("end");
        JSONObject programme = ((JSONObject) broadcastJson.get("programme"));
        JSONObject display_titles = ((JSONObject) programme
                .get("display_titles"));
        String title = display_titles.getString("title");
        String subtitle = display_titles.getString("subtitle");
        String pid = programme.getString("pid");
        String synopsis = programme.getString("short_synopsis");
        return new RadioShow(type, title, subtitle, start, end, synopsis, pid);
    }

}

If I copy this class into my test project and locate in the same package then I can run against a different test copy. What I want to do now is run the code against an implemented version of the JSON library. If I import the JSON jar (JSON lib) into my project along with its dependencies then I can use this instead of the Android stubbed version. Because the package structures between the external library and the Android version are different I need to switch the import to use the alternative library:

import org.json.JSONObject;
becomes
import net.sf.json.JSONObject;

And because Android are using a slightly different version I need to change the constructor:

JSONObject json = new JSONObject(body);
becomes
JSONObject json = JSONObject.fromObject(body);

The class then compiles. Running my test is now successful:

This approach allowed me to quickly find several bugs in the RadioStationLoader before copying over the fixes to the original and deploying to the emulator. This is clearly not a sustainable approach but it provides a solution to developing code in a test-driven, dependency injected way.

Finally, I needed a way to wire my services together to provide them to the main Activity classes. It’s hard to write applications using inversion of control without somewhere to invert the control to. In Java web applications this would be Spring . In Android, no such framework is available. This seems unfortunate as Spring (core) is lightweight (in many senses), even more so with a stripped-down alternative like Spring ME. Without such a framework, a developer is limited in leveraging the power of IOC architectures. My crude alternative was to configure my system and inject my dependencies in an ‘application context’ Java class.

client = new DefaultHttpClient();
httpService = new HttpService(client);
radioStationLoader = new RadioStationLoader(httpService);
rssFeedParser = new RssFeedParser();
lastFmRssReader = new LastFmRssReader(rssFeedParser);

public RadioStationLoader getRadioStationLoader(){
    return radioStationLoader;
}

public LastFmRssReader getLastFmRssReader(){
    return lastFmRssReader;
}

finally…

The one main recommendation that I would make as a result of this work would be for Google to provide an alternative android.jar that is stubbed as little as possible. This is particularly important where common open-source libraries have been integrated into the platform. This would make unit testing, and therefore test-driven development, much less painful. Oh, and Spring on Android would be nice too.

MPs in the Media Mash-up

At the end of July the Guardian held an internal hackday at their offices in King’s cross. They invited me back, and another engineer from BBC Radio’s A&Mi department, Chris Lowis, came along with me. We teamed up with Leigh Dodds & Ian Davis from Semantic Web specialists, Talis to produce an ‘Interactive-MP-Media-Appearance-Timeline’ by mashing up data from BBC Programmes and the Guardian’s website.

Before the event Talis extracted data about MPs from the Guardian’s Open Platform API and converted it into a Linked Datastore. This store contains data about every British MP, the Guardian articles in which they have appeared, a photo, related links and other data. Talis also provide a SPARQL endpoint to allow searching and extraction of the data from the store.

Coincidentally, the BBC programmes data is also available as a linked datastore. By crawling this data using the MP name as the search key we were able to extract information about the TV and radio programmes in which a given MP had appeared. A second datastore was created from the combination of these two datasets, and by pulling in some related data from dbpedia. Using this new datastore we created a web application containing an embedded visualisation of the data.

We created the web-app using the lightweight ruby web-framework Sinatra. A simple RESTful URL schema provided access to a web page showing basic information about an MP.

Nick Clegg: a busy man in 2009

Nick Clegg: a busy man in 2009

In addition we queried the datastore to give a list of all of the MPs appearances across Guardian and BBC content. This was returned as a JSON document, and passed to an embedded Processing Java applet. A Java applet may seem like an unusual choice in 2009, but Processing is an excellent choice for the rapid development of responsive graphics applications, due to its integration with existing JAVA libraries, and its powerful graphics framework.

Leigh at Talis put together a screencast showing the app in action:

The Processing applet shows a month-by-month scrollable timeline. The user can move back and forward in time, at variable speeds, by pressing the mouse either side of the applet frame. In each month slot, a stack of media appearances is displayed, represented by the logo of the BBC brand, or in the case of Guardian articles, the Guardian brand. Moving the mouse over a media appearance reveals the headline or programme description and clicking a media appearance will navigate the browser to the episode page on the /programmes or the article page on guardian.co.uk.

We demonstrated the application to the hackday audience, and in the prize giving ceremony were awarded the ‘Best use of third-party data’ award. We think that the application demonstrates some of the ways the structured RDF data provided by BBC’s /programmes website can be used. This project shows how powerful the linked-data concept is when used in conjunction with other data that has been exposed in a similar way. As more media organisations expose their domains in this manner, more interesting and wide-reaching visualisations and web-applications can be built.

Thanks to Chris Lowis for contributions to this post. Photos courtesy of Mat Wall.

Now also published on the BBC’s Internet blog