Writing maintainable code
Tuesday 27 September 2011 20:14
I heartily endorse this fine article on writing maintainable code. What do you mean I'm biased because I wrote it?
Topics: Software design, Software development, Testing
Design Minimalism or the Dinosaur Snaggletooth
Tuesday 7 December 2010 00:00
While looking for a templating language for node.js, I came across an article On Design Minimalism, which gels well with my view that reusable code is all about small components that do as little as possible to get the job done, rather than frameworks that do everything you can think of.
Topics: Software design
Naming Functional/Procedural Functions
Thursday 9 September 2010 14:15
It was recently pointed out to me (unfortunately, I can't remember where) that a good way to distinguish functional and procedural versions of similar functions is to use adjectives for the functional version and verbs for the procedural version.
For instance, consider functions that sort a list. The functional version would be called sorted, and returns a sorted copy of the list:
some_list = [2, 0, 1] sorted_list = sorted(some_list) # some_list == [2, 0, 1], sorted_list == [0, 1, 2]
On the other hand, the procedural function would be called sort, and modifies the list in-place:
some_list = [2, 0, 1] sort(some_list) # some_list == [0, 1, 2]
This is probably obvious to everybody else, but I'd never really thought it too much before.
Topics: Software design
Naming Test Methods
Thursday 13 May 2010 14:37
Picking names sensibly in code is crucial to readability. For instance, let's say you're storing prices in your database, and they're represented by a Price class. What does a PriceHelper do? Naming something a helper isn't helpful in the least. On the other hand, the class PriceFetcher probably fetches prices from somewhere. Others have written about what names not to use for classes, but I want to talk about a different set of names. I want to talk about how we name our tests. For instance, let's say I'm implementing a join method in Java that will glue together a list of strings. Being a good programmer, I write a test to describe its behaviour:
class JoinTest {
@Test
public void testJoin() {
assertEquals("", join(", ", emptyList()));
assertEquals("Apples", join(", ", asList("Apples")));
assertEquals(
"Apples, Bananas, Coconuts",
join(", ", asList("Apples", "Bananas", "Coconuts"))
);
}
}
So, what's wrong with calling the method testJoin? The answer is that the method tells us nothing about what we expect to happen, just that we're testing the join method. By picking a name that describes what we expect, we can help the next reader of the code to understand what's going on a little better:
class JoinTest {
@Test public void
joiningEmptyListReturnsEmptyString() {
assertEquals("", join(", ", emptyList()));
}
@Test public void
joiningListOfOneStringReturnsThatString() {
assertEquals("Apples", join(", ", asList("Apples")));
}
@Test public void
joiningListOfStringsConcatenatesThoseStringsSeparatedBySeparator() {
assertEquals(
"Apples, Bananas, Coconuts",
join(", ", asList("Apples", "Bananas", "Coconuts"))
);
}
}
Splitting up tests in this fashion also means that it's easier to pin down failures since you can easily work out exactly what test cases are failing.
This works well with Test Driven Development (TDD). Each time you want to add a new behaviour and extend the specification a little further by adding a failing test, the test method describes the new behaviour in both the test code and its name. In fact, having rearranged the code so that the test method names are on their own line without modifiers, we can read a specification of our class simply by folding up the bodies of our test methods:
class JoinTest {
joiningEmptyListReturnsEmptyString() {
joiningListOfOneStringReturnsThatString() {
joiningListOfStringsConcatenatesThoseStringsSeparatedBySeparator() {
}
This reflects the view that test driven development is just as much about specification as testing. In fact, you could as far as saying that regression tests are a happy side-effect of TDD. Either way, the intent of each test is now that much clearer.
Topics: Testing
Orders of Magnitude
Sunday 21 February 2010 20:57
Improving performance is often a desirable goal. Sometimes you'll have a precise number for just how much performance needs to be improved by, particularly in real-time systems. More often, though, the request for improved performance is far more vague. So, what sort of numbers should we aim for when we want things to go faster? This depends on why you want faster performance -- do you just want to save a bit of time, or do you really want things to change?
Take, for instance, the time it takes to run your entire test suite. This can vary wildly, depending on the application, from seconds to days. Let's say we're working on a small project, and we have a test suite that covers the entire application in one minute. This is fast enough that we can run the entire suite every time before we commit, but we won't be running it every time we make a small change. If we made it go, say, twice as fast, we'd definitely save ourselves some time -- thirty seconds for every commit, if we really do run all the tests before every commit. This is still too slow to be running each time we make a small change, but what if we sped up the suite by an order of magnitude instead, so it takes only a few seconds to run? Now, running the entire suite every minute or two is practical, rather than just before every commit.
Sometimes, we really can get performance improvements of an order of magnitude, by improvements in technology or a clever new algorithm. Otherwise, we might still be able to do what all programmers do -- cheat. If we can get most of the benefit in a much shorter time, then this is often good enough. Going back to our test suite, if we can identify some subset of the tests that run in 10% of the time with 90% of the coverage, then most bugs we might introduce are still picked up, while our tool, the test suite, becomes more flexible.
By improving performance not by small amounts, but by orders of magnitude, we can change the way we use and think about our tools. Performance really does matter.
Topics: Software development
Object orientation without classes
Saturday 9 January 2010 21:47
So, in the last post, I decided that object orientation was about a couple of key concepts: encapsulation and polymorphism. Revisiting our NameGenerator example, we had this code in Java:
public class NameGenerator {
private final FirstNameGenerator firstNameGenerator;
private final LastNameGenerator lastNameGenerator;
public NameGenerator(FirstNameGenerator firstNameGenerator,
LastNameGenerator lastNameGenerator) {
this.firstNameGenerator = firstNameGenerator;
this.lastNameGenerator = lastNameGenerator;
}
public String generateName() {
return firstNameGenerator.generate() + " " + lastNameGenerator.generate();
}
}
However, there's an awful lot of duplication here -- in the fields and constructor, we mention a first name generator no less than six times. There's also another form of duplication present -- this constructor is identical to every other constructor I write in Java in that all it does is assign arguments to fields. After all, doing work in the constructor is a bad idea. In an ideal world, we might decide a more succinct syntax for our class would be:
public class NameGenerator(FirstNameGenerator firstNameGenerator,
LastNameGenerator lastNameGenerator) {
public String generateName() {
return firstNameGenerator.generate() + " " + lastNameGenerator.generate();
}
}
Now, the constructor arguments are parameters for the class itself. This is functionally equivalent to the example above, while cutting down on the duplication we saw. Incidentally, Scala users will be familiar with this style of constructing objects.
The example is still in a statically typed language -- so what happens if we remove the static typing?
public class NameGenerator(firstNameGenerator, lastNameGenerator) {
public generateName() {
return firstNameGenerator.generate() + " " + lastNameGenerator.generate();
}
}
I'm not sure this has really helped readability or clarity. Certainly, we haven't gotten the same gains that we got by changing from a Java constructor to using class parameters. However, if we inspect our dynamically typed example closely, we might notice that there's very little that distinguishes it from a couple of function definitions. This leads us to write code that looks something like this:
function nameGenerator(firstNameGenerator, lastNameGenerator) {
return {
generateName: function() {
return firstNameGenerator.generate() + " " + lastNameGenerator.generate();
}
};
}
In a few simple transformations, we've moved from Java to Javascript. Instead of a class, we have a function that takes a couple of generators. This function returns a new object with a single field called generateName. It just so happens that this field is a function, which uses the two generators to generate a full name. So, to give an example usage of the class:
// Assume we already have first and last name generators var generator = nameGenerator(firstNameGenerator, lastNameGenerator); var name = generator.generateName();
Hopefully this is enough to persuade you that we don't need classes in order to capture what we mean by object orientation. Indeed, there are only really two language features that we've used here. The first is the object, which allows us to group together the various parts of some idea (although in this case, we only have a single field).
The second feature is functions. But just having functions is not enough -- there are many languages that have functions where this wouldn't be possible to write. To be more specific we need to have first class functions, so that we can assign a function to a field as easily as we would assign a number or a string to a field. It also helps to have anonymous functions so that we can create functions easily, in the same way that Javascript's object literal notation makes it straightforward to build objects. The other piece of the puzzle is closures. Although firstNameGenerator and lastNameGenerator were not defined in the function assigned to generateName, they can still access these values since Javascript supports closures. This means that the references to the generators are still valid even after the function nameGenerator has returned.
This raises the question of what features you actually need in a language, and what features you can implement by composing other features. Another example might be namespaces. Many languages have namespaces explicitly built into the language, but by using layers of objects, you can easily build namespaces in Javascript despite the lack of explicit support for namespaces in the language. For instance, if were writing real Javascript code, we should put our nameGenerator method into the proper namespace, rather than in the global namespace:
ZWOBBLE = ZWOBBLE || {};
ZWOBBLE.names = ZWOBBLE.names || {};
ZWOBBLE.names.nameGenerator = function(firstNameGenerator, lastNameGenerator) {
...
}
We can easily write a function to replace this first couple of lines, so we have:
ZWOBBLE.define("ZWOBBLE.names");
ZWOBBLE.names.nameGenerator = function(firstNameGenerator, lastNameGenerator) {
...
}
Again, I hope this enough to convince that there's no need in Javascript for explicit support for namespaces. I could go on, but my broader point is that by giving a language a few basic, but powerful, constructs, one can build on top of them to reach the same features that are explicitly supported in other languages.
Incidentally, I've recently read a couple of articles on object orientation that I found interesting. The first distinguishes between abstract data types and objects, while the second makes the case for tail calls in object orientation languages.
Topics: Software design
Object orientation, inheritance and the template pattern (oh my!)
Monday 30 November 2009 19:43
Recently, I've been discussing with a variety of people the merits of various styles of programming. One person's complaint about object orientated programming is that it leads to huge hierarchies of code, where you constantly have to move up and down the class hierarchy to find out what's going on. I've encountered such code on plenty of occasions, but I disagree that it's a problem with object orientation. Instead, I believe it's a problem with inheritance.
Of course, this all depends on what we mean by object orientation. So, let's take a simple example in Java. In this case, we're going to make a class that generates names.
public class NameGenerator {
public String generateName() {
return generateFirstName() + " " + generateLastName();
}
protected String generateFirstName() {
// Generate a random first name
...
}
protected String generateLastName() {
// Generate a random last name
...
}
}
So, NameGenerator will generate names using the default implementations of generateFirstName and generateLastName. If we want to change the behaviour of the class, say so that we only generate male names, we can override the generateFirstName method to only generate male names. This is known as the template pattern, the idea being that by putting the overall algorithm in one method that delegates to other methods, we avoid code duplication while still allowing different behaviours in different subclasses.
However, in my experience, the template pattern is a terrible way to write code. In some piece of code, if we see that the NameGenerator is being used, we might take a quick look at the class to see how it behaves. Unfortunately, the fact that subclasses may be overriding some methods to change behaviour isn't immediately obvious. This can make debugging difficult, since we start seeing behaviour we don't expect according to the class definition we have right in front of us. This becomes especially confusing if one of the methods affects control flow, for instance if we use one of the protected methods in a conditional, such as an if statement.
Testing also becomes more difficult. When we subclass NameGenerator, what methods should we be testing? We could always test the public method, but then we'll be testing the same code over and over again. We could test just the protected methods, but then we aren't testing the public method fully since it's behaviour changes depending on the protected methods. Only testing the public method if we think its behaviour will have changed is error-prone and brittle to changes made in behaviour in NameGenerator.
The solution is to use composition rather than inheritance. In this case, the first step is to pull out the two protected methods into two interfaces. The first interface will generate first names, and the second interface will generate last names. We then use dependency injection to get a hold of some implementation of these classes. This means that, rather than creating classes themselves, we have them passed in via the constructor. So, our NameGenerator now looks like this:
public class NameGenerator {
private final FirstNameGenerator firstNameGenerator;
private final LastNameGenerator lastNameGenerator;
public NameGenerator(FirstNameGenerator firstNameGenerator,
LastNameGenerator lastNameGenerator) {
this.firstNameGenerator = firstNameGenerator;
this.lastNameGenerator = lastNameGenerator;
}
public String generateName() {
return firstNameGenerator.generate() + " " + lastNameGenerator.generate();
}
}
Now we've made the dependencies on the generation of first and last names clear and explicit. We've also made the class easier to test, since we can just pass in mocks or stubs into the constructor. This also encourages having a number of smaller classes interacting, rather than a few enormous classes. The code is also more modular, allowing it be reused more easily. For instance, if there's somewhere in the code where we just want to generate first names, we now have an interface FirstNameGenerator that we can use straight away.
For more on dependency injection, Misko Hevery has a good blog post on the subject, as well as a tech talk on dependency injection.
Notice how the new version of our code is completely devoid of inheritance. That's not to say that we have a completely flat class hierarchy -- we need implementations of FirstNameGenerator and LastNameGenerator, but it's important to keep inheritance and implementation distinct. Despite the lack of inheritance, I believe that this is still an example of object orientated programming in action. So, if not inheritance, what characterises object orientation?
I would first suggest encapsulation -- in this case, we don't care how the first and last name generators work, we just call their methods to get a result out. In this way, we can concentrate on one small part of the program at a time.
This doesn't tell the whole story though. If we didn't allow any subtyping, or rather polymorphism, then our example would not be able to use different implementations of FirstNameGenerator. By having polymorphism, we allow different implementations to be used so that we can use the same algorithm without rewriting it.
So, there we have my view of object orientation -- composition of small objects using encapsulation and polymorphism, while avoiding inheritance and, in particular, the template pattern.
Topics: Software design, Testing
Dynamic languages and testing
Friday 23 October 2009 19:57
The debate between static and dynamic typing has gone on for years. Static typing is frequently promoted as effectively providing free tests. For instance, consider the following Python snippet:
def reverse_string(str):
...
reverse_string("Palindrome is not a palindrome") # Works fine
reverse_string(42) # 42 is not a string
reverse_string() # Missing an argument
In a static language, those last two calls would probably be flagged up as compilation errors, while in a dynamic language you'd have to wait until runtime to find the errors. So, has static typing caught errors we would have missed?
Not quite. We said we'd encounter these errors at runtime -- which includes tests. If we have good test coverage, then unit tests will pick up these sorts of trivial examples almost as quickly as the compiler. Many people claim that this means you need to invest more time in unit testing with dynamic languages. I've yet to find this to be true. Whether in a dynamic language or a static language, I end up writing very similar tests, and they rarely, if ever, have anything to do with types. If I've got the typing wrong in my implementation, then the unit tests just won't pass.
Far more interesting is what happens at the integration test level. Even with fantastic unit test coverage, you can't have tested every part of the system. When you wrote that unit, you made some assumptions about what types other functions and classes took as arguments and returned. You even made some assumptions about what the methods are called. If these assumptions were wrong (or, perhaps more likely, change) in a static language, then the compiler will tell you. Quite often in a static language, if you want to change an interface, you can just make the change and fix the compiler errors -- “following the red”.
In a dynamic language, it's not that straightforward. Your unit tests as well as your code will now be out-of-date with respect to its dependencies. The interfaces you've mocked in the unit tests won't necessarily fail because, in a dynamic language, they don't have to be closely tied to the real interfaces you'll be using.
Time for another example. Sticking with Python, let's say we have a web application that has a notion of tags. In our system, we save tags into the database, and can fetch them back out by name. So, we define a TagRepository class.
class TagRepository(object):
def fetch_by_name(self, name):
...
return tag
Then, in a unit test for something that uses the tag repository, we mock the class and the method fetch_by_name. (This example uses my Funk mocking framework. See what a subtle plug that was?)
...
tag_repository = context.mock()
expects(tag_repository).fetch_by_name('python').returns(python_tag)
...
So, our unit test passes, and our application works just fine. But what happens when we change the tag repository? Let's say we want to rename fetch_by_name to get_by_name. Our unit tests for TagRepository are updated accordingly, while the tests that mock the tag repository are now out-of-date -- but our unit tests will still pass.
One solution is to use our mocking framework differently -- for instance, you could have passed TagFetcher to Funk, so it will only allows methods defined on TagFetcher to be mocked:
...
tag_repository = context.mock(TagFetcher)
# The following will raise an exception if TagFetcher.fetch_by_name is not a method
expects(tag_repository).fetch_by_name('python').returns(python_tag)
...
Of course, this isn't always possible, for instance if you're dynamically generating/defining classes or their methods.
The real answer is that we're missing integration tests -- if all our tests pass after renaming TagRepository's methods, but not changing its dependents, then nowhere are we testing the integration between TagRepository and its dependents. We've left a part of our system completely untested. Just like with unit tests, the difference in integration tests between dynamic and static languages is minimal. In both, you want to be testing functionality. If you've got the typing wrong, then this will fall out of the integration tests since the functionality won't work.
So, does that mean the discussed advantages of static typing don't exist? Sort of. While compilation and unit tests are usually quick enough to be run very frequently, integration tests tend to take longer, since they often involve using the database or file IO. With static typing, you might be able to find errors more quickly.
However, if you've got the level of unit and integration testing you should have in any project, whether its written in a static or dynamic language, then I don't think using a static language will mean you have fewer errors or fewer tests. With the same level high level of testing, a project using a dynamic language is just as robust as one written in a static language. Since integration tests are often more difficult to write than unit tests, they are often missing. Yet relying on static typing is not the answer -- static typing says nothing about how you're using values, just their type, and as such make weak assertions about the functionality of your code. Static typing is no substitute for good tests.
Funk 0.2 released
Monday 19 October 2009 15:08
Funk 0.2 has just been released -- you can find it on the Cheese Shop, or you can always get the latest version from Gitorious. You can also take a peek at Funk's documentation.
The most important change is a change of syntax. Before, you might have written:
database = context.mock()
database.expects('save').with_args('python').returns(42)
database.allows('save').with_args('python').returns(42)
database.expects_call().with_args('python').returns(42)
database.allows_call().with_args('python').returns(42)
database.set_attr(connected=False)
Now, rather than calling the methods on the mock itself, you should use the functions in funk:
from funk import expects
from funk import allows
from funk import expects_call
from funk import allows_call
from funk import set_attr
...
database = context.mock()
expects(database).save.with_args('python').returns(42)
allows(database).save.with_args('python').returns(42)
expects_call(database).with_args('python').returns(42)
allows_call(database).with_args('python').returns(42)
set_attr(database, connected=False)
If you want, you can leave out the use of with_args, leading to a style very similar to JMock:
from funk import expects
from funk import allows
from funk import expects_call
from funk import allows_call
...
database = context.mock()
expects(database).save('python').returns(42)
allows(database).save('python').returns(42)
expects_call(database)('python').returns(42)
allows_call(database)('python').returns(42)
To help transition old code over, you can use funk.legacy:
from funk.legacy import with_context
@with_context
def test_view_saves_tags_to_database(context):
database = context.mock()
database.expects('save')
One final change in the interface is that has_attr has been renamed to set_attr. Hopefully, the interface should be more stable from now on.
There's also a new feature in that you can now specify base classes for mocks. Let's say we have a class called TagRepository, with a single method fetch_all(). If we try to mock calls to fetch_all(), everything will work fine. If we try to mock calls to any other methods on TagRepository, an AssertionError will be raised:
@with_context
def test_tag_displayer_writes_all_tag_names_onto_separate_lines(context):
tag_repository = context.mock(TagRepository)
expects(tag_repository).fetch_all().returns([Tag('python'), Tag('debian')]) # Works fine
expects(tag_repository).fetch_all_tags() # Raises an AssertionError
Two words of caution about using this feature. Firstly, this only works if
the method is explicitly defined on the base class. This is often not the case
if the method is dynamically generated, such as by overriding
__getattribute__ on the type.
Secondly, this is no substitute for integration testing. While its true that the unit test above would not have failed, there should have been some integration test in your system that would have failed due to the method name change. The aim of allowing you to specify the base class is so that you can find that failure a little quicker.
If you find any bugs or have any suggestions, please feel free to leave a comment.
Funk – A Python mocking framework
Monday 28 September 2009 17:42
Roll up, roll up! That's right folks, I've written a Python mocking framework.
Another mocking framework?
Yup. As for why, there are a few reasons.
The simplest is to see just how difficult it was to write a usable mocking framework. It turns out not to be all that difficult – I managed to write the core of the framework over a weekend, although plenty of time was spent tweaking and adding behaviour afterwards.
A somewhat better reason is that none of the existing Python mocking frameworks that I could find really did what I wanted. The closest I found was Fudge. My criteria were something like:
- Not using the record/replay pattern.
- Allowing the expected calls and their arguments to be set up beforehand.
- Allowing differentiation between the methods that have to be called, and methods that can be called.
So what's wrong with Fudge?
Fudge meets all of these expectations. So what went wrong?
Firstly, I found Fudge too strict on ordering. Imagine I have a TagRepository that returns me tags from a database. I want to mock this object since I don't want to make a database trip in a unit test. So, in Fudge, I would set up the mock like so:
@with_fakes
def test_gets_python_and_debian_tags():
tag_repository = fudge.Fake()
tag_repository.expects('with_name').with_args('python').returns(python_tag)
tag_repository.next_call().with_args('debian').returns(debian_tag)
# Rest of the test
This would require me to get the Python tag before the Debian tag – yet I really didn't care which method I called first. I'm also not a fan of the syntax – for the first expectation, expects is used, yet for the second expectation, next_call is used.
The second problem I had was that, if you only set up one expectation on a method, you could call it many times. So, with the example above, if you had only set up the expectation for the Python tag, you could get the Python tag any number of times, so long as you asked for it at least once.
I dislike this since, by adding a second expectation, we have now changed the behaviour of the first. This does not lend itself to being able to modify or refactor the test quickly.
Finally, Fudge used a global context for mocks. The ramification of this is that, when using the decorator @with_fakes, each test inherits the mocks set up for the previous test. For instance:
@with_fakes
def test_tag_is_saved_if_name_is_valid():
database = fudge.Fake()
database.expects('save').with_args('python')
tag_repository = TagRepository(database)
tag_repository.save('python')
@with_fakes
def test_tag_is_not_saved_if_name_is_blank():
tag_repository = TagRepository(None)
tag_repository.save('')
The second test above would fail since it does not save the Python tag to the database created in the first test. This seemed somewhat unintuitive to me, so I ended up rolling my own decorator. At the start of each test, it would remove all of the mocks currently set up so that I could start from a blank slate.
The other effect is that it makes it difficult to nest mock contexts – admittedly, something I rarely need to do, but it can be useful to make a quick assertion that requires some amount of mocking.
Okay, what does Funk do differently?
Let's take a look at how we'd write that first test using Funk:
@with_context
def test_gets_python_and_debian_tags(context):
tag_repository = context.mock()
tag_repository.expects('with_name').with_args('python').returns(python_tag)
tag_repository.expects('with_name').with_args('debian').returns(debian_tag)
# Rest of the test
The first difference is that using the @with_context decorator means that we get a context passed in. If you want to build your own context, you can do so simply by calling funk.Context().
Secondly, Funk doesn't care what order you call the two methods in.
Finally, even if you only set one expectation, Funk expects the method to be called once. Setting up further expectations will not affect the existing expectations.
Show me the code!
You can see the Git repository over on Gitorious, or grab version 0.1 from the Cheese Shop. Feel free to try it out, although the API isn't 100% stable yet. The source includes some documentation, but you might also want to take a look at some of the tests to get an idea of what you can do.