Mike's corner of the web.

Nope: a statically-typed subset of Python that compiles to JS and C#

Sunday 22 March 2015 15:55

Nope is a statically-typed subset of Python that uses comments as type annotations. For instance, here's the definition of a Fibonacci function:

#:: int -> int
def fib(n):
    seq = [0, 1]
    for i in range(2, n + 1):
        seq.append(seq[i - 1] + seq[i - 2])
    
    return seq[n]

print(fib(10))

And here's a generic identity function:

#:: T => T -> T
def identity(value):
    return value

Since the types are just comments, any Nope program is directly executable as a Python program without any extra dependencies.

Having written your program with type annotations, you can now compile it to some horrible-looking JavaScript or C# and run it:

$ python3 fib.py
55
$ nope compile fib.py --backend=dotnet --output-dir=/tmp/fib
$ /tmp/fib/fib.exe
55
$ nope compile fib.py --backend=node --output-dir=/tmp/fib
$ node /tmp/fib/fib.js
55

Why?

A little while ago, I wrote mammoth.js, a library for converting Word documents to HTML. Since some people found it useful, I ported it to Python. Both implementations are extremely similar, and don't heavily exploit the dynamic features of either language. Therefore, a third port, even to a statically-typed language such as C# or Java, is likely to end up looking extremely similar.

This led to the question: if I annotated the Python implementation with appropriate types, could I use it to generate vaguely sensible C# or Java code? Nope is an experiment to find out the answer.

Should I use it?

This project is primarily an experiment and a bit of fun. Feel free to have a play around, but I'd strongly suggest not using it for anything remotely practical. Not that you'd be able to anyway: many essential features are still missing, while existing features are likely to change. The type-checking is sufficiently advanced to allow partial type-checking of the Python port of Mammoth.

I discuss a few alternatives a little later on, and explain how Nope differs. In particular, there are a number of existing type checkers for Python, the most prominent being mypy.

Examples

Simple functions

Functions must have a preceding comment as a type annotation.

#:: int -> int
def triple(value):
    return value * 3
Functions with optional and named arguments

Optional arguments are indicated with a leading question mark, and must have a default value of None. For instance:

#:: name: str, ?salutation: str -> str
def greeting(name, salutation=None):
    if salutation is None:
        salutation = "Hello"
    
    print(greeting + " " + name)

print(greeting("Alice"))
print(greeting("Bob", salutation="Hi"))

Note that the type of salutation changes as it is reassigned in a branch of the if-else statement. It is initially of type str | none, which is the union of the formal argument type and none (since it's optional). After the if-else statement, it is of the narrower type str, which allows it to be safely used in the string concatenation.

Variables

Most of the time, Nope can infer a suitable type for variables. However, there are occasions where an explicit type is required, such as when inferring the type of empty lists. In these cases, an explicit type can be specified in a similar manner to functions:

#:: list[int]
x = []
Classes

When annotating the self argument of a method, you can use the explicit name of the class:

class Greeter(object):
    #:: Greeter, str -> str
    def hello(self, name):
        return "Hello " + name

For convenience, Nope also introduces Self into the scope of the class, which can be used instead of referring to the containing class directly:

class Greeter(object):
    #:: Self, str -> str
    def hello(self, name):
        return "Hello " + name

As with local variables, instance variables assigned in __init__ can have type annotations, but will often be fine using the inferred type:

class Greeter(object):
    #:: Self, str -> none
    def __init__(self, salutation):
        self._salutation = salutation

    #:: Self, str -> str
    def hello(self, name):
        return self._salutation + " " + name

Generic classes are also supported, although the exact syntax might change:

#:generic T
class Result(object):
    #:: Self, T, list[str] -> none
    def __init__(self, value, messages):
        self.value = value
        self.messages = messages
Code transformations

To preserve some of the advantages of working in a dynamic langauge, Nope supports code transformations: given the AST of a module, a transformation returns an AST that should be used for type-checking and code generation. So long as the transformation and the original runtime behaviour are consistent, this allows you to use code such as collections.namedtuple:

import collections

User = collections.namedtuple("User", [
    #:: str
    "username",
    #:: str
    "password",
])

The current implementation is a bit of a hack, but the ultimate goal is to let a user specify transformations to apply to their code. Ideally, this would allow Python libraries such as SQLAlchemy to be supported in a type-safe manner.

And the rest

Nope also supports while and for loops, try statements, raise statements, destructuring assignment and plenty more. However, I've left them out of this post since they look the same as they do in Python. The only difference is that Nope will detect when inappropriate types are used, such as when trying to raise a value that isn't an exception.

I've started using Nope on a branch of Mammoth. Only some modules are currently being type-checked by Mammoth, such as html_generation.

If you're feeling brave, Nope has a set of execution tests that check and compile sample programs. It's the not the greatest codebase in the world with many unhanded or improperly handled cases, but feel free to read the tests if you want to dive in and see exactly what Nope supports. At the moment, Nope compiles to Python (which means just copying the files verbatim), JavaScript and C# with varying degrees of feature-completeness. The C# implementation in particular has huge scope for optimisation (since it currently relies heavily on (ab)using dynamic), but should already be fast enough for many uses.

Type system

Nope ends up with a few different kinds of type in its type system. It would be nice to be able combine some of these, but for the time being I've preferred to avoid introducing extra complexity into existing types. At the moment, Nope supports:

  • Ordinary classes, made by using class in the usual way. Since inheritance is not yet supported, a type T is a subclass of itself and no other ordinary classes.
  • Function types, such as:
    • int, str -> str (Requires two positional arguments of type int and str respectively, and returns a str.)
    • int, x: int, ?y: str -> int (Has two required arguments, the second of which can be passed by the name x. Has an optional third argument called y.)
  • Structural types. For instance, we might define a structural type HasLength with the attribute __len__ of type -> int (takes no arguments, returns an integer). Any type with an appropriately typed attribute __len__ would be a subtype of HasLength. Currently not definable by users, but should be.
  • Generic types. Nope currently supports generic functions, generic classes and generic structural types. For instance, the generic structural type Iterable has a single formal type parameter, T, and an attribute __iter__ of type -> Iterator[T], where Iterator is also a generic structural type.
  • Type unions. For instance, a variable of type int | str could hold an int or a str at runtime.

What about IronPython or Jython?

Both IronPython and Jython aim to be Python implementations, rather than implementing a restricted subset. This allows them to run plenty of Python code with little to no modification.

Nope differs in that it aims to allow the generation of code without any additional runtime dependencies. Since the code is statically typed, it should also allow better integration with the platform, such as auto-complete or Intellisense in IDEs such as Visual Studio, Eclipse and IntelliJ.

What about mypy?

mypy is an excellent project that can type check Python code. If you want a project that you can practically use, then mypy is by far more appropriate. There are other type checkers that also use Python annotations to specify types, such as obiwan.

At this point, Nope differs in two main regards. Firstly, Nope's scope is slightly different in that I'm aiming to compile to other languages.

The second main difference is that Nope aims to have zero runtime dependencies. Since mypy uses Python annotations to add type information, programs written using mypy have mypy as a runtime dependency. This also allows some meta-programming with somewhat consistent type annotations, as shown in the collections.namedtuple from earlier:

import collections

User = collections.namedtuple("User", [
    #:: str
    "username",
    #:: str
    "password",
])

What next?

I'm only working on Nope in my spare time, so progress is a little slow. The aim is to get a C# version of Mammoth working by using Nope in the next few months. Although there might not be feature parity to begin with, I'd be delighted if I got far enough to get the core program working.

If, in a surprising turn of events, this turns out to work, I'd be interested to add effects to the type system. Koka is the most notable example I know of such a system, although I'll happily admit to being some unknowledgeable in this area. Thoughts welcome!

Topics: Software design, Language design

Code smears: code smells that spread across your codebase

Monday 5 January 2015 21:34

The term "code smells" gives us a handy shorthand for signs that our code might not be especially clean. In my experience, the worst sort of code smell are "code smears": those smells that tend to spread their badness across your codebase, which also makes them more difficult to fix as time goes on. For instance, suppose a function has too many arguments, making it difficult to see the purpose of each argument. That code smell isn't confined to the function definition: it's going to appear every time that the function is called.

Code smells tell us that there's probably room for improvement. Ideally, we'd avoid all code smells, but there often comes a point where our time is better spent elsewhere. Yet not all code smells are created equal. Some code smells can be fixed later with little or no impact on the rest of the codebase: for instance, if I take an extremely long function and break it up into smaller functions that compose together nicely, nobody who calls the function needs to be aware that anything has changed.

On the other hand, some code smells tend to make their presence felt across a codebase: for instance, if a function takes four boolean arguments, this leads to somewhat mysterious function calls such as update(true, false, true, true). Fixing this requires changing everywhere that calls that function. This task can range from tedious, if the function is called in many places, to impossible, if third parties are calling into our code. As time goes on, it's likely that there will be more callers of the our function, and so more work to clean it up. I'd suggest that this second group of code smells is more severe than the first, and therefore deserves the more severe name of code smears.

As a further incentive to avoid code smears moreso than code smells, I've found that code smears are often quicker and easier to fix so long as you do so as soon as they appear. Splitting up a long method can be time-consuming and difficult. Giving a function a better name or grouping together related arguments into a single object can often be accomplished in a couple of minutes provided that the function is only used in one or two places under your control.

Sometimes, it's a good trade-off not to fix a code smell. It's rarely a good trade-off to leave behind a code smear.

Topics: Software design

The Dangerous Allure of Code Reuse

Tuesday 20 May 2014 20:42

Back when I was learning to code, one of the ideas that kept cropping up was the idea of code reuse. If one of the main benefits of functions is the ability to reuse a bit of code, doesn't it follow that we should have future code reuse in mind when writing functions? My experience says: nope. It's this thinking that leads me to create functions with six Boolean arguments, or to create functions with complex, deeply nested switching logic. By following principles such as "Don't repeat yourself" (DRY) and "You ain't gonna need it" (YAGNI), and creating short functions that operate at a single level of abstraction, I find that useful functions pop out without consciously thinking about code reuse.

If I try to write a function with code reuse in mind, it becomes tempting to write features that might be useful. In other words, it encourages the opposite of YAGNI (you ain't gonna need it). Far better to have a tightly focused function that does exactly what I need, no more nor less. Sometimes that function will already exist exactly as we need it. Great! Use it.

But what if there's an existing function that's similar, but would require some changes for our required functionality? Should I create a new function or should I modify the existing function? It's tempting to tweak the existing function in the name of code reuse, but I'd suggest that we should err on the side of creating new functions.

One of my guiding principles is also one of the main benefits of functions: abstraction. Imagine that there's already a function that does exactly what you need. What would it be called? What concept does it represent? Now take a look at the existing function. Are they the same concept? If the concepts are related but distinct, that suggests they belong in different functions. If it happens that they have similar implementations, then you'll find the same consideration happening recursively: is this sub-concept in my new function actually the same sub-concept in the existing function? And so on. Although you might have created a new function, it might not actually lead to much new code if it shares many of the same underlying concepts.

I try to write short functions that operate at a single level of abstraction. The result is that I write plenty of short functions that only get used in one place, at least initially. Such functions might end up getting used in other places, but they weren't really designed with reuse in mind: instead, the foremost consideration was whether that function represented a coherent abstraction of a concept.

This isn't to say that I completely ignore code reuse when writing functions. Sometimes, when weighing up different design choices that are similarly elegant for my current use-case, I'll pick a design based on which one is likely to be best suited to usages that I anticipate in the future. However, I won't add anything specifically for code reuse: I've still chosen a design which does only what I need, but is also amenable to anticipated future change. If that future change never happens, then the function still makes sense and doesn't have extra unused functionality.

To summarise: I try to avoid crowbarring in functionality to existing functions just because it's similar. Separate concepts should go in separate functions, even if they share underlying concepts. Better to have two coherent, small, simple concepts than a larger but ill-defined concept. Code reuse is a good thing, but it's not something that I actively design for.

Footnote: for brevity, I've only talked about functions in this post, but it applies equally to classes and other such constructs for the same reasons.

Topics: Software design

Beware Common Sense

Monday 19 May 2014 13:42

My hodge-podge page on test-driven development originally had a part that read:

Even when following techniques such as TDD, common sense must prevail!

This was in reference to making new test cases pass by just hard-coding new values, something that I refuted by claiming that the implementation was "obviously" not the one we wanted.

After Graham Helliwell reminded me of Uncle Bob's Transformation Priority Premise, I changed the page to something a little more helpful.

In retrospect, appealing to common sense was a copout. A few years ago, I decided to stop using phrases such as "obviously" and "of course" in my writing. The reason? If it is actually obvious, then it doesn't need stating. If it does need stating, then it's not as obvious as I think, and I run the risk of making the reader feel stupid for no good reason.

An appeal to common sense is much the same. When I say common sense, which I really mean is: from my own experience and knowledge, the "right thing" to do is second nature, and I don't have to think too hard about it. This is the "curse of knowledge": it might be common sense to me, but one of the points of writing is to explain ideas and concepts to others. An appeal to common sense is giving up that point while running the risk of insulting the reader. Being forced to consciously go through a process is a useful exercise in and of itself, making explanations of common sense useful to both listener and explainer.

Topics:

Converting docx to clean HTML: handling the XML structure mismatch

Tuesday 17 December 2013 08:11

One of my recent side projects is Mammoth, which converts docx files produced by Microsoft Word into HTML. It aims to produce clean HTML by using semantic information in the original document, such as the styles applied to each paragraph, rather than trying to exactly copy the font, size, colour, and so on. I wrote Mammoth so that editors wouldn't have to spend hours manually converting Word documents into HTML. Although we're converting XML to XML, there's quite a mismatch in structure. This blog post describes how Mammoth handles the mismatch. If you're interested in trying it out, you can find a Python version (including a CLI) and a JavaScript version.

The docx format stores each paragraph as a distinct w:p element. Each paragraph optionally has a style. For instance, the following docx XML represents a heading followed by an ordinary paragraph [1].

<w:p style="Heading1>A Study in Scarlet</w:p>
<w:p>In the year 1878 I took my degree</w:p>

We'd like to convert this to an h1 element and a p element:

<h1>A Study in Scarlet</h1>
<p>In the year 1878 I took my degree</p>

This seems fairly straightforward: we take each paragraph from the docx XML, and convert it to an HTML element depending on the style. We can use a small DSL to let the user control how to map docx styles to HTML elements without having to write any code. In this case, we might write:

p.Heading1 => h1:fresh
p => p:fresh

To the left of the arrow, we have a paragraph matcher. p.Heading1 from the first rule matches any paragraph with the style Heading1, while p from the second rule matches any paragraph. To the right of the arrow, we have an HTML path. To process a docx paragraph:

  • Find the first rule where its paragraph matcher matches the current docx paragraph
  • Generate HTML to satisfy the HTML path. h1 is satisfied if there's a top-level h1 i.e. an h1 with no parents. h1:fresh means generate a fresh (i.e. newly-opened) top-level h1 element. We'll see a little later why this notion of freshness is useful.

Things become a bit more tricky when we'd expect to generate some nested HTML, such as lists. For instance, consider the following list:

  • Apple
  • Banana

One way of representing this in docx is:

<w:p style="Bullet1">Apple</w:p>
<w:p style="Bullet1">Banana</w:p>

Note that there's no nesting of elements, even though the two docx paragraphs are part of the same structure (in this case, a list). The only way to tell that these bullets are in the same list is by inspecting the style of sibling elements. Compare this to the HTML we expect to generate:

<ul>
  <li>Apple</li>
  <li>Banana</li>
</ul>

To generate this HTML, you can write the following rule:

p.Bullet1 => ul > li:fresh

The HTML path uses > to indicate children. In this case, the HTML path is satisfied when there's a top-level ul with a fresh li as a child. Let's see how this example works by processing each docx paragraph.

The first paragraph matches p.Bullet1, so we require a top-level ul with a fresh li as a child. Since we have no open elements, we open both elements followed by the text of the paragraph:

<ul>
  <li>Apple

The second paragraph also requires a top-level ul with a fresh li as a child. We close and open the li since it needs to be fresh, but leave the ul alone:

<ul>
  <li>Apple</li>
  <li>Banana

Finally, we close all elements at the end of the document:

<ul>
  <li>Apple</li>
  <li>Banana</li>
</ul>

The key is that HTML elements aren't closed after processing a docx paragraph. Instead, HTML elements are kept open in case following docx paragraphs are actually part of the same structure. An element will eventually be closed either by processing a docx paragraph that isn't part of the same structure, or by reaching the end of the document.

A more complicated case is that of nested lists. For instance, given the list:

  • Fruit
    • Apple
    • Banana
  • Vegetable
    • Cucumber
    • Lettuce

This would be represented in docx by:

<w:p style="Bullet1">Fruit</w:p>
<w:p style="Bullet2">Apple</w:p>
<w:p style="Bullet2">Banana</w:p>
<w:p style="Bullet1">Vegetable</w:p>
<w:p style="Bullet2">Cucumber</w:p>
<w:p style="Bullet2">Lettuce</w:p>

And we'd like to generate this HTML:

<ul>
  <li>
    Fruit
    <ul>
      <li>Apple</li>
      <li>Banana</li>
    </ul>
  </li>
  <li>
    Vegetable
    <ul>
      <li>Cucumber</li>
      <li>Lettuce</li>
    </ul>
  </li>
</ul>

In this case, we need two rules: one each for Bullet1 and Bullet2:

p.Bullet1 => ul > li:fresh
p.Bullet2 => ul > li > ul > li:fresh

To see how this works, let's follow step by step. We start by processing the first docx paragraph. This has the style Bullet1, which requires a ul and li element to be open. This generates the following HTML:

<ul>
  <li>
    Fruit

The second paragraph has the style Bullet2, which means we need to satisfy the HTML path ul > li > ul > li:fresh. Since the ul and li from processing the first docx paragraph have been left open, we only need to generate the second set of ul and li elements, giving the HTML:

<ul>
  <li>
    Fruit
    <ul>
      <li>Apple

The third paragraph also has the style Bullet2. The first three elements of the style rule (ul > li > ul) are already satisfied, but the final li needs to be fresh. Therefore, we close the currently open li, and then open a new li:

<ul>
  <li>
    Fruit
    <ul>
      <li>Apple</li>
      <li>Banana

The fourth paragraph has the style Bullet1. The first element of the style rule (ul) is satisfied, but the li needs to be fresh. Therefore, we close the outer li, along with its children, before opening a fresh li:

<ul>
  <li>
    Fruit
    <ul>
      <li>Apple</li>
      <li>Banana</li>
    </ul>
  </li>
  <li>
    Vegetable

The processing of the final two paragraphs proceeds in the same way as before, giving us the HTML:

<ul>
  <li>
    Fruit
    <ul>
      <li>Apple</li>
      <li>Banana</li>
    </ul>
  </li>
  <li>
    Vegetable
    <ul>
      <li>Cucumber</li>
      <li>Lettuce

Since we've reached the end of the document, all that remains is to close all open elements:

<ul>
  <li>
    Fruit
    <ul>
      <li>Apple</li>
      <li>Banana</li>
    </ul>
  </li>
  <li>
    Vegetable
    <ul>
      <li>Cucumber</li>
      <li>Lettuce</li>
    </ul>
  </li>
</ul>

I've left plenty of details out, such as handling of hyperlinks and images, but this gives an overview of how Mammoth deals with the greatest mismatch between the structure of docx XML and HTML.

[1] If you go and look at an actual docx file, you'll discover that the XML is more complicated than what I've presented. I've only included the bits that matter for an overview.

Topics: Algorithms, Programs

Fun with Prolog: write an algorithm, then run it backwards

Sunday 10 November 2013 21:21

Compared to most other languages, Prolog encourages you to write code in a highly declarative style. One of the results is that you can write an algorithm, and then run the same algorithm "backwards" without any additional code.

For instance, suppose you want to find out whether a list is a palindrome or not. We write a predicate like so:

palindrome(L) :- reverse(L, L).

We can read this as: palindrome(L) is true if reverse(L, L) is true. In turn, reverse(L1, L2) is true when L1 is the reverse of L2. We try out the palindrome predicate in the interpreter:

?- palindrome([]).
true.

?- palindrome([1]).
true.

?- palindrome([1, 1]).
true.

?- palindrome([1, 2]).
false.

?- palindrome([1, 2, 1]).
true.

So far, not that different from any other programming language. However, if we set some of the elements of the list to be variables, Prolog tries to fill in the blanks -- that is, it tries to find values for those variables so that the predicate is true. For instance:

?- palindrome([1, A]).
A = 1.

In the above, Prolog tells us that the list [1, A] is a palindrome if A has the value 1. We can do something a bit more fancy if we use a variable for the tail of the list, rather than just one element. [1 | A] means a list with 1 as the first element, with any remaining elements represented by A.

?- palindrome([1 | A]).
A = [1]

Prolog tells us that [1 | A] is a palindrome if A has the value [1]. However, if we hit the semicolon in the interpreter, Prolog gives us another value for A that satisfies the predicate.

?- palindrome([1, 2 | A]).
A = [1] ;
A = [2, 1]

Now Prolog is telling us that [2, 1] is another value for A that satisfies the predicate. If we hit semicolon again, we get another result:

?- palindrome([1, 2 | A]).
A = [1] ;
A = [2, 1] ;
A = [_G313, 2, 1]

This time, Prolog says A = [_G313, 2, 1] satifies the predicate. The value _G313 means that any value would be valid in that position. Another hit of the semicolon, and another possibility:

?- palindrome([1, 2 | A]).
A = [1] ;
A = [2, 1] ;
A = [_G313, 2, 1] ;
A = [_G313, _G313, 2, 1]

We still have _G313, but this time it appears twice. The first and second element of A can be anything so long as they're the same value. We can keep hitting semicolon, and Prolog will keep giving us possibilities:

?- palindrome([1, 2 | A]).
A = [1] ;
A = [2, 1] ;
A = [_G313, 2, 1] ;
A = [_G313, _G313, 2, 1] ;
A = [_G313, _G319, _G313, 2, 1] ;
A = [_G313, _G319, _G319, _G313, 2, 1] ;
A = [_G20, _G26, _G32, _G26, _G20, 2, 1] ;
A = [_G20, _G26, _G32, _G32, _G26, _G20, 2, 1]

In each of these possibilities, Prolog correctly determines which of the elements in the list must be the same. Now for one last example: what if we don't put any constraints on the list?

?- palindrome(A).
A = [] ;
A = [_G295] ;
A = [_G295, _G295] ;
A = [_G295, _G301, _G295] ;
A = [_G295, _G301, _G301, _G295] ;
A = [_G295, _G301, _G307, _G301, _G295] ;
A = [_G295, _G301, _G307, _G307, _G301, _G295] ;
A = [_G295, _G301, _G307, _G313, _G307, _G301, _G295]

Once again, Prolog generates possibilities for palindromes, telling us which elements need to be the same, but otherwise not putting any restrictions on values.

In summary, we wrote some code to tell us whether lists were palindromes or not, but that same code can be used to generate palindromes. As another example, we might want to implement run-length encoding of lists:

?- encode([a, a, a, b, b, a, c, c], X).
X = [[3, a], [2, b], [1, a], [2, c]] .

Once we've written encode, to work in one direction (turning ordinary lists into run-length-encoded lists), we can use the same predicate to work in the other direction (turning run-length-encoded lists into ordinary lists):

?- encode(X, [[3, a], [2, b], [1, a], [2, c]]).
X = [a, a, a, b, b, a, c, c] .

For the interested, the implementation can be found as a GitHub gist. One caveat is that the implementation of encode has to be written carefully so that it works in both directions. Although this might be harder (and much less efficient) than writing two separate predicates, one for encoding and one for decoding, using a single predicate gives a high degree of confidence that the decode operation is correctly implemented as the inverse of the encode operation. Writing a version of encode that actually works in both directions is an interesting challenge, and also the topic of another blog post.

(Thanks to Ninety-Nine Prolog Problems for inspiration for examples.)

Topics: Prolog, Language design

Relocatable Python virtualenvs using Whack

Saturday 7 September 2013 17:25

One of the uses for Whack is creating relocatable (aka path-independent) Python virtualenvs. Normally, a virtualenv is tied to a specific absolute path, meaning that moving the virtualenv causes errors:

$ virtualenv venv
$ venv/bin/pip install glances
(Snipping pip output)
$ mv venv venv2
$ venv2/bin/glances -v
bash: venv2/bin/glances: /tmp/venv/bin/python: bad interpreter: No such file or directory

Copying the entire virtualenv has similar but subtler problems. Rather than getting a straightforward error, the scripts in the new virtualenv will use the Python interpreter and libraries in the original virtualenv.

Whack allows virtualenvs to be created, and then moved to any other location:

$ whack install \
    git+https://github.com/mwilliamson/whack-package-python-virtualenv-env.git \
    venv
$ venv/bin/pip install glances
(Snipping pip output)
$ whack deploy venv --in-place
$ # Now we can copy the virtualenv to any other path,
$ # and it will continue to work
$ mv venv venv2
$ venv2/bin/glances -v
Glances version 1.7.1 with PsUtil 1.0.1

The whack deploy command is necessary to add any newly-installed scripts in the virtualenv to the bin directory.

One question is: why not use the --relocatable argument that virtualenv itself provides? This works in many cases, and doesn't require installation of Whack, but it also comes with a warning from virtualenv's documentation:

The --relocatable option currently has a number of issues, and is not guaranteed to work in all circumstances. It is possible that the option will be deprecated in a future version of virtualenv.

Topics: Python, Whack, Programs

An experiment in reusable web widgets

Wednesday 31 July 2013 10:28

For the same reasons that breaking down programs into short, composable functions is a good idea, it seems like breaking down web code into short, composable web widgets would be a good idea. (By web widget, I mean the HTML, CSS and JavaScript that go together to implement a particular piece of functionality.) Having shorter snippets makes code easier to understand and change, with the potential for reuse.

Yet it feels like there's no good way of sharing the HTML, CSS and JavaScript that go together to implement a particular piece of functionality. For instance, the usual way of creating a web widget using JQuery is to create a JQuery plugin, but there's no natural way of using such a JQuery plugin from Knockout. Over the past few days, I've tried an experiment in creating web widgets that can be written and consumed independently of technology.

First of all, I've defined a widget as being a function that accepts a single options argument. That options argument must contain an element property, which is the element that will be transformed into the widget (for instance, we might turn an <input> element into a date picker). The options argument can also contain any number of other options for that widget. The interface is kept simple so it's easy to implement, while still being sufficiently general. It's not exactly something to write home about, but the value is in choosing a fixed interface.

Now that we've defined the notion of web widget, we'll want to start consuming and creating widgets. For instance, we can create a widget that will turn its message option to uppercase wrapped in <strong> tags:

function shoutingWidget(options) {
    var element = options.element;
    var contents = options.message.toUpperCase();
    // Assuming that we've defined htmlEscape elsewhere
    element.innerHTML = "<strong>" + htmlEscape(contents) + "</strong>";
}

We can use it like so:

shoutingWidget({
    element: document.getElementById("example"),
    message: "Hello!"
});

which will transform the following HTML:

<span id="example"></span>

into:

<span id="example"><strong>HELLO!</strong></span>

However, most of the time, I'm not writing web code using raw JavaScript. So, for any given web framework/library, we can start to answer two questions:

  • What's the easiest way we can consume a widget?
  • What's the easiest way we can create a widget?

In particular, when a widget is used, we shouldn't care about the underlying implementation. Whether it was created using jQuery or Knockout or something else, we should be able to use it with the same interface.

Let's see how this works with Knockout. To create a web widget, I call the function knockoutWidgets.widget() with an object with an init function, and I get back a widget (which is just a function). The init function is called with the options object each time the widget is rendered. The init function should return the view model and template for that widget. For instance, to implement the previous example using Knockout:

var shoutingWidget = knockoutWidgets.widget({
    init: function(options) {
        var contents = options.message.toUpperCase();
        return {
            viewModel: {contents: contents},
            template: '<strong data-bind="text: contents"></strong>'
        }
    }
});

To consume widgets from Knockout, we have to explicitly specify dependencies. By avoiding putting all widgets into a single namespace, we avoid collisions without using long, unwieldy names. For instance, to create an emphatic greeter widget that transforms:

<span id="example"></span>

into:

<span id="example">Hello <strong>BOB</strong>!</span>

we can write:

var emphaticGreeterWidget = knockoutWidgets.widget({
    init: function(options) {
        return {
            viewModel: {name: options.name},
            template: 'Hello <span data-bind="widget: \'shout\', widgetOptions: {message: name}"></span>!'
        }
    },
    dependencies: {
        shout: shoutingWidget
    }
});

emphaticGreeterWidget({
    element: document.getElementById("example"),
    name: "Bob"
});

Importantly, although we've created the widget using Knockout, any code that supports our general notion of a web widget should be able to use it. Similarly, emphaticGreeterWidget can use shoutingWidget regardless of whether it was written using Knockout, raw JavaScript, or something else altogether.

Although I've successfully used this style with Knockout for some small bits of work, there are still two rather major unsolved problems.

The first problem: how should data binding be handled? All the above examples have data flowing in one direction: into the widget. What if we want data to flow in both directions, such as a date picker widget?

The second problem: should content within widgets be allowed? Our shouting widget had the message passed in via the options argument, but it could have been specified in the body of the element that the widget was applied to. Using raw JavaScript, that means a definition that looks something like:

function shoutingWidget(options) {
    var element = options.element;
    var message = "message" in options ? options.message : element.textContent;
    var contents = message.toUpperCase();
    // Assuming that we've defined htmlEscape elsewhere
    element.innerHTML = "<strong>" + htmlEscape(contents) + "</strong>";
}

If we allow content within widgets, then we have to work out how the widget interacts with the content and the web library in use. For instance, if we're using Knockout, do we apply the bindings before or after the widget is executed? How should the widget detect changes when its children change as a result of those Knockout bindings?

Also notably absent from the examples was any mention of CSS, despite my earlier mention. The reason: it hasn't been needed in my small experiments so far, so I haven't thought that much about it! It's something that will need dealing with at some point though.

Thoughts on the overall idea or those specific problems are welcome! You can take a look at the code on GitHub.

Topics: HTML, JavaScript

The Cthulhu Effect, or what happens to old programmers

Sunday 28 July 2013 13:27

There's a perception in at least some parts of the world of software development that coding is a game for the young. I'm not sure whether it's actually true or not, but after a conversation with a friend, I've decided to chalk up the phenomenon to "the Cthulhu Effect". Apparently, the more you stare at Cthulhu, the more insane you go. Similarly, I expect a decade or two of staring at awful code is enough to drive any programmer to madness, leaving you with the choice of either running from the code or embracing madness. (That also goes a long towards explaining the demeanor of many experienced programmers.)

Topics: Nonsense

Adding git (or hg, or svn) dependencies in setup.py (Python)

Wednesday 29 May 2013 21:02

You can specify dependencies for your Python project in setup.py by referencing packages on the Python Package Index (PyPI). But what if you want to depend on your own package that you don't want to make public? Using dependency_links, you can reference a package's source repository directly.

For instance, mayo is a public package on PyPI, so I can reference it directly:

setup(
    install_requires=[
        "mayo>=0.2.1,<0.3"
    ],
    # Skipping other arguments to setup for brevity
)

But suppose that mayo is a private package that I don't want to share. Using the dependency_links argument, I can reference the package by its source repository. The only way I could get this working with git was to use an explict SSH git URL, which requires a small transformation from the SSH URLs that GitHub or BitBucket provide. For instance, if GitHub lists the SSH URL as:

git@github.com:mwilliamson/mayo.git

then we need to explictly set the URL as being SSH, which means adding ssh:// at the front, and replacing the colon after github.com with a forward slash. Finally, we need to indicate the URL is a git URL by adding git+ to the front. This gives a URL like:

git+ssh://git@github.com/mwilliamson/mayo.git

To use a specific commit, add an at symbol followed by a commit identifier. For instance, if we wanted to use version 0.2.1, which has the tag 0.2.1 in git:

git+ssh://git@github.com/mwilliamson/mayo.git@0.2.1

Then, we can use the URL in setup.py like so:

setup(
    install_requires=[
        "mayo==0.2.1"
    ],
    dependency_links=[
        "git+ssh://git@github.com/mwilliamson/mayo.git@0.2.1#egg=mayo-0.2.1"
    ]
    # Skipping other arguments to setup for brevity
)

Note that we depend on a specific version of the package, and that we use the URL fragment (the bit after #) to indicate both the package name and version.

Topics: Python