For testability reasons, we’ve begun using a lot of dependency injection in our Python projects. Imagine that I have a function that does some work and then ultimately calls an API over-the-wire:


import simplejson
from google.appengine.api import urlfetch # App Engine example
def my_method(user_id):
  url = 'http://www.example.com/lookup-user/%d.json' % user_id
  result = urlfetch.fetch(url)
  if result.status_code == 200:
    return simplejson.loads(result.content)
  return {}


This is bad for testability because when my test code executes my_method(), it will either go out over the wire (super bad!) unless I monkey-patch urlfetch to be some other class (really inconvenient).

So, instead, we can accept an alternate urlfetch mechanism optionally in the method:


from google.appengine.api import urlfetch as urlfetch_builtin
def my_method(user_id, urlfetch=urlfetch_builtin):
  url = 'http://www.example.com/lookup-user/%d.json' % user_id
  result = urlfetch.fetch(url)
  if result.status_code == 200:
    return simplejson.loads(result.content)
  return {}


So now the test client can pass in an alternate implementation of urlfetch that can return instrumented results and exceptions. This proves to be extremely handy.

One problem with this particular implementation is that it forces the client to do some odd things. Imagine that I have a piece of client code that itself is dependency injected:


def client_function(urlfetch=None):
  # here, I must check for an injection so that I know to provide it to my_method
  if urlfetch:
    return my_method(123, urlfetch=urlfetch)
  else:
    return my_method(123)


This is unfortunate. A better way to implement dependency injection and prevent this client if statement is to adjust my_method:


from google.appengine.api import urlfetch as urlfetch_builtin
def my_method(user_id, urlfetch=None):
  urlfetch = urlfetch or urlfetch_builtin
  # ...


Since urlfetch is required by the method, we can take None to mean “use the default implementation”. Now the client code becomes simply:


def client_function(urlfetch=None):
  return my_method(123, urlfetch=urlfetch)


Make sure you setup your dependency injection params in this way and save your clients some headaches!

One of the great things about templating systems is the ability to encapsulate repeatable chunks of code. One of the tough things about templating systems is management of a global “state” to ensure the the overall document that is emitted is as efficient as possible – especially in the HTML world were bandwidth is a performance consideration.

An example of this is where the templating chunks require a particular external resource, like JavaScript. You don’t want each invocation of a repeated sub-template to cause a tag to be emitted; you only need it once for the entire document.

Asp.Net has some handling for JavaScript in their UserControl mechanism (RegisterClientScriptBlock()) that uses a server-side registration mechanism to emit an appropriate document.

Dave Mosher and Brett Zabos, here at VendAsta, have used a pure JavaScript approach to get the JavaScript resources loaded efficiently, built atop the YUI loader. Make sure you check out their escapades.

Google App Engine doesn’t have a unique constraint in the classical sense of relational databases. This is a favourite construct of application developers and it’s unfortunate that it’s not present. At the same time, a basic understanding of the underlying datastore points to why it’s tough, or at least inefficient.

There are places where you’re willing to sacrifice some performance in order to guarantee a unique value. Datastore does guarantee uniqueness on its keys, so we can use a secondary helper model to guarantee uniqueness.


class Unique(db.Model):
  @classmethod
  def check(cls, scope, value):
    def tx(scope, value):
      key_name = "U%s:%s" % (scope, value,)
      ue = Unique.get_by_key_name(key_name)
      if ue:
        raise UniqueConstraintViolation(scope, value)
      ue = Unique(key_name=key_name)
      ue.put()
    db.run_in_transaction(tx, scope, value)
class UniqueConstraintViolation(Exception):
  def __init__(self, scope, value):
    super(UniqueConstraintViolation, self).__init__("Value '%s' is not unique within scope '%s'." % (value, scope, ))


This class simply leverages the key uniqueness aspect of datastore to ensure that the value doesn’t exist. A call to check() where a matching scope and value already exists will raise a UniqueConstraintViolation.

To use this, you need to build out a common create method on your models. Below, I’ve used a classmethod to achieve this:


class Account(db.Model):
  name = db.StringProperty()
  email = db.StringProperty() # unique=True - wouldn't that be great?
  @classmethod
  def create(cls, name, email):
    Unique.check("email", email)
    a = Account(name=name, email=email)
    a.put()
    return a


I need to make a call to Unique.check() first to ensure uniqueness (which causes at least a lookup, but likely a lookup and a put – both are a performance hit relative to doing nothing at all), and then I create my own account. Unique.check() will throw if the value is not unique, preventing the Account from being created.

Note that this technique lugs around an additional dictionary in datastore (costing some $$, though realistically not much). Also note that if you jam too many scopes into this class, you’ll get degrading performance (though it’s still just a key lookup which is very efficient).

Recently, I discovered something surprising about Google App Engine’s datastore and ReferenceProperty. Imagine I have a class like the following:


  from google.appengine.ext import db
  class Home(db.Model):
    address = db.StringProperty()
    room = db.ReferenceProperty(Room)


where Room is also a db.Model.

Datastore uses a proxying technique such that db.Model objects are created lightly (i.e., with only their key) and any hit to an attribute causes the entire object to be inflated by looking it up in the datastore.

I thought this was achieved with a lazy load technique, meaning that Room in the above example contained the logic to load itself (or more accurately Home would populate room with a proxy to Room that would know how to inflate). To do this, the proxy class would hold only its key and use that key to look itself up. From this, it follows that I could access that key without inflating the object.

It doesn’t work this way!

Instead, the referencing class (Home in the above example) holds the key (in a protected attribute, which you shouldn’t touch) and is responsible for inflating reference types.

I’ll say that again: the referenced object does not inflate itself lazily, but the referencing object does the inflation.

So, I previously thought code like this would not cause a datastore lookup:


  # assume home is an instance of Home
  room_key = home.room.key()


but I was very wrong. Simply hitting home.room causes home to lookup the entire room.

But what if you only want to get at the key (which the home has, but is protecting)?

This post suggests that the safe way to get this key is to access it via the class attribute, not the instance attribute. Here is the resulting code for my example:


  room_key = Home.room.get_value_for_datastore(home)


Note carefully the capitalization (indicating the classes versus objects).

This gives me the room key without causing a room lookup. Over and out.

We have a templatetag, called scurl, that needs to look at the HttpRequest object. Django’s templating system provides a straight-forward, yet wordy, mechanism to pass the request object in to the template:


  def my_view(request):
    return render_to_response("myview.html", context_instance=RequestContext(request))

The render method of our scurl templatetag gets access to this context and thus access to the HttpRequest.

So far, so good.

In our project, we also use inclusion_tags to include common chunks of HTML into pages. This sort of tag looks something like this:


  @register.inclusion_tag("html-to-include.html")
  def my_include_tag(myparam):
    return {"inclusion_param":myparam}

inclusion_tag is a nice time saving decorator that automatically pulls up the appropriate template and renders it. However, when we included our scurl tag (remember, this tag depends on the HttpRequest) inside the html-to-include.html template, everything blew up. Looking at the Django source, I was surprised to see that when processing the inclusion_tag, a new context is created and the parent context is not used. At first, this seemed crazy, but in thinking about it more, it’s the right thing to do: we don’t necessarily want our parent context colliding with expectations of the inclusion_tag. That is, because of this separate context, there is a way to formally adapt our parent context to the context of the inclusion_tag.

So now, the challenge is to adapt the context. The inclusion_tag decorator provides a handy flag takes_context that tells Django to provide the context to the template tag. To do this, we need to alter our signature slightly; the context must be the first parameter:


  @register.inclusion_tag("html-to-include.html", takes_context=True)
  def my_include_tag(context, myparam):
    return {"inclusion_param":myparam}

The parameter context is now passed the parent RequestContext.

Now, misunderstanding #2 came along: we thought that this context would automatically be propagated when rendering the template. It was not; a new Context instance was still being created. Diving into the source, inclusion_tag has an undocumented keyword parameter context_class that allows you to specify what context that Django instantiates. So this led to this trial:


  @register.inclusion_tag("html-to-include.html", takes_context=True, context_class=RequestContext)
  def my_include_tag(context, myparam):
    return {"inclusion_param":myparam}

This failed because the __init__ signature for RequestContext looks nothing like the Context __init__ signature. And in hindsight, it’s a naive trial because, unless there’s some serious magic under the hood, how would the actual request get passed through to the RequestContext that was being instantiated?

But wait! The intent of the inclusion_tag is to provide an adapter between contexts. AND python is dynamically typed. The latter is relevant because my initial tag scurl doesn’t actually care that it has a RequestContext, only that context['request'] returns what it needs.

So, all we had to do was implement the adapter (dumping the erroneous context_class bit):


  @register.inclusion_tag("html-to-include.html", takes_context=True)
  def my_include_tag(context, myparam):
    return {"inclusion_param":myparam, "request":context['request']}

Pretty cool, but rife with a pretty deep understanding of Django internals that is vital for any templatetag author. Would have never sorted this out without access to the Django source…

So, here we sit, uncertain of our first steps….

Here at VendAsta, we’ve just kicked off our Friday afternoon jam sessions. Starting at noon, we bring in lunch and all get together in adhoc groups and work on projects of interest. Anything. Well, anything software.

It’s an odd feeling though to have all constraints lifted and actually believe that we can just go out and play, experiment, learn, research, anything.

But now the group is starting to self-organize and some ideas are emerging. I’m confident that these sessions will rock as people get going on things.

It’s gonna be great!

Code review is a vital mechanism for ensuring conventions and patterns, as well as spreading knowledge of solutions and technologies throughout a team.

We have been evaluating Crucible as a formal tool for tracking comments and laying the comments alongside the actual code. The cool thing is that it integrates (somewhat) with Fisheye, Jira, and Subversion – all tools that we use. However, it comes with a steep price tag, and its baked-in process is too heavyweight for our small Scrum teams.

So I recently went on a search for an alternative.

In short order, I came across rietveld, which has come out of Guido van Rossum’s 20% Google project (originally Mondrian). This app is built on Google App Engine and integrates with Subversion (along with Perforce from the original Mondrian days).

We’ll try it out and see how it goes.

The best side-effect: in rietveld, we have Guido showing us the best practices in building a Django app on App Engine! Best practice information in this space is hard to come by.