Both Safari and Internet Explorer have default privacy settings that are quite harsh on third-party cookies, and both browsers need some consideration in order to get them working correctly. With the proliferation of widgets on the internet, third-party cookies are quite common, so I’m surprised there isn’t a more thorough treatment of the issue (though don’t expect one here!).

Internet Explorer

Internet Explorer (I tested with IE7) has a default privacy setting of “Medium”, which requires that sites issuing third-party cookies must have a Compact Privacy Policy (P3P) header. The best documents I’ve found on this subject are here and here. Pay particular attention to the “Unsatisfactory Cookies” section in the first document; use the second document to determine your appropriate P3P header. Once you are correctly serving an appropriate P3P header, third-party cookies should start to work in Internet Explorer.

Safari

Safari (I tested with Safari 4) has a default third-party cookie setting of “Accept cookies: only from sites I visit.” I took this to mean sites that are the topmost frame (e.g., url showing in the address box). A colleague of mine, Dave Mosher, saw me struggling with this and casually mentioned that the user must first interact with the frame in order to set third-party cookies. This is an interesting, Apple-ish approach; I suppose if I interact with the frame, I have explicitly “visited” it. Further, this solution would work out for many third-party cookie scenarios (e.g., a login) because the cookie isn’t set until after some sort of interaction. In our case however, we need to set the cookie on the very first load before the user has a chance to interact. So, to make this work, the best I could come up with was: for Safari, if our cookie wasn’t present, I introduced an interstitial screen that effectively asked the user to click a link, which allowed us to set a third-party cookie. Unfortunately, I could not find anything more elegant, but this is better than many of the other proposed solutions I found (including popping new windows, using HTML5 Local Storage, etc.). Finally, as long as this cookie is present, you are free to read it before the user interacts with your frame; i.e., if it is a permanent cookie, the user will see the interstitial screen only once (of course depending on your particular cookie setting logic).

Caveat: you may also require the P3P header for Safari. I had it in place while I discovered the Safari solution, so I’m not sure if it works without it.


import simplejson
from google.appengine.api import urlfetch # App Engine example
def my_method(user_id):
  url = 'http://www.example.com/lookup-user/%d.json' % user_id
  result = urlfetch.fetch(url)
  if result.status_code == 200:
    return simplejson.loads(result.content)
  return {}


This is bad for testability because when my test code executes my_method(), it will either go out over the wire (super bad!) unless I monkey-patch urlfetch to be some other class (really inconvenient).

So, instead, we can accept an alternate urlfetch mechanism optionally in the method:


from google.appengine.api import urlfetch as urlfetch_builtin
def my_method(user_id, urlfetch=urlfetch_builtin):
  url = 'http://www.example.com/lookup-user/%d.json' % user_id
  result = urlfetch.fetch(url)
  if result.status_code == 200:
    return simplejson.loads(result.content)
  return {}


So now the test client can pass in an alternate implementation of urlfetch that can return instrumented results and exceptions. This proves to be extremely handy.

One problem with this particular implementation is that it forces the client to do some odd things. Imagine that I have a piece of client code that itself is dependency injected:


def client_function(urlfetch=None):
  # here, I must check for an injection so that I know to provide it to my_method
  if urlfetch:
    return my_method(123, urlfetch=urlfetch)
  else:
    return my_method(123)


This is unfortunate. A better way to implement dependency injection and prevent this client if statement is to adjust my_method:


from google.appengine.api import urlfetch as urlfetch_builtin
def my_method(user_id, urlfetch=None):
  urlfetch = urlfetch or urlfetch_builtin
  # ...


Since urlfetch is required by the method, we can take None to mean “use the default implementation”. Now the client code becomes simply:


def client_function(urlfetch=None):
  return my_method(123, urlfetch=urlfetch)


Make sure you setup your dependency injection params in this way and save your clients some headaches!

A desirable, and obvious, feature is to identify the experts that my friends trust, or conversely, which of my friends trust a particular set of experts. From a use case perspective, a lot of this information is jammed on to the screen at one time; that is, we cannot simply query for a single friend or a single expert.

This is a classic graph walk style problem that is relatively easy to scale in a parallel processing environment, but nasty where you can only do serialized queries.

Initially, we felt a bit limited by the AppEngine query model. It is very fast, but only allows for serialized queries.

In reaction, we began investigations into using a MapReduce mechanism to compute these “trust networks” offline. Specifically, we were planning on using MapReduce deployed into Amazon’s Elastic MapReduce engine. The results of these computations were to be stored on Amazon’s S3 (as JSON docs) and simply accessed statically from our AppEngine applications.

This solution, while sound, suffered some drawbacks. We had a difficult time with dealing with the recomputation of the trust networks. We could choose to recompute them daily, but this was wasteful of resources: many of the networks would be completely unchanged. Alternatively, we could queue jobs when new trusts are formed and process just those, but then we have to develop another job to determine which trust networks cascade from the new trust and need recomputation. The absolute nail in the coffin of this approach is that we (typically) use Facebook to define friend networks and their T&C do not allow the storage of friend relationships for any great amount of time, making offline processing difficult and incomplete at best.

We needed another solution that allowed for the real time computation of trust networks.

An architect at our company, Kevin Pierce, also happens to be one of the best reverse engineers I know. He started poking around the gory depths of the AppEngine source and discovered that all of the API calls stub out in MakeSyncCall. This inevitably led to the discovery of the partner MakeCall which yields an RPC object that you can wait on. Of course, if I can wait on one RPC object, I can also wait on many of them. A subsequent release of AppEngine capitalized on this fact allowing the developer to easily make parallel UrlFetch requests. There were some earlier code libraries that also facilitated parallel UrlFetch requests leveraging the same mechanism.

At Google I/O, Ryan Baldwin and I had the opportunity to corner Ryan Barrett (AppEngine’s supersmart Data Store guru) and chat with him about using this approach for other API interaction. Ryan seemed pretty excited about our exploration of this area and indicated that this was something they had intended to expose to developers but, like any successful software project, competing priorities kept getting in the way.

Encouraged, we set about developing code to allow developers to queue up a set of AppEngine API calls and then kick them off in parallel. In particular, for the use case that we were looking to solve was the ability to perform multiple Data Store Queries in parallel; this would allow us to query for each friend’s trusts in parallel and collect the results together.

An example invocation might look something like this:


# set up the async queries
runner = MultiTask()
for uid in user_ids:
    query = db.GqlQuery("SELECT __key__ FROM Account WHERE facebook_id = :1", uid)
    runner.append(QueryTask(query, limit=1, client_state=uid))

# peel out the results
for task in runner:
    task_result = task.get_result() # will raise any exception that occurred for the given query
    print '%s: %s' % (task.client_state, task_result[0])


Here, we’re making multiple parallel queries, though the mechanism allows for mixing around different API calls.

As an aside, this can yield some pretty scary/awesome looking values in the logs files:

Three seconds can chew up a lot of API time!

I’m excited to announce that we’ve bundled up this code into a package and released it to the wild on Google Code as asynctools. So far, we’ve wrapped Query (including GqlQuery) and UrlFetch. Others will come as we have time/motivation. Note that you can combine these together and execute arbitrary combinations of them in parallel.

Send us any feedback, comments, encouragement you have.

An example of this is where the templating chunks require a particular external resource, like JavaScript. You don’t want each invocation of a repeated sub-template to cause a tag to be emitted; you only need it once for the entire document.

Asp.Net has some handling for JavaScript in their UserControl mechanism (RegisterClientScriptBlock()) that uses a server-side registration mechanism to emit an appropriate document.

Dave Mosher and Brett Zabos, here at VendAsta, have used a pure JavaScript approach to get the JavaScript resources loaded efficiently, built atop the YUI loader. Make sure you check out their escapades.

I got to wondering how it came to be.

Scanning the View Source of the GMail page, I didn’t see any tags that were relevant, so I thought it might be handled by iPhone OS, similar to the favicon.ico .

So I pointed my iPhone browser at a web site, added the site to the home screen, and then checked out the web logs to see what was going on.

Sure enough, there were requests for /apple-touch-icon-precomposed.png and then /apple-touch-icon.png. It seems that the precomposed does not apply any processing on the iphone itself (it is used raw), while the second one applies a glass effect. Presumably, the latter one is only requested if the first one 404s.

A nice finishing touch if you have a mobile version of your site. Or perhaps regardless. Add this to the list of “standard web site items” like favicon.ico, robots.txt, 404, 500, sitemap.xml, etc.

There are places where you’re willing to sacrifice some performance in order to guarantee a unique value. Datastore does guarantee uniqueness on its keys, so we can use a secondary helper model to guarantee uniqueness.


class Unique(db.Model):
  @classmethod
  def check(cls, scope, value):
    def tx(scope, value):
      key_name = "U%s:%s" % (scope, value,)
      ue = Unique.get_by_key_name(key_name)
      if ue:
        raise UniqueConstraintViolation(scope, value)
      ue = Unique(key_name=key_name)
      ue.put()
    db.run_in_transaction(tx, scope, value)
class UniqueConstraintViolation(Exception):
  def __init__(self, scope, value):
    super(UniqueConstraintViolation, self).__init__("Value '%s' is not unique within scope '%s'." % (value, scope, ))


This class simply leverages the key uniqueness aspect of datastore to ensure that the value doesn’t exist. A call to check() where a matching scope and value already exists will raise a UniqueConstraintViolation.

To use this, you need to build out a common create method on your models. Below, I’ve used a classmethod to achieve this:


class Account(db.Model):
  name = db.StringProperty()
  email = db.StringProperty() # unique=True - wouldn't that be great?
  @classmethod
  def create(cls, name, email):
    Unique.check("email", email)
    a = Account(name=name, email=email)
    a.put()
    return a


I need to make a call to Unique.check() first to ensure uniqueness (which causes at least a lookup, but likely a lookup and a put – both are a performance hit relative to doing nothing at all), and then I create my own account. Unique.check() will throw if the value is not unique, preventing the Account from being created.

Note that this technique lugs around an additional dictionary in datastore (costing some $$, though realistically not much). Also note that if you jam too many scopes into this class, you’ll get degrading performance (though it’s still just a key lookup which is very efficient).


  from google.appengine.ext import db
  class Home(db.Model):
    address = db.StringProperty()
    room = db.ReferenceProperty(Room)


where Room is also a db.Model.

Datastore uses a proxying technique such that db.Model objects are created lightly (i.e., with only their key) and any hit to an attribute causes the entire object to be inflated by looking it up in the datastore.

I thought this was achieved with a lazy load technique, meaning that Room in the above example contained the logic to load itself (or more accurately Home would populate room with a proxy to Room that would know how to inflate). To do this, the proxy class would hold only its key and use that key to look itself up. From this, it follows that I could access that key without inflating the object.

It doesn’t work this way!

Instead, the referencing class (Home in the above example) holds the key (in a protected attribute, which you shouldn’t touch) and is responsible for inflating reference types.

I’ll say that again: the referenced object does not inflate itself lazily, but the referencing object does the inflation.

So, I previously thought code like this would not cause a datastore lookup:


  # assume home is an instance of Home
  room_key = home.room.key()


but I was very wrong. Simply hitting home.room causes home to lookup the entire room.

But what if you only want to get at the key (which the home has, but is protecting)?

This post suggests that the safe way to get this key is to access it via the class attribute, not the instance attribute. Here is the resulting code for my example:


  room_key = Home.room.get_value_for_datastore(home)


Note carefully the capitalization (indicating the classes versus objects).

This gives me the room key without causing a room lookup. Over and out.


  def my_view(request):
    return render_to_response("myview.html", context_instance=RequestContext(request))

The render method of our scurl templatetag gets access to this context and thus access to the HttpRequest.

So far, so good.

In our project, we also use inclusion_tags to include common chunks of HTML into pages. This sort of tag looks something like this:


  @register.inclusion_tag("html-to-include.html")
  def my_include_tag(myparam):
    return {"inclusion_param":myparam}

inclusion_tag is a nice time saving decorator that automatically pulls up the appropriate template and renders it. However, when we included our scurl tag (remember, this tag depends on the HttpRequest) inside the html-to-include.html template, everything blew up. Looking at the Django source, I was surprised to see that when processing the inclusion_tag, a new context is created and the parent context is not used. At first, this seemed crazy, but in thinking about it more, it’s the right thing to do: we don’t necessarily want our parent context colliding with expectations of the inclusion_tag. That is, because of this separate context, there is a way to formally adapt our parent context to the context of the inclusion_tag.

So now, the challenge is to adapt the context. The inclusion_tag decorator provides a handy flag takes_context that tells Django to provide the context to the template tag. To do this, we need to alter our signature slightly; the context must be the first parameter:


  @register.inclusion_tag("html-to-include.html", takes_context=True)
  def my_include_tag(context, myparam):
    return {"inclusion_param":myparam}

The parameter context is now passed the parent RequestContext.

Now, misunderstanding #2 came along: we thought that this context would automatically be propagated when rendering the template. It was not; a new Context instance was still being created. Diving into the source, inclusion_tag has an undocumented keyword parameter context_class that allows you to specify what context that Django instantiates. So this led to this trial:


  @register.inclusion_tag("html-to-include.html", takes_context=True, context_class=RequestContext)
  def my_include_tag(context, myparam):
    return {"inclusion_param":myparam}

This failed because the __init__ signature for RequestContext looks nothing like the Context __init__ signature. And in hindsight, it’s a naive trial because, unless there’s some serious magic under the hood, how would the actual request get passed through to the RequestContext that was being instantiated?

But wait! The intent of the inclusion_tag is to provide an adapter between contexts. AND python is dynamically typed. The latter is relevant because my initial tag scurl doesn’t actually care that it has a RequestContext, only that context['request'] returns what it needs.

So, all we had to do was implement the adapter (dumping the erroneous context_class bit):


  @register.inclusion_tag("html-to-include.html", takes_context=True)
  def my_include_tag(context, myparam):
    return {"inclusion_param":myparam, "request":context['request']}

Pretty cool, but rife with a pretty deep understanding of Django internals that is vital for any templatetag author. Would have never sorted this out without access to the Django source…

Here at VendAsta, we’ve just kicked off our Friday afternoon jam sessions. Starting at noon, we bring in lunch and all get together in adhoc groups and work on projects of interest. Anything. Well, anything software.

It’s an odd feeling though to have all constraints lifted and actually believe that we can just go out and play, experiment, learn, research, anything.

But now the group is starting to self-organize and some ideas are emerging. I’m confident that these sessions will rock as people get going on things.

It’s gonna be great!

We have been evaluating Crucible as a formal tool for tracking comments and laying the comments alongside the actual code. The cool thing is that it integrates (somewhat) with Fisheye, Jira, and Subversion – all tools that we use. However, it comes with a steep price tag, and its baked-in process is too heavyweight for our small Scrum teams.

So I recently went on a search for an alternative.

In short order, I came across rietveld, which has come out of Guido van Rossum’s 20% Google project (originally Mondrian). This app is built on Google App Engine and integrates with Subversion (along with Perforce from the original Mondrian days).

We’ll try it out and see how it goes.

The best side-effect: in rietveld, we have Guido showing us the best practices in building a Django app on App Engine! Best practice information in this space is hard to come by.

The investment was in essence to allow us to go much, much faster on our plan. As such, we are now swimming in job openings.

Get your resumes in! Or come by our office Friday’s at 4p for a beverage and to chat more about who we are and what we’re up to. Or give me a call (306-230-3322).

Calling all Sask-ex-pats! Saskatchewan is booming! Saskatoon is leading the way! Got a new family – raise, em in Sask! You turned out awesome after all!

I use Mac OS X 10.5.3 right now with python 2.5.2. Very often, python simply dies with the cryptic “Bus Error,” which I can only assume means that something very dramatic has happened. It’s surprising and somewhat frustrating to realize just how much drama there is on my computer.

I thought I had it licked by installing DarwinPorts and compiling python with a different version of readline (the word is that the readline lib that ships with Mac is bad).

However, as I was writing this post and used the interpretter to check my python version, it crapped out when I exited with a simple “exit()”.

Back to the drawing board I guess.

Update 2008-07-08: I found another resource on the topic. I’m not sure which of the steps in this post helped, but my python is much more stable now.

I still like the music though, so it was a really weird dynamic.

The Spades opened for him. They are a power trio from Peterborough Ontario and did a fabulous job. The drummer has his snare drum tuned about as low as it will go, and paired with two bass toms, he laid a powerful backbeat for the other guys. Good vocals, good guitar, and a solid bass line rounded it all out. A double “marching band” drum piece, that the bassist described to me as a “good break,” pulled the crowd right in. Good work, The Spades.

Matt Good, you know music, but take some notes from The Spades on stage presence. Oh, and drop the attitude.

Update: Upon reflection, this show was not “One of the Strangest Shows I’ve Seen” – I’ve seen some pretty strange stuff and this wasn’t one of them. Instead there was just a really weird range of emotions going on.

I’ve decided to save the old title for something that really deserves it.

In the opening day keynote, Vic Gundotra (VP Engineering) had a freshingly blunt answer to the question as to why Google was developing so many free, open source products: Google makes more money. That is, by enabling developers to build better, faster, larger applications targeted at the Internet, these applications will attract more users and more Internet users means more $$ for Google. (Aside: we had a short debate as to whether or not this was true, but the converse definitely holds: fewer Internet users means less $$ for G.)

The primary themes for Google are centred around making their compute cloud more accessible (via Google App Engine), making it easier to develop rich client apps (via Google Web Toolkit) and bringing human interaction to bear in applications (via OpenSocial).

They also have an entire track devoted to Android (their mobile phone platform). Gundotra stated that G feels that eventually, the mobile browser will be their target platform, but until mobile browser capabilities are enhanced, Android is an important project. Android looks cool to be sure, but I couldn’t help thinking that I was looking at a Zune: in countries well connected, the iPhone seems to have so much steam. Ironically, Gundotra pointed out that he felt the only phone that had a sufficiently powerful browser was in fact the iPhone. So perhaps G has no interest in competing with that platform, but only to help the other phone manufacturers get their shit together.

Google App Engine, in particular, is very interesting to me. In some ways, it’s super advanced (scalability is “automatic”), but in other ways, it’s so immature (there is no real way to migrate your data models to new versions). I can’t tell you how many times in the last day I’ve heard people ask for “offline processing.” That said, of course it’s brand new, and they are working hard at it.

All in all, it’s been a great conference with the usual mix of lots of stuff you can find online along with a number of nuggets of information that you just couldn’t get anywhere else. The other important bits are always in the panel / fireside chats – vision/philosophy stuff – so many conference-goers miss these. At the meta level, this conference is very under-stated. Everything is in black/grey/white, there is little to no marketing anywhere, the swag consisted of a t-shirt (black and white) and a water bottle (black and white). I guess, like the G homepage, it’s about simplicity and content

Bottom line: it’s very cool. It’s based on python with a very Django-like framework.

By far and away the coolest thing is the Google-proprietary Datastore/BigTable. This is Google’s approach to a database, except indexes, data replication, scalability, etc. are all managed for you. As an application developer, you can concentrate on your app logic and let the framework and infrastructure deal with these issues. Oh, and by the way, all the indexes and interfaces work exactly the same on the local dev environment as they do in this expansive Google compute cloud.

To be clear, in my experience with large webapps, data scalability and availability is precisely the place where a team will spend most of their time. The Google App Engine eliminates this for you.

This is a real testament to Google’s insistence and persistence on building out a world-class infrastructure and a super-solid architecture layer to build upon. Remember, this company did almost nothing other than search for the first 4 or 5 years. They had lots of employees back them too (not like today, of course), but instead of building out gobs of applications, they focused on the underlying architecture pieces and produced things like Datastore. This sort of investment has a massive flywheel effect over time.

Back to Google App Engine: it will be great to find out what G’s roadmap is for this product if one is seriously considering hosting their corporate apps. In the mean time, Google App Engine is a pretty serious endorsement for python/Django.

I love Ruby as a language, but damn I’m starting to have a tough time finding a place for Rails.

Allan

So, it’s a MacBook Pro – the 2.4GHz Core2 Duo model. First impressions: the hardware is, of course, extremely sexy. Sleek, solid; the keyboard is awesome. There are some amazingly beautiful apps that come loaded, notably Front Row and Expose/Spaces. I had a slight difficulty getting my wifi (aka AirPort) to work with my D-Link DIR-625, but got that sorted out (I had to “downgrade” it away from WPA2 and force it to stay only on WPA).

Toughest thing so far: window management. I hadn’t realized how reliant I had become on the Windows Task Bar. Mac OS X has a very different conceptual model that I’m starting to get the swing of. For example, the little blue, almost imperceptible dot below applications on the dock means that the program is running – even though there’s no window to be found. Very helpful to know that; thanks Kevin.

Anyway, things are going well so far, though I’m not taking it to work until the end of our current iteration, as I think right now, it would be a productivity loss (just the windows mgmt thing). End of iteration is only a few days away though…

VendAsta seems to be slowly being overrun with Macs. Stay tuned to see how that all turns out.

He’s currently working on introducing Scrum to his new organization. I encourage everyone who is dubious to check their trepidation and give it a whole-hearted whirl; Zach has the experience and credentials to know what he’s talking about.

I must say that I’m a bit jealous of their 25 linear feet of whiteboard. Our current temporary space is a bit whiteboard challenged:

Thankfully, we’ve begun work with the architecture design team on our new permanant space.

As I see it, the problem with this philosophy is one of opportunity cost. In most cases the statement above is pretty much accurate – we can build it, and we can do a damn good job of it. In doing so, however, what have you not been able to do? That is, in the time you’re building project B, how much time has been taken from your primary project A?

Just because you can do something is not a good enough reason to actually undertake the work. Even if there is a small quality hit you take by outsourcing (which, by the way, likely won’t happen assuming the people that you enlist to help you are experts in their field), it is worth it to outsource in order not to rob time, energy and focus away from your primary goal. In everything you undertake, you should always be asking yourself, “Is this the highest value add work that I can be doing right now? By doing this work, am I taking away from work where I am truly the only one in the world who can do it?”

After answering those questions, hopefully there’s not an arbitrary cash flow reality that gets in your way of doing the right thing.