App Engine: JSON Objects in the Google Datastore

Following up on my previous post about Python Objects in the Google Datastore where I’ve shown an easy way of storing any python object inside the Google Datastore by pickling it. Pickling works fine and it can literally store functions, classes and class instances in the datastore which is great, but as I discussed earlier, JSON could work too, so here I’ll introduce a similar approach at storing JSON objects (strings, lists and dictionaries) inside Google.

The problem with pickling objects is that they’re very difficult to maintain. Google provides a front-end for their Datastore where you can run GQL queries, watch statistics and even manage data on the fly. Since pickled objects are stored in bytes as opposed to JSON strings, I figured that when editing an entity using the Google admin, pickled objects tend to break, since the values sent back to the server by your browser are usually strings. I haven’t found a good way around this yet (update: I did find a way), which is why I temporarily switched to JSON, which is easier to edit and maintain.

Here’s the draft code I’m using to store JSON objects in Google App Engine, which I called (surprise) JsonProperty:

from django.utils import simplejson
from google.appengine.ext import db

class JsonProperty(db.TextProperty):
	def validate(self, value):
		return value

	def get_value_for_datastore(self, model_instance):
		result = super(JsonProperty, self).get_value_for_datastore(model_instance)
		result = simplejson.dumps(result)
		return db.Text(result)

	def make_value_from_datastore(self, value):
		try:
			value = simplejson.loads(str(value))
		except:
			pass

		return super(JsonProperty, self).make_value_from_datastore(value)

Note that Google still runs Python 2.5 hence the simplejson library is only available through the django.utils package (it ships with Python starting from version 2.6).

So the logic is quite clear here, we use dumps and loads to dump and load strings from and to the Datatstore. Perhaps it’s lacking some code in the validate method, we do after all need to check if the given value is convertible to JSON and then raise an exception if not. But I’ll leave that to the next version of this snippet, as I’m currently only using it to prototype stuff and maintain small and medium-sized projects — so do use this at your own risk ;)

As a usage sample you can take this simple code which can be run from the App Engine console:

class MyEntity(db.Model)
    name = db.StringProperty()
    obj = JsonProperty()

my_obj = {'key-1': 'value-1', 'key-2': 'value-2'}
entity = MyEntity(name="my-name", obj=my_obj)
entity.put()

entities = MyEntity.all()
entities = entities.fetch(10)
for entity in entities:
    print entity.obj # outputs the dictionary object

And don’t forget that you can manually browse the Datastore using your App Engine dashboard and see how such entities are actually stored.

I wrote about performance in that previous post, but honestly, I didn’t have the time to measure it, so if you guys do feel free to leave your benchmark results in the comments section. Some graphs would be cool too. Oh and thanks for retweeting!