Approach: Building a toy template engine in Python.

TL;DR Let's build a simplistic template engine and explore how it works under the hood – in case you were ever wondering.

If you wish to dive straight into the code, then Github is your friend.

Language design

Our language is pretty basic. We will have two types of tags, variables and blocks.

<!-- variables are surrounded by `{{` and `}}` -->
<div>{{my_var}}</div>

<!-- blocks are surrounded by `{%` and `%}` -->
{% each items %}
    <div>{{it}}</div>
{% end %}

Most blocks need to be closed as in the example and the closing tag is just the word end.

Our template engine will be able to handle basic loops and conditionals. We will also add support for callables in blocks – personally, I find it handy being able to call arbitrary Python functions in my templates.

Loops

Loops allow for iterations over collections or iterable objects.

{% each people %}
    <div>{{it.name}}</div>
{% end %}

{% each [1, 2, 3] %}
    <div>{{it}}</div>
{% end %}

{% each records %}
    <div>{{..name}}</div>
{% end %}

In the example above, people is the collection and it refers to the current item in the iteration. Dotted paths in names will resolve to nested dictionary attributes. Using ’..’ we can access names in the parent context.

Conditionals

Conditionals need no explanation. Our language will support if and else constructs, and the following operators: ==, <=, >=, !=, is, >, <.

{% if num > 5 %}
    <div>more than 5</div>
{% else %}
    <div>less than or equal to 5</div>
{% end %}

Callables

Callables can be passed via the template context and get called with positional or keyword arguments in the template. Call blocks do not need to be closed.

<!-- supports positional arguments... -->
<div class='date'>{% call prettify date_created %}</div>
<!-- ...and keyword arguments -->
<div>{% call log 'here' verbosity='debug' %}</div>

Theory

Before delving into the details of how our engine will compile and render the templates we need to talk a bit about how we will represent a compiled template in memory.

Compilers use Abstract Syntax Trees to represent the structure of a computer program. ASTs are the product of lexical analysis on the source code. An AST has a lot of advantages over source code since it does not contain unnecessary textual elements such as delimiters; furthermore, nodes in the tree can be enhanced with attributes without altering the actual code.

We will parse and analyse the template and build such a tree to represent the compiled template. To render the template we will walk the tree, pass the correct context into each node and spit out HTML.

Tokenizing the template

The first step in parsing the template is splitting the contents into fragments. Each fragment can either be arbitrary HTML or a tag. To split the contents we will use regular expressions and the split() function.

VAR_TOKEN_START = '{{'
VAR_TOKEN_END = '}}'
BLOCK_TOKEN_START = '{%'
BLOCK_TOKEN_END = '%}'
TOK_REGEX = re.compile(r"(%s.*?%s|%s.*?%s)" % (
    VAR_TOKEN_START,
    VAR_TOKEN_END,
    BLOCK_TOKEN_START,
    BLOCK_TOKEN_END
))

Let’s analyze TOK_REGEX a bit. We can see that there’s an option between a variable tag and a block tag. That makes sense – we want to split on either variables or blocks. We surround the option with capturing parenthesis to include the matching text in the resulting tokens. The ? inside the options is for non-greedy repetition. We want our regex to be lazy and stop on first match, so, for example, we can extract variables inside blocks. This is a very nice explanation of how to control the greediness of regular expressions.

The following example shows our regular expression in action:

>>> TOK_REGEX.split('{% each vars %}<i>{{it}}</i>{% endeach %}')
['{% each vars %}', '<i>', '{{it}}', '</i>', '{% endeach %}']

We will encapsulate each fragment of text in a Fragment object. This object will determine the fragment type and prepare the fragment for consumption by the compile function. Fragments can be one of four types:

VAR_FRAGMENT = 0
OPEN_BLOCK_FRAGMENT = 1
CLOSE_BLOCK_FRAGMENT = 2
TEXT_FRAGMENT = 3

Building the AST

Once we’ve tokenized the template content it’s time to go through each fragment and build the syntax tree. We will use a Node class to serve as the base class for tree nodes, and then create concrete subclasses for each possible node type. A subclass should provide implementations for process_fragment() and render(). process_fragment() is used to further parse the fragment contents and store necessary attributes on the Node object. render() is responsible for converting the node to HTML using the provided context.

A subclass could optionally implement enter_scope() and exit_scope() which are hooks that get called by the compiler during compilation and should perform further initialization and cleanup. enter_scope() is called when the node creates a new scope (more on this later), and exit_scope() is called when the node’s scope is about to get popped off the scope stack.

This is our base Node class:

class _Node(object):
    def __init__(self, fragment=None):
        self.children = []
        self.creates_scope = False
        self.process_fragment(fragment)

    def process_fragment(self, fragment):
        pass

    def enter_scope(self):
        pass

    def render(self, context):
        pass

    def exit_scope(self):
        pass

    def render_children(self, context, children=None):
        if children is None:
            children = self.children
        def render_child(child):
            child_html = child.render(context)
            return '' if not child_html else str(child_html)
        return ''.join(map(render_child, children))

As an example of a concrete subclass, this is a Variable node:

class _Variable(_Node):
    def process_fragment(self, fragment):
        self.name = fragment

    def render(self, context):
        return resolve_in_context(self.name, context)

In order to determine the node type (and hence instantiate the correct node class) we will look at the fragment type and text. Text and Variable fragments directly translate to text nodes and variable nodes. Block fragments need a bit more processing – their type is denoted by the block command which is the first word in the fragment text. For example, the fragment:

{% each items %}

is a block node of ‘each’ type since the block command is each.

A node can also create a scope. During compiling, we keep track of the current scope and new nodes are added as children of that scope. Once we encounter a correct closing tag we close the scope, pop it off the scope stack and set the top of the stack as the new current scope.

def compile(self):
    root = _Root()
    scope_stack = [root]
    for fragment in self.each_fragment():
        if not scope_stack:
            raise TemplateError('nesting issues')
        parent_scope = scope_stack[-1]
        if fragment.type == CLOSE_BLOCK_FRAGMENT:
            parent_scope.exit_scope()
            scope_stack.pop()
            continue
        new_node = self.create_node(fragment)
        if new_node:
            parent_scope.children.append(new_node)
            if new_node.creates_scope:
                scope_stack.append(new_node)
                new_node.enter_scope()
    return root

Rendering

The last step in the pipeline is to render the AST to HTML. We visit all the nodes in the AST and call render() passing as a parameter the context given to the template. During rendering we will need to deduce whether we are dealing with literals or context variable names that need to be resolved. For this, we will use ast.literal_eval() which can safely evaluate strings containing Python code:

def eval_expression(expr):
    try:
        return 'literal', ast.literal_eval(expr)
    except ValueError, SyntaxError:
        return 'name', expr

If we are dealing with a context variable name instead of a literal then we need to resolve it by searching for its value in the context. We need to take care of dotted names and names that reference the parent context. Here’s our resolving function which is the final piece of the puzzle:

def resolve(name, context):
    if name.startswith('..'):
        context = context.get('..', {})
        name = name[2:]
    try:
        for tok in name.split('.'):
            context = context[tok]
        return context
    except KeyError:
        raise TemplateContextError(name)

Conclusion

I hope this academic exercise gave you a taste of how the internals of a templating engine might work. This is far and away from production quality but it can serve as the basis of better things.

You can find the full code on Github, and you can discuss this further on Hacker News.

Huge thanks to Nassos Hadjipapas, Alex Loizou, Panagiotis Papageorgiou and Gearoid O’Rourke for reviewing.

How to win your next hackathon: 8 top tips.

TL;DR We won this Fall's AngelHack London and we'd like to share our experience with you.

On the weekend of 10-11th of November, we took part in the first ever AngelHack London. For those who haven’t heard of it, AngelHack claims to be the world’s biggest hackathon and holds events from San Francisco to Tel Aviv.

Judging from the amount of people participating in the London hackathon at Bloomberg’s offices, their claim could quite possibly be true. Over 350 developers, designers and ‘idea guys’ attended with around 60 teams surviving until the end. It was a deeply impressive event from Greg and the rest of the AngelHack team.

What we did

We designed, hacked, shipped and pitched a browser app called Penny. All this happened in about 22 hours.

We also ‘won’ the hackathon, along with two other great teams: BnkPositive and Testlio.

At the time, we were exhausted and winning came as a complete shock. Looking back now at the array of 60 hacks presented that day, it’s easy to think “damn, we got lucky”.

Probably most of it was luck, but we think we did a few things right. Eight things, to be specific. We’d like to share them with you.

1. Prepare beforehand, have a plan

Have a clear plan of what you want to build before you arrive, if that’s within the rules of the hackathon. If you’re not allowed to plan the product, at least plan who will take what roles. Who’s the front-end guy? Who’s handling deployment?

We met and talked for about an hour or so the day before AngelHack. We didn’t actually do much sketching or wireframing, but did talk about what was feasible and who would Penny’s potential users be.

When it comes to planning, do whatever you think will help your team hit hard on the day. Just remember to play fair and don’t break the rules.

2. Dream big, build small

Have a big idea. Tackle a big problem. Dream big dreams.

Now that you’ve done that, also accept that you will not be able to build everything you want to. You have to make sure that the portion of the product you’ve planned to build is feasible in 24 hours.

At a hackathon you need to build the subset of features that maximize your chances of someone understanding what you’re trying to achieve and provide the most value to your end user.

For us, that meant abandoning slick user profile pages and not building some cool viral mechanics. Instead we focused on getting the main value proposition of Penny in place — alerting users when garments discount.

3. Be disciplined & committed

You’re at a hackathon to have fun for sure, but you’re also there to build. That means knowing when to put the nerf gun down and put your headphones on. Looking back, we had roughly a ‘code-for-1-hour-then-check-in’ cycle going. That helped us achieve small wins and plan ahead.

We know it’s easy to get distracted and let time slip by, but you just need to think about how great it will feel to stand on stage at the end of the day and demo a product actually that works.

When the team at the table next to you start calling you the “hardcore guys”, you’re probably on the right track.

4. Use the clouds

We deployed a pretty decent infrastructure for Penny in just a day. We used Github pages to set up a marketing site for people to sign up for the beta, EC2 servers for the application and the crawlers, SQS queues for communication between the parts of the system, a hosted MongoDB powered by the great folks at MongoHQ and SES for sending out notification emails.

And we seriously suck at ops.

You’ve got to minimize the time spent on things other than your application and maintain developer happiness. If you’re like us, going all sysadmin will probably not help that much.

5. Make it look & work good

It shouldn’t come as a surprise that design counts. Even if your team is all developers who don’t know their Arial from their Helvetica, it still counts. At AngelHack, we got the impression that design was a substantial differentiator for the judges.

If you have design skills in your team, take the time to craft an awesome interface. We were lucky in that regard and people were stopping by our desks to comment on how nice the app looked.

If you don’t have the skills, try to find a designer on the day or use a framework and keep it simple. Either way, there are few excuses for not considering your hack’s design.

6. Deploy

The first thing we did when we started hacking on Saturday was to make a Fabric script for deploying Penny. Pushing code to “production” (let’s not kid ourselves, hackathon code is hardly production code) feels like progress and keeps you motivated during the deep hours of the night.

As an added benefit, if your project is accessible from the internets you shield yourself from laptop blunders during your pitch, since you can use any other available machine for demoing.

7. Have a demo, show it to people

Demo, demo, demo! You’re being judged on your technical chops. Your idea might be brilliant, but you need to show that you can build it. Even just a frontend with ridiculous, fake numbers will get you a long way.

The most common question from the judges was “what did you actually build today?”. If you can’t show them something, your chances of winning are going to be impaired.

Also remember to demo to anyone you can grab throughout the day. We were lucky to know a few other people at the day who we could show our progress to. This not only feels good, but it can throw light on a problem that you’ve totally overlooked.

8. Pitch like a badass

At a hackathon like AngelHack, it’s not enough to just build something, you’ve also got to pitch it. This can be super intimidating to a lot of people because it means standing up in front of a room full of very intelligent people and trying to sell your idea.

You have to consider a few things before you get up to pitch: do you know how you would extend your idea given time and money; do you know how it could make money; have you considered all the significant technical challenges (even if you haven’t solved them); why are you the right people to be doing this.

Ultimately, if faced with tough questions, honest answers are the best. One of our judges turned out to be an expert in the space we’d positioned Penny. He asked some very good questions and at one point we even had to answer “yeah that’s going to be a huge challenge”.

But that’s okay – when you think about it, it’s those “huge challenges” that get us out of bed in the morning.

Originally posted on http://usepenny.com/

Hack Cyprus 2012: A tale of hacking in Cyprus.

TL;DR Hack Cyprus 2012 was the first ever hackathon in Cyprus. Here are some lessons learned from organizing it.

First, some geography. Do you see that little island in the middle of Turkey, Lebanon and Egypt? That’s Cyprus. Its population? Less than a million people. According to Google, it doesn’t even deserve a country tag at this zoom level.

Cyprus experienced its first hackathon last weekend, and it was awesome. More than 80 people got together and hacked for around 30 hours building stuff ranging from event aggregators, underwater explorer robots, augmented reality games and mobile apps. For Cyprus, a country starving for technological innovation, this is a huge thing - and it’s all part of a bigger movement trying to create a tech startup scene on the island.

When me, @geracleous and @gmakkoulis, set out to organize Hack Cyprus 2012 we had no clue on how to go about it. Sure, we’ve been to a fair share of hackathons ourselves, but organizing one entails a completely different set of requirements. This post is an attempt to share our experiences, dos and dont’s, with everyone that’s asking the same questions we did, a couple of months ago.

Ask, a lot

When you don’t know, ask. We scoured the web and our social networks and found people that have done this before. The information and advice they gave us was invaluable.

Give yourself plenty of time

We first committed to the idea of organizing this hackathon in mid-June and, naively, thought that 2.5 months were enough. As it turns out, hackathons take a lot of time and resources to organize, especially if all of you are occupied full-time with either work or university. If we had another two weeks to prepare, we would have saved ourselves from a lot of stress.

Educate people

This is mostly location-depended. If you are planning a hackathon in a country that’s not at the forefront of current technology trends (Cyprus, for example), then people are likely to never have heard of a hackathon before. What’s even worse, they might correlate it to security hacking and get the wrong impression. Be prepared to educate them, explain in simple terms what a hackathon is and give them examples of previous hackathons and what came out of them. For example, we created a small series of posts showcasing past hackathons in the UK and the US and circulated them regularly amongst our followers.

Validate your intuition

You think that people will respond well to your hackathon. Hell, you are certain they will. Well, have fun explaining that to any potential sponsor! They need to see numbers, they need to feel safe that they will get their money’s worth back.

Before even coming up with any structure for the hackathon, our designer friend @orourkedesign had this idea that we should set up a simple website with a bit of explanatory text and a big button saying “I want this to happen”. Clicking on the button would register a user’s interest (by adding them to a Facebook event and in a Firebase database), and would then display their profile pic alongside all the others that expressed their support. The result was great. We got over 160 supporters and we validated our idea - all with a couple of hours of work.

Don’t let sponsors stall you or play you

If you are looking for sponsorship, prepare to negotiate. If a potential sponsor is slow to respond, go to the next one. Don’t waste time with people that don’t get it. Also - and this is important - do not take anything for granted until you actually have the money. We made that mistake and we paid for it.

Venue

The venue is the single most important thing you need to secure. Make sure there’s enough room for double the amount of people you expect. Otherwise it’s going to get too crowdy and, after a while, smelly. Pick your venue wisely as it will play a role in your hackathon’s success. It needs to have toilets (more than one), projectors, lots of desk space, windows, air conditioning (if you live in a hot climate), fast internet, strong and reliable wi-fi and loads of wall jacks and ethernet ports. If you are going to have workshops or presentations during the event, it’s best if you have a second room for that so you don’t disturb the teams.

Food and drinks

Hackathons tend to be on the unhealthy side of food. Red Bulls, chocolates, crisps are all essential I know, but, take some time to think different. For example, have some fruits, juices, cereal and milk - people will appreciate that. Eating well will keep them focused for more time. Of course, everything should be free if your budget allows for it.

Prizes

People come to hackathons mostly for the fun of it but it wouldn’t hurt if they got rewarded for their hard work! A prize is always a good motivator, even though some might say it’s the wrong kind of motivator.

In any case, if you plan to have prizes you should cater for different categories of hacks. For example, we offered the Hack Cyprus Prize and the Crowd Favourite Prize that every team was eligible for and then we had prizes for mobile apps and hardware.

Another important point is to be transparent with your scoring scheme. We got that wrong, so please get it right. At the start of the event make sure you tell the teams what they will be judged upon and how important each score dimension is.

Outputs

You will have a good time, but any half-descent event needs to leave its mark in internet history. Have a live feed that aggregates tweets, pics and even commit messages! When the event is over freeze the feed to a static page and serve it to the Googlebot for consumption. Take lots of pictures and videos, make a public group on Flickr and tell everyone to upload their material from the event. Have a live stream during presentation time for people that didn’t make it to the venue. We went a step further and got everyone to sign our banner. That was sweet!

I’ll leave you with some pictures from the event - you can discuss on Hacker News.