0.9.5 Almost There, But Stumped On Templates

Since the 0.9.4 release I’ve rewritten the main part of the decoding parser so that it’s much cleaner and handles more edge conditions. If there’s one word that defines what makes MIME horrible it would be “edge”.

It’s amazing the kind of stupid crap clients are allowed to send out in headers. I’ve actually got test cases from Mutt where it breaks headers up into completely different encodings across multiple lines for no damn reason at all. Here’s a test I had to write just to cover the one worst case I found:

def test_dumb_shit():
    # this is a sample of possibly the worst case Mutt can produce
    idiot = '=?iso-8859-1?B?SOlhdnkgTel05WwgVW7uY/hk?=\n\t=?iso-8859-1?Q?=E9?='
    should_be = u'H\xe9avy M\xe9t\xe5l Un\xeec\xf8d\xe9'
    assert_equal(encoding.header_from_mime_encoding(idiot), should_be)

If you can decode what that actually is and why it’s retarded then you’ll win a prize. I’ll give you a hint: B != Q.

The example of the above working is on this page at oneshotblog.com so that’s solved (for now).

Now About Those Templates

I originally wanted to make writing templates as easy as writing an email or a web page, so I let you do things like this:

To: {{ message['from'] }}
From: {{ confirm_address }}@{{host}}
Subject:  Please confirm your new blog {{ post_name }}
Reply-To: {{ confirm_address }}@{{host}}

Hi there, you requested that I create a blog with a post named:

{{ post_name }} : {{ message['subject'] }}
...

This is a Jinja2 that oneshotblog.com sends out when it makes you confirm creating a blog.

This seems all normal, since it’s the headers you want and the body, like an email, but it is a total fantasy. The reason is you can’t actually decode the above template if any part of it has non-ascii chars. I’ll explain, and then hopefully I can get some suggestions on what people would like.

First, Lamson does the right thing and in your templates you are getting perfectly usable and future proof unicode objects to work with in your handlers. Also, Jinja2 has no problem with taking this unicode and processing the above template. It works fine, and Jinja2 or Mako aren’t to blame for the problem with templates now.

The problem is actually in how email has to be parsed, since the email library expects you to hand it a ascii encoded string, no unicode allowed. Nothing else. Lamson’s new code can handle the unicode fine, the problem is actually in this one function:

def respond(template, variables, html=False):
    results = render(template, variables)

    data = email.message_from_string(results.encode('ascii', 'replace'))

    if html:
        msg = mail.MailResponse(Html=data.get_payload())
    else:
        msg = mail.MailResponse(Body=data.get_payload())

    msg.update(data)

    return msg

The above function has to go, but where I’m stuck is that it’d be way more elegant to ditch this whole mess completely and remove headers from the template.

It’s too hard to write these headers by hand unless they are simple ones, and there needs to be a way to pass in more complex headers that you want to add. If I’m providing a way to pass in headers to lamson.view.respond then why not take the headers out of the template and simplify the whole rendering process?

I’d like to hear from people using Lamson or those who might have ideas on this. If taking your headers out of the template and moving them to the render call will totally screw up your day, then let me know. If it comes down to it, and I have to do it, I’ll probably write a conversion tool to help.

Also, if you have a suggestion for doing the above better but keeping the headers then let me know.

Thanks for your help folks, and I’ll be hanging out in the #lamson channel on irc.freenode.org if people want to hash out a design.