Lamson 1.0pre2, HTML Email, Standalone
Lamson 1.0pre2 features two features that might signal the end of the beginning or the beginning of the end, depending on your perspective: HTML Email and Lamson Standalone. HTML Email support comes from a new module lamson.html that gives a nice template method to send out HTML to victims…uh…customers. Lamson Standalone will be a way to run Lamson as your customized email server instead of another server like Postfix.
Both of these features will be done for Lamson 1.0, but are currently just getting started as of 1.0pre2. Once they’re done that will be the last two requested features people had before 1.0.
Getting This Release
As usual, you can get this release from /releases or if you are too lazy to read that page then do this:
$ sudo easy_install -U lamson
In addition, if you want play with the new features you’ll need to install these (optional) Python libraries:
$ sudo easy_install -U pydns $ sudo easy_install -U beautifulsoup $ sudo easy_install -U markdown2 $ sudo easy_install -U clevercss
I’ll decide if I should make these mandatory or optional with the next release based on people’s feedback.
Lamson Standalone
Quite a few people have asked for a way to install Lamson as their primary email server. Usually they want it for just their own email on a machine just for them where installing something like Postfix is overkill or just too hard. This is entirely possible with Lamson, but it’s kind of not the right use for it since simple delivery of mail is much better handled and implemented by an older server like Postfix.
Rather than resist giving people a way to setup Lamson as their primary mail server, I’ve decided to start making it possible for them. By the next release (1.0pre3) you’ll be able to run Lamson as your only email server for small installations handling mail for one person.
I’ll also include a screen cast showing people how to use the standalone functionality to build a Personal Mail Management Server to filter and control their mail without any other mail server. I’m going to treat this as a first project for most people wanting to get started using Lamson for their email.
As of 1.0pre2, the functionality that’s available is code in the lamson.server.Relay that will use PyDNS to query up the MX host for recipient addresses. It’s fairly primitive right now, but you use it by creating a Relay with the host explicitly None:
relay = server.Relay(host=None, port=25, debug=1)
Once you do this the Relay will lookup the hosts rather than trying to send through a relay host.
Don’t go crazy with this yet, since it has to be tested with various kind of nasty email addressing out there, and it needs to have a way to generate a bounce when it has an error.
HTML Email
Well, I broke down and implemented my idea for making HTML Email easy as hell to generate. In the past I didn’t want to include simple HTML support because, well, HTML Email is annoying as hell and I didn’t want to deal with the support headaches. I knew once I threw HTML generation into Lamson I’d have an army of marketing people using Lamson poorly to generate their marketing materials. I also think that HTML formatting in email doesn’t work as a customer development strategy.
Yet, every time I tell someone about Lamson, the very first, second, third, and 300th thing they ask is if it does HTML Email. Over and over and over this was the most important feature, above spam blocking, filtering, building applications, intelligent state management, or anything else that Lamson supports.
Well, if the people want HTML email generation, then the people will get it. I introduce you to lamson.html which makes it trivial to produce HTML in your email and doing it in a nice clean way using CleverCSS and Jinja2 templates.
Let’s start with the simplest little example that will send out a disgusting html template:
import sys from lamson import html, server from config import testing relay = server.Relay(host=None, port=25, debug=1) hs = html.HtmlMail("style.css", "html_test.html") title = "Test Message HTML" msg = hs.respond(locals(), "content.markdown", From=sys.argv[1], To=sys.argv[2], Subject="Test %(title)s") relay.deliver(msg)
You could run this right out of the Lamson source tree like this:
export PYTHONPATH=tests python sender.py thedude@thedude.com victim@gmail.com
The result would look like this in victim’s email:
How lamson.html.HtmlMail Works
The rationale behind the HtmlMail class is that you’ll typically have a template you want to send, some CSS, and then body content that you’ll plug into the template as a slug for each person. What HtmlMail does is let you setup the template as HTML with Jinja2, and then specify your CSS using CleverCSS (which is way easier).
When you go to send, you generate a Markdown template as the body. This lets you write the actual body of your HTML mail the same way you would a regular email, but still let’s you get a good HTML output using the HTML/CSS templates. With markdown regular folks can write the marketing copy for your spam while keeping the design separated.
HtmlMail then glues this all together by doing the following:
- Parses your CleverCSS template (after running it through Jinja2).
- Runs your content markdown through Jinja2 and markdown2 just like all your other Lamson templates.
- Injects your CSS into your HTML tags so that you get your styles even though many clients rip out the style tags from your HTML. See the output sample below.
- Knits your generated content into your HTML template as the {{content}} variable.
The end result is that these two lines of code from the above sample:
... hs = html.HtmlMail("style.css", "html_test.html") ... msg = hs.respond(locals(), "content.markdown", From=sys.argv[1], To=sys.argv[2], Subject="Test %(title)s")
Will let you blast out well formatted HTML emails that have a high probability of displaying in most mail clients.
How The HTML And CSS Looks
Here’s what each of the files in the above sample look like. First the style.css
:
body: margin: 10 padding: 20 background: green - 30 color: blue h1: font-size: 3em h2: font-size: 2em color: yellow h2: font-size: 1em p: padding: 0.3em background: red h2: color: yellow #bright: background: black color: white .dull: background: gray color: black
Notice that CleverCSS supports quite a few very cool features, like calling functions, nesting, variables, and calculations.
Next you have the outer template html_template.html
that wraps your content markdown template. Notice
that there’s no style tag where the above CleverCSS is placed. That gets done
later by lamson.html.HtmlMail.
<html> <head> <title>{{ title }}</title> </head> <body style="background: magenta"> <h1 class="bright">{{ title }}</h1> {{ content }} <h2 id="dull">All done.</h2> </body> </html>
Finally, you have the content.markdown
file that has your basic markdown formatted email:
Hello ===== I would *love* for you to tell me what is going on here joe. NOW! Alright ------- This is the best I can come up with. Zed
You can use any format you want by changing a setting when you construct your HtmlMail object, but markdown seems to be the closest to what people would send as plain text for their email. In fact, I’d like to find a way to send the raw markdown as the plaintext version of each HTML email that goes out.
With the above three components, you then get the following output:
<html> <head> <title>Test Message HTML</title> </head> <body style="margin: 10; padding: 20; background: #006200; color: blue"> <h1 class="bright" style="background: black; color: white">Test Message HTML</h1> <h1 style="font-size: 3em">Hello</h1> <p style="padding: 0.3em; background: red">I would <em>love</em> for you to tell me what is going on here joe. NOW!</p> <h2 style="color: yellow">Alright</h2> <p style="padding: 0.3em; background: red">This is the best I can come up with.</p> <p style="padding: 0.3em; background: red">Zed</p> <h2 id="dull" style="background: gray; color: black">All done.</h2> </body> </html>
Like magic HtmlMail has taken your CleverCSS file and merged it into the tags of your HTML to style it, making it work in most clients without forcing you to do it manually. It does this by walking the CSS and generated HTML with BeautifulSoup and setting the style attribute as it goes.
Librelist JSON Archives
There’s been a ton of work on librelist.com to make it work better, but the big feature is a complete JSON dump of the mail archives and a new Archive Browser for reading the archives.
The browser is just a fast one day hack I did to prove that the JSON archive format was good. It uses JQuery UI to build the interface, and has no servers side software other than Nginx. Yes, you read that right, there is no Django behind this, just a pure Nginx setup serving files and directory indexes.
The real meat of this setup is the Lamson server on librelist and an Nginx module.
First, the librelist server generates a JSON version of each archived email which you can see here with your browser. This JSON actually loads as an object in JavaScript, so you have code like this:
function appendMessage(msg) { display = '<div id="message"><div id="header">' + '<div>' + msg.headers['Date'] + '</div>' + '<div id="addressing">' + msg.headers['From'] + '</div>' + '<div id="subject">' + msg.headers['Subject'] + '</div>' + '</div><div id="body"><pre>' + summarize(msg) + '</pre></div></div>' $("#messages").append(display); }
Second, there’s a tiny modification to Nginx’s autoindex I made so that it will generate a JSON version of the index instead of HTML so that you can “browse” the stored JSON files via JavaScript:
function loadDay(date) { if(window.MAILING_LIST) { $.getJSON("/archives/" + window.MAILING_LIST + date + "/json/", {}, function(data) { $("#messages").empty(); base = data[0] msgs = data[1] for(i in msgs) { fetchMessage(base + msgs[i]) } }); } else { $("#messages").html("<h2>Now pick a list to look at for that day.</h2>"); } }
With those two components I can export a JSON version of the mailing list archives without having to run a large web framework.
ChangeLog
For those who are interested, here’s the Bazaar logs from this release, separated into changes to Lamson and changes to the included librelist.com sample.
Lamson Changes
- Cleanup and tuning of the html code.
- Documenting the HtmlMail API.
- New HTML email generation with CleverCSS support and merging the CSS into the HTML results.
- Additional tests for the new MX query style of Relay.
- Implemented a feature to let the relay do its own delivery, rather than needing a smart host. Requires PyDNS.
- Cleaned up the build so that it removes junk from the examples.
- Fixed up the lamson default help command so that it is more succinct.
- Properly skips spam filtering if the spam databse doesn’t exist.
- Made the spam filter not barf if there’s no spam db, which helps when testing.
- Now don’t bother storing the START state since it’s the default.
- Cleaned up mail some more and deprecated the MailRequest.msg and MailResponse.msg to be replaced with .base (since that’s what it is).
- Added a simple method to copy the attachments of a MailRequest to a MailResponse.
- Got decent attachment copying implemented, mail api needs a bit of cleanup.
- Return an empty string when trying to encode a none.
- Minor bug in the safequeue, added logging so you can track oversize.
- Getting the headers right for replies to work, removed the first message.
- Added options to Queue and QueueReceiver to limit the size of incoming mail.
- Added a version command finally, with some good information.
librelist.com Changes
- Force the permissions to be correct for serving up the archives.
- Make sure that bounces are always saved for later.
- Json convert script can’t go there.
- Need to install simplejson to make the json work on librelist.
- Prevent loop back of messages from lamson to itself, and stop messages that should not be considered lists like unbounce and noreply.
- Implements a json version of the mailing list archives in addition to the regular MIME encoded ones.
- Update of changes from the live server for setting up the spam system.
- Migration to install the spam database on deployment.
- Added spam filtering to the admin handler where it counts.
- Confirmations are removed after they are verified.
- Template for bad list name help.
- Tweaked the formatting for the bad list name helper, and made sure that it returns to START.
- Now giving out an error message for people who get the name format wrong.
- Added additional headers needed for achives, and try to maintain the date and message-id headers.
- Production and staging deployment scripts setup and a migration to make those work.
- Implemented a simple bash based deployment setup for automating librelist deploys and migrations.
- Nope, can’t add the – yet without some regex wizardry.
- Small change to allow – in the mailing list names.
- Updated the messages librelist sends out for readability.