Deploying a Nesta CMS Blog with Pygments Syntax Highlighting ∞

Ryan Daigle relaunched his website using a new set of tools:

Having wrestled with quite a few blogging engines in the past I had several requirements of a new setup. Firstly it had to support a workflow that lets me write on my local machine using the tools I prefer, namely markdown formatted articles composed with IA Writer or a basic text editor.

Second, it had to support a git-based workflow. My content is going to live in git and there’s no reason the publishing platform shouldn’t build on top of that as well. This also plays well with Heroku deployments.

He’s using the Sinatra-based Nesta CMS in combination with hosting on Heroku. The post I’m linking to contains an in-depth walk-through of his setup and template, including Pygments-based syntax-highlighting, which is a topic I just wrote about on this very blog.

Ryan’s setup differs from mine insofar that I decided to build something from scratch¹ and not build it as a static site generator. I currently use a combination of iA Writer for article drafting and an online posting form for the actual posting of articles. This way, I can also post from my iOS devices and without access to the computer that runs the static site generator. But Ryan’s approach has definitely picked my interest and I’ll surely play around with what Nesta has to offer.

1. cf. NIH-syndrome ↩

The Little Launch Center That Could've Been

App Cubby’s Launch Center is piece of art. It’s beautifully crafted, delightful to look at, and incorporates innovative features.

And yet, earlier today, I deleted it from my iPhone.

Launch Center’s basic premise, taken from their website, is:

Launch apps and trigger actions from a quick, easy to use app! Call a friend, start an email, send a text, post to Twitter, turn on a flashlight — get to where you’re going faster than ever before.

Turns out, I was able to get to where I was going faster using other means after all.

My Setup

When Launch Center 1.2 debuted it received rave reviews all over the place. And whenever something gets so much attention, the nerd in me wants in on the action and give it a spin as well.

The much toted headline feature of the 1.2 update was an easier way to add shortcuts to apps supporting URL schemes for some of the deeper nested screens, without having to go through the regular menu structure. Think Instagram’s camera or OmniFocus‘ “Add Inbox Item” screen. So I started adding actions launching a few frequently used apps, calling my wife, and, obviously, the flash light, among others.

I ended up with a screenful of shortcuts like this:

I added Launch Center to my iPhone’s first home screen and forced myself to go through it whenever I wanted to accomplish the task that could’ve been accomplished with one the shortcuts shown above.

So much for the theoretical part. Let’s go through the list of shortcuts to discuss my results in practice.

Phonecalls and Messages

Calling my significant other or sending her a message is a common task for me during the day. Using Launch Center shortcuts to accomplish those didn’t really give me a boost in getting where I wanted faster though. I keep the “Phone” icon in my iPhone’s dock and have my wife’s phone numbers added to my list of “Favorites” so they’re never more than one or two taps away.

For messages, I never close or delete conversations in the “Messages” app and since most recent conversations are located at the top neither were those very much out of reach.

Furthermore, while out and about I tend to use Siri to make calls and send messages, which tends to be much faster still than either the “tappy” way or Launch Center.

Todos and Tweets

Launch Center supposedly strives with apps that support a deeper URL scheme and both OmniFocus (my task manager of choice) and Tweetbot (my iOS Twitter client of choice) are such brethren.

As shown above, I added shortcuts for launching OmniFocus straight to the “Add Item to Inbox” screen and to launch Tweetbot straight to the “New Tweet” screen.

In practice, both OmniFocus and Tweetbot have pretty much omnipresent buttons in their UI to accomplish the same task. You have to be navigated pretty deeply into Tweetbot’s view hierarchy to not be able to locate its bright blue “New Tweet” icon in the upper right corner. OmniFocus’ icon bar at the bottom of the screen includes an “Add to Inbox” icon that even works while the app is syncing with the server (when pretty much everything else doesn’t work).

Launching Apps

The next type of shortcut was to launch apps that didn’t support any special URL scheme. For me, this included Reeder and 1Password.

I’ll make it short. Even on an iPhone 4S swiping to the second home screen was faster than going through the overhead of launching Launch Center, locating the app to launch in the scrollable list, and then sitting through its launch time.

(The way my brain is wired up, I know exactly where a frequently used icon sits on any given home screen, so there’s no “hunting for the spot to tap”.)

System Settings

On iOS 5, the built-in Settings app also has some of these URL scheme shortcuts available for Launch Center to use. I added shortcuts to the VPN pane (there’s no way to specifically launch one VPN connection straight through a shortcut as far as I can tell) and to toggle “Airplane Mode” (which isn’t really a toggle after all, since it’ll only ever switch it on, never off).

Those settings shortcuts worked, but they’re going the way of the dodo if you believe the rumors to be true of what changes are coming in iOS 5.1.

Conclusion

Admittedly, I may just be using the wrong apps to get something out of Launch Center. Or my workflow may just not be appropriate for it to fit in. (As opposed to Clear, which I did manage to fit in.)

Also, I didn’t really get the appeal of Launch Center’s “scheduling” feature, which debuted in version 1.1. It uses a “Notification Center”-based approach to push shortcuts you have defined and scheduled to your lock screen for easy access at a given time.

One could argue that you can use it as a gentle reminder for daily sweeps of your OmniFocus inbox for example, but due items in OmniFocus also show up on your lock screen and swiping them brings you directly to that respective view.

All in all, for me personally, Launch Center seems to be more trouble than it’s worth. I applaud App Cubby for their efforts and I’m sure the majority of my criticism isn’t to be blamed on them in the first place. Things like Settings losing its URL scheme for example is totally not their fault.

But, as of this morning, my home screen looks like this again and I have one less thing to worry about.

Update: App Cubby’s David Barnard responded on Twitter (mere seconds after I’ve posted this) with the reassuring information that they’re aware of some of the issues and working on solutions. I’m certainly keeping an eye out on their future updates.

Where Clear Fits In

The web was abuzz when Realmac Software’s Clear for iPhone was released. The buzz was comprised of both ends of the spectrum: High praise for its innovative, chrome-less, gesture-based UI on one end, criticism for its lack of features and being limited to just the iPhone app without a desktop counterpart on the other. 

Like Ben Brooks, I am a long-time OmniFocus user on my Mac(s), iPhone, and iPad – Clear ain’t nowhere near replacing that. 

What I found, however, is that because of the sheer speed of using Clear it’s perfectly usable as a pure list-making tool to complement the full-blown OmniFocus experience. 

One obvious use case is maintaining a grocery shopping list in Clear. I have tried to do that in OmniFocus once. And boy have I tried to like it. But it has way too much friction to be even remotely useful for such a simple task. 

In a way, OmniFocus is just overqualified for that. And I mean that in an entirely positive fashion. It’s just impossible to excel at any and all potential situations you could use a todo list for, so it’s unreasonable to expect that. 

Another scenario that I found myself preferring Clear over its bigger, more fully-featured brother is meeting notes, both as preparation and note taking during a meeting, like Scrum standups and one-on-one meetings. 

It was also more effective to jot down the things I wanted to remember to talk about during the meetings in Clear than in, say, a plain-text (or Markdown-flavored) note-taking tool such as Elements – not only due to the fact that Clear only supports lists, but also because it artificially limits you to 30 characters per list item¹, encouraging clear (pun!) and concise bullet points. 

In general, Clear is great at simple lists of non-repeating tasks that also don’t require a lot of context-switching (and, hence, context specification per task) to work through. 

What’s your use-case for Clear?

PS: Some of the tweets Clear’s @useclear Twitter account retweeted recently are quite hilarious.

1. Except when you use Siri-dictation, where this limit apparently does not apply. ↩

Syntax Highlighting for a Rails Blog

Sparing you the boring details of the holy language war between Python and Ruby, the Python world has one great thing going for it: Proper syntax highlighting libraries. While there are notable Ruby implementations like Coderay, even popular Ruby-based shops like GitHub rely on the elephant in the room: Pygments – which is, if you couldn’t tell from the name, based on Python.

Now, my blog is written in Ruby on Rails and runs on Heroku and, let’s put it this way, Heroku’s Ruby VMs and Python aren’t exactly sending each other Valentine’s cards.

However, there’s an unofficial API for Pygments, accessible over HTTP, that you can use from your Rails application to do the syntax highlighting for you. You just POST a blob of text to it and get back a blob of text with syntax highlighting classes applied.

With that in mind, you can totally go down the route of passing your blog post’s body to that service when somebody views a page on your blog, just like Austin Vance blogged about¹. And while I adore the moving clouds, it’s not something I would advocate for performance and scalability reasons. Plus, as @hmans put it, you don’t really want to be directly dependent on a third-party service to render a page on your site².

So I’ve done things in a slightly different way around here.

The Background

My posts are written in Markdown, rendered to HTML through Redcarpet 2 in a before_save callback whenever a post is created or updated. The syntax highlighting through the aforementioned unofficial Pygments API happens right after my Markdown prose has been transformed into HTML and stored in a separate database column along with the original, unhighlighted Markdown.

That way, I’m killing multiple birds with one stone.

First of all, displaying a page on my blog displays pre-generated HTML. It doesn’t slow down the request with calling third-party services or executing expensive regular expressions. While there’s a way to counteract many of these side-effects with caching, this approach also makes sure that future changes in APIs or syntax won’t change any of my old posts without me noticing. I try to be very attentive with my posts and old posts being a potential moving target isn’t an option.

Lastly, doing all the syntax highlighting server-side saves clients the execution of expensive syntax highlighting in JavaScript for example, using plugins like jquery-syntax, improving rendering times of your pages noticeably.

The Implementation

As I mentioned before, the transformation from Markdown to syntax-highlighted HTML happens in a before_save callback³ in my Post model.

class Post < ActiveRecord::Base
  # [..]
  before_save :generate_body_html

  protected

  def generate_body_html
    return if body.blank?
    markdown = Redcarpet::Markdown.new(
      Redcarpet::Render::HTML.new(:hard_wrap => true),
      :no_intra_emphasis => true,
      :autolink => true,
      :fenced_code_blocks => true)

    self.body_html = Redcarpet::Render::SmartyPants.render(
      SyntaxHighlighter.new(markdown.render(body)).to_s)
  end
end

Basically, a new renderer is instantiated with a variety of my preferred options (fenced_code_blocks being the most important, more on that in a bit). The contents of the post (stored in the body attribute) are then rendered to HTML, passed to the to-be-disclosed SyntaxHighlighter class, and, lastly, the result of that gets the SmartyPants rules applied.

The whole shebang finally gets assigned to the body_html attribute, which will henceforth contain the fully-baked, ready to display HTML code of the post body.

Now, for the meat of it, let’s take a look at the SyntaxHighlighter class.

require 'net/http'
require 'uri'

class SyntaxHighlighter
  PYGMENTS_URI = 'http://pygments.appspot.com/'

  def initialize(html)
    doc = Nokogiri::HTML(html)
    doc.search("pre > code[class]").each do |code|
      request = Net::HTTP.post_form(URI.parse(PYGMENTS_URI), 
                  { 'lang' => code[:class],
                    'code' => code.text.rstrip })
      code.parent.replace request.body
    end
  end

  def to_s
    doc.search("//body").children.to_s
  end
end

I’m using Nokogiri (in my book still the de-facto standard in HTML parsing) to disassemble the post’s HTML structure (remember that we’re passing in a copy of the body that has already been transformed from Markdown into HTML) to find blocks of code wrapped in <pre><code></code></pre> tag pairs, which also have a value assigned to the class attribute of the <code> tag.

And this is where the fenced_code_blocks option to Redcarpet comes in. With this enabled, you can write your code-blocks like this in Markdown:

``` ruby
class Foo
end
```

That means, no more 4-spaces indentation for every line and, more importantly, you can pass along a value for the class attribute to assign to the <code> tag it’s wrapped in, specifying the programming language of the code to highlight.

But back to the code.

Once those code blocks have been located and extracted along with their classes, they’re posted to pygments.appspot.com individually (because you could have multiple code blocks in a post, each in a different programming language) and the result is inserted back into the document structure, replacing the previously extracted code block.

Lastly, the to_s method simply returns the document converted back to a string, which, as you recall, then gets saved in the body_html attribute for later use when displaying a post.

Not to beat a dead horse here, but all of this only ever happens when you create or update a post. It never happens when a post is merely viewed.

For styling, all the default Pygments styles rules apply. If you’re happy with the defaults, Trevor supplies a default.css in his pygments repository. In addition to that, more styles are available all over the web. Also, Favio Manriquez Leon has a way to preview the built-in styles on his blog.

Conclusion

So there you have it. A way to re-use Python’s great syntax highlighting for your Heroku-powered Rails blog in a way that will survive getting fireballed.

1. Sadly, I couldn’t really tell when, as his post doesn’t seem to have a date. But it must’ve been some time after July 2011, since he references a Railscasts episode from that time. ↩

2. Even when it’s being run by the commendable Trevor Turk. ↩

3. Brush up your callbacks knowledge in Rails with this guide ↩

Proper Language Detection on the Web

A couple of days ago I quipped:

Why can’t we agree that polling the preferred language from browsers trumps GeoIP six ways to Sunday? Travelers of the world will thank you.

— Patrick Lenz (@patricklenz) February 10, 2012

Which, apparently, struck some sort of nerve as it got favorited and retweeted above my usual average. And that prompted me to explore this topic a little more from a technical standpoint.

But, first, a little bit of history.

What originally angered me was the fact that some websites insist on offering me a German language experience of their site, despite the fact that I deliberately switch all of my computers, phones, televisions, and vacuum cleaners to use the English language, because, well, that’s how I roll.

The Difference Between Shopping and Information

Shopping sites like Amazon have every right in the world to send me to their local offerings if they detect that I’m coming from a certain country where they have a local presence in. (More on that detection in a bit.)

Yet, they don’t. They simply offer a friendly banner at the top of Amazon.com that informs me of a local offering I might be interested in. (And I am.)

Compare and contrast that with sites like Softonic, which slap me in the face with a modal dialog – requiring an immediate decision on my part – suggesting I use their site in German instead of the version that I chose to access and which matches the language of my browser.

While I appreciate the fact that they may be employing a CDN¹ for speedy downloads from a local server, this is an implementation detail independent from the actual language used on their site, which I don’t really happen to care about to be in the language of my geographic location (as opposed to the language of my browser and operating system).

(And I’m not trying to single out Softonic here. This forced language detection based on geographic location instead of browser language seems to be common practice. Softonic just happened to be the site that prompted the tweet embedded above.)

Odd Birds and Travelers

I may well be the odd bird in this case in that my geographical location does not match the preferred language set in my browser.

But what about travelers?

Everyone traveling abroad is probably in the same boat as I am. Potentially even worse, given that their foreign language skills may not be as fluent and a foreign language version of a site (that is not a shopping site) may be of even less interest.

Technical Hurdles

Detecting the geographic location of a visitor to your site is a non-trivial task. It usually involves some kind of third-party service doing a geolocation lookup, which means a round-trip to at least one other server before you can even render the first page view to your visitor, delaying the often crucial first impression the visitor gets by a couple hundred milliseconds.

While this is a necessary evil for DRM-plagued services such as Netflix and targeted online advertising to use geotargeting as part of their offering, using a geolocation for the display language of your website is just plain overkill.

Why? Because every modern browser sends its language preference (or, more appropriately, that of its user) along with every HTTP request being made in the form of the Accept-Language header that is part of the HTTP/1.1 RFC.

The Accept-Language request-header field is similar to Accept, but restricts the set of natural languages that are preferred as a response to the request.

It’s even possible to supply multiple languages with appropriate weightings:

Each language-range MAY be given an associated quality value which represents an estimate of the user’s preference for the languages specified by that range.

So without round-tripping to (potentially non-free) third-party services in order to find out where your visitor is coming from, you can simply use the Accept-Language header supplied by the visitor in most situations to make an even more informed decision what their language preference may be.

Lead by example

In Rails, the value of the Accept-Language can be found in request.headers[‘HTTP_ACCEPT_LANGUAGE’].

This is, for example, the value supplied by Chrome 17 when set to the English language:

    Accept-Language: en-US,en;q=0.8

If you switch it to German, it’ll send:

    Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4

Basically, this list specifies German as the most desired language, US-English as the middle ground, and, if all else fails, any dialect of English will do.

Other browsers like Firefox and Safari supply similar values for Accept-Language, albeit not as detailed.

Great further reading on the topic of internationalizing a Rails application is the Guide on the Rails I18n API and Iain Hecker’s http_accept_language gem.

If you’d like to use this header in JavaScript and don’t have a way to pass it to the Javascript layer from your app (I heard those cases exist), there’s a helpful jQuery plugin from Dan Singerman called jQuery Browser Language, which gives you access to this header otherwise unavailable to JavaScript.

Conclusion

Don’t force people to use your site or web application in a language they didn’t request just because they happen to sit in a spot on this planet where people usually speak it. Don’t use geotargeting for language detection just because you can. Instead, make use of established practices such as the Accept-Language header to deliver a much better user experience and much more likely matching their expectations.

1. A Content Delivery Network, supposedly serving files from a geographically close location with better latency to your neck of the woods. ↩