Webzero and Postgres
I learned two things yesterday browsing the web idly: Webzero as a name for the minimalist style of the new web applications we are seeing, that so far I’ve just been referring to as “a new Rails app with great UX”: light, easy on the eyes, easy to use, without functionality added by a committe (or the demands of growth long after the software fulfills the needs of 80% of its users)
The other thing: strings, technically varchar columns, in PostgreSQL and in MariaDB only take up storage depending on their length, not the declared maximum size of the column. Having grown up on crufty Oracle databases, I was used to choosing the length of string columns very carefully. Not necessary anymore.
The latter has a teeny tiny Webzero implication: it’s not necessary to annoy the user anymore because you thought 30 characters ought to be enough for anybody’s email, when your user deals with john.jacob.jingleheimer@alltheschmidts.com every day.
Deltacloud Core goes to the Incubator
We’ve been working on it for a while, but today, I finally managed to import the existing code for Deltacloud Core into Apache’s subversion repository. From now on, development of the Deltacloud API will happen within the Apache Incubator. The main reason for this move is that we’ve heard from several people that they really liked the idea of Deltacloud, and the concept of a true open source cloud API, but didn’t like it as a Red Hat project. Apache Incubator is a well-established place for parties with different interests to collaborate on a specific code base, so it seemed the logical place to address these concerns. We’ve also has great success with putting qpid into the Incubator.
The process was a little longer than I would have liked, but there were several administrativa to handle, plus interrupts from vacations and conferences.
But it’s all done now, except for the project page at Apache. Importing the code into subversion proved a little more difficult than I would have liked, but I was very keen on preserving the history of the existing git repository. The two big issues were that the git repo contained several copies of rails that had been committed and later, when we switched to Sinatra, deleted; the master branch also contained a good number of merge commits. The first issue, uploading megabytes of useless history from committing and deleting rails, was easily addressed with git filter-branch. The second was a little hairier, but thanks to the quite incredible possibility of grafting one repository on top of another and this excellent tutorial it went quite smoothly; if you can call resolving lots of merge failures manually smooth.
From now on, all Deltacloud API work will be done in the subversion repository. We’re still waiting for the official readonly git mirror to be set up — for now, you will have to make do with git svn. If you have an existing git checkout, you will unfortunately have to throw that away, and check out from scratch:
mkdir deltacloud
cd deltacloud
git svn init -s https://svn.apache.org/repos/asf/incubator/deltacloud
git svn fetchMailing lists will also change for Deltacloud Core — we’re moving to the lists hosted at Apache, in particular, to deltacloud-dev@incubator.apache.org. Apache lists are managed by ezmlm, so send email to deltacloud-dev-subscribe@incubator.apache.org to subscribe to the list.
REST practices
We have several projects at Red Hat that are currently implementing REST API’s, from . At first, REST is nice and easy, and has a lot of appeal because it’s not SOAP. When you get into the nitty-gritty of designing a REST API though, there are numerous niggling issues; to help address them across all our REST API’s, and simply to have a forum for people who are interested in best practices around REST, we now have a mailing list — if you are interested in designing good REST API’s, join the list and jump in on the discussion; this is a topic that goes way beyond Red Hat’s own applications and web services.
Mark compiled the discussions so far into an excellent guide, a great resource for the issues to look out for when writing a REST API and a good list of possible solutions to those issues.
Quick Tip: Add users to a group with augeas
When you use Augeas to add users to groups, you might do that by first adding a new user node to the corresponding group (we’ll use kvm for the sake of this example), and then set its value to the name of the user:
augtool> ins user after /files/etc/group/kvm/user[last()]
augtool> set /files/etc/group/kvm/user[last()] joeThat works great for groups that already contain at least one user. If there are no users in the group yet, the ins will fail since there is no match for /files/etc/group/kvm/user[last()].
Instead of saying ‘add a new user after the last user’, you need to say ‘add a new user either after the gid field or after the last user’, which turns the above ins command to
augtool> ins user after /files/etc/group/kvm/*[self::gid or self::user][last()]
augtool> set /files/etc/group/kvm/user[last()] joeDeltacloud sings a new tune
A few weeks ago, I came across Sinatra, a minimalist Ruby web framework, much leaner, meaner and simpler than Rails — while it’s probably not a good fit for traditional database-backed web applications, it seemed like an ideal framework for Deltacloud Core.
Deltacloud Core is a cross-cloud API providing abstraction and compatibility across multiple clouds. There are quite a few such API’s out there; what makes Deltacloud unique is that it is not a client-side library tied to a specific language. Instead, it is a RESTful web service, initially written in Rails by Bob McWhirter.
While Rails was a good initial framework for Detlacloud Core, and is the default Ruby web framework for a reason, there’s a few things that make it overkill for a simple webservice:
- Deltacloud Core is stateless, and therefore has no need to store anything in a database; that means we’ll never use Rails’
ActiveRecord - Deltacloud Core isn’t really a MVC app; it’s simply an adapter around existing API’s, and the split of responsibility that MVC implies doesn’t really fit with how an API adapter works.
- Deltacloud Core hooks into a few places in the handling of requests, most importantly to do parameter validation based on the operation being performed in the API. Much easier to do with a simple framework like Sinatra
Of course, the biggest draw of Sinatra is that it concentrates all the controller code in one place (currently ~ 300 lines in server.rb), which makes it much easier to understand how the whole application is put together. More generally, using Sinatra makes it possible to concentrate code in fewer files, increasing readability.
Sinatra Additions
Out of the box, Sinatra fell short of our needs in two areas: RESTful request validation and responding with a variety of content types.
Michal and I took care of request validation with a small custom DSL for describing RESTful operations, cleverly called Rabbit. That DSL lets us talk about collections and operations on collections, and generates validation logic and online documentation on-the-fly. You can browse an instance of Deltacloud Core and immediately see what operations are available, and what parameters a specific operation like create instance accepts. (Why and how that is important is a topic for another blog post)
Deltacloud Core can be used in two ways: you can either point your web browser at it and browse a variety of information about your objects in a specific cloud, or make RESTful requests that return XML responses. That dual nature makes it necessary to respond to some requests with HTML, and to others with XML. Sinatra doesn’t have a mechanism that’s as convenient as Rails’ respond_to for this, though there’s a nice extension that does exactly that, so that we could write listing of a collection, any collection, (model in the code below), simply as
singular = model.to_s.singularize.to_sym
@elements = driver.send(model.to_sym, credentials, filter)
instance_variable_set(:"@#{model}", @elements)
respond_to do |format|
format.xml { return convert_to_xml(singular, @elements) }
format.html { haml :"#{model}/index" }
endThe Sinatra respond_to extension actually didn’t do content negotiation based on the Accept header; we added that based on a couple of existing Rack extensions. And discovered in the process that Chrome’s Accept header is utterly insane.
All in all, the move to Sinatra worked out very well; existing functionality was easy to preserve, and we added validation and self-documentation to Deltacloud Core, both in human-readable format and as (experimental) XML. Next up: more features for the API.
Static Blogging
For a while now, I’ve been unhappy with my blog setup; when I started blogging, the only blog my hosting provider had was b2evolution, a monster of a traditional blogging engine, more suitable for hosting many blogs than just the one I needed. Add to that the fantastico, allegedly a package manager, and you have a blog that’s hard to use and an engine that’s complicated to update, since you’ll invariably make small changes to the stock engine.
I looked around for a while, and toyed with the idea of using a dirt-simple blog engine based on Sinatra like nesta, when I came across static site generators like Jekyll and webby, and finally settled on Jekyll — it’s a little more tuned towards a blog, even though the Liquid templating is something you need to get used to, to put it mildly.
A static site generator suits my needs much better than the general database-backed blogging engines: I can manage my posts in git, I can write them with my favorite text editor in any number of markups (for now, it’s markdown), and the blog can be hosted as a bunch of static web pages. The biggest reason to make a blog dynamic is to enable comments — with a service like Disqus that’s not necessary any longer.
Migration
Since b2evolution is definitely a niche blog engine, there’s few migration scripts out there, and none for migrating to Jekyll. But b2evolution uses MySQL as its database, and armed with an XML dump of the database, and my mad Ruby scripting skills it wasn’t too hard to transform that XML dump into Markdown pages for Jekyll.
Two things were really important in the migration: preserving permalinks, and keeping the Atom feed from spitting out old posts as new ones. I took care of the permalinks with a little script that generates a .htaccess file with permanent Redirect instructions from old permalinks to new ones. The Atom feed is similarly easy to address by adding the old Atom ID’s as additional frontmatter metadata in the Jekyll pages, and preferring that over the Jekyll-generated atom ID’s, which are used for new, non-migrated posts.
Deltacloud and Eucalyptus
The Eucalyptus folks yesterday gave the Deltacloud API a spin against a Eucalyptus instance. No surprises there, seeing how Eucalyptus strives for compatibility with the EC2 API — Deltacloud API just works. All you need to do is use the Deltacloud EC2 driver and make sure it has the EC2_URL environment variable set and pointing to your Eucalyptus instance.
