Watzmann.Blog2014-06-13T16:29:09-07:00http://watzmann.net/David Lutterkortlutter@watzmann.netUsing Puppet's policy-based autosigning2014-06-13T00:00:00-07:002014-06-13T00:00:00-07:00http://watzmann.net//blog/2014/06/puppet-autosign-policy<p>Handling SSL certificates is not a lot of fun, and while Puppet’s use of
client certificates protects the server and all its deep, dark secrets very
well from rogue clients, it also leads to a lot of frustration. In many
cases, users would configure their <code>autosign.conf</code> to allow any (or almost
any) client’s certificate to be signed automatically, which isn’t exactly
great for security. Since Puppet 3.4.0, it is possible to use
<a href="http://docs.puppetlabs.com/puppet/latest/reference/ssl_autosign.html#policy-based-autosigning">policy-based autosigning</a>
to have much more control over autosigning, and to do that in a much more
secure manner than the old autosigning based solely on client’s hostnames.</p>
<p>One of the uses for this is automatically providing certificates to
instances in EC2. Chris Barker
<a href="https://github.com/mrzarquon/mrzarquon-certsigner">wrote a nice module</a>,
based on a <a href="https://gist.github.com/jbouse/8763661">gist</a> by
<a href="https://github.com/jbouse">Jeremy Bouse</a> that uses policy-based
autosigning to provide EC2 instances with certificates, based on their
<code>instance_id</code>.</p>
<p>I recently got curious, and wanted to use that same mechanism but with
preshared keys. Here’s a quick step-by-step guide of what I had to do:</p>
<h2 id="the-autosign-script">The autosign script</h2>
<p>When you set <code>autosign</code> in <code>puppet.conf</code> to point at a script, Puppet will
call that script every time a client request a certificate with the
client’s certname as the sole command line argument of the script and the
CSR on stdin. If the script exits successfully, Puppet will sign the
certificate, and refuse to sign it otherwise.</p>
<p>On the master, we’ll maintain a directory <code>/etc/puppet/autosign/psk</code>; files
in that directory must have the certname of the client and contain the
preshared key.</p>
<p>Here is the <code>autosign-psk</code> script; the OID’s for Puppet-specific
certificate extensions can be found
<a href="http://docs.puppetlabs.com/puppet/latest/reference/ssl_attributes_extensions.html#recommended-oids-for-extensions">here</a>:</p>
<pre><code>#! /bin/bash
PSK_DIR=/etc/puppet/autosign/psk
csr=$(< /dev/stdin)
certname=$1
# Get the certificate extension with OID $1 from the csr
function extension {
echo "$csr" | openssl req -noout -text | fgrep -A1 "$1" | tail -n 1 \
| sed -e 's/^ *//;s/ *$//'
}
psk=$(extension '1.3.6.1.4.1.34380.1.1.4')
echo "autosign $1 with PSK $psk"
psk_file=$PSK_DIR/$certname
if [ -f "$psk_file" ]; then
if grep -q "$psk" "$psk_file"; then
exit 0
else
echo "File for '$psk' does not contain '$certname'"
exit 1
fi
else
echo "Could not find PSK file for $certname"
exit 1
fi
</code></pre>
<h2 id="puppet-master-setup">Puppet master setup</h2>
<p>On the Puppet master, we put the above script into
<code>/usr/local/bin/autosign-psk</code>, make it world-executable, and point
<code>autosign</code> at it:</p>
<pre><code>cp somewhere/autosign-psk /usr/local/bin
chmod a+x /usr/local/bin/autosign-psk
mkdir -p /etc/puppet/autosign/psk
puppet config set --section master autosign /usr/local/bin/autosign-psk
</code></pre>
<p>A PSK for client <code>$clientname</code> can easily be generated with</p>
<pre><code>tr -cd 'a-f0-9' < /dev/urandom | head -c 32 >/etc/puppet/autosign/psk/$certname
</code></pre>
<h2 id="puppet-agent-setup">Puppet agent setup</h2>
<p>On the agent, we create the file <code>/etc/puppet/csr_attributes.yaml</code> with the
PSK in it:</p>
<pre><code>---
extension_requests:
pp_preshared_key: @the_psk@
</code></pre>
<p>With all that in place, we can now run the Puppet agent and have it get its
certificate automatically; that process is as secure as we keep the
preshared key.</p>
Don't hate the HATEOS2012-12-20T00:00:00-08:002012-12-20T00:00:00-08:00http://watzmann.net//blog/2012/12/dont-hate-the-hateos<p><a href="http://37signals.com/svn/writers/dhh">DHH</a> has a
<a href="http://37signals.com/svn/posts/3373-getting-hyper-about-hypermedia-apis">post</a>
on some of the hoopla around hypermedia API’s over at
<a href="http://37signals.com/svn/">SvN</a>, complete with a cool picture of the
<em>WS-*</em>. While I agree with most of his points, he’s missing the larger
point of API discoverability.</p>
<p>The reason discoverability is front and center in RESTful API’s isn’t some
naive belief that the semantics of the API will just magically be
discovered by the client — instead, it’s a strategy to keep logic
that belongs on the server out of clients. When a client is told that they
have to discover the URL for posting a comment to an article, they are also
told to prepare that that operation might not be available. There are lots
of reasons why that operation may not be possible for the client; none of
them need to interest the client, all it cares about is whether that
operation is advertised in the article or not.</p>
<p>DHH also puts up a nice strawman, and then ceremoniously burns it to the
ground:</p>
<blockquote>
<p>The idea that you can write one client to access multiple different APIs
in any meaningful way disregards the idea that different apps do
different things.</p>
</blockquote>
<p>Again, that misses the point, especially of discoverability. Not every API
has exactly one deployment. Many clients need to work with multiple
different deployments of the same API; the
<a href="http://deltacloud.apache.org/">Deltacloud API</a> is a good example of how
discoverability lays down clear guidelines for clients on what they can
assume, and what they have to be prepared for being different with each
different endpoint they want to talk to. You can look at that as making the
contract between server and client explicit in the API. Discoverability
makes conditional promises to the client: if you see X, you may safely do
Y.</p>
<p>We are all in agreement though that overall we want to tread very lightly
when it comes to standardizing API mechanisms - I think there are some
areas around RESTful API’s where some carefully crafted standards might
help, but staying out of range of the <em>WS-*</em> is much more important.</p>
CIMI v1.0 released2012-08-29T00:00:00-07:002012-08-29T00:00:00-07:00http://watzmann.net//blog/2012/08/cimi-released<p>This morning, the <a href="http://dmtf.org/">DMTF</a> officially
<a href="http://is.gd/wmTCAQ">announced</a> the availability of
<a href="http://dmtf.org/standards/cloud">CIMI v1.0</a>. After two years of hard work,
heated discussions, and many a vote on proposed changes, CIMI is the best
shot the fragmented, confusing, and in places legally encumbered, landscape
of IaaS API’s has at a universally supported API. Not just because of the
impressive number of industry players that are part of the working group
but also because it has been designed from the ground up as a modular
<a href="http://en.wikipedia.org/wiki/Representational_state_transfer">RESTful API</a>,
taking the breadth of existing IaaS API’s into account.</p>
<p>While the name suggests that CIMI is 75% CIM, the two have actually no
relation to each other, except that they are both DMTF standards. CIMI
covers most of the familiar concepts from IaaS management: instances
(called machines), block storage (volumes), images, and networks. The
standard itself is big, though most of the features in it are optional, and
I don’t expect that any one provider will support everything mentioned in
the standard. To get started, I highly recommend reading the
<a href="http://dmtf.org/sites/default/files/standards/documents/DSP2027_1.0.0.pdf">primer</a>
first, as a gentle introduction to how CIMI views the world and how it
models common IaaS concepts. The
<a href="http://dmtf.org/sites/default/files/standards/documents/DSP0263_1.0.0.pdf">standard</a>
itself then serves as a convenient reference to fill in the details.</p>
<p>One of the goals of CIMI is that providers with widely varying feature sets
can implement it, and it therefore puts a lot of emphasis on making what
exactly a provider supports discoverable, using the
<a href="http://en.wikipedia.org/wiki/HATEOAS">well-known mechanisms</a> that a
<a href="http://fedoraproject.org/wiki/Cloud_APIs_REST_Style_Guide">RESTful style makes possible</a>
, and that we’ve also
<a href="http://deltacloud.apache.org/rest-api.html">used in the Deltacloud API</a> to
expose as much of each backend’s features as possible. This emphasis on
discoverability is one of the things that sets CIMI apart from the popular
vendor-specific API’s, where the API has to be implemented in its entirety,
or not at all.</p>
<p>We’ve been involved in the working group for the last two years, bringing
our experience in designing <a href="http://deltacloud.apache.org/">Deltacloud</a> to
the table. We’ve also been busy adding various pieces to Deltacloud, and
that implementation experience has been invaluable in the CIMI
discussion. We’ll continue to improve our CIMI support, and build out what
we have; in particular, we are working on</p>
<ul>
<li>the CIMI frontend for Deltacloud; when you run <code>deltacloudd -f cimi</code>, you
get a server that speaks CIMI, with the antrypoint at
<code>/cimi/cloudEntryPoint</code>. You can try out the latest code at
<code>https://dev.deltacloud.org/cimi/cloudEntryPoint</code></li>
<li>the CIMI client app (in <code>clients/cimi/</code> in our
<a href="https://github.com/apache/deltacloud">git repo</a> — the app makes it
both easier to experiment with the CIMI API, and serves as an example of
CIMI client code.</li>
<li>a CIMI test suite; as part of our test suites, we are adding tests that
can be run against any CIMI implementation and will eventually be a
useful tool to informally qualify such implementations</li>
</ul>
<p>As with all open source projects, we always have way more on the todo list
than we actually have time to do. If you are interested in contributing to
Deltacloud’s CIMI effort, have a look at
<a href="http://deltacloud.apache.org/how-to-contribute.html">our Contribute page</a>,
stop by the <a href="http://deltacloud.apache.org/contact.html">mailing list</a>, or
drop into our IRC channel <code>#deltacloud</code> on freenode.</p>
Evolution for REST API's2012-08-03T00:00:00-07:002012-08-03T00:00:00-07:00http://watzmann.net//blog/2012/08/rest-api-evolution<p>Like everything,
<a href="http://en.wikipedia.org/wiki/Representational_State_Transfer">REST</a> API’s
change over time. An important question is how these changes should be
incorporated into your API, and how your clients should behave to survive
that evolution.</p>
<p>The first reflex of anybody who’s thought about API’s and their evolution
is to stick a version number on the API, and use that to signal to clients
what capabilities this incarnation of the API has, and maybe even let
clients use that to negotiate how they talk to the
server. <a href="http://www.mnot.net/blog/2011/10/25/web_api_versioning_smackdown">Mark</a>
has a very good post explaining why, for the Web, that is not just
undesirable, but often not feasible.</p>
<p>If versioning is out, what else can be done to safely evolve REST API’s ?
Before we dive into specific examples, it’s useful to recall what our
overriding goal is. Since it is much easier to update a server than all the
clients that might talk to it, the fundamental aim of careful evolution of
REST API’s is:</p>
<blockquote>
<p>Old clients must work against new servers</p>
</blockquote>
<p>To make this maxim practical, clients need to follow the simple rule:</p>
<blockquote>
<p>Ignore any unexpected data in the interaction with the server</p>
</blockquote>
<p>In particular, clients can never assume that they have a complete picture
of what they will find in a response from the server.</p>
<p>Let’s look at a little toy API to make these ideas more tangible, and to
explore how this API can change while adhering to these rules. The API is
for a simplistic blogging application that allows posting articles, and
retrieveing them. For the sake of simplicity, I will omit all HTTP request
and response headers.</p>
<h3 id="a-simple-rest-api">A simple REST API</h3>
<p>In sticking with good
<a href="http://fedoraproject.org/wiki/Cloud_APIs_REST_Style_Guide">REST practice</a>,
the API has a single entrypoint at <code>/api</code>. Issuing a <code>GET /api</code> will result
in the response</p>
<pre><code><api>
<link rel="articles" href="/api/articles"/>
</api>
</code></pre>
<p>The articles collection can be retrieved with a <code>GET /api/articles</code>:</p>
<pre><code><articles>
<article href="/api/articles/1">
<title>Evolution for REST API's</title>
<content>
Like everything, ....
</content>
</article>
<article href="/api/articles/2">
...
</article>
<actions>
<link rel="create" href="/api/articles"/>
</actions>
</articles>
</code></pre>
<p>Each article consists of a title and some content; the <code>href</code> on each article
gives clients the URL from which they can retrieve that article, and serves as
a unique identifier for the article.</p>
<p>The actions element in the articles collection tell the client that they can
create new articles by issuing <code>POST</code> requests to <code>/api/articles</code>:</p>
<pre><code><article>
<title>How to version REST API's</title>
<content>...</content>
</article>
</code></pre>
<p>It’s worth pointing out a subtlety in including a link for the <code>create</code>
action: one reason for including that link is to tell clients the URL to
which they can <code>POST</code> to create new articles, and keep them from making
assumptions about the URL space of the server. A more important reason
though is that we use the presence of this link to communicate to the
client that it may post new articles. This, following the
<a href="http://en.wikipedia.org/wiki/HATEOAS">HATEOS</a> constraint for REST API’s,
is the more important reason to include an explicit link: clients should
not even assume that they are allowed to create new articles.</p>
<h4 id="adding-information-from-the-server">Adding information from the server</h4>
<p>Readers might want to know when a particular article has been made
available. We therefore add a <code>published</code> attribute to the representation
of articles that a <code>GET</code> on the articles collection or on an individual
article’s URI returns:</p>
<pre><code><article href="/api/articles/2">
<title>How to version REST API's</title>
<content>...</content>
<published>2012-08-03T13:00</published>
</article>
</code></pre>
<p>This does not break old clients, because we told them to ignore things they
do not know about. A client that only knows about the previous version of
our API will still work fine, it just won’t do anything with the
<code>published</code> element.</p>
<h4 id="allowing-more-data-when-creating-an-article">Allowing more data when creating an article</h4>
<p>Some articles might be related to other resources on the web, and we’d want
to let authors call them out explicitly in their articles. We therefore change
the API to accept articles with some additional data on <code>POST
/api/articles</code>:</p>
<pre><code><article>
<title>Great REST resources</title>
<content>...</content>
<related>
<link rel="background" href="http://en.wikipedia.org/wiki/Representational_state_transfer"/>
<link rel="background" href="http://en.wikipedia.org/wiki/HATEOAS"/>
</related>
</article>
</code></pre>
<p>As long as our new API allows posting of articles without any <code>related</code>
links, old clients will continue to work.</p>
<h4 id="blogging-apis-everywhere">Blogging API’s everywhere</h4>
<p>If our blogging software is so successful that clients must be prepared to
deal with both servers that support adding related reosurces, and ones that
do not, we need a way to indicate that to those clients that know about
related resources. While there are many ways to do that, one that we’ve
found works well for <a href="http://deltacloud.org/">Deltacloud</a> is annotating the
collections in the toplevel API entrypoint. When a client does a <code>GET /api</code>
from a server that supports related resources, we’d send them the following
XML back:</p>
<pre><code><api>
<link rel="posts" href="/api/posts">
<feature name="related_resources"/>
</link>
</api>
</code></pre>
<h4 id="updating-articles">Updating articles</h4>
<p>Authors want to revise their articles from time to time; we’d make that
possible by allowing them to <code>PUT</code> the updated version of an article to its
URL. This won’t introduce any problems for old clients, but new clients
will need to know whether the particular instance of the API they are
talking to supports updating articles. We’d solve that by adding <code>actions</code>
to the article itself, so that a <code>GET</code> of an article or the articles
collection will return</p>
<pre><code><article href="/api/posts/42"/>
<title>...</title>
...
<actions>
<link rel="update" href="/api/posts/42"/>
</actions>
</article>
</code></pre>
<p>Not only does the <code>update</code> link tell clients that they are talking to a
version of the blogging API that supports updates, it also lets us hide
complicated business logic that decides whether an article can be updated
or not by simply showing or suppressing the <code>update</code> link.</p>
<h4 id="merging-blogs">Merging blogs</h4>
<p>Because of its spectacular content, our blog has been so successful that
we want to turn it from a personal blog into a group blog, supporting
multiple authors. That of course calls for adding the name of each author
(or their avatar or whatnot) to each post — in other words, we want
to make passing in an author mandatory when creating or updating an
article. Rather than break old clients by silently slipping in the author
requirement, we add a new action to the articles collection:</p>
<pre><code><articles>
<post>...</post>
<actions>
<link rel="create_with_author" href="/api/articles_with_author"/>
...
</actions>
</articles>
</code></pre>
<p>Old clients will ignore that new action; the remaining question is if we
can still allow old clients to post new articles. If we can, for example,
by defining a default author out-of-band with this API, we’d still show the
old <code>create</code> action in the articles collection. If not, we’d take the
ability to post away from old clients by not displaying the <code>create</code> action
anymore — but we haven’t broken them, since they can still continue
to retrieve posts, we’ve merely degraded them to readonly clients.</p>
<p>While this seems like an extreme change, consider that we’ve changed our
application so much that existing clients can simply not provide the data
we deem necessary for a successful post. It’s much more realistic that we’d
find a way to let old clients still post articles using the old <code>create</code>
link.</p>
<h3 id="some-consequences-for-xml">Some consequences for XML</h3>
<p>There are two representations that are popular with REST API’s: JSON and
XML. The latter poses an additional challenge for the evolution of REST
API’s because the use of XML in REST API’s differs subtly from that in many
other places. Since clients can never be sure that they know about
everything that might be in a server’s response, it is not possible to
write down a <a href="http://www.w3.org/XML/Schema">schema</a> (or
<a href="http://relaxng.org/">RelaxNG</a> grammar) that the client could use to
validate server responses, since responses from an updated server would
violate that schema, as the simple example of adding a <code>published</code> date to
articles above shows.</p>
<p>It’s of course possible to write down RelaxNG grammars for a specific
version of the API, but they are tied to that specific version, and must
therefore be ignored by clients who want to happily evolve with the server.</p>
<h3 id="questions-">Questions ?</h3>
<p>I’ve tried to cover all the different scenarios that one encounters when
evolving a RESTful API — I’ve left out HTTP specific issues like
status codes (must never change) and headers (adding new optional headers
is ok) as the Openstack folks have decided for their
<a href="http://wiki.openstack.org/APIChangeGuidelines">API Change Guidelines</a>.</p>
<p>I’d be very curious to hear about changes that can not be addressed by one
of the mechanisms described above.</p>
Deltacloud 1.02012-06-15T00:00:00-07:002012-06-15T00:00:00-07:00http://watzmann.net//blog/2012/06/deltacloud-on-dot-oh<p>The upcoming release of Deltacloud 1.0 is a huge milestone for the project:
even though no sausages were hurt in its making, it is still chockful of
the broadest blend of the finest IaaS API ingredients. The changes and
improvements are too numerous to list in detail, but it is worth
highlighting some of them. TL;DR: the
<a href="http://people.apache.org/~lutter/deltacloud/1.0.0/rc1/">release candidate</a>
is available now.</p>
<h2 id="ec2-frontend">EC2 frontend</h2>
<p>With this release, Deltacloud moves another step towards being a universal
cloud IaaS API proxy: when we started adding support for
<a href="http://dmtf.org/cloud">DMTF CIMI</a> as an alternative to the ‘classic’
Deltacloud API, it became apparent that adding additional frontends could
be done with very little efforts. The new
<a href="http://mifo.sk/deltacloud-with-ec2-frontend">EC2 frontend</a> proves that
this is even possible for API’s that are not RESTful. With that, Deltacloud
allows clients that only know the EC2 API to talk to various backends,
including <a href="http://openstack.org/">OpenStack</a>, vSphere, and
<a href="http://ovirt.org/">oVirt</a>.</p>
<p>The EC2 frontend supports the most commonly needed operations, in
particular those necessary for finding an image, launching an instance off
of it and managing that instance’s lifecycle. In addition, managing SSH key
pairs is also supported. We hope to grow the coverage of the API in future
releases to the point where the EC2 frontend is good enough to support the
majority of uses.</p>
<p>The debate around the ‘right’ cloud IaaS API is heated and continues,
especially around standards, and we still see the right answer to this
debate in a properly standardized, broadly supported, and openly governed
API such as DMTF’s CIMI — yet it is undeniable that EC2 is the front
runner in this space, and that large investments into EC2’s API exist; it
is Deltacloud’s mission to alleviate the resulting lockin, and the addition
of the EC2 frontend allows users to experiment with different backend
technologies while migrating off the EC2 API on their own pace.</p>
<p>One issue that the EC2 frontend brings to the forefront is just how
unsuitable that API is for fronting different backend implementations: IaaS
API’s that are designed for this purpose provide extensive capabilities for
client discovery of various features. EC2 on the other hand provides no way
for providers to advertise their deviation from EC2’s feature set, and no
possibilities for clients to discover them.</p>
<h2 id="cimi-frontend">CIMI frontend</h2>
<p>We continue our quest to support the fledgling CIMI standard as broadly and
as faithfully as possible. With this release, we introduce support for the
CIMI networking API; for now only for the Mock driver, but we are looking
to expand backend support for networking as clouds add the needed features
for them.</p>
<p>Besides the core CIMI API, which is purely a RESTful XML and JSON API, work
also continues on the simple human-consumable HTML interface for it; we’ve
learned from designing the Deltacloud API and helping others using that
API, that a web application that stays close to the API, but is easy to use
for humans is an invaluable tool. With this release, that application can
now talk to OpenStack, RHEV-M/oVirt, and EC2 via Deltacloud’s CIMI proxy.</p>
<h2 id="operational-and-code-enhancements">Operational and code enhancements</h2>
<p>With three frontends, it’s become even more urgent that the three frontends
can be run from the same server instance to reduce the number of daemons
that need to be babysat. Thanks to an extensive revamp of the guts of
Deltacloud to turn it into a
<a href="http://www.sinatrarb.com/intro#Modular%20vs.%20Classic%20Style">modular Sinatra app</a>
it is now possible to expose all three frontends (or only one or two) from
the same server.</p>
<p>We now also base our RESTful routes and controllers on
<a href="https://github.com/mifo/sinatra-rabbit">sinatra-rabbit</a> — only
fitting since sinatra-rabbit started life as the DSL we used inside
Deltacloud for our RESTful routing and our controllers.</p>
<p>A lot of work has gone into rationalizing the HTTP status codes that
Deltacloud returns, especially when errors occur; in the process, we
learned quite a bit about just how fickle and moody vSphere can be.</p>
<p>Other drivers have seen major updates, not least of which the OpenStack
driver, which now works against the OpenStack v2.0 API; in particular, it
works against the <a href="http://hpcloud.com">HP cloud</a> — with the EC2
frontend, Deltacloud provides a capable EC2 proxy for OpenStack. We’ve also
added a driver for the
<a href="http://www.fujitsu.com/global/solutions/cloud/solutions/global-cloud-platform/">Fujitsu Global Cloud Platform</a>,
which was mostly written by Dies Köper of Fujitsu.</p>
<p>The
<a href="http://people.apache.org/~lutter/deltacloud/1.0.0/rc1/">release candidate</a>
for version 1.0.0 is out now, packages for rubygems.org, Fedora and other
distributions will appear as soon as the release has passed the vote on the
mailing list.</p>
Sinatra Rabbit - a RESTful DSL2012-03-13T00:00:00-07:002012-03-13T00:00:00-07:00http://watzmann.net//blog/2012/03/sinatra-rabbit<p>TL;DR: have a look at
<a href="https://github.com/mifo/sinatra-rabbit">sinatra-rabbit</a>.</p>
<p>When we converted <a href="http://deltacloud.apache.org">Deltacloud</a> from
<a href="http://rubyonrails.org/">Rails</a> to <a href="http://www.sinatrarb.com/">Sinatra</a>,
we needed a way to conveniently write the controller logic for RESTful
routes with Sinatra. On a lark, I cooked up a DSL called ‘Rabbit’ that lets
you write things like</p>
<pre><code>collection :images do
description "The collection of images"
operation :index do
description "List all images"
param :id, :string
param :architecture, :string, :optional
control { ... controller code ... }
end
operation :show do
description 'Show an image identified by "id" parameter.'
param :id, :string, :required
control { ... show image params[:id] ... }
end
operation :destroy do
description "Remove specified"
param :id, :string, :required
control do
driver.destroy_image(credentials, params[:id])
status 204
respond_to do |format|
format.xml
format.json
format.html { redirect(images_url) }
end
end
end
end
</code></pre>
<p>That makes supporting the common REST operations convenient, and allows us
to auto-generate documentation for the REST API. It has been very useful in
writing the
<a href="https://github.com/apache/deltacloud/blob/master/server/lib/cimi/server.rb">two</a>
<a href="https://github.com/apache/deltacloud/blob/master/server/lib/deltacloud/server.rb">frontends</a>
for Deltacloud.</p>
<p>The DSL has lots of features, for example, validation of input parameters,
conditionally allowing additional parameters, describing subcollections,
autogenerating <code>HEAD</code> and <code>OPTIONS</code> routes and controllers, and many more.</p>
<p><a href="http://mifo.sk/">Michal Fojtik</a> has pulled that code out of Deltacloud and
extracted it into its own github project as
<a href="https://github.com/mifo/sinatra-rabbit">sinatra-rabbit</a> In the process,
there were quite a few dragones to slay: for example, in Deltacloud we
change what parameters some operations can accept based on the specific
backend driver. For example, in some clouds, it is possible to inject
user-defined data into instances upon launch. In Deltacloud, the logic of
what routes to turn on or off is based on introspecting the current driver,
which means that Deltacloud’s Rabbit knows about drivers. That, of course,
has to be changed for the standalone <code>sinatra-rabbit</code>. Michal just added
route conditions that look like</p>
<pre><code>collection :images do
operation :create, :if => lambda { complicated_condition(request) } do
...
end
end
</code></pre>
<p>Hopefully, <code>sinatra-rabbit</code> will grow to the point where we can remove our
bundled implementation from Deltacloud, and use the standalone version;
there’s still a couple of features missing, but with enough people
sending patches, it can’t be very long now ;)</p>
Public Deltacloud Instances2011-10-13T00:00:00-07:002011-10-13T00:00:00-07:00http://watzmann.net//blog/2011/10/deltacloud-public-instances<p>Installing Deltacloud is work. Not a lot of work, in fact it is
<a href="http://incubator.apache.org/deltacloud/drivers.html#h1">very easy</a>, but it
still involves installing a package/gem and starting a server. For simple
development and test uses, even that is not necessary any more.</p>
<p>There’s two of them: one,
<a href="https://api.deltacloud.org/">https://api.deltacloud.org/</a> runs the latest
stable release, currently release
<a href="http://www.apache.org/dyn/closer.cgi?path=incubator/deltacloud/">0.4.1</a>. The
other, <a href="https://dev.deltacloud.org/">https://dev.deltacloud.org/</a>, runs the
bleeding edge code from the
<a href="https://github.com/apache/deltacloud">git repo</a>.</p>
<p>Both use the same self-signed SSL certificate. Its SHA-1 fingerprint is
<code>D3:3D:13:73:37:88:59:F1:FE:08:51:70:A0:BA:60:99:F1:E9:DD:45</code>.</p>
<p>If you’ve been scratching your head, wondering what all this Deltacloud
business is about, just head over to one of these two servers and explore
the API. There’s a friendly HTML interface for just that, or, of course,
the obligatory XML and JSON, variants. The public servers run the EC2
driver as their default; when prompted for a username and password, just
enter your Amazon AWS access key ID and secret.</p>
Deltacloud 0.4.0 released2011-09-14T00:00:00-07:002011-09-14T00:00:00-07:00http://watzmann.net//blog/2011/09/deltacloud-0.4.0<p>We just released <a href="http://deltacloud.org/">Apache Deltacloud</a>
<a href="http://incubator.apache.org/deltacloud/download.html">0.4.0</a>, part of the
<a href="http://incubator.apache.org/">Apache Incubator</a>. The release contains a
huge number of enhancements and additions. The full list can be found in
<a href="http://www.mail-archive.com/deltacloud-dev@incubator.apache.org/msg02564.html">the release announcement</a>,
but some of them bear highlighting separately.</p>
<p>The biggest new feature is probably a driver for
<a href="http://www.vmware.com/products/vsphere/overview.html">VMWare’s vSphere</a>. This
makes it possible to turn any vSphere 4.0 installation into a simple
cloud. To use the driver against a vSphere API at
<code>https://vsphere.example.com/sdk</code>, start the Deltacloud server with</p>
<pre><code>API_PROVIDER=vsphere.example.com deltacloudd -i vsphere -t 200
</code></pre>
<p>Besides the basics of image and instance management, the driver supports a
few nifty features, in particular injection of user data.</p>
<p>A second new driver adds support for ‘condor-cloud’, a simple cloud
implementation that uses the
<a href="http://www.cs.wisc.edu/condor/">Condor grid manager</a> as its backend. While
it’s probably not enough to replicate EC2, it is certainly good enough to
build a simple cloud out of a few machines, Condor, Deltacloud, and a few
glue scripts.</p>
<p>We (well, <a href="http://www.mariosandreou.com/">Marios</a>) added support for
firewalls to the Deltacloud API. Since not all clouds offer firewalling,
this is currently only supported for EC2 and Eucalyptus, via their security
groups. The model that the API exposes though represents a fairly generic
firewall with rules and sets of rules. We will expand that to other drivers
in future releases.</p>
<p>We (well, <a href="http://mifo.sk/">Michal</a>) reworked the entire HTML UI using
<a href="http://jquerymobile.com/">jquery-mobile</a> — if you want to explore the
Deltacloud API from your smartphone or tablet, that just got a whole lot
easier. I find the result a much cleaner UI; since the Deltacloud HTML
interface follows the API very closely, as it is meant as a tool to test
and explore the API, pages have always been fairly sparse, something that
the mobile interface makes less annoying.</p>
<p>For more details, read
<a href="http://www.mail-archive.com/deltacloud-dev@incubator.apache.org/msg02564.html">the release announcement</a>
or <a href="http://incubator.apache.org/deltacloud/download.html">download the release</a></p>
Shuttleworth declares goat rodeo over, picks AWS as winner2011-09-12T00:00:00-07:002011-09-12T00:00:00-07:00http://watzmann.net//blog/2011/09/goat-rodeo-over<p>As we all know by now, cloud computing is a veritable
<a href="http://unplugged.rcrwireless.com/index.php/20110908/mobile-software/10428/redhat-calls-the-cloud-a-goat-rodeo/">goat rodeo</a>,
an unseemly sight for anybody’s stomach. Disconcerted by these proceedings,
<a href="http://www.markshuttleworth.com/">Mark Shuttleworth</a> lets his stomach have
the better of him, and
<a href="http://www.markshuttleworth.com/archives/765">declares it over</a> by picking
the winner. That, of course, is not how you end a goat rodeo: you end it by
pointing out that there are goats, not horses, involved.</p>
<p>The goat in question is the undisputable fact that, in North America,
Amazon’s Web Services are the front runner in the IaaS cloud
computing space. Which makes Amazon’s API the most used and most widely
studied IaaS API. Mark’s recommendation for the OpenStack project, which
currently has multiple API’s, but should, for a number of reasons, settle
on one is to just adopt the AWS API. And does this with arguments that are
simultaneously breathtaking and misleading. In particular, the discussion
of HTTP belittles how crucial it was that the definition and evolution of
HTTP was <em>not</em> controlled by a single vendor.</p>
<p>I completely agree with Mark that whether the AWS API is a good or bad API
is completely besides the point for this discussion — where we disagree
is his assertion that API’s aren’t a place where meaningful innovation is
going to happen. Anybody who has followed the evolution of the API’s of any
single cloud provider, let alone the wider cloud ecosystem, will
immediately see the folly of this assertion. And there is no reason to
assume that this API evolution will stop any time soon, there are still way
too many important debates on modeling familiar concepts for cloud, and on
streamlining cloud usage to be had.</p>
<p>The big problem with adopting the AWS API wholesale though is that it puts
the future of OpenStack, and if Mark had his way, the whole IaaS space,
into the hands of a single vendor. There is absolutely no way for anybody
to get a feature into that API, unless they can get Amazon to agree with
them and implement it in their version. Mark addresses this concern with a
brusque</p>
<blockquote>
<p>It’s true that those API’s would better be defined in a clean,
independent forum analogous to the W3C than inside the boiler-room of
development at any single cloud provider, but that’s a secondary
issue. And over time, it can be engineered to work that way.</p>
</blockquote>
<p>This is not a matter of ‘better’ or ‘cleaner’ — it is a question of how
we, as a community, want to see the world of cloud to shape up. Especially
from a North American viewpoint, that landscape looks disheartening right
now, as if IaaS cloud will sooner or later be the domain of two or three
large vendors; that is not the case in Europe, nor does it have to be the
future. The much-used analogy with the early ISP market comes to mind, with
AOL’s seemingly unstoppable march to world dominance. One of the reasons
this never happened and we all can choose from a large number of ISP’s
today of course is the dogged pursuit of truly open standards.</p>
<p>Sticking to one vendor’s offering is the exact opposite of an open
standard; and there is no reason to believe that Amazon can be engineered
into a more open process for API innovation. For one, I am not aware of a
single standards effort, be it through an SDO or an open source project,
that Amazon is involved in. Adopting AWS API’s wholesale and widely will
reduce the incentive for Amazon to join any open API effort, not increase
it.</p>
<p>Amazon’s API’s should be studied because Amazon is a pioneer, a first-mover
in the IaaS space, who were in many cases the first ones to think through
specific issues. Whatever the technical merits, their lack of participation
in open processes makes their API a dangerous bet, no matter how benign
they might be today.</p>
<p>P.S.: There’s some talk of <a href="http://deltacloud.org/">Deltacloud</a> in the
<a href="http://www.markshuttleworth.com/archives/765#comment-368605">comments</a>;
the analogy with JDBC is completely oblique, since we are talking about
loosely coupled web services here, and not tightly coupled in-process
API’s. Which also makes the second point, that Deltacloud is and has to be
a lowest common denominator, moot. There is enough flexibility in a RESTful
API to avoid that. Interestingly though, the comment directly contradicts
Mark’s assertion that API’s aren’t a place to innovate — certainly, there
is <em>some</em> innovation in all these different models.</p>
My git workflow2011-09-08T00:00:00-07:002011-09-08T00:00:00-07:00http://watzmann.net//blog/2011/09/git-workflow<p>Somehow, I find myself writing the same email to introduce people to git
over and over again. But no more ! Now, I will only send out links to this
blog entry.</p>
<p><a href="http://git-scm.com/">Git</a> can be intimidating at first, even though it is
probably the most forgiving source control system out there. There are
plenty of great <a href="http://git-scm.com/documentation">tutorials</a> on git these
days. I assume that you have read enough of them to understand the absolute
basics of cloning a repo, pulling to update your repo, and how to
commit. In other words, you’ve run
<a href="http://www.kernel.org/pub/software/scm/git/docs/git-clone.html"><code>git clone</code></a>,
<a href="http://www.kernel.org/pub/software/scm/git/docs/git-pull.html"><code>git pull</code></a>
and
<a href="http://www.kernel.org/pub/software/scm/git/docs/git-commit.html"><code>git commit</code></a>
before.</p>
<p>The key to working with git happily is to use
<a href="http://progit.org/book/ch3-0.html">branches</a> liberally; if in doubt,
branch. And the key to working with branches is understanding <code>git rebase</code>.
In particular, <code>git rebase -i</code> will make you fall in love with git. It
lets you not just edit committed patches, it also lets you combine patches,
reorganize them etc. Once the initial excitement over interactive rebase
wanes, try out interactive add (<code>git add -i</code>) to renew the bliss.</p>
<p>Once the basics are out of the way, you will want to implement some
extension to whatever you’ve cloned and pulled, and then submit that back
upstream for inclusion. That usually involves working on your own for a
bit, and then generating and sending out patches of your work for review
and merging upstream. Changes you make should always go onto private
(‘topic’) branches; create a new branch for each piece of distinct
work. The overall workflow for this is</p>
<pre><code> git checkout master
git pull # make sure we have the latest bits
git checkout -b dev/feature
... edit/add/commit until happy, with an eye towards having your
branch constitute an easily reviewable patch series; when
working on the branch for longer, pull master repeatedly and
rebase your branch ...
</code></pre>
<p>Once your work is ready to be shared with the rest of the world, do the
following to generate and mail out patches</p>
<pre><code> git checkout master
git pull
git rebase master dev/feature
git format-patch -o /tmp/patches master
git send-email --to=hackers@example.org --compose --subject 'Awesome feature' --thread /tmp/patches
</code></pre>
<p>When changes need to be made to address review comments, work them into
your <code>dev/feature</code> branch, using interactive rebase to add them where
needed in the patch series, then repost.</p>