Watzmann.Blog - Entirely too many words about idempotency in systems management

Entirely too many words about idempotency in systems management

21 May 2018

Idempotency is one of the fundamental ideas when managing systems: it’s both convenient and natural to demand that any management action has the same result whether it’s performed once or multiple times. For example, if the management action is ‘make sure that httpd is running’, we only want to say that once, and if, for some reason, that action gets performed multiple times, the result should stay the same. In this post, I’ll use ‘classical’ management actions on long-lived, stateful servers as examples, but the arguments apply in the same way to management actions that manipulate cloud services or kubernetes clusters, or really any other system that you’d want to manage.

It has always bothered me that it’s not obvious that stringing such idempotent actions together will always be idempotent, too.

Formally an action is a function f that turns some system state x into an updated state f(x). Idempotent then means that f(f(x)) = f(x), which I’ll also write as f f = f, dropping the argument x to f. For two actions f and g to be idempotent when we string them together then means that f g f g = f g. Clearly, if f and g commute, for example because f is ‘httpd must be running’ and g is ‘crond must be running’, the result of combining them is f g f g = f f g g = f g because both f and g are idempotent.

But what if they are not ? What if f is ‘make sure httpd is running’ and g is ‘make sure httpd.conf has this specific content’ ? How can we convince ourselves that combining these two actions is still idempotent ? When we look at real-life management actions, they are actually more than just idempotent: they are constant functions. No matter what state the system is in, we want the result of f to be that httpd is running. That means that f is not just idempotent, i.e. that f f = f, but that for any other management action g, we have f g = f. And if f and g are constant functions, f g = f and therefore f g f g = f f = f, which makes the combination f g idempotent, too, but is much stronger than mere idempotency.

In practice, there are of course other considerations. For example, the action ‘make sure httpd is running’ will generally fail if httpd is not even installed, but that does not really affect the argument above, we’d just have to get more technical and talk of management actions as partial functions, where concatenating them only makes sense where they are both defined. Similarly, we can work around order-dependency by getting a little more formal about what we consider the system state x and that management actions should actually be constant on the ‘interesting’ part of the state, and the identity everywhere else.

It therefore seems misleading to harp so much on idempotency when we talk about systems management. What we really want are constant functions, not just idempotent ones, a fact that the notion of ‘desired-state management’ nicely alludes to, but doesn’t make quite clear enough.

Watzmann.Blog Varying amounts of fiber

Entirely too many words about idempotency in systems management