When you have a bunch of machines involved in a process, you need to ensure that various stages in this process have executed. If the target host is unavailable, you want a guarantee that the job will execute when the host becomes available again. This is well beyond the capabilities of ssh in a for-loop. In trying to solve this problem, we had assessed tools like mcollective, but came to the conclusion that they were inappropriate for our environment. mcollective in particular was removed from consideration as it was designed for a more homogeneous environment than the one here at Anchor. When we realised we needed a different solution, a few of us gathered around a whiteboard and started enumerating our requirements. The result was Orchestra, which we’re releasing today under the BSD License….