Second strike with Lightning!

We put Kyoto Cabinet under the gun recently as a means to improve Redis. The Anchor Propulsion/Internet Laboratory validated Kyoto Cabinet as “fresh”, but extended live testing has revealed sub-optimal behaviour in some situations.

To recap, we used Kyoto Cabinet to give Redis near-realtime disk persistence with a greatly reduced memory footprint; we called this “NDS” and published the code. Dirty keys are flushed from memory periodically into Kyoto Cabinet’s backing store.

This works fantastically most of the time, but we’ve discovered that some operations cause a massive blowout in the on-disk files. Kyoto Cabinet is a key-value store. When you update a value in Kyoto Cabinet it can be rewritten in-place, unless the new value is longer than the old one. In this case, Kyoto Cabinet makes a new copy of the value and leaves the old chunk of space unused. NDS is conservative about flushing to disk, but there’s no way to avoid Kyoto Cabinet allocating more and more space if your values keep growing in length.

So how does this apply to Redis? Setting a new value for string, like updating “foo” to “foobar”, will do it. It’s a similar story with numbers, which are also stored as strings: “1000” is twice as big as “10”. In our particular situation, a Redis list is also a value. Repeatedly appending elements to a list is a perfect recipe for this problem.

Kyoto Cabinet’s own defragmentation methods weren’t doing it for us and it became apparent that we needed an alternative, so we went looking, and found the Lightning Memory-Mapped Database (LMDB).

LMDB is part of the OpenLDAP project so it’s got some decent pedigree behind it, and extensive testing in the lab showed that it doesn’t have the same issues as Kyoto Cabinet or any of the previously tested options.

On paper it’s very good: high performance, multi-version concurrency so clients don’t tread on each other, transactional and fully thread-safe. In practice it seems to hold up, too. We don’t actually need all the fancy features, and it keeps track of free pages for reuse, which addresses our qualm with Kyoto Cabinet.

Integrating LMDB into the NDS codebase isn’t quite a walk in the park, as the API is very baroque and requires a lot of jumping through hoops to get simple things done. That said, we’re pretty excited about it; we’ve rewritten NDS to use it without too much difficulty, and performance is definitely up to scratch. Once the very smart people down in the lab verify that it doesn’t cause problems under production loads we’ll be eagerly rolling it out into production.