Bughunting in Ceph’s radosgw: ETags
RADOS Gateway (henceforth referred to as radosgw) is an add-on component for Ceph, large-scale clustered storage now mainlined in the Linux kernel. radosgw provides an S3-compatible interface for object storage, which we’re evaluating for a future product offering.
We’ve spent the last few days digging through radosgw source trying to nail a some pesky bugs. For once, the clients don’t appear to be breaking spec, it’s radosgw itself.
We’re using DragonDisk as our S3-alike client – what works? PUTing and GETing files works, obviously. Setting the Content-Type metadata returns a failure, and renaming a directory almost works – it gets duplicated to the new name, but the old copy hangs around.
Wireshark to the rescue! We started pulling apart packet dumps, and it quickly became evident that setting Content-Type on objects was actually working fine, so what gives? We put it side by side with some known-good traffic to S3 and quickly spotted the quirks.
radosgw wasn’t returning the new object’s ETag in the HTTP response headers. The Amazon S3 docs are hopelessly imprecise in some areas, but it seems the ETag is mandatory. Aha!
Quotes around my ETag? Why!?
We weren’t done yet though, DragonDisk was still reporting “invalid response”. So we looked a bit closer and discovered that the ETag is also meant to be in the response body, which is a brief XML document. Easy enough to fix, but somehow this had never cropped up in development.
That’s not a typo – apparently the official way to do this is to steal the ETag directly from the headers, leave the quotes in (which is pointless for both HTTP and XML), and then convert them to HTML entities! Someone at Amazon really wasn’t paying attention when they developed the initial spec and code.
We’re not quite done yet. Our ETags are all sorted, but DragonDisk is still reporting invalid response from radosgw sometimes. That’s going to be another fun instalment.