2011 June | Anchor Web Hosting Blog

Elegantly reverting unintentional LVM changes

Published June 30th, 2011 by benoit

This is the first in a series of articles we’ll eventually publish about various interesting tricks relating to LVM – the Linux Logical Volume Manager. Stay tuned for more coverage.

Occasionally you find yourself having committed a change to LVM (e.g. extending a logical volume, removing a logical volume, etc.) that you didn’t intend to make.

In some cases, it’s possible to revert the change by simply performing the opposite action (e.g. lvextend with a positive offset after an unintentional lvreduce). This is by no means elegant, but it almost always works.

However, there are unintentional changes which can’t be elegantly reverted in this fashion. For instance, assume I have a logical volume foo split across multiple physical segments:

root@cyllene:~# lvdisplay -m

— Logical volume —
LV Name                /dev/cyllene/foo
LV UUID                yaLJM6-ZzWi-cR89-ZJ4T-lUiI-tnqk-9biVeM
LV Size                40.00 GiB
Current LE             10240
Segments               2
Block device           252:3

— Segments —
Logical extent 0 to 7679:
Type                linear
Physical volume     /dev/md1
Physical extents    36441 to 44120

Logical extent 7680 to 10239:
Type                linear
Physical volume     /dev/md1
Physical extents    51801 to 54360

There is no guarantee that if I accidentally delete this logical volume and re-create it again with the exact same size, that precisely the same physical extents (and the same two above segments) would once again be used to satisfy the logical extent allocation. If different physical extents are used, this will make it impossible to read the partition that originally existed within the logical volume before it was deleted (you’d like to be able to use it again, right?). In addition to this, the UUID and block device of the re-created logical volume will be different, possibly affecting your ability to mount the device by UUID (or by device mapper minor number, but most people don’t do this). The lvcreate utility simply doesn’t give you the flexibility to re-create the logical volume in exactly the state that it previously existed.

This is where LVM’s wonderful automated configuration archiving feature comes in handy.

Let’s delete the LV from the above example, and re-create it with the same size (40 GiB):

root@cyllene:~# lvremove /dev/cyllene/foo
Do you really want to remove active logical volume foo? [y/n]: y
Logical volume “foo” successfully removed

root@cyllene:~# lvcreate -n foo -L40G cyllene
Logical volume “foo” created

Now, let’s check which physical extents were allocated to satisfy our requested allocation of 10240 logical extents (or roughly 40 GiB):

root@cyllene:~# lvdisplay -m

— Logical volume —
LV Name                /dev/cyllene/foo
LV UUID                E4hnBR-ODdX-HwOs-1ssl-C40v-vaus-XefvQe
LV Size                40.00 GiB
Current LE             10240
Segments               1
Block device           252:3

— Segments —
Logical extent 0 to 10239:
Type                linear
Physical volume     /dev/md1
Physical extents    51801 to 62040

Woops. Suddenly our mapping of 10240 logical extents (previously in two physical segments with PE mappings 36441-44120 and 51801-54360) has turned into one large monolithic segment (with PE mapping 51801-62040). This is definitely not what we wanted.

Luckily, LVM takes a snapshot of the volume group’s configuration state before any action on the volume group is taken. These are safely stored in files named /etc/lvm/archive/<volumegroup>_<serial>.vg, where serial is the iteration of the state of the volume group.

Let’s pretend I just deleted the logical volume foo from the above example. I’d now like to roll this change back so that the volume group is returned to the state it was before I executed the lvremove command.

Let’s first list the available restore points, using the vgcfgrestore –list <volume group> command:

root@cyllene:~# vgcfgrestore –list cyllene

<snip>

File:         /etc/lvm/archive/cyllene_00008.vg
VG name:      cyllene
Description: Created *before* executing ‘lvremove /dev/cyllene/foo’
Backup Time: Thu Jun 30 17:00:53 2011

File:         /etc/lvm/backup/cyllene
VG name:      cyllene
Description: Created *after* executing ‘lvremove /dev/cyllene/foo’
Backup Time: Thu Jun 30 17:00:53 2011

Ok. It seems like we want to restore the state of the volume group to that which is described in the file /etc/lvm/archive/cyllene_00008.vg. Let’s execute another vgcfgrestore command, this time with the –file <restore file> argument:

root@cyllene:~# vgcfgrestore –file /etc/lvm/archive/cyllene_00009.vg cyllene
Restored volume group cyllene

root@cyllene:~# lvdisplay -m

— Logical volume —
LV Name                /dev/cyllene/foo
LV UUID                yaLJM6-ZzWi-cR89-ZJ4T-lUiI-tnqk-9biVeM
LV Size                40.00 GiB
Current LE             10240
Segments               2

— Segments —
Logical extent 0 to 7679:
Type                linear
Physical volume     /dev/md1
Physical extents    36441 to 44120

Logical extent 7680 to 10239:
Type                linear
Physical volume     /dev/md1
Physical extents    51801 to 54360

Excellent! Our logical volume has been re-created and returned to precisely the state it existed in before it was removed by our inadvertent lvremove command. Note that you still need to make it active before you’re allowed to mount it again:

root@cyllene:~# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
foo cyllene -wi— 40.00g

root@cyllene:~# lvchange -ay /dev/cyllene/foo
root@cyllene:~#

You can now make use of the logical volume exactly as you would have previous to it being deleted.

Tags: lvm, recovery
Posted in FTW

For the lulz (Building Secure Websites)

Published June 28th, 2011 by Chris Collins

With all of the #antisec love going around, we felt was a good time to discuss some of the key principles in writing secure webcode. Today’s topic is unsanitised input.

A great piece of philosophy for designing secure systems is that any piece of information that comes from an external source is inherently untrustworthy. This applies quite strongly to the web – every detail in a web request comes from a user somewhere, and can easily be forged and specially crafted to manipulate poorly written code.

This is particularly notable in languages like PHP, where there is a low barrier to entry for writing code, but doesn’t often provide good frameworks or assistance to avoid being caught out by these problems.

Validate All Input

The first rule is to validate all input. You can’t trust anything that comes in from the request to conform to a certain format.

Just because your pretty HTML form only allows certain values to be sent back doesn’t mean that some nasty cracker somewhere can’t craft up a request to your webserver that doesn’t conform to this.

The way we deal with this is to clean these parameters, always. If we expect the parameter to be an integer, we ensure that the parameter only contains an integer, and treat everything else as an error. If it’s a string, we have to assume we don’t know how long that string is, or what sort of characters it may contain.

Now, I know some of you are thinking “Ah, but we can avoid all of this by using javascript to validate our forms!”. That doesn’t work as the javascript has no bearing on requests sent by our crafty cracker – they can only influence the result from a user using a web browser in the way you intended.

Tread with Care

With strings in particular, filtering the contents is not always an option – in these cases, we need to ensure that when we use the string elsewhere, its contents do not interfere with the other operation.

One of the most common places where this gets fouled up is in SQL queries.

A common (erroneous) construct you’ll find used is:

$query = sprintf("SELECT * FROM articles WHERE id="%s"', $_REQUEST['id']);
$myresults = mysql_query($query);

// do stuff with $myresults...

The problem with this is that $id is joined verbatim with the rest of the SQL query. If $id contains characters with special significance in SQL, such as the ” (double-quote) character, they will be interpreted as such.

If Little Johnny Cracker were to send our PHP page a request with id set to:

"; DELETE FROM articles; --

the next thing we know, our DB server no longer has anything in its articles table!

We can protect against this by modifying the string so that characters that are special to SQL are appropriately “escaped”. In this specific instance, there is a mysql_real_escape_string function to do this for us, but most other database libraries and other languages offer different ways to perform this important task.

If we amend our example above, the code becomes:

$query = sprintf('SELECT * FROM articles WHERE id="%s"',
            mysql_real_escape_string($_REQUEST['id'])) ;
$myresults = mysql_query($query) ;

// do stuff with $myresults...

This ensures that the contents of id will be interpreted correctly as part of the string being compared to id, and won’t contain characters that could be interpreted as part of the SQL command.

But this seems all too hard/too manual!

Now, you’re all bound to say, “There must be an easier way!”, and there usually is.

There are plenty of libraries and wrappers available which are aware of the problem with trying to prepare safe SQL queries and provide simple, convenient, parameter escaping to ensure that user input can be safely combined with your queries.

In PHP, there is the PEAR MDB library which can do the above safely. An example:

$sth = $db->prepare('SELECT * FROM articles WHERE id=?') ;
$myresult = $db->execute($sth, $_REQUEST['id']) ;

// do stuff with $myresults...

Other languages, such as Ruby, Python and Perl, all have their own safe and convenient methods for doing this too.

In Summary

Don’t trust the input into your web code. Make sure you never mix input with strings you’re handing off elsewhere, such as via exec, eval or to a database. And finally, use convenience methods for correcting/filtering user input when possible to make your life easy.

Tags: pwned, security, sql injection
Posted in FTW

Implementing proper trust relationships in backup systems

Published June 21st, 2011 by Keiran Holloway

As many people may have read today, a certain web hosting company which operates in Australia suffered an attack which resulted in significant amounts of data loss. Not only was their live production data lost, but all backups were also unrecoverably lost in the process.

Whilst significant technical details have yet to be released as to the nature of these attacks, now is an opportunity as good as ever to explain how the trust relationships exist between the backup server and the system being backed up across our entire server infrastructure.

Our backup servers are separated with there being a minimal trust relationship between the backup server and the system being backed up:

All backups happen on an independent network which is inaccessible from the Internet.
The backup server can only read data from the machine being backed up (and not alter it).
The machine being backed up can only send new data and not alter the data that is already backed up.

Using this methodology should protect any attacks from the public, customer facing servers from damaging the independently stored backups.

Tags: backups, redundancy, security
Posted in FTW

Blog Posts from June, 2011

Elegantly reverting unintentional LVM changes

For the lulz (Building Secure Websites)

Implementing proper trust relationships in backup systems

Archives

Blogroll

Recent Posts

Meta