A great piece of philosophy for designing secure systems is that any piece of information that comes from an external source is inherently untrustworthy. This applies quite strongly to the web – every detail in a web request comes from a user somewhere, and can easily be forged and specially crafted to manipulate poorly written code.

This is particularly notable in languages like PHP, where there is a low barrier to entry for writing code, but doesn’t often provide good frameworks or assistance to avoid being caught out by these problems.

Validate All Input

The first rule is to validate all input. You can’t trust anything that comes in from the request to conform to a certain format.

Just because your pretty HTML form only allows certain values to be sent back doesn’t mean that some nasty cracker somewhere can’t craft up a request to your webserver that doesn’t conform to this.

The way we deal with this is to clean these parameters, always. If we expect the parameter to be an integer, we ensure that the parameter only contains an integer, and treat everything else as an error. If it’s a string, we have to assume we don’t know how long that string is, or what sort of characters it may contain.

Now, I know some of you are thinking “Ah, but we can avoid all of this by using javascript to validate our forms!”. That doesn’t work as the javascript has no bearing on requests sent by our crafty cracker – they can only influence the result from a user using a web browser in the way you intended.

Tread with Care

With strings in particular, filtering the contents is not always an option – in these cases, we need to ensure that when we use the string elsewhere, its contents do not interfere with the other operation.

One of the most common places where this gets fouled up is in SQL queries.

A common (erroneous) construct you’ll find used is:

$query = sprintf("SELECT * FROM articles WHERE id="%s"', $_REQUEST['id']);
$myresults = mysql_query($query);

// do stuff with $myresults...

The problem with this is that $id is joined verbatim with the rest of the SQL query. If $id contains characters with special significance in SQL, such as the ” (double-quote) character, they will be interpreted as such.

If Little Johnny Cracker were to send our PHP page a request with id set to:

"; DELETE FROM articles; --

the next thing we know, our DB server no longer has anything in its articles table!

Copyright © Randall Munroe - xkcd.com

We can protect against this by modifying the string so that characters that are special to SQL are appropriately “escaped”. In this specific instance, there is a mysql_real_escape_string function to do this for us, but most other database libraries and other languages offer different ways to perform this important task.

If we amend our example above, the code becomes:

$query = sprintf('SELECT * FROM articles WHERE id="%s"',
            mysql_real_escape_string($_REQUEST['id'])) ;
$myresults = mysql_query($query) ;

// do stuff with $myresults...

This ensures that the contents of id will be interpreted correctly as part of the string being compared to id, and won’t contain characters that could be interpreted as part of the SQL command.

But this seems all too hard/too manual!

Now, you’re all bound to say, “There must be an easier way!”, and there usually is.

There are plenty of libraries and wrappers available which are aware of the problem with trying to prepare safe SQL queries and provide simple, convenient, parameter escaping to ensure that user input can be safely combined with your queries.

In PHP, there is the PEAR MDB library which can do the above safely. An example:

$sth = $db->prepare('SELECT * FROM articles WHERE id=?') ;
$myresult = $db->execute($sth, $_REQUEST['id']) ;

// do stuff with $myresults...

Other languages, such as Ruby, Python and Perl, all have their own safe and convenient methods for doing this too.

In Summary

Don’t trust the input into your web code. Make sure you never mix input with strings you’re handing off elsewhere, such as via exec, eval or to a database. And finally, use convenience methods for correcting/filtering user input when possible to make your life easy.

Whilst significant technical details have yet to be released as to the nature of these attacks, now is an opportunity as good as ever to explain how the trust relationships exist between the backup server and the system being backed up across our entire server infrastructure.

Our backup servers are separated with there being a minimal trust relationship between the backup server and the system being backed up:

Using this methodology should protect any attacks from the public, customer facing servers from damaging the independently stored backups.

Depending on your host and product used, your data could be sitting on anywhere from locally attached disks, a NAS/SAN or some clustered distributed block device/filesystem with no way to easily determine who has access to it, what snapshots exist, what will happen to failed media, etc. For certain customers with certain sensitive applications, that is simply not an acceptable risk.

To protect your data at rest, the most reassuring option is to utilise disk encryption within your operating system where only you possess the key. Whilst this can present a bit of an operational challenge (i.e. how does the key get entered if the server is rebooted), the show stopper question is going to be “Can I actually run a production database with heavy I/O load on encrypted storage?”. This post seeks to answer that question or to at least help you figure it out for your circumstances.

Database server

Our test server is a Dell PowerEdge R410 with:

• 2 × Intel Xeon E5620 CPUs
• 16 GB of RAM
• Dell PERC H700 hardware RAID controller
• 2 × Dell 50GB SSDs in a hardware RAID 1 array for the database
• 2 × 300GB 15K SAS in a hardware RAID 1 array for the operating system
• Red Hat Enterprise Linux Server 6.0 (kernel 2.6.32-71.18.1.el6.x86_64)

The interesting thing to note in this setup is that the CPUs contain hardware accelerated AES encryption via the AES instruction set (Intel’s marketing name is AES-NI). You can check if your CPU and kernel support this feature by running at the command line:

grep aes /proc/cpuinfo

if you get output, then congratulations, you have the feature supported!

Encrypted storage

The recommended way to do block device level encryption under Linux is to use the device mapper target dm-crypt. This is a straightforward component that is built-in to the Linux kernel and well supported in all modern Linux distributions. To set up full disk encryption, you should use your distribution’s installer when you initially provision your server. If you are not encrypting your entire OS (sans /boot) and data, then you will need to use the utility cryptsetup on the storage device you want to encrypt. For our testing, we are just encrypting the SSDs as follows:

cryptsetup luksFormat /dev/sdb -c aes-xts-plain -s 512

This command formats the storage device with a LUKS meta-data header. This header contains useful information including:

• the fact that it contains encrypted data (rather than just random data)
• a UUID
• the  algorithm used
• the key size
• the key (securely encrypted with a passphrase that you provide at format time)
• checksums and other useful bits

For high security, we are using a 512 bit AES key in an XTS cipher mode (requires kernel >= 2.6.24). A small word of warning: whilst cryptsetup allows you to setup encryption on devices without any sort of identifying meta-data header, DON’T ever do that. If you need to hide the fact that you are using encryption there are better ways to do that. Contact us for details if you really want to go down that path.

Once you have formatted the device, you then need to activate it by running the command:

cryptsetup luksOpen /dev/sdb testing

this will then make the unencrypted block device ready for you to use (i.e mkfs) as /dev/mapper/testing. To have the operating system setup the device at boot, you will need to put an entry into /etc/crypttab. Check the man page for details.

Finally, for best performance you want the implementation of AES that is most optimised for your hardware. You can check /proc/crypto to see which algorithms and drivers are available and their priority (highest priority number indicates higher priority). In order of performance, you want AES-NI, AES-x86_64 and the generic AES implementation last. The highest priority crypto implementation at the time of an encrypted block device being activated (via cryptsetup luksOpen) is used. Higher priority drivers installed into a running kernel have no effect on active devices.

Benchmarking

The simple benchmark tool zcav is fine for our purposes. It only measures the sustained read/write speed. For an I/O device, this benchmark would normally be of limited value as the device bottleneck is usually with the rate of I/O requests that can be handled. For an encrypted block device, the bottleneck is instead in the throughput (i.e. the sustained read/write speeds) as the higher the throughput, the more work that the CPU has to do.

Results

The output from zcav was fed into gnuplot to generate the following graphs:

The obvious thing to note in this diagram is that the new AES instructions make a huge difference to performance. The impact of the encryption is relatively low and should be quite acceptable for a production system. An interesting thing to note is the relatively low performance of the RAID array. Being RAID 1, the RAID controller should have been able to balance the read requests over both drives to get about double what it is getting. I haven’t looked into the RAID settings though to see if this can be tuned.

The throughput here with the AES instructions in use is almost identical.

Conclusion

The results certainly look promising for even relatively high end work loads. One thing that is not measured here is the latency difference. It should be relatively straightforward to calculate the latency that will be added to every I/O request.

The numbers above can also be massively improved upon with a newer kernel featuring the dm-crypt scalability patches (the benchmark above was limited to using 1 core out of 8!).

The nice thing about using the built-in dm-crypt solution is that you can use it on physical as well as virtual machines. If you are after something a bit more turn-key, you may wish to look at a system with drives (and a controller and BIOS) that support self encrypting drives as an alternative.

Whilst our office building already has excellent security as it houses a credit union, an additional security layer for our office floor can’t hurt (this is unrelated to our servers; they are housed in the secure Global Switch data centre). Now sure, we could easily just get a commercial off the shelf alarm system and be done with it, but it can be pretty fun and interesting to get your hands dirty and do a bit of DIY hacking.

Using some simple components, we can quickly and easily build a basic RFID security system. One of the advantages of doing it ourselves (besides the fun!) is that we can integrate it into our standard monitoring and internal chat systems. The parts list for this project come from the excellent Australian electronics store Little Bird Electronics. The parts:

• Freetronics TwentyTen (Arduino compatible)
• Ethernet Shield with PoE
• Power over Ethernet injector
• RFID module
• Passive Infra-red Detector
• Misc parts like a power supply, LEDs, buzzer

UPDATE 7th March 2011:

We thought it would be good to re-use the proximity cards we already carry around with us for this project, but we couldn’t get any of them to work with this reader.  There are a lot of proprietary standards out there and documentation on compatibility is really quite poor.  We are currently sourcing another reader that can handle a wider range of 125kHz proximity cards.

For the curious, Indala FlexCard cards by HID Global use a totally proprietary air interface.

Whilst doing some research, an awesome open source hackers RFID module, the Proxmark3 was discovered. A very nice learning tool.

More posts to follow with details as the project progresses…

Anchor’s been doing this for what must be about ten years now. That’s way longer than I’ve been employed here, but what with our tech-director-and-co-founder being busy stuntjumping his scooter over rows of parked cars, it’s fallen to me to write this one up.

We use apache’s mod_suexec to run PHP scripts as though they’re CGI scripts, and it works great. There’s lots of guides out there about how to do this for yourself, but for us one of the most important things when deploying a solution is to ensure that it Just Works, for everyone. When we do it right, nothing breaks and noone notices a thing, because it works exactly as everyone expects. That’s what we’ve done.

It’s also one of the reasons that apache isn’t going anywhere in a hurry. Newer tech like nginx is an absolute performance-demon compared to apache, but the barrier to entry is way too high if your goal is “throw the site on the server and make it work”. Everyone writes apps assuming they’ll be deployed to apache. As a hosting company, if you’re not offering something compatible you’d better have a good ace up your sleeve.

So, we give you a rundown on how Anchor does PHP; it’s secure and it just works. There are many others like it, but this one is ours. If you’re an existing customer, we hope you’ve never had to think about it. If you’ve got any questions we’d love to hear them.

These days, anti-virus protection for e-mail is fairly thorough, and nobody’s really swapping floppies full of 16 colour games at recess. Malware authors have moved on to new and more fertile ground — embedding their junk in web pages, and relying on browser exploits to gain access to computers. Of course, with this method, you can only get infected if you actually visit a page that has an infestation, so the malware authors have two options: either entice you to visit their sites, or modify existing websites that users will visit in the course of their day — legitimate sites that people know and trust, but with a little added infection.

Enticing people to a whole dodgy site is usually just a matter of providing something people love to look at and sticking it in search engines. Since the attacker has to have a stable, identifiable presence for the search engines to direct users to, that can also be used by anti-malware lists like stopbadware.org to protect web users, so this isn’t a particularly effective means of attack, and is waning somewhat in popularity. Far more effective is infecting a legitimate website with some form of malware. How does it happen, though? In our experience, there are four vectors for infection:

1. Brute-force password guessing, where the attacker has a botnet they control to just repeatedly try lots and lots of usernames and passwords. They’re bound to get lucky sooner or later.
2. Some sort of web-based exploit, typically a vulnerability in the web application that allows the attacker to run code of their choosing; this is then either used directly to edit files, or bootstrapped into sufficient access to edit files via another method.
3. Password “scraping”, where the attacker gets direct access to the FTP password for your site. This can either be some sort of malware on the workstation of the web developer (or someone else related to the management of the website) that gets the password off the local machine (in a saved password file, or via a keylogger), or else via the “lost password” functionality provided by the hosting provider. Once the attacker has the FTP password for the site, they are free to login to the live site and make whatever changes they like.
4. Direct modification of the website code on the client-side computer, relying on the developer not to notice it and then upload the compromised content to the live site. We recently had our first “confirmed” case of this (where the web developer found the malicious modifications in their local copy), and they swear blind they didn’t download the HTML from the live site (which would bring the “infection” onto the local machine from the infected live site — which we’ve seen before, and categorise under vectors 1 and/or 2).

The countermeasures required to combat all these vectors boil down to a few simple precautions.

• Use strong passwords. (Protects against vector 1) Yes, they’re a pain to manage, but a weak password is just an open invitation to getting repeatedly and painfully owned. Of course, the strongest password is a keypair, which leads us to…
• Don’t use FTP. (Protects against vectors 1 and 3) The list of reasons for this is long, but for securing your website, FTP is a pain because you can only use passwords[1]. Switch to using SFTP (the file transfer component of the SSH protocol) and you can use public keys, which are, for all practical purposes, unguessable. You should also encrypt your keys with a passphrase, which means that even if the attacker does get access to your workstation and copies the key, it’ll be useless to them — unless they keylog your passphrase, which brings us to…
• Keep your workstation secure. (Vectors 3 and 4) It seems that attackers have realised that the weakest link in the website security chain is still the Windows desktop, and they’re increasingly hitting it as the first step in taking over websites (if you get the right workstation, you can get the credentials to hundreds or thousands of websites, because one web developer often works on many different sites). So, on any machine you connect to webservers from, you need to be doubly, triply sure that it’s rock solid — and that’s just a matter of following all the good advice out on the web. Antivirus, antispyware and firewall software, constantly running, well-configured, and kept up to date; keep up to date with your application patches, especially for your web browser, e-mail client, and core OS; don’t visit dodgy web sites; and so on.
• Protect your e-mail. (Vector 3) If someone can get access to your e-mail, they can also get access to your website, by using the password recovery feature (or impersonating you to your hosting company). If they delete the e-mails that are coming in before you notice them, you’ll never know what’s going on, and all the password changes and workstation security in the world won’t help you.
• Don’t use shared hosting. (Vectors 1 and 3). This might seem an odd thing to say, given that we sell shared hosting, but it is a legitimate way to reduce your vulnerability. If you use a dedicated server (including a VPS), you (or we) can configure it to only allow logins from certain IP addresses, rather than the entire Internet. This means that even if an attacker does get your password (or SSH key), via brute force or sniffing it off your workstation, they can’t login from their own machine because it won’t have an authorized IP address. On shared hosting, this configuration is impractical, because hundreds of people have legitimate access to the server, from a great many different IP addresses.
• Code responsibly. (Vector 2) It is said that “PHP is great, because its ease of use means that any idiot can produce a security hole — and most of them do”. Whilst this is a little derogatory to the many (several? few? one? PLEASE?) good PHP programmers out there, it is certainly fair to say that the capabilities of many people who write dynamic code aren’t up to the challenge of writing code that is exposed to the extremely hostile conditions that are the public Internet.

  Thus, if you are not familiar with the common security practices and problems with the language or environment that you are developing for, stop right now and go learn a little. There’s plenty of good information out there on the Internet from people who have learnt the lessons the hard way. Celebrate the benefits of literacy by learning from their mistakes rather than having to educate yourself by cleaning up an infected website. If you feel that isn’t something you can commit to, then please, for the sake of the Internet, find someone else to write the code.

• Keep your web applications patched. (Vector 2) Whilst some sites do use custom-built web applications, many sites choose to use a standard CMS or other application to manage their website. This is great, because hopefully someone else is taking a bit of responsibility for the security of the software, but that doesn’t do you any good if you don’t keep it up-to-date. Far, far too many people install a CMS once, then forget about it. Almost all of these applications have a vulnerability at some point, and not keeping them up to date is absolutely fatal, because once a vulnerability is found in a piece of software, an attacker can typically use Google to find all of the publicly-available instances of the vulnerable software, and quickly attack them all.

  This means that you need to keep yourself well-informed of any security updates for your off-the-shelf web applications. Subscribe to a relevant security announcements mailing list, or ensure that your vendor sends them to you. (If your commercial CMS vendor doesn’t have this ability, find a new CMS vendor).

Websites get compromised all the time, by a variety of methods. You should reinforce your defences, lest you’re the next target.

1. The first person to mention Kerberos or other unused-in-practice authentication schemes in a comment gets a free laughing at. If you think SFTP and SCP aren’t supported in widely used web development programs, try finding something that supports GSSAPI…

I found this earlier this evening; it’s a little old (~18 months), but that doesn’t make it any less relevant, so it’s a good read. There’s the odd inaccuracy here and there, but it’s solid stuff, and relevant to anyone writing webapp code to handle authentication.
http://www.matasano.com/log/958/enough-with-the-rainbow-tables-what-you-need-to-know-about-secure-password-schemes/

The article focuses heavily on one particular authentication methodology, because it’s something a lot of people do. After you read it, you’ll understand that it’s something a lot of people do poorly. It assumes a reasonable degree of knowledge about what you’re doing (ay, there’s the rub), but the best lesson to be learnt there is that you shouldn’t be writing that code! Just stop! Someone’s already done it better than you, and it’s been checked by a lot more people than you and your colleague, who reckons it “looks pretty secure”. If you’ve got time to read all the comments on the article you’ll find a rollercoaster of people saying, “wouldn’t it be secure if I just..?” Just don’t do it. Don’t write my security code, bro!

Just using a proper authentication mechanism means you can spend your valuable time on more important things.

• Like not building SQL queries in a piecemeal manner, using unsanitised user-supplied data
• Like not making your whole site world-writeable so you can handle file uploads (you don’t have to do this on our shared hosting servers)
• Like fixing your newsletter signup process so it requires double opt-in, saving you from filling your database with spurious accounts and causing massive headaches when you do a campaign mailout
• Like optimising your database schema for bitchin’ performance – do you retain a DBA on a six-figure salary to do this for you? Nah I didn’t think so, this is cheaper and much more fun
• Like checking that your code doesn’t spew 14 PHP Warns/Errors to the apache logs for every page access, and fill the partition
• Like trying to get nice round corners on your page elements

The script does good things, but MySQL could probably be doing things better to begin with – make it more secure out-of-the-box, and the last thing they should be doing is shipping it with an empty root password. >_< It pains me to say it, but I think MSSQL probably comes with a more-secure initial configuration. Things like no remote connections, and no test database unless you explicitly ask for it.

The documentation for mysql_secure_installation is brief but functional. One of the things I thought it was doing was writing out a convenient .my.cnf file for the user, but it turns out it just deletes it once it’s finished. This is a shame, because the per-user config file is really cool.

If you’re not already taking advantage of them, you should. MySQL calls them “option files”, and it lets you set default parameters for the client apps you use at the command line, like mysql and mysqldump, etc. We use them to store root’s login credentials so we don’t have to lookup the password for a given machine, then mess around with mysql’s asinine option-handling.

It’s best to at least read up briefly on option files before using them, but they’re pretty straightforward: you have “groups” of options in the file written as [groupname], then one option on each line after that. An instructive example:

[mysql]
database=mysql



[client]
user=root
password=verySecurePassword


If you’re going to keep your password here, you must make sure the file is adequately protected from other users who might try to view it.

The group-names I mentioned correspond to the client program that will use the options. [client] is a special case; those options will be used by all the mysql client programs, so it’s the ideal place for your username and password.

Having the database option in the [mysql] group means you’re automatically connected to a database once you start the mysql command-line client. We don’t put this in the [client] group because it would affect other utilities like mysqldump.

The enquiries we get relate to a security scan carried out by an Approved Scanning Vendor (ASV). The usual report format is a list of potential “vulnerabilities” detected, with a severity rating of 1 to 5 assigned to each. Anchor’s shared hosting servers never have any problems with this, so the report reads like a missal of mundanity.

TCP port 21 is open, an FTP service appears to be running! Crazy, I know…

The thing is, this scan is really just one small part of a much larger framework. The core requirements of the PCIDSS don’t specify at all how the scan should be performed; it’s really about secure storage and transmission of data, and accountability and auditing.

Do our customers’ applications really encrypt the data they store in the database? I don’t know, but it sure isn’t checked as part of the scan. Requirement 6 is “Develop and maintain secure systems and applications”. Mm-hmm, that’s a good idea…

Security is really a commodity nowadays, a fact highlighted most perfectly in the vending of SSL certificates. In case you hadn’t guessed, the PCIDSS scans we’ve seen can proudly join the ranks. Thankfully there are scanners who really know where their towel is, looks good to me!

http://www.scanlesspci.com/

http://blogs.zdnet.com/security/?p=1114

http://jeremiahgrossman.blogspot.com/2008/04/my-blog-is-pci-certified-by-scanless.html