Amazon Web Services: Cloud Computing Free of Charge

Howly shmoly, just read the announcement of Amazon’s Free Usage Tier offering an EC2 micro instance free of charge for a whole year! Sounds cool, doesn’t it? Well let’s go back a few months and analyze the reason why I left Amazon in favor of Media Temple’s (ve) service: Amazon is way too expensive for a young geek like me, barely having the money to pay rent for my lousy apartment in Moscow ;)

Well that’s not the only reason, but I’m now quite comfortable with (mt)’s services, except their tech support, but that’s not what this post is about. I must have flooded my Twitter with messages about Django and Python. Honestly, I fell in love with Python a few months ago (in theory) then started scratching code in the beginning of October, but then again, this is not what the post is about (can somebody tell me why I’m going off-topic today?)

Back to AWS. The news is good, but the fact that they mention “new customers” frightens me:

These free tiers are only available to new AWS customers and are available for 12 months following your AWS sign-up date. When your free usage expires or if your application use exceeds the free usage tiers, you simply pay standard, pay-as-you-go service rates (see each service page for full pricing details).

I’ve been with them for over a year, paying a bunch of money every month, so I’m not a new customer for them anymore, unfortunately. So they’re not really targeting old customers which were unsatisfied with something, but new ones which have never tried EC2 (S3 and all the rest). Again, this is only a trial, unlike Google AppEngine, which happens to love Python code.

So my thinking is – is this all a coincidence, or is it a light for me towards AppEngine, Python and Google? Stay tuned: @kovshenin :)

Amazon Web Services: EC2 in North California

January is going crazy for me down here in Moscow, lot’s of stuff happening, loads of work. No time to tweet, not time to blog. As I mentioned in my earlier post, I quit my job at GSL and now working at a new local startup. I’ll make sure to announce it as soon as the website is alright, so stay tuned ;) Anyways, as I wrote back in December, I’m moving all my stuff to the new EC2 in the Northern California region, and I guess I can say that I’m finally done.

The process is not too different from simply moving to a new dedicated hosting or to a new EC2 instance in the same region, though there are a few nuances. I was surprised to note that the S3 Fox plugin for Firefox haven’t yet added the new region (Europe is present though). I thought it might not work for some reason (S3 and EC2 being in different regions), but hopefully it does. I also considered using the good old mod_php for Apache instead of running mod_suphp which gave me a tiny boost in performance. All the configurations were straightforward, copy from one EC2, paste into the other. Not without a few changes of course.

I also had a change in the Elastic IP address, but hey that was whitelisted by Twitter! So I guess I’ll have to write to them again for the new whitelisting. Oh well.. One more interesting thing is that I’m now running on an EBS-backed instance, which was introduced by Amazon not so long ago. I wouldn’t have to worry about getting my stuff lost on a terminate or a rebooted machine as the whole drive is being dumped into an EBS. So backups are now completely instant via the AWS Management Console, they’re called Snapshots, takes one click and a few minutes ;) Now if I’d like to terminate one EC2 instance and start the whole thing over on another one, I’d just restore from EBS or Snapshot! Unless, of course, I decide to move to another region. I believe EBS blocks and Snapshots are restricted to regions, furthermore, EBS and EC2 compatibility are restricted to a certain zone in one region, which is obvious. I wouldn’t like to run an EC2 instanced in one data center, backed by a hard drive located in a different one.

Another good question would be Amazon CloudFront. Well, since the S3 buckets haven’t changed, CloudFront should work the way it used to despiting the move. Or at least I hope so ;)

Amazon Web Services: Moving to a New Region

I wrote about Optimizing Your Amazon Web Services Costs back in November, where I mentioned some of the upsides of Reserved Instances at Amazon, but haven’t mentioned any downsides, and here we are. Two weeks later Amazon announced the Northern California Region opening. I thought it wouldn’t differ from the Virginia data center, but still decided to give it a shot for a few hours.

I didn’t do much benchmarking but hey, I’m running a Twitter app.. Foller.me, remember? This means that access times to the Twitter API are crucial, so I started off with some basic pinging, and the pings from California seemed to be a few times faster than the ones from Virginia. Next, I ran Xdebug and analyzed the cache grind sheets for a few requests to different profile pages. Sweet to know that 95% of the time taken to load a page is curl accessing the Twitter API ;) this means that my code is well optimized. The overall results in the California region was ~40% better than Virginia, so I thought of moving there. The problem was that I already had a 1 year contract with Amazon for an instance in the Virginia region.

I wrote to Amazon via their contact form and asked about reservation transfers from one region to another, of course with additional charges (the California region is slightly more expensive) and soon got a negative reply. They mentioned that reserved instances are not transferrable from one region to another but I can always cancel my reservation in one region and open up a new one in the other. They didn’t mention any refunds so I decided to ask, but soon, scrolling through their FAQ I found this:

Q: Can I move a Reserved Instance from one Region or Availability Zone to another?
A: No. Each Reserved Instance is associated with a specific Region and Availability Zone, which is fixed for the lifetime of the Reserved Instance and cannot be changed.

Q: Can I cancel a Reserved Instance?
A: The one-time payment for a Reserved Instances is not refundable. However, you can choose not to run or entirely stop using your Reserved Instance at any time, at which point you will not incur any further usage charges.

So I asked myself, why the heck would anybody want to cancel a reserved instance if they don’t get refunded? The conversation kept going on Twitter. Friends mentioned that I could purchase an additional reserved instance in the California region and then sell computing time on the one I have in Virginia, but that sounded too sarcastic. I felt unlucky and sad, and thought I thought should stick to the instance I had in Virginia. If only I had waited a few more weeks before making the purchase…

This morning I received another email from Amazon, stating that although they don’t usually do this sort of stuff, they got approval to process the cancellation with a refund just this one time, so now I’m free to reserve an instance in Northern California, happy holidays! Well, on Christmas Eve, this feels like a gift and I’m very excited about launching all my stuff in the new region, hopefully in January. So, thank you Amazon and Happy Holidays to all of you.

Cloud Tips: Amazon EC2 & Rejected Email

A few weeks ago I’ve setup my email in the /etc/aliases for user root (and the others) and started to actually read my root email from time to time (I wonder why I never did that before). Anyways, what bugged me straight away is that I had some rejected emails that were not being delivered, yielding the following errors (I removed some numbers):

Deferred: 450 4.7.1 : Helo command rejected: Host not found
421 invalid sender domain 'domU.compute-1.internal' (misconfigured dns?)

And some others that looked alike. Tonnes of them, every four hours! The emails to other addresses were delivered fine though. I had WordPress notification messages delivered to my email, never lost a message. I also tried sending out a few using the mail command via SSH, everything okay. For a second I thought that maybe those addresses were simply invalid, but wouldn’t the server reply with an “Invalid recepient” error? Probably.. Here’s what I got from the Amazon Web Services support forums:

It seems that some remote mail servers complain about your server
identifying itself in the SMTP dialogue as domU.compute-1.internal,
while its external name is ec2.compute-1.amazonaws.com

Makes total sense. Perhaps some servers do try to see where the e-mail is coming from and of course the .internal domain is unresolvable (thus the “dns” misconfiguration error). I had to identify myself with an external, resolvable name. So I copied the external name into the /etc/mailname file and hmm.. Well, it’s been a week now and I haven’t received anymore delivery errors, so that must have worked.

Optimizing Your Amazon Web Services Costs

I’ve been with Amazon for quite a long time now and you must have heard that their web hosting services aren’t very cheap. The average total of one instance per month (including EBS, S3 and all the others) was around $120 at the start. That was back in July 2009 when I had no idea about how all this stuff works. With a lot of experimenting I managed to drop my instance per month costs down by around 40%. Below are a few tips that can help you lower your Amazon Web Services charges:

  • Use reserved EC2 Instances where possible. Amazon charges $0.085 per hour for an m1.small Linux instance in the US, that’s around $61 per month and $734 per year. A reserved instance costs me $227 for one year, plus $0.03 per running hour, that makes it around $490 per year for an m1.small instance. Use reserved instances only if you’re sure that you’ll be using it for a whole year. You can save even more if you purchase a reserved instance for three years.
  • Storage: EBS vs EC2. Pick EC2! That’s right, EC2! EBS charges you for provisioned storage, IO requests and snapshots. These may rise pretty quickly if you’re running MySQL on an EBS block – very risky! Run your MySQL on EC2. The php files and everything else should preferably be on EC2 aswell. You can use your EBS block for tiny backups of core PHP files if you’re running more than one EC2 instance.
  • EBS is cheaper than S3. S3 should only be used in cases where you have to serve your static content from different servers (perhaps through CloudFront), and maybe store some backups there too (don’t forget to remove the old ones!), but EBS is cheaper, even with snapshots.
  • CloudFront is okay. It does speed up your website, but you have to know that it’s more expensive for requests to Japan and Hong Kong

There you go. With these tips you should be able to get the Amazon hosting services for around $90/month, unless of course you have a 3 million visitors per day website ;) Also, for those of you wondering.. I haven’t used RackSpace, but I did compare their prices to Amazon’s and they’re more expensive.

Cloud Tips: Automatic Backups to S3

In a previous post about backing up EC2 MySQL to an Amazon S3 bucket we covered dumping MySQL datasets, compressing them and uploading to S3. After a few weeks test-driving the shell script, I came up with a new version that checks, fixes and optimizes all tables before generating the dump. This is pretty important as mysqldump will fail on whatever step would cause an error (data corruption, crashed tables, etc), thus your uploaded to S3 archive would be kind of corrupt. Here’s the script:

filename=mysql.`date +%Y-%m-%d`.sql.gz
echo Checking, Fixing and Optimizing all tables
mysqlcheck -u username -p password --auto-repair --check --optimize --all-databases
echo Generating MySQL Dump: ${filename}
mysqldump -u username -p password --all-databases | gzip -c9 > /tmp/${filename}
echo Uploading ${filename} to S3 bucket
php /ebs/data/s3-php/upload.php ${filename}
echo Removing local ${filename}
rm -f /tmp/${filename}
echo Complete
<pre>

There you go. If you remember my previous example I stored the temporary backup file on Amazon EBS (Elastic Block Storage) which is quite not appropriate. Amazon charges for EBS storage, reads and writes, so why the extra cost? Dump everything into your temp folder on EC2 and remove afterwards. Don't forget to make changes in your upload.php script ($local_dir settings). Also, just as a personal not and to people who didn't figure out how to upload archives with data to S3, here's another version of the script which takes your public_html (www, htdocs, etc) directory, archives it, compresses and uploads to an Amazon S3 bucket:

<pre>filename=data.`date +%Y-%m-%d`.sql.gz
echo Collecting data
tar -czf /tmp/${filename} /ebs/home/yourusername/www
echo Uploading ${filename} to S3 bucket
php /ebs/data/s3-php/upload.php ${filename}
echo Removing local ${filename}
rm -f /tmp/${filename}
echo Complete

Oh and have you noticed? Amazon has changed the design a little bit, and woah! They’ve finally changed the way they show the Access Secret without a trailing space character! Congrats Amazon, it took you only a few months.

FTP Breaking on FEAT (vsftpd on Fedora Core 8)

It’s been a while since I connected to my Amazon EC2 running Fedora Core 8 via FTP and for no reason I tried connecting there today and badaboom! Strange though, it worked fine about a month ago, I was able to upload and download files, but this time I got a little crash. On one version of FileZilla FTP client I received a simple “Unable to connect” error. On a newer version I noticed that the FEAT (features list, or whatever) command was breaking the connection so I googled that.

People say that the server is broken but they don’t mention any tips on how to fix that. I logged on via SSH, rebooted the vsftpd daemon, with no luck. Then I tried connecting to localhost via FTP (in SSH) using the ftp command. I got a connection, LS and CWD commands worked just fine and I was able to see the files. So I sent a FEAT command and got an “invalid command” error. Humm?

Somebody on the Ubuntu forums mentioned that it’s an encodings issue. Client unable to handle UTF8 though server runs only UTF8. Does that make any sense? Guess not. Well before you go digging into your encoding settings and messing up your configuration files, or shutting down the server and starting a new instance (I’m on Amazon EC2) you might wanna try this fix.

I have no idea how it got there, but in my /etc/vsftpd.conf I found a new strange line saying:

connect_from_port_20=YES

For one second there I thought that it’s fair enough. But hey, wasn’t FTP supposed to work on port 21? Right. Comment out that line, restart your vsftpd daemon (service vsftpd restart) and voila! Worked for me.

I still think it’s strange though.. Ghosts? ;)

Cloud Tips: Backing Up MySQL on Amazon EC2 to S3

Now that I’m all set up in the Amazon cloud I’m starting to think about backups. Elastic Block Storage (EBS) on Amazon is great and the Snapshots (backups) can be generated with a few clicks from the Management Console, but, for a few reasons I’d like to set up my own backup scripts and here’s why:

  • Amazon EBS snapshots are cool, but there might be a situation where I’d like to restore only one file, or a few rows from a MySQL dump or whatever. EBS works in bundled mode, this means that you store an image of the hard-drive you’re working it, no matter how many files there are. It might be painful setting up an extra hard-drive just for backups and work with its snapshots
  • I don’t believe there’s a feature to schedule or automate EBS snapshots
  • I’d like a simple way to download backed up date onto my local PC
  • Some of my clients like to get their weekly backups by FTP

I don’t really care about my php files, images and other stuff that’s located on my EBS, cause I’m sure I have local copies of all that. The most important part in all my projects is the data stored in my MySQL database, thus I’m going to show you how to setup a simple shell script to generate daily MySQL backups and a simple php script to upload them to a secure S3 bucket.

Take a look at this simple shell script:

filename=mysql.`date +%d.%m.%Y`.sql.gz
echo Generating MySQL Dump: ${filename}
mysqldump -uyour_username -pyour_password --all-databases | gzip -c9 > /ebs/backups/${filename}
echo Uploading ${filename} to S3 bucket
php /ebs/data/s3-php/upload.php ${filename}
echo Removing local ${filename}
rm -f /ebs/backups/${filename}
echo Complete

I’m assuming you’re familiar with shell scripting, thus there’s no need to explain the first few lines. Don’t forget to type in your own username and password for MySQL access, also, I used the path /ebs/backups/ for my daily MySQL backups. You choose your own.

There are a few scripts located in /ebs/data/s3-php/ including upload.php, which takes a single parameter – filename (don’t put your path there, let’s keep things simple). The script simply reads the given file and uploads it to a preset path into your preset S3 bucket. I’m working with the S3-php5-curl class by Donovan Schonknecht. It uses the Amazon S3 REST API to upload files and it’s just one php file called S3.php, which in my case is located in /ebs/data/s3-php right next to my upload.php script.

Before going on to upload.php take a look at this file which I called S3auth.php:

$access_id = 'your_access_id';
$secret_key = 'your_secret_key';
$bucket_name = 'bucket';
$local_dir = '/ebs/backups/';
$remote_dir = 'backups/';

This is the settings file which I use in upload.php. These particular settings assume your backups will be located in /ebs/backups and will be backed up to an Amazon S3 bucket called ‘bucket’ and in the ‘backups’ directory within that bucket. Using single quotes is quite important, especially with the secret_key, as Amazon secret keys often include the backslash symbol. Here’s the upload.php script:

require("S3.php");
require("S3auth.php");

$s3 = new S3($access_id, $secret_key);
$s3->putBucket($bucket_name, S3::ACL_PRIVATE);
$s3->putObjectFile($local_dir.$argv[1], $bucket_name, $remote_dir.$argv[1], S3::ACL_PRIVATE);

All simple here according to the S3.php class documentation. Notice the $argv[1], that’s the first argument passed to the upload.php script, thus the filename of the backup file.

That’s about everything. Try a few test runs with the shell script (remember to chmod +x it otherwise you’ll not be able to execute it) and finally setup a cron running the script daily. Mine works like a charm!

Cloud Tips: Amazon EC2 Email & S3 CNAME Issues

So you moved your blog or website (or whatever) to Amazon EC2 and wondering why your e-mail notices have stopped working? Now I know there’s bunch of articles about the EC2 email issues, and most of them state that the letters are getting into the spam boxes or aren’t getting delivered at all, because Amazon’s IP pool has been blacklisted by most e-mail providers.

Don’t panic! Not just yet.. You might as well try the postfix via google mail or perhaps some paid mail relay servers, but hey, the php mail function requires the sendmail daemon to be running, and if you’re using the Fedora Core 8 AMI on EC2, you might as well try to turn it on:

service sendmail start

Worked for me, and the messages aren’t being marked as spam, while I’m still getting messages from my WordPress installation on MediaTemple marked as Junk by Windows Live Mail ;) I don’t believe Amazon’s in the blacklists… Really… Anyone, but not Amazon .. Right?

The next AWS issue a novice is going to bump into is the CNAME dillema. It’s so straightforward though, really… Let’s say I want an S3 bucket on s3.foller.me instead of the good old s3.amazonaws.com address. Create a new bucket called s3.foller.me, go to your DNS editor and add a CNAME record for s3.foller.me pointing to s3.foller.me.s3.amazonaws.com. Done. The bucket name and the CNAME have to be the same and this is the one and only trick.

Happy clouding, cheers!

Working With Amazon EC2: Tips & Tricks

It’s been a while now since I’ve been hosting on Amazon Web Services and I’d just like to point out some issues I had and quick ways of solving them. We’re gonna talk about setting up a server that would serve not only you, but your clients too, cause $100/mo is quite expensive, isn’t it? So let’s begin and keep this as straightforward as possible. If you don’t understand something, it’s probably because you haven’t read the official EC2 docs and haven’t searched the forums. This is not a tutorial, it’s just a set of rules you may want to follow to make things right.

Once you start a new instance from an Amazon predefined AMI (Fedora Core 8 for example) I suggest you start building your structure right straight away. Attach an EBS volume to you instance (I mount it to /ebs) and start creating your users with their home directories in /ebs/home/kovshenin not the regular /home/kovshenin. Also point your MySQL server to keep your database files in /ebs/mysql. There are plenty tutorials out there on how to do that.

Now, edit your httpd.conf, add your vhosts, point them to the right users dirs, install an ftp server and make sure you chroot the users to their home directories. That way they won’t be able to mess up with eachothers files and folders, peek passwords etc. You might want to change the root user’s home directory to / instead of /root in case you’ll want to use ftp via your root user (which is quite dangerous).

Now comes the fun part. The HTTP server runs under the apache user by default in FC8 and I recommend you don’t touch this. Damn it took me quite some time to figure out how the heck can the apache user execute and write to files not belonging to apache. I messed up big time with the groups, adding apache to all my client’s users groups, but thank god I found mod_suphp in the end. Install that one and make sure you use it and there’s no need to change the users umasks anymore.

Note: There’s a little issue with the mod_suphp in Fedora as far as I know, which doesn’t let you use the suPHP_UserGroup directive in the httpd.conf yelling that it does not exist. Most of the man pages on the net say you have to use that directive, but I’m good without it. It seems that suphp can figure out what user to run on its own, look closely at the config files, and also make sure you’re running php-cgi, not the CLI version. By the way, this is the part where WordPress stops asking you your FTP credentials on plugins/themes update, install, remove and core upgrade too. Speeds up the whole process ;)

I used the following code to test how mod_suphp works (or doesnt):

<?php echo system("id"); ?>

Which should output what’s the current user. Make sure you check everything works before going public, and do not set your min_uid and min_gid in suphp lower than 50. It’s safer to chown -R files and folders than to let suphp run your scripts via root or some other powerful user.

Backing up your EC2 and EBS

This is very important. Once you have everything set up and running, DO backup. Backing up the EBS is quite simple, just create a snapshot from the Amazon EC2 Management Console. Backing up the running AMI (instance) is a little bit mroe complex. You have to use the ec2 command line tools to bundle a new volume, upload it to an Amazon S3 bucket and register the AMI. There are plenty tutorials on the net on how to do that. Shouldn’t take you more than half an hour to figure it out.

Just make sure you have copies of all the major config files (httpd.conf, crontab, fstab, ..) backed up on your /ebs/config for instance. You might need them in the future (when you loose everything, haha ;) Restoring a backed up AMI instance is simple. Launch a new instance using the AMI you generated, attach the Amazon Elastic IP address to it and voila. Way too simple.

About the EBS, there are quite a few things you should be able to do with it before continuing. Restoring a backed up Snapshot: Create Volume from Snapshot, umount /ebs, deattach old volume, attach new volume, mount /ebs. Cool? Be careful when you’re resizing your EBS. The xfs filesystem automatically grows as far as I know, but in my case I use the ext3 filesystem. So if you need to grow your ext3 EBS you’ll go:

  1. Create a Snapshot
  2. Create a new EBS Volume from that Snapshot you created (say 10 GB if you were running 5 GB)
  3. Attach it to your Instance, say /dev/sdg
  4. Use the resize2fs command to resize the partition to 10GB
  5. Mount it to /ebs2 or whatever
  6. Check to see if everything’s in place
  7. Unmount /ebs2, deattach /ebs2, unmount /ebs, deattach /ebs
  8. Attach the 10GB volume to where /ebs was attached (/dev/sdf)
  9. Mount /ebs and start your services

There you go, back to work, server. By the way, when working with Amazon AWS, note that you should be working in the same region where your AMI is (us, eu, east, 1c, …) otherwise some of the options (when attaching, etc) might just not come up. Beware of that.

Well, I guess those are pretty much all the basics. Don’t forget to read the Amazon S3 tutorials and API, pretty sweet stuff! Good luck.