Lesson Learned: Backup Before Upgrading

Friday started with off quite crazy with a surprise from my web hosting server. I don’t really know why but I decided to run an apt-get upgrade last night and everything seemed fine until this morning when I was unable to log back on via SSH. So what did I do? Reboot, obviously and it broke everything. Ping was lost, services are down, websites not working, oh my!

I know my web hosting provider wouldn’t help much since I’m renting a virtual private server with root access which comes with a lot of responsibility, besides I was quite sure I’d handle it faster than I would have opened a support request and described the details, and I did!

Luckily I was able to use the Change OS feature and grabbed Ubuntu 11.04 which was installed within 20 minutes with all my old files in the /old directory. The rest was up to my typing speed — install necessary software, copy the old MySQL databases, restore the users, set up nginx and php-fpm and voila. I was back online. Approximate downtime: 45 minutes.

Lesson learned: backup your full VPS container before upgrading. Thank you Media Temple for providing the freedom to break things and the tools to fix them :)

How To Deploy & Publish a .NET MVC Application

This is the second post in the series called .NET MVC From a PHP Developer’s Perspective where I discuss the pros and (mostly) cons of jumping onto the .NET bandwagon. Keep in mind that I’m a php/unix guy. I’m now at week 5 and I recently deployed my .NET MVC application to a web hosting provider. This was done for a testing and debugging purpose, since I had to become familiar with the whole process.

We begin with picking the right hosting plan for your .NET MVC application, then proceed to deploying your application and the contents (images, javascript, etc). Finally I show you a little trick on how to deploy your database in a format (.bak) most hosting providers will (hopefully) accept.

Picking the Right Hosting Plan

I picked Parking.ru – a Russian Windows hosting provider, which were generous enough to provide me with a two weeks trial period. My choice was based on a few calls to several different hosting providers, their technical teams had no clue about if my .NET Framework 4, MVC 2.0 application with an SQL Express database would run on their hosting plans. The answers were all similar to:

Well we’re not sure, you may try this hosting plan, but we’re not sure. And you still have to pay in order to use it. It’s running Windows Server, but we don’t know what MVC is, so we can’t guarantee you anything, sorry.

Heh, there was always a problem with Russian hosting providers, but overall the service is getting better. Tech teams now at least know whether they’re offering Apache or nginx ;) But anyways, I was quite happy to get a free trial at Parking.ru, and since they had no FAQs on how to publish my MVC application, I ran off to Google.

Publishing Your Application and Contents

It’s not really a big deal after you get it the first time. Visual Studio can automatically publish your application and the contents to a remote FTP server, which in my case worked like a charm. The only prob I had is with the external contents such as images, stylesheets and some javascript, which all sumed up to around 500 files. I didn’t add them as resources to my project, thus they were never published. Some common sense was required here, so I fired up my favorite FTP client and flushed them all manually.

Publishing Your SQL Express Database

The major issue was of course with the database. As I wrote in my previous article, the MVC Music Store is the tutorial my application was based on. The tutorial creates the models from a database scheme, and all the data is stored in an SQL Express file with the MDF extension. My hosting provider supported backups (.bak) which I could upload to their servers, so I googled around for ways to generate such backups.

Most articles I came across mention SQL Server Management Studio, so I installed that, and after a few trial and error, I managed to get my SQL Express database inside the Studio. Now what’s weird is that the backup option didn’t work, yielding something about incorrect filenames. Of course my database was called [C:UserskovsheninDocumentsVisual Studio 2010…], but SQL queries worked quite okay, so did an SQL export which generated an SQL dump of all my data. Unfortunately that was not good enough for my hosting provider.

With a few more experimenting and searching, I came across the following query, which could be issued within Visual Studio itself (does not require the SQL Server Management Studio). It generates a .bak file which is exactly what I was looking for:

BACKUP DATABASE [C:PATHTOYOURDATABASE.MDF] TO DISK = N'C:my-database.bak' WITH NOFORMAT, NOINIT,  NAME = N'My Database Backup', SKIP, NOREWIND, NOUNLOAD, STATS = 10

Honestly, I have no idea what some of the options mean, but I’m sure Google can help. One thing you should pay attention to is the square brackets in the database name, especially if it contains the space character (which it will, if you saved your database in the Visual Studio directory by default). The command could not be parsed by Visual Studio, but the execution generates a .bak file on your hard drive. I instantly uploaded the file to my hosting provider, and voila!

I spent 3-4 hours in total on deploying my MVC application, but now that I got it right, takes me a couple of minutes. Then again, I really miss mysqldump, subversion and the rest. They seem to be a little more lightweight than .NET and IIS.

Driving the (ve) Server at Media Temple

It’s been a few weeks now since Media Temple launched their new (ve) Server and I’ve been testing it out for a few days now. I’m actually hosting my blog there to experience some real traffic load and my first impressions are awesome!

I started off with the simplest 512 MB server and transferred a few websites to the new platform. I’m not too used to the Ubuntu Linux operating system but I found my way around quickly. They do have other operating systems options, but Ubuntu is the one they recommend. First few tests showed that my load time decreased dramatically compared to my Amazon EC2 instance, which I was quite happy with. Next step was to run a few load tests using the Apache Benchmark tool (ab), and very soon I realized that I got quite a few failed requests, memory shortage and other strange stuff.

Media Temple’s (ve) servers are hosted on the Virtuozzo platform by Parallels, and after browsing their documentation I found out that there’s no swap space available for Virtuozzo containers. They do allow around 80% of burstable RAM (so you get around 1 GB when running 512 MB) but when that runs out, you’re left with nothing, not even some swap space on your hard drive. Some heavy load tests showed 30% request failure, which is quite horrible.

Media Temple don’t give much information on the new platform via the support system and in memory shortage questions in their user forums they advice you to upgrade, of course! Well, I wouldn’t like to upgrade to just run a couple of load tests, and what about Digg-traffic? Should I predict that and upgrade before the spike? Then downgrade again to save some cash? Of course not.

A good option I found here is to tune Apache a little bit, reduce it’s resources limits. This will not increase performance, but may guarantee a 100% fail-safe workflow. We wouldn’t like our users to see a blank page (or a memory shortage error) when a spike hits, but we would rather want them to wait more than often and still load the requested page. The settings mostly depend on what software you’re running, which services and the RAM available in your container.

You might want to reduce the KeepAliveTimeout in your apache settings (mine’s now set to 5), and the rest is up to the mpm prefork module. You’ll have to modify your settings and then run some tests until you’re comfortable with the results. Mine are the following:

<IfModule mpm_prefork_module>
    StartServers 3
    MinSpareServers 2
    MaxSpareServers 5
    MaxClients 10
    MaxRequestsPerChild 0
</IfModule>

This is on a 512 MB (~ 400 more burstable) container. An Apache Benchmark test showed that 100 concurrent (simultaneous) requests performed in 26 seconds with 0% failed requests, this makes 3.84 requests per second, which is quite good. To give a comparison, the same test ran on the mashable.com website gave 30 seconds with 3.32 requests per second, and of course a 0% failure. Also check out other MPMs for Apache which could give results too.

This definitely requires more fine-tuning and if the page load time becomes too high then yes, there is a reason to upgrade, but don’t forget about other performance tricks such as CDNs, gzip (deflate) and others. When you’re done with Apache, proceed to MySQL fine-tuning & php configuration, there are some tricks there too to give you some extra speed & performance.

I’ll keep playing around with this server, plus I’ve purchased a 1GB (ve) this morning, so there’s quite lot of tests that have to be run. Anyways, if you’re looking for a good, high-performance VPS, then Media Temple is definitely a choice to consider. For only $30/mo you can get quite a good looking virtual server. It is more interesting than their old dedicated virtual servers (although still in beta). Cheers, and don’t forget to retweet this post ;)

Optimizing Your Amazon Web Services Costs

I’ve been with Amazon for quite a long time now and you must have heard that their web hosting services aren’t very cheap. The average total of one instance per month (including EBS, S3 and all the others) was around $120 at the start. That was back in July 2009 when I had no idea about how all this stuff works. With a lot of experimenting I managed to drop my instance per month costs down by around 40%. Below are a few tips that can help you lower your Amazon Web Services charges:

  • Use reserved EC2 Instances where possible. Amazon charges $0.085 per hour for an m1.small Linux instance in the US, that’s around $61 per month and $734 per year. A reserved instance costs me $227 for one year, plus $0.03 per running hour, that makes it around $490 per year for an m1.small instance. Use reserved instances only if you’re sure that you’ll be using it for a whole year. You can save even more if you purchase a reserved instance for three years.
  • Storage: EBS vs EC2. Pick EC2! That’s right, EC2! EBS charges you for provisioned storage, IO requests and snapshots. These may rise pretty quickly if you’re running MySQL on an EBS block – very risky! Run your MySQL on EC2. The php files and everything else should preferably be on EC2 aswell. You can use your EBS block for tiny backups of core PHP files if you’re running more than one EC2 instance.
  • EBS is cheaper than S3. S3 should only be used in cases where you have to serve your static content from different servers (perhaps through CloudFront), and maybe store some backups there too (don’t forget to remove the old ones!), but EBS is cheaper, even with snapshots.
  • CloudFront is okay. It does speed up your website, but you have to know that it’s more expensive for requests to Japan and Hong Kong

There you go. With these tips you should be able to get the Amazon hosting services for around $90/month, unless of course you have a 3 million visitors per day website ;) Also, for those of you wondering.. I haven’t used RackSpace, but I did compare their prices to Amazon’s and they’re more expensive.

Cloud Tips: Amazon EC2 Email & S3 CNAME Issues

So you moved your blog or website (or whatever) to Amazon EC2 and wondering why your e-mail notices have stopped working? Now I know there’s bunch of articles about the EC2 email issues, and most of them state that the letters are getting into the spam boxes or aren’t getting delivered at all, because Amazon’s IP pool has been blacklisted by most e-mail providers.

Don’t panic! Not just yet.. You might as well try the postfix via google mail or perhaps some paid mail relay servers, but hey, the php mail function requires the sendmail daemon to be running, and if you’re using the Fedora Core 8 AMI on EC2, you might as well try to turn it on:

service sendmail start

Worked for me, and the messages aren’t being marked as spam, while I’m still getting messages from my WordPress installation on MediaTemple marked as Junk by Windows Live Mail ;) I don’t believe Amazon’s in the blacklists… Really… Anyone, but not Amazon .. Right?

The next AWS issue a novice is going to bump into is the CNAME dillema. It’s so straightforward though, really… Let’s say I want an S3 bucket on s3.foller.me instead of the good old s3.amazonaws.com address. Create a new bucket called s3.foller.me, go to your DNS editor and add a CNAME record for s3.foller.me pointing to s3.foller.me.s3.amazonaws.com. Done. The bucket name and the CNAME have to be the same and this is the one and only trick.

Happy clouding, cheers!

Working With Amazon EC2: Tips & Tricks

It’s been a while now since I’ve been hosting on Amazon Web Services and I’d just like to point out some issues I had and quick ways of solving them. We’re gonna talk about setting up a server that would serve not only you, but your clients too, cause $100/mo is quite expensive, isn’t it? So let’s begin and keep this as straightforward as possible. If you don’t understand something, it’s probably because you haven’t read the official EC2 docs and haven’t searched the forums. This is not a tutorial, it’s just a set of rules you may want to follow to make things right.

Once you start a new instance from an Amazon predefined AMI (Fedora Core 8 for example) I suggest you start building your structure right straight away. Attach an EBS volume to you instance (I mount it to /ebs) and start creating your users with their home directories in /ebs/home/kovshenin not the regular /home/kovshenin. Also point your MySQL server to keep your database files in /ebs/mysql. There are plenty tutorials out there on how to do that.

Now, edit your httpd.conf, add your vhosts, point them to the right users dirs, install an ftp server and make sure you chroot the users to their home directories. That way they won’t be able to mess up with eachothers files and folders, peek passwords etc. You might want to change the root user’s home directory to / instead of /root in case you’ll want to use ftp via your root user (which is quite dangerous).

Now comes the fun part. The HTTP server runs under the apache user by default in FC8 and I recommend you don’t touch this. Damn it took me quite some time to figure out how the heck can the apache user execute and write to files not belonging to apache. I messed up big time with the groups, adding apache to all my client’s users groups, but thank god I found mod_suphp in the end. Install that one and make sure you use it and there’s no need to change the users umasks anymore.

Note: There’s a little issue with the mod_suphp in Fedora as far as I know, which doesn’t let you use the suPHP_UserGroup directive in the httpd.conf yelling that it does not exist. Most of the man pages on the net say you have to use that directive, but I’m good without it. It seems that suphp can figure out what user to run on its own, look closely at the config files, and also make sure you’re running php-cgi, not the CLI version. By the way, this is the part where WordPress stops asking you your FTP credentials on plugins/themes update, install, remove and core upgrade too. Speeds up the whole process ;)

I used the following code to test how mod_suphp works (or doesnt):

<?php echo system("id"); ?>

Which should output what’s the current user. Make sure you check everything works before going public, and do not set your min_uid and min_gid in suphp lower than 50. It’s safer to chown -R files and folders than to let suphp run your scripts via root or some other powerful user.

Backing up your EC2 and EBS

This is very important. Once you have everything set up and running, DO backup. Backing up the EBS is quite simple, just create a snapshot from the Amazon EC2 Management Console. Backing up the running AMI (instance) is a little bit mroe complex. You have to use the ec2 command line tools to bundle a new volume, upload it to an Amazon S3 bucket and register the AMI. There are plenty tutorials on the net on how to do that. Shouldn’t take you more than half an hour to figure it out.

Just make sure you have copies of all the major config files (httpd.conf, crontab, fstab, ..) backed up on your /ebs/config for instance. You might need them in the future (when you loose everything, haha ;) Restoring a backed up AMI instance is simple. Launch a new instance using the AMI you generated, attach the Amazon Elastic IP address to it and voila. Way too simple.

About the EBS, there are quite a few things you should be able to do with it before continuing. Restoring a backed up Snapshot: Create Volume from Snapshot, umount /ebs, deattach old volume, attach new volume, mount /ebs. Cool? Be careful when you’re resizing your EBS. The xfs filesystem automatically grows as far as I know, but in my case I use the ext3 filesystem. So if you need to grow your ext3 EBS you’ll go:

  1. Create a Snapshot
  2. Create a new EBS Volume from that Snapshot you created (say 10 GB if you were running 5 GB)
  3. Attach it to your Instance, say /dev/sdg
  4. Use the resize2fs command to resize the partition to 10GB
  5. Mount it to /ebs2 or whatever
  6. Check to see if everything’s in place
  7. Unmount /ebs2, deattach /ebs2, unmount /ebs, deattach /ebs
  8. Attach the 10GB volume to where /ebs was attached (/dev/sdf)
  9. Mount /ebs and start your services

There you go, back to work, server. By the way, when working with Amazon AWS, note that you should be working in the same region where your AMI is (us, eu, east, 1c, …) otherwise some of the options (when attaching, etc) might just not come up. Beware of that.

Well, I guess those are pretty much all the basics. Don’t forget to read the Amazon S3 tutorials and API, pretty sweet stuff! Good luck.

Have You Tried the Amazon Web Services?

Amazon EC2, EBS, S3.. I’ve been looking for the perfect web hosting for over two years now. Is this it?

A few months ago I really liked MediaTemple cause they offered pretty good US hosting starting from $20/mo, which was quite good for the Foller.me project, so at the starting point I chose them. Their service is cool, definitely worth the money, but. A few weeks have passed, along with some major development on the service update and I got stuck with MySQL and overall server performance. It’s pretty tough to scan through 2,000,000 relations from @cnnbrk and then geocode their locations so I thought that I need to fine-tune MySQL and work out a more powerful caching system.

Yes, MediaTemple do offer dedicated MySQL grids for $50/mo, so that’s $70/mo overall. Not that bad, but thinking ahead, I’d also like to tweak up my http server, so that’d be a virtual dedicated plan for $50/mo, which makes $100/mo in total. Woah! And that’s just the start (around 500 megs RAM, 20 GB disk space and 1 TB bandwidth).

Now the Amazon Web Services offers a 2 GB RAM, 1.6 GHz virtual machine for only $0.10/hr, that makes ~$70/mo. Put up an Elastic Block Store (EBS) up to 1 TB and attach it to the instance around $20/mo. and perhaps an Amazon S3 bucket $10/mo. That makes about $100/mo in total. It’s not just the price though, I loved the way you’re in total control of whatever is happening on your server. You tune it however you like, whenever you like. Save bundled volumes and start over at any time. One-click EBS volume backups, elastic IP address and up to 20 instanced running simultaneously (you can increase this number by contacting Amazon). You also get to pick whatever OS you’d like to run (they’re called AMIs). You can build your own bundled OSs and make them available public.

Oh, and one of the best things about Amazon EC2 (Elastic Cloud) is that it’s so flexible! Switching servers has never been so easy. Start a new instance, attach an EBS, tune it up. Associate your old Elastic IP address to the new instance and voila! Go ahead and terminate your old instance, cause you’re riding your new mustang now!

I’m also sure that you can setup multiple servers and network balancers.. Like clustered computing y’know, the possibilities are endless! But I’m too far away from that at the moment, though I’m sure that whenever I have some free time, I will throw some experiments in that field ;) I’ve already setup Trac and SVN server a few days ago, works great!

Virtual Private Servers, Dedicated Servers, blah blah blah. Those are from the past. It’s Amazon Web Services. Go get your account right now ;)

Linux Dummy: Unscheduled Maintenance

If anyone of you have tried to access the blog yesterday night, you might have noticed that nothing was working. Sorry! I’ll say it straight, it’s completely my fault. Yesterday evening I decided to set up a cron job for automatic backups on my VPS – a full MySQL dump and a compressed archive of the www directory. So I got a couple of error messages stating that I don’t have the right to access some files which were in the wp-content/upload and wp-content/cache folders… I was frustrated!

Next… Never attempt to do this, okay? I logged in as root, changed owner on all files and folders including sub-folders of the www directory, set it to kovshenin:kovshenin. Voila, the backup worked! In a couple of minutes my VPS ran out of memory and I couldn’t even logon via SSH to reboot the server!

Now that’s funny! I called my hosting provider this morning and asked them what happened? They said everything’s fine, rebooted my server. I managed to logon by SSH, ran the “top” command, and looked at my memory usage growth! 100% was reached in 17 minutes, and bang! Disconnect. Two more calls to my provider didn’t help. They said that the only thing they can do is reset my yesterday’s VPS state completely.

So what really happened? I’m not sure but I bet it’s the WP-Super Cache plugin for WordPress! You see, cached files were created by the user that the httpd (apache) daemon ran – thus, one called “webmaster”. The user “kovshenin” apperantly didn’t have access to those files, and the change owner command spoiled all the cache! Now the static files were owned by “kovshenin”, and “webmaster” (apache) didn’t have any rights for those files. WP-Super Cache must have been in an infinite loop trying to access those, and of course, with no luck – therefore memory leak.

After another reboot I managed to quickly get into the WordPress control panel, enable Maintenance Mode and disable all the other plugins. Enabled them one by one. Setting 0777 as the rights for the cache directory and two WP-Super Cache config files solved the problem. The site was working fine again, and the new generated cache files were owned by “webmaster”… The day has been saved.

But what about the backups? Finally, I came to a thought that both “kovshenin” and “webmaster” users should be in the same groups. So I added “webmaster” to the “kovshenin” group, and “kovshenin” to the “webmaster”. Everything’s great! Apart from the fact that my Google Analytics now shows 0 visitors for 21.05.2009. Jeez, what a dummy…

Benchmarking: Your Web Hosting is Not That Perfect

Today I realized that the VPS I’m renting for $20/mo is not as good as it seemed at first. Ever thought about high loads? Okay, this may sound like some DDoS hacking tools, but no! 100 requests with 10 simultaneous made my virtual private server think for ~ 1,5 minutes. Jeez!

It took me quite some time to find good software for running some load tests on my webserver, linux has some good utilities (linux.com/feature/143896), but I suggest you start from ApacheBench which is a command line utility bundled with the Apache distribution. It’s cross-platform, therefore you can use it on Windows (I did). Anyways, here’s how you launch a test:

ab -n 100 -c 10 http://www.microsoft.com/

Why did I pick Microsoft? Well, if I get like 10,000 views tomorrow and everybody tries that command, that’d be a DDoS attack on Microsoft servers and I think they’re good enough to handle it. My server would just explode :)

Anyways, take a look at what the results may be like:

Benchmarking www.kovshenin.com (be patient).....done

Server Software:        Apache/2.2.8
Server Hostname:        www.kovshenin.com
Server Port:            80

Document Path:          /
Document Length:        84 bytes

Concurrency Level:      10
Time taken for tests:   90.984 seconds
Complete requests:      100
Failed requests:        1
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 0)
Write errors:           0
Non-2xx responses:      100
Total transferred:      36564 bytes
HTML transferred:       8674 bytes
Requests per second:    1.10 [#/sec] (mean)
Time per request:       9098.438 [ms] (mean)
Time per request:       909.844 [ms] (mean, across all concurrent requests)
Transfer rate:          0.39 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   15   3.4     16      16
Processing:  2203 8866 8879.2   6188   48797
Waiting:     1969 8532 8664.9   5891   48750
Total:       2219 8880 8879.6   6203   48813

Percentage of the requests served within a certain time (ms)
  50%   6203
  66%   7281
  75%   8141
  80%   8313
  90%  17078
  95%  32266
  98%  43813
  99%  48813
 100%  48813 (longest request)

Ah.. And a failed request there, how sad… You might also want to check out your load on the server while benchmarking. Use the ‘top’ command, it should produce similar output:

Yup, although the super cache plugin is working, wordpress consumes a lot of memory… I also ran this with a 500/100 requests, that made my server go down for like 6 minutes, I had over 200 failed requests and my blog kept saying database connection error until the test had finished. Free memory dropped down to 0! Scary? For more information about how ab works, read Apache HTTP server benchmarking tool documentation at apache.org.