Ok, so I moved to Amazon EC2 from the old host, www.apthost.com. Apthost was great (and I still have them of course for a while), but oddly enough my sessions started having issues and I couldn't log in to the admin back end of my application...I thought I broke my code...naturally I check it locally then on another server of mine. It worked. So I'm going to assume (without spending a ton of time) that Apthost did something to their php.ini or other settings that created and issue.

There's a lot of problems like this with shared hosting and that's ultimately the reason why I'll never go back...after I start moving off all my things from Apthost, Lunarpages, and some other host that I really shouldn't have. Now, EC2 has it's drawbacks of course as well. I hope to outline some of these and other topics related to EC2 in this blog along with posting updates about what I'm doing with Minervablog, which is of course a media blog. Maybe I'll even have some screencasts. Important to note though is that Minervablog isn't dead in the water (for the very few that have seen it so far) and it is probably going to take on some new changes due to EC2...I can now program it to infinitely scale and test that ability out without incurring a ton of charges. Then who knows, you might see an AMI out there of Minervablog =)

So as many already know, EC2 doesn't save data if the instance is turned off...in fact, as of right now my data isn't backed up. I have an image, but it's dated, meaning this post won't be "safe" if something were to happen.

While I don't expect my instance to crash and while I'm not doing anything that would make it crash, you never know. So everyone is figuring out backup and failover solutions. In fact, it's a requirement for any production site.

So I'm going to jot down a list of possible solutions and tips over time so I can go back to figure out which to employ.

First, Subversion is your friend. I already have this site on SVN. I don't use FTP (bleh) any longer in fact. All changes to this site are done via SVN post commit scripts. Meaning after I make a change and submit it to the repository for versioning, it updates this site with just the changes. This really saves a boat load of time.

We can use Subversion and it's great hook scripts for more though. In addition to running an update command on the working copy that runs this site, we can also mirror our SVN repository to an S3 account. Not only are we backing up to S3 now (on each change to the site), but we're getting it all under version control.

The downside in my mind to this is that I make a bunch of changes (especially in development) so this will give us an increased number of requests which can add up. I really think requests are the silent killer when it comes to S3. So perhaps instead of running on the post-commit, we can simply setup a cron job to run the command to sync the repository every day or whatever time period.

Remember with SVN: It only updates what has changed, so the transfer in will be much less than if you sent all the files or even the changed file. SVN is jus sending the changes.

So that's two options.

1. Keeping the site's code completely backed up from the minute it changes.

2. Backing it up on a daily/hourly/etc. basis.

It will completely depend on the level of risk you can or wish to take. I'll most likely (I haven't set this up yet) run a cron job every day. Also important to note: If there are no changes, nothing happens. So this backup is efficient.

Drawbacks: It won't copy over the database or any images/files that were uploaded outside the SVN. For example, I can attach an image with this post and SVN won't know about it. So it would be lost along with this post. A mySQL replication/backup solution would have to be figured out along with some sort of rsync of the files (ignoring .svn directories) to catch the images. This does backup the site's scripts as well...but doesn't put them under version control which is a very nice thing to have. So in the event of a crash on a larger site, not only would there be recovery, but all the changes that were made to the site would still be saved...Very handy if it was a scripting error that caused the instance to crash somehow. You could pinpoint it easier.

Here's a more info about how from the free SVN book.

Blah. Finally get the master - slave setup working here. It's just two instances of mysql writing to two different locations on the EC2 instance. The next step will be to move the location of the datadir over to an S3 mount.

That I'll save for the weekend though because I am simply out of steam for tonight. Too many late nights lately. The cool thing is I figured out how to setup the replication after many many bad tutorials and carelessly reading tutorials meant for mysql 4. MySQL 5 is much different and, in my opinion from what I saw, is much easier to setup master - slave replication.

I simply made a new cnf file for the slave, setup new datadir, different port to run on, told it was the 2nd mysql server, installed base tables, did a dump from the current db to sync up this blog's database. Setup user on the master...and after a few easy SQL queries (in phpMyAdmin actually since I also setup a new admin for the slave) I was able to start the slave and voila.

What can I do with a master slave? Well I may actually want to setup another slave for the application. This slave is for backup purposes to S3. If for whatever crazy reason S3 has a problem (because THAT never happened before hehe) then I wouldn't be able to read from it...so we want to read from the instance I think...but if I want to be efficient I may write to the master only and ALWAYS read from one fo the slaves (the one on the EC2 intance). This is SUPER easy with CakePHP, there's a great post on the CakePHP Google Groups for master - slave database. This type of setup probably wouldn't matter too much for me since I don't really do a ton of writing. I don't have to balance and disperse the load between writes and reads...Unless I started getting millions of comments every day I guess.

More to come...and if changing the datadir to an S3 mount through ElasticDrive is as easy as I think it is...I'll have some great news soon. 

So it worked. From everything i could tell...and it was very easy to setup. In fact, if you already have a MySQL slave running you can even move it to an S3 mount with ease. Here's what I did:

1. Setup MySQL slave by making a "my_slave.cnf" file and making a new folder in the same area the mysql master is, but just named it mysql_slave instead. Then matched that in the cnf file so that the datadir was pointed there and the socket. Then also included the necessary settings to make it a slave (and the my.cnf file a master). Started it, started slave, voila. So if you follow MySQL's site's intstructions you'll be ok.
2. Then...I setup ElasticDrive and mounted S3 as a partition. Also fairly straight forward. I mounted using the "direct network storage" method/use case as described in the readme for ElasticDrive (not RAID).
3. Then I created a directory on the S3 mount, chown it to mysql:mysql, shut down the slave, copied the entire directory on the EC2 instance where the slave was to the new directory on the S3 mount (again making sure the ownership is mysql:mysql and making sure all paths matched up as I defined in the my_slave.cnf file.
4. Then I restarted the MySQL slave server. It automatically started up where it left off...no issuing of any commands to change master or start slave. It just picked up as if it never was shut off. Of course I also didn't do any writing to the master during this time period. If I had...I'm sure I would have had to resync the databases.
5. I tested. I wrote the post below.
6. I shut down the slave on S3...I tried starting up the slave server restoring the my_slave.cnf I created to run off the EC2 instance location...of course it had an error and couldn't start (because it was out of sync).
7. I then removed the mysql_slave diretory on the EC2 instance and copied it over from the S3 mount. Restarted the slave server and voila. It picked up like nothing ever went wrong.

So that's it folks. It's EXTREMELY easy...but what wouldn't be easy is if you were doing this on a site that was constantly being updated. You'd have to stop all writing to the master DB for the brief period of time it would take to setup a new slave. If no data has been written since the time the slave went down...you'd be in great shape. It took less than 5 minutes to copy files and setup that slave again. On the other hand, if the master is out of sync with the slave...you'd have to do a database dump and setup the slave all over again (following the instructions off the MySQL site). The change master  and start slave commands, etc. Not a super big deal, but also not as quick.

What I really liked about this was that changing the location of the slave was extremely easy (stopping, editing 2 lines in a cnf file, starting again) and fast.

What I don't like so far about having the MySQL slave on the S3 mount using ElasticDrive is that the requests are racking up. To me it seems like I'll rack up S3 costs in requests and not from storage or transfer. It seems like ElasticDrive makes a rediculous amount of requesets (during this test period and moving of files, etc. I seemd to have charged about 1-2 cents in requests..but that's for what? 30 min tops?)

The next thing I need to find out is if there's a more efficent way ElasticDrive (or another solution) can work with S3 so the requests aren't out of hand...or what I REALLY wish was that there would still be support for AOL's XDrive (which has 5GB free)...or that Google/Yahoo/whoever would jump on the bandwagon and some sort of free solution ... or at least free requests, would be available.

Pretty cool and pretty powerful stuff though if I was running a site and had the budget to tack on another 2-4 cents an hour on top of the 10 cents an hour from the EC2 instance...and of course the transfer/storage (which wouldn't amount to anything unless the database was huge). 

Ok so the game plan here. This is a version of my application, Minevablog. It needs some cleaning up though. Once I get that all set, I'm going to start stress testing. Well backup...First, well, a backup. Just to be safe. After that, I'll be setting up multiple instances and setting them up to be all load balanced and clustered.

Ok once that multi-instance environment is setup, THEN I'll stress test. Probably running load testing from another instance actually. I will hammer the ever living christ out of this blog and push it. I hope I can get an instance to crash. I want to see how I can keep the site alive under extreme conditions...How I might be able to turn on more instances for support, etc. I also want to benchmark things to see how speedy everything is.  

Mmmm, exciting.