Success...MySQL Slave Database on S3 Mount


April 11, 2008 by Tom (2 Comments)

So it worked. From everything i could tell...and it was very easy to setup. In fact, if you already have a MySQL slave running you can even move it to an S3 mount with ease. Here's what I did:

  1. Setup MySQL slave by making a "my_slave.cnf" file and making a new folder in the same area the mysql master is, but just named it mysql_slave instead. Then matched that in the cnf file so that the datadir was pointed there and the socket. Then also included the necessary settings to make it a slave (and the my.cnf file a master). Started it, started slave, voila. So if you follow MySQL's site's intstructions you'll be ok.
  2. Then...I setup ElasticDrive and mounted S3 as a partition. Also fairly straight forward. I mounted using the "direct network storage" method/use case as described in the readme for ElasticDrive (not RAID).
  3. Then I created a directory on the S3 mount, chown it to mysql:mysql, shut down the slave, copied the entire directory on the EC2 instance where the slave was to the new directory on the S3 mount (again making sure the ownership is mysql:mysql and making sure all paths matched up as I defined in the my_slave.cnf file.
  4. Then I restarted the MySQL slave server. It automatically started up where it left off...no issuing of any commands to change master or start slave. It just picked up as if it never was shut off. Of course I also didn't do any writing to the master during this time period. If I had...I'm sure I would have had to resync the databases.
  5. I tested. I wrote the post below.
  6. I shut down the slave on S3...I tried starting up the slave server restoring the my_slave.cnf I created to run off the EC2 instance location...of course it had an error and couldn't start (because it was out of sync).
  7. I then removed the mysql_slave diretory on the EC2 instance and copied it over from the S3 mount. Restarted the slave server and voila. It picked up like nothing ever went wrong.

So that's it folks. It's EXTREMELY easy...but what wouldn't be easy is if you were doing this on a site that was constantly being updated. You'd have to stop all writing to the master DB for the brief period of time it would take to setup a new slave. If no data has been written since the time the slave went down...you'd be in great shape. It took less than 5 minutes to copy files and setup that slave again. On the other hand, if the master is out of sync with the slave...you'd have to do a database dump and setup the slave all over again (following the instructions off the MySQL site). The change master  and start slave commands, etc. Not a super big deal, but also not as quick.

What I really liked about this was that changing the location of the slave was extremely easy (stopping, editing 2 lines in a cnf file, starting again) and fast.

What I don't like so far about having the MySQL slave on the S3 mount using ElasticDrive is that the requests are racking up. To me it seems like I'll rack up S3 costs in requests and not from storage or transfer. It seems like ElasticDrive makes a rediculous amount of requesets (during this test period and moving of files, etc. I seemd to have charged about 1-2 cents in requests..but that's for what? 30 min tops?)

The next thing I need to find out is if there's a more efficent way ElasticDrive (or another solution) can work with S3 so the requests aren't out of hand...or what I REALLY wish was that there would still be support for AOL's XDrive (which has 5GB free)...or that Google/Yahoo/whoever would jump on the bandwagon and some sort of free solution ... or at least free requests, would be available.

Pretty cool and pretty powerful stuff though if I was running a site and had the budget to tack on another 2-4 cents an hour on top of the 10 cents an hour from the EC2 instance...and of course the transfer/storage (which wouldn't amount to anything unless the database was huge). 

Simpe Test...


April 11, 2008 by Tom (0 Comments)

So my first attmept here to write data to the master database while the slave's config is pointed to an S3 mount using ElasticDrive...The slave database started...and the status shows it as being on and awaiting data. So here's some data.

We'll see what the delay is (if I can tell without any kind of exact measurement provisions in place) and if it works period. 

Edit: Looks like it worked exactly how I'd hope it would. The slave database picked up the change and I double checked the old directory where the mysql sock file and data was vs the new location on the S3 mount and sure enough a mysql sock file is on the S3 mount directory. So I'm assuming (without removing the slave data on the EC2 directory) that it's working. Though I will probably turn off S3 and check out some other backup solutions since I don't make a whole lot of changes to my site...I don't need an instant backup. I also have to make sure my files are backed up anyway, part of which I'm hoping I can use a simple command for in the SVN post-commit hook script. 

MySQL Master - Slave


April 11, 2008 by Tom (0 Comments)

Blah. Finally get the master - slave setup working here. It's just two instances of mysql writing to two different locations on the EC2 instance. The next step will be to move the location of the datadir over to an S3 mount.

That I'll save for the weekend though because I am simply out of steam for tonight. Too many late nights lately. The cool thing is I figured out how to setup the replication after many many bad tutorials and carelessly reading tutorials meant for mysql 4. MySQL 5 is much different and, in my opinion from what I saw, is much easier to setup master - slave replication.

I simply made a new cnf file for the slave, setup new datadir, different port to run on, told it was the 2nd mysql server, installed base tables, did a dump from the current db to sync up this blog's database. Setup user on the master...and after a few easy SQL queries (in phpMyAdmin actually since I also setup a new admin for the slave) I was able to start the slave and voila.

What can I do with a master slave? Well I may actually want to setup another slave for the application. This slave is for backup purposes to S3. If for whatever crazy reason S3 has a problem (because THAT never happened before hehe) then I wouldn't be able to read from it...so we want to read from the instance I think...but if I want to be efficient I may write to the master only and ALWAYS read from one fo the slaves (the one on the EC2 intance). This is SUPER easy with CakePHP, there's a great post on the CakePHP Google Groups for master - slave database. This type of setup probably wouldn't matter too much for me since I don't really do a ton of writing. I don't have to balance and disperse the load between writes and reads...Unless I started getting millions of comments every day I guess.

More to come...and if changing the datadir to an S3 mount through ElasticDrive is as easy as I think it is...I'll have some great news soon. 

EC2 Backup Ideas: SVN


April 10, 2008 by Tom (0 Comments)

So as many already know, EC2 doesn't save data if the instance is turned off...in fact, as of right now my data isn't backed up. I have an image, but it's dated, meaning this post won't be "safe" if something were to happen.

While I don't expect my instance to crash and while I'm not doing anything that would make it crash, you never know. So everyone is figuring out backup and failover solutions. In fact, it's a requirement for any production site.

So I'm going to jot down a list of possible solutions and tips over time so I can go back to figure out which to employ.

First, Subversion is your friend. I already have this site on SVN. I don't use FTP (bleh) any longer in fact. All changes to this site are done via SVN post commit scripts. Meaning after I make a change and submit it to the repository for versioning, it updates this site with just the changes. This really saves a boat load of time.

We can use Subversion and it's great hook scripts for more though. In addition to running an update command on the working copy that runs this site, we can also mirror our SVN repository to an S3 account. Not only are we backing up to S3 now (on each change to the site), but we're getting it all under version control.

The downside in my mind to this is that I make a bunch of changes (especially in development) so this will give us an increased number of requests which can add up. I really think requests are the silent killer when it comes to S3. So perhaps instead of running on the post-commit, we can simply setup a cron job to run the command to sync the repository every day or whatever time period.

Remember with SVN: It only updates what has changed, so the transfer in will be much less than if you sent all the files or even the changed file. SVN is jus sending the changes.

So that's two options.

1. Keeping the site's code completely backed up from the minute it changes.

2. Backing it up on a daily/hourly/etc. basis.

It will completely depend on the level of risk you can or wish to take. I'll most likely (I haven't set this up yet) run a cron job every day. Also important to note: If there are no changes, nothing happens. So this backup is efficient.

Drawbacks: It won't copy over the database or any images/files that were uploaded outside the SVN. For example, I can attach an image with this post and SVN won't know about it. So it would be lost along with this post. A mySQL replication/backup solution would have to be figured out along with some sort of rsync of the files (ignoring .svn directories) to catch the images. This does backup the site's scripts as well...but doesn't put them under version control which is a very nice thing to have. So in the event of a crash on a larger site, not only would there be recovery, but all the changes that were made to the site would still be saved...Very handy if it was a scripting error that caused the instance to crash somehow. You could pinpoint it easier.

Here's a more info about how from the free SVN book.

Cleaning Up...


April 9, 2008 by Tom (0 Comments)

There will be some cleaning up going on to this blog application. I will also be changing up the way it looks, probably in line and on track with the templating system...which of course just uses CakePHP's "themes" approach.

Though the blog's functionality is pretty far along, so I'm going to be working on some refinements to what I have now. More importantly, I'm going to be working with setting up load balancing solutions and take care of other scalability tasks. This way I can move forward with a more prepared system.

The overall goal here, in case anyone was wondering, is to build a blog (media blog) application that is infinitely scalable and hopefully *impossible* to crash. Of course with the ability to extend the base code and have more than just a media blog in the form you see here.