linux.conf.au 2014

Perth, Western Australia - 6th to 10th January 2014

linux.conf.au


<-- Back to schedule

Abusing DRBD for low-downtime, multi-terabyte database migrations

How would you architect a database migration for a 1 GB Joomla website? How about a high-traffic 200 GB JBoss cluster? Now what about a 3 TB MySQL database for a large, well-known cloud service supporting more than ten frontends, dozens of asynchronous workers and millions of users worldwide? What if I told you it could be done in 10 minutes without the use of MySQL replication?

I will begin by providing a brief overview of the technologies in play here, including DRBD, Pacemaker and MySQL. While this is a technical talk, there will not be assumed prior knowledge of the details of high availability (HA) clusters or MySQL server administration.

The body of my talk will explain our solution for a very large, high-traffic MySQL database migration. I will also expand on various hurdles we encountered, including the reasons why MySQL replication is unsuitable for this use case. I will explain why we finally chose DRBD as the mechanism for syncing MySQL data to the new cluster, and how this idea was refined to minimise risk and downtime.

I also intend to discuss some of the challenges encountered during the implementation of this idea, ranging from ensuring consistency and correctness of the data to effectively testing the functionality of the new cluster prior to going live. Some trivial yet blocking problems were uncovered in the testing phase, which is why thorough testing in advance is so crucial for minimising downtime.

Finally, I will be concluding with an analysis of the effectiveness of this technique, both for the purpose it was used and in general. This will include a comparison of the resulting 10-minute downtime window with that expected of alternative migration options, as well as an explanation of why this turned out to be the best solution for our particular use case.

Steven McDonald

Steven is a systems administrator and geek in Sydney, Australia. He currently works for Anchor as part of a large team of sysadmins, and in his spare time makes the occasional small contribution to an open-source project. He also enjoys writing and playing music; going for long drives; and eating bacon, egg and anchovy sandwiches for breakfast.