With this revision, My xtrabackup branch has been merged into trunk.
What does this mean? It means that we now get a drizzlebackup.innobase binary which is the xtrabackup port for Drizzle. Exciting times.
“WL4271 Encrypted Online Backup: Preview 3†branch in Launchpad
Now with Windows support. Many thanks to Chuck Bell for helping get the code going on Windows.
We can however, all sit around dumbfounded as to how Windows has so little of a POSIX like layer and yet doesn’t define ENOTSUP.
As a refresher, this tree implements:
(world peace not included)
UPDATE: If you’re wondering why the branch isn’t there, it’s still pushing to launchpad. Yes, that’s over 7 hours to push a branch. ick. Can’t be too much longer, surely. I cannot wait until lp uses shared repos.
Quick experiment with online changing of enabling/disabling compressed backups and local checkpoints (LCPs).
Backup is incredibly trivial and correct (even have some nodes do compressed, some not).
LCPs are a bit trickier when it comes to restore… currently how the code sits is that a block using the compressed file interface in NDBFS must specify if it wants to use the compressed read/write interface or not. So when you have LCPs that differ in compressed/non-compressed than the current config file setting, you’re not going to be able to restore them (although setting CompressedLCP=1 should let you restore either compressed or non-compressed LCPs).
At some point, I’ll probably move AsyncFile (our async file IO class) to just use azio alway, and modify azio to be transparent for non-compressed files…. I just have to fix up azio for direct io.
Over the past N weeks/couple of months, we’ve been making a number of improvements to how backups are done in MySQL Cluster.
Once you get to large data sets, you start to really care about how long a backup takes.
Traditionally, MySQL Cluster has been in-memory only. The way to back this up is to just write from memory to disk (rate limited) and synchronised across the cluster. Since memory is really fast (compared to the rate we’re writing out to disk) – never had a problem.
In MySQL 5.1 (and Cluster Carrier Grade Edition- CGE), disk based attributes are supported. This means that a row has both in memory and disk based parts. As we all (should) know, disk seeks take a very long time. We don’t want to seek.
So, at some point recently we changed the scanning order from in-memory order (which previously made perfect sense) to on disk order. Randomly seeking through RAM is much cheaper than all the disk seeks. This greatly improved backup performance.
We also did some read-ahead work, which again, greatly improved performance.
Today, I see mail from Jonas about changing the way we read tuples for backup (and LCP) to make it even more efficient (READ_PACKED). This should also reduce CPU usage for LCP/Backup… which is a casual issue. I should really take the time to look closely at this and review.
I also wrote a patch to the code in NDB that writes files to disk to write a compressed gzio stream instead of an uncompressed one. This happens in a different thread, so potentially using one of those CPU cores that ndb wouldn’t otherwise use… and also dramatically reducing the amount of data written to disk…. this patch isn’t in any tree yet, and I’ve yet to try it with the READ_PACKED patch, which together should work rather well.
I also need to grab Brian at some point and find out why azio (as used by the ARCHIVE engine) doesn’t work the same way as gzio for basic stream writing…