Arrived early morning, not much sleep, took forever to get to hotel and then had to wait for room (double not happy), got some work done in the office and am now in desperate need of sleep before a way too early start.
(The weird thing is that the reason behind this post has nothing to do with being in Japan)
For a start, you shouldn’t be using a web forum. Are you too stupid to use email or what?
Ask yourself, does your question boil down to “pls do my job 4 me. kthxbye” (because it’s in stupid speak because it’s an obviously stupid question to be asking). If it does, don’t post it. Instead, RTFFM (where F is for Fine…. at least one of them is anyway), look at the archives. You will then find approximately eleventy-billion messages pointing you to exactly the right tool and docs to let you answer your question – easily.
So, if anybody is crazy enough to grab patches from the commits@lists.mysql.com list and try and use them… you may have run into this problem (which I do every few weeks/months): new files aren’t created. That’s right kids, BitKeeper (or at least the post-commit hook that mails out the patches) produces patch files where new files are in a completely different format than GNU patch expects. Not even BK can import these patches (bk import -temail or -tpatch just don’t do it).
So… I present this script – unfuck_bk_patches.pl which once run across a patch that includes new files, allows you to apply it using patch (or, for example, quilt).
I release it under the “sworn at software license”… which means you’re allowed to do anything you want with it as long as at some point you have sworn at computer software for being crappy.
Thoughts on latest changes:
However, I have other thoughts (that are a lot more interesting to discuss):
We should:
Past problems:
There are a lot of things looking good and being done right too (or if not right, a lot better than a year or two ago). e.g.
Things we should fix with commits, code review and all that:
Things we should fix internally:
Things I don’t think will happen but could be useful…:
What if it all goes proprietary:
My thoughts on the non-free Network Monitoring and Advisory Service:
P.S. there’s nothing here I wouldn’t say to anybody… and they’re welcome to disagree (and they do… sometimes even for good reasons).
* dchinner hands MacPlusG3 a bigger knife….
(on #xfs yesterday)
Know how to do C/C++ on Win32 and Linux/UNIX? In Melbourne? Want some contract work (we also have permanent positions that you may be suitable for) – email me your resume (in plain text) NOW!
So, last night a group of us sat down and went through all the mini-conf proposals for linux.conf.au 2008
There were a lot of proposals. There were also a lot of good ones.
We’re not announcing anything yet… but in the interest of openness… here’s the procedure.
We started out as any responsible group of selectors would…. looking at the proposals over beer:
a few jokes thrown in… frank discussion and all that. But really, we came to the conclusion that it’d been all done before and we needed to somehow narrow down all the excellent suggestions…
Luckily, the pub we were meeting at had the right facilities!
And we went about selecting a few more…
Of course, there are simply some mini-confs that we all agreed were a must have…Â although nothing was certain…
One of the more hilarious suggestions of the evening was to force somebody to organise a PostgreSQL miniconf, convince Marten to hold a MySQL company meeting in Melbourne around Jan 2008 and have all of MySQL AB come and sit in the back of the room for the PostgreSQL miniconf.
I’m not quite sure how to feel about this but I feel today has been more productive than all of last week. hrrm..
(Filling 1 through 41 is an exercise left to the reader… I just like the number 42)
handler::info can be called on a handler that has never had ::exeternal_lock called. So if you rely on a call to handler::external_lock to set up something (e.g. a pointer to a transaction object), you may explode in a heap.
See: Bug#26793
A little while ago now, I did some tech reviewing of a book called Backup & Recovery
… specifically MySQL related things (and MySQL Cluster). Curtis was kind enough to send me a copy of the book as well – and I’ve been reading the rest of it bits at a time since I got it.
I’m rather impressed… it gives a good mix of overview and digging deeper on just about every way to back up and recover systems. It also discusses several products that I didn’t know about (and have partly investigated now because of it).
It also has good sections on process: as in how to decide what to backup, encouraging the use of checklists and all that. Heck… recently when doing a restore I realised I never backed up /boot (annoying, not catastrophic… as I know my way around).
I recommend getting a copy of it if you need to back up and restore systems and don’t know everything already.
ndb_mgm> all report MemoryUsage
Node 1: Data usage is 11%(632 32K pages of total 5440)
Node 1: Index usage is 22%(578 8K pages of total 2592)
Node 2: Data usage is 61%(3331 32K pages of total 5440)
Node 2: Index usage is 40%(1039 8K pages of total 2592)
ndb_mgm>
Oh, and that’s coming from saved command history.
(as seen when upgrading my cluster here to mysql-5.1.19 ndb-6.2.3 – i.e. MySQL Cluster Carrier Grade Edition – i.e. the -telco tree)
Over the past N weeks/couple of months, we’ve been making a number of improvements to how backups are done in MySQL Cluster.
Once you get to large data sets, you start to really care about how long a backup takes.
Traditionally, MySQL Cluster has been in-memory only. The way to back this up is to just write from memory to disk (rate limited) and synchronised across the cluster. Since memory is really fast (compared to the rate we’re writing out to disk) – never had a problem.
In MySQL 5.1 (and Cluster Carrier Grade Edition- CGE), disk based attributes are supported. This means that a row has both in memory and disk based parts. As we all (should) know, disk seeks take a very long time. We don’t want to seek.
So, at some point recently we changed the scanning order from in-memory order (which previously made perfect sense) to on disk order. Randomly seeking through RAM is much cheaper than all the disk seeks. This greatly improved backup performance.
We also did some read-ahead work, which again, greatly improved performance.
Today, I see mail from Jonas about changing the way we read tuples for backup (and LCP) to make it even more efficient (READ_PACKED). This should also reduce CPU usage for LCP/Backup… which is a casual issue. I should really take the time to look closely at this and review.
I also wrote a patch to the code in NDB that writes files to disk to write a compressed gzio stream instead of an uncompressed one. This happens in a different thread, so potentially using one of those CPU cores that ndb wouldn’t otherwise use… and also dramatically reducing the amount of data written to disk…. this patch isn’t in any tree yet, and I’ve yet to try it with the READ_PACKED patch, which together should work rather well.
I also need to grab Brian at some point and find out why azio (as used by the ARCHIVE engine) doesn’t work the same way as gzio for basic stream writing…
somewhere in these 1.6GB of debug log files and signal traces… somewhere… really.
I’m going and stealing Jay’s idea (who stole it off Brian Duff… but his was for Oracle so obviously doesn’t count :)
So, my five wishes for MySQL Are:
5. Six-monthly release cycles
Getting a release out there takes way too long. There’s a variety of reasons, but seeing the amazing success of other free software projects taking the shorter release cycle, with each release not being too ambitious, I’m pretty convinced.
Although I think our increased use of pushbuild has helped immensely with the general quality of the tree, there’s a lot more that can be done…
4. Much more in depth automated testing
For MySQL Cluster we have (in parts) an insanely detailed test suite… lots of error injection to test failure (or, more importantly, recovery), verifying data, transactional consistency etc. Also, tests that run multiple connections, threads, simultaneous transactions and test the results programatically (comparing strings is generally not very useful… even when it comes to something as simple as IPv4 or IPv6 address!).
Part of getting here is the SoC project I’m mentoring… which should have an output of some utilities for helping write better tests (rather like we have for NDB… but for SQL instead of NDB API).
3. Sane build system
Something that not just the build team knows how to produce a binary that’s released. The other side of this is a way to build your local working tree (and test it) on BLAH platform, without pushing.
3.5. (yes, i have a 3.5)
Kill HPUX. There’s a reason the name sounds like a disease.
2. Increased liberal use of asserts
This is just a wish from a bug I’ve been tracking down where the code to build an I_S table didn’t check that we opened all the tables successfully before calling handler::info()… which promptly goes fooey because the handler hasn’t opened the table. Luckily here it’s ended in a crash… IMNSHO it’s better to bail out on an assert than possibly return crap to the user…
1. Pluggable data dictionary
There are so many oddities and “bugs” around related to the casually strange mix of data dictionaries around the MySQL Server (and Federated, NDB, InnoDB, Falcon etc) that we could do a lot better if we had a “Virtual Data Dictionary Layer”… where engines who really do have a reason to not be simple (e.g. NDB) could plug in and the MySQL Server could get all the metadata consistently from the cluster (with out own form of sane-ish caching).
This would also solve the online ALTER TABLE frm consistency problem with engines that may roll back the alter table on crash recovery (or not support querying the index until it has finished being built).
Perhaps this is something for either the next engine summit or dev conf…
So, I hear you ask (ask dagnammit!), what does a MySQL Cluster hacker do when not hacking MySQL Cluster?
Well,
Today, a box arrived:
I eagerly opened the box… eager for what was inside:
Hrrm… this does look promising! Quickly I turned it all up the right way:
It’s here! It’s here! Only about a month after Michael got his copies! Gah international shipping (and moving addresses) can be casually annoying sometimes. But anyway, it’s here, it looks sweet and now I can go set up a menu option to automatically transcode MythTV recordings to a good format for the iPod. Brilliant!
Of course, this is about how many copies you should go and buy:
So kids, go order your copy of Practical MythTV today! You too, like me, can then go “hrrm… my MythTV MySQL database isn’t Clustery enough… perhaps I should run it on NDB”. :)
You should also go and get a copy of the MySQL Cluster Certification Study Guide as I have writing in it too! Yes, this past year has kept me busy!
Today, immediately after lunch, I got IO errors from my laptop hard drive (ironically while attempting a file system dump). Words to the affect of “oh gosh and bother” exiting my mouth and the decision was made to go get a new drive.
Well… one “shortcut” to the computer store later, have new HD (will travel).
Backup from previous night, xfsrestore here I come. And a good number of hours later… about 1.5million files restored.
I do wish file systems had REPEATABLE_READ though… that would be nice.
Linus is right when he talks about the performance of SCMs…. and that BitKeeper was about the first one to be worth using at all (really).
But as an interesting speed comparison… I’ve managed to pull the latest git (with git) and build it in less time than BitKeeper has taken to pull the latest NDB tree…. hrrm..
One of the reasons I’m so enjoying quilt for every day hacking is that it is blindingly fast.
As Roland pointed out, recently a bunch of us took the MySQL Cluster Certification Exam. I was strongly encouraging people to take it (and, in fact, got the majority of people to take it… except those working on urgent customer issues or were rather new to the product). That’s one of my cunning plans (number 8 IIRC) – to have anybody doing anything to do with MySQL Cluster inside MySQL AB to be certified (in cluster, not insane).
Response from the exam is overwhelmingly positive and as it’s new, afterwards I discussed a few questions with Roland (in fact, all the ones I got wrong). The (good!?) news is that I legitimately brain-farted on some.
As for bragging rights… I currently share the top score. I’ll also (happily) buy a (large) beer for the first person to score higher than I did. So get cracking!
(oh, warning.. this isn’t some trivial know-a-tiny-bit-and-score-really-high certification… this is a serious one).
P.S. I may have heard at some point one of our people saying they thought it was the best certification exam we have…
P.P.S. Yes, this means you should go strive towards getting it if you do anything with MySQL Cluster.
P.P.P.S. If you’re an employer/manager with people doing stuff (or will be doing stuff… or evenly possibly going to do stuff) with MySQL Cluster, get them certified!