If you ever get something like this:
bk clone -lq ndb bug25567
clone: unable to readlock /home/stewart/Documents/MySQL/5.1/ndb
then try this:
bk unlock -s
to remove stale locks.
I have no idea how anybody is meant to come to the command from the error message… blindly guessing ‘bk help unlock’ worked for me though.
lca2007 already totally rocks.
did the speaker thing this morning, which was awesome (and i won’t spoil the surprise)
mysql miniconf today, should rock.
Today I set up a mysqld connected to my not-quite-HA cluster at home here to replicate from my MythTV database into cluster. The idea behind this is to eat an increasing amount of my own dogfood around the house.
To do this, I also set up the MySQL Instance Manager to manage the now multiple instances of MySQL Servers on a box here. I found it a pain to do, it should be a lot simpler, but isn’t. At least now things are going okay…. but the feature wish list I have is rather long (perhaps I should hack some stuff up in this “spare time” i’ve been hearing so much about).
I’m also about 10 minutes (or however long the build takes) off moving one of the data nodes off the machine so it will be a real 2 node system (but I still have to move the management server to a third machine to have any real HA… and I have a PowerPC machine marked for that, I just have to await some patches to make it work :)
Currently though, my Gallery is being served off this. There are so many more photos I should add, I just haven’t come up with a decent way to interface f-spot and Gallery together – especially when I go back and retouch, delete or tag photos.
I’m involved in the authoring of two books at the moment – both using different tools, neither of which would be my choice if it was up to me. One is using DocBook, writing raw incredibly verbose XML… which honestly, isn’t that much fun. The other is in Microsoft Word (well, OpenOffice.org Writer for me). The last time I really used Microsoft Word really seriously was probably around 1998/1999 with Office 98 on the Mac. It was a pretty awesome suite of software actually. Especially after the update that fixed a few crashing bugs :)
One thing I do notice though is the collaboration tools in OOo Writer are nowhere near good enough. The notes are small yellow rectangles where you either have to hold the mouse cursor over them to read them (ick, slow) or double click them and scroll right forever to read the whole or in conjunction with the last way, use the object browser.
Also, track changes doesn’t really track changes to things you’ve changed. i.e. i cannot edit the same thing twice and keep both changes. urgh. I’m pretty sure MS Word let me do that… It certainly allowed me to use versions in a Microsoft Word format document – which, unfortunately, a lot of the world still primarily deals with.
If these few things were fixed it would be a much better word processor.
It’s always frustrated me how poorly word processors handle large files too (except perhaps Nisus Writer… that was certainly a neat app). Add a bunch of images and your file now takes ages to open and save. blergh.
As for figures in DocBook, there seems to not be much input and output processing… i.e. if you put in SVG and you output to HTML it doesn’t output very nicely (puts in SVGs into small windows) and probably completely doesn’t work on browsers that don’t do SVG.
I sometimes wonder if we’ve really moved on to something better than LaTeX and xfig…. okay, there are better tools than xfig for a large number of diagram types.
Over at the DevZone, MySQL 5.1.14 Downloads the cool kids are grabbing the latest 5.1 beta. Lots of Cluster fixes in this release too. We’re getting to a much more polished state for NDB with each release and that’s a good thing to see.
On a totally different topic, i bought a really sweet smelling mango today and cannot wait for the right time sometime this afternoon to eat it. All the summer fruits are really nice at the moment (benefit of being in a warm December I guess) and I’m loving it.
Although 37-41 degrees (Celcius, duh) can be less fun with a rather warm laptop.
I often see things go past my eyes where customers (and users – i.e. those that don’t send wads of cash our way and hence are not financially supporting my beer, curry and photography habits) have amazing uptime and reliability requirements.
When talking to businesses that use MySQL, it’s not uncommon to have the “if the DB is down, our business doesn’t operate” line bandied around. How people make sure this never happens can differ (hint: it often involves replication and good sysadmin practices).
One thing I like doing is making things easier for people. Sometimes it’s also a much more complicated problem than you’re initially led to believe.
I think configuration files are obsolete. Okay, maybe just for databases. Everything should be changable as an online operation. This should also be able to be done via a standard interface – in our case, SQL. This means it’s suddenly really easy to write portable UIs around the admin functionality (no getting the parsing and generation – most trickily, the modification of text based config files right) just the issuing of SQL to the server, relativly simple. This even enables web apps to tune the database a bit, opting for various amounts of automation for various applications – in a cross platform way!
One of my visions for NDB (MySQL Cluster) is to get rid of the (user visible) configuration file and manage everything through SQL (or management client, something like that). This way you could ALTER CLUSTER ADD NODE, ALTER CLUSTER SET DataMemory=4GB etc and things should “just work”, take however long it needs – without downtime.
In a clustered environment, we could do these operations transactionally so that in the event of node or system failure we have some hope of being in a nicely consistent state and that during system recovery (or node recovery) we’re not performing a configuration change in addition to restarting (e.g. if you edited a config file and then had a crash).
Config changes could also have EXPLAIN, a non-modifying operation that would EXPLAIN what would be done – e.g. rolling restart, taking approximately X minutes per node and Y minutes total. This could help in planning and scheduling of configuration changes.
(i wonder if that made any sense)
I normally read Planet MySQL from RSS, but for some reason I ended up on the actual site today in a Web Browser (Epiphany to be exact) and saw this:
And thought, “wow, ugly”. I don’t keep my browser “maximised” because I think it’s a stupid way to work – I often switch between tasks or like to have an editor open while referring to something in a browser (e.g. some tech details of some source module), or monitor IRC or IM. I remembered that Epiphany has an Ad Blocking extension, so in an effort to de-uglify, I enabled it. I now see:
Hrrm… much better. Notice how the links on the left to the most active are actually useful now (I can read them).
Note that this isn’t a rant on adverts on web sites – I can handle them (the google ones which aren’t obtrusive) – I’m against the uglyweb.
Spoke with Brian the other day on what was required to get NDB to be a pluggable engine – and started hacking.
The tricky bits invole dependencies of things like mysqldump and ndb_restore on some headers to determine what tables shouldn’t be dumped (hint: the cluster database used for replication).
Also, all those command line parameters and global variables – they’re fun too. It turns out InnoDB and PBXT are also waiting on this. In the meantime, I’ve done a hack that puts config options in a table.
Currently blocked on getting the embedded server (libmysqld) to build properly – but i now have a sql/mysqld binary with pluggable NDB. All libtool foo too.
Hopefully i’ll be able to post soon with a “it works” post
Inspired by PeterZ’s Opening Tables scalability post, I decided to try a little benchmark. This benchmark involved the following:
I wanted to test file system impact on this benchmark. So, I created a new LVM volume, 10GB in size. I extracted a ‘make bin-dist’ of a recent MySQL 5.1 tree, did a “mysql-test-run.pl –start-and-exit” and ran my script, timing real time with time.
For a default ext3 file system creating MyISAM tables, the test took 15min 8sec.
For a default xfs file sytem creating MyISAM tables, the test took 7min 20sec.
For an XFS file system with a 100MB Version 2 log creating MyISAM tables, the test took 7min 32sec – which is within repeatability of the default XFS file system. So log size and version made no real difference.
For a default reiserfs (v3) file system creating MyISAM tables, the test took 9m 44sec.
For a ext3 file system with the dir_index option enabled creating MyISAM tables, the test took 14min 21sec.
For an approximate measure of the CREATE performance…. ext3 and reiserfs averaged about 100 tables/second (although after the 20,000 mark, reiserfs seemed to speed up a little). XFSÂ averaged about 333 tables/second. I credit this to the check for if the files exist being performed by a b-tree lookup in XFS once the directory reached a certain size.
Interestingly, DROPPING the tables was amazingly fast on ext3 – about 2500/sec. XFS about 1000/sec. So ext3 can destroy easier than it can create while XFS keeps up to speed with itself.
What about InnoDB tables? Well…
ext3(default): 21m 11s
xfs(default): 12m 48s
ext3(dir_index): 21m 11s
Interestingly the create rate for XFS was around 500 tables/second – half that of MyISAM tables.
These are interesting results for those who use a lot of temporary tables or do lots of create/drop tables as part of daily life.
All tests performed on a Western Digital 250GB 7200rpm drive in a 2.8Ghz 800Mhz FSB P4 with 2GB memory running Ubuntu 6.10 with HT enabled.
At the end of the test, the ibdata1 file had grown to a little over 800MB – still enough to fit in memory. If we increased this to maybe 200,000 tables (presumably about a 3.2GB file) that wouldn’t fit in cache, then the extents of XFS would probably make it perform better when doing INSERT and SELECT queries as opposed to the list of blocks that ext3 uses. This is because the Linux kernel caches the mapping of in memory block to disk block lookup making the efficiency of this in the file system irrelevant for data sets less than memory size.
So go tell your friends: XFS is still the coolest kid on the block.
I’ve talked about disk space allocation previously, mainly revolving around XFS (namely because it’s what I use, a sensible choice for large file systems and large files and has a nice suite of tools for digging into what’s going on).Most people write software that just calls write(2) (or libc things like fwrite or fprintf) to do file IO – including space allocation. Probably 99% of file io is fine to do like this and the allocators for your file system get it mostly right (some more right than others). Remember, disk seeks are really really expensive so the less you have to do, the better (i.e. fragmentation==bad).
I recently (finally) wrote my patch to use the xfsctl to get better allocation for NDB disk data files (datafiles and undofiles).
patch at:
http://lists.mysql.com/commits/15088
This actually ends up giving us a rather nice speed boost in some of the test suite runs.
The problem is:
– two cluster nodes on 1 host (in the case of the mysql-test-run script)
– each node has a complete copy of the database
– ALTER TABLESPACE ADD DATAFILE / ALTER LOGFILEGROUP ADD UNDOFILE creates files on *both* nodes. We want to zero these out.
– files are opened with O_SYNC (IIRC)
The patch I committed uses XFS_IOC_RESVSP64 to allocate (unwritten) extents and then posix_fallocate to zero out the file (the glibc implementation of this call just writes zeros out).
Now, ideally it would be beneficial (and probably faster) to have XFS do this in kernel. Asynchronously would be pretty cool too.. but hey :)
The reason we don’t want unwritten extents is that NDB has some realtime properties, and futzing about with extents and the like in the FS during transactions isn’t such a good idea.
So, this would lead me to try XFS_IOC_ALLOCSP64 – which doesn’t have the “unwritten extents” warning that RESVSP64 does. However, with the two processes writing the files out, I get heavy fragmentation. Even with a RESVSP followed by ALLOCSP I get the same result.
So it seems that ALLOCSP re-allocates extents (even if it doesn’t have to) and really doesn’t give you much (didn’t do too much timing to see if it was any quicker).
I’ve asked if this is expected behaviour on the XFS list… we’ll see what the response is (i haven’t had time yet to go read the code… i should though).
So what improvement does this patch make? well, i’ll quote my commit comments:
BUG#24143 Heavy file fragmentation with multiple ndbd on single fs If we have the XFS headers (at build time) we can use XFS specific ioctls (once testing the file is on XFS) to better allocate space. This dramatically improves performance of mysql-test-run cases as well: e.g. number of extents for ndb_dd_basic tablespaces and log files BEFORE this patch: 57, 13, 212, 95, 17, 113 WITH this patch : ALL 1 or 2 extents (results are consistent over multiple runs. BEFORE always has several files with lots of extents). As for timing of test run: BEFORE ndb_dd_basic [ pass ] 107727 real 3m2.683s user 0m1.360s sys 0m1.192s AFTER ndb_dd_basic [ pass ] 70060 real 2m30.822s user 0m1.220s sys 0m1.404s (results are again consistent over various runs) similar for other tests (BEFORE and AFTER): ndb_dd_alter [ pass ] 245360 ndb_dd_alter [ pass ] 211632
So what about the patch? It’s actually really tiny:
--- 1.388/configure.in 2006-11-01 23:25:56 +11:00 +++ 1.389/configure.in 2006-11-10 01:08:33 +11:00 @@ -697,6 +697,8 @@ sys/ioctl.h malloc.h sys/malloc.h sys/ipc.h sys/shm.h linux/config.h \ sys/resource.h sys/param.h) +AC_CHECK_HEADERS([xfs/xfs.h]) + #-------------------------------------------------------------------- # Check for system libraries. Adds the library to $LIBS # and defines HAVE_LIBM etc --- 1.36/storage/ndb/src/kernel/blocks/ndbfs/AsyncFile.cpp 2006-11-03 02:18:41 +11:00 +++ 1.37/storage/ndb/src/kernel/blocks/ndbfs/AsyncFile.cpp 2006-11-10 01:08:33 +11:00 @@ -18,6 +18,10 @@ #include #include +#ifdef HAVE_XFS_XFS_H +#include +#endif + #include "AsyncFile.hpp" #include @@ -459,6 +463,18 @@ Uint32 index = 0; Uint32 block = refToBlock(request->theUserReference); +#ifdef HAVE_XFS_XFS_H + if(platform_test_xfs_fd(theFd)) + { + ndbout_c("Using xfsctl(XFS_IOC_RESVSP64) to allocate disk space"); + xfs_flock64_t fl; + fl.l_whence= 0; + fl.l_start= 0; + fl.l_len= (off64_t)sz; + if(xfsctl(NULL, theFd, XFS_IOC_RESVSP64, &fl) < 0) + ndbout_c("failed to optimally allocate disk space"); + } +#endif #ifdef HAVE_POSIX_FALLOCATE posix_fallocate(theFd, 0, sz); #endif
So get building your MySQL Cluster with the XFS headers installed and run on XFS for sweet, sweet disk allocation.
class Guard {
public:
Guard(NdbMutex *mtx) : m_mtx(mtx) { NdbMutex_Lock(m_mtx); };
Guard(NdbLockable & l) : m_mtx(l.m_mutex) { NdbMutex_Lock(m_mtx);
};
~Guard() { NdbMutex_Unlock(m_mtx); };
private:
NdbMutex *m_mtx;
};
The Programme for linux.conf.au 2007 has hit the streets (err.. web) and it’s looking pretty neat.
I’m glad to see the MySQL and PostgreSQL miniconfs on different days – means I should be able to pop into the PostgreSQL one as well. Kernel could be interesting too… I guess it can depend on the sessions and stuff though.
Greg Banks’ session on “Making NFS Suck Faster” should be interesting. Tridge’s session on “clustering tdb – a little database meets big iron” should be really interesting (after all, I hack on a clustered database for a crust). After lunch, I’m a bit torn between a few sessions – but Matthew Garrett‘s “Fixing suspend for fun and profit” could be a laugh.
The next session will involve last minute jitters for my session (which is next: “eat my data: how everybody gets file IO wrong” – which will be great fun as there will no doubt be a bunch of smart people about ready to expand and clarify things.
By the end of the day I’ll be torn between Keith Packard’s “X Monitor Hotplugging Sweetness” (Hopefully the extension will be called XBLING – as I keep tryning to convince him to call an X extension that) and Garbage Collection in LogFS by Jorn Engel.
On Thursday, I’ll want to be in all the sessions at once – including Practical MythTV as presented by Mikal Still and myself. If you’re not in our session (and damn you for not being :) you should check out the no doubt other great things on: Dave Miller on Routing and IPSEC Lookup scaling in the linux kernel should be great fun, OzDMCA by Kim Weatherall will no doubt bring a tear to the eye, Rasmus on Faster and Richer Web Apps with PHP 5 (aparrently the aim when coding PHP is to not suck… so a lot of PHP “programmers” should take note – and ask to see how fast he can down a beer in), Andrew Cowie is talking on writing rad GTK apps (always fun when you can see something from your coding efforts). My photographer side of my brain is telling me to go to the GIMP Tutorial too. Hrrm… busy day (but our MythTV tute will ROCK – so show up and be converted).
After a morning berocca (err… tea), the NUMA sessions sound interesting (especially on memory mapped files – going to be thinking about this and databases odly enough). Lunch, then the Heartbeat tutorial sounds interesting (from a “we have an internal one and i wonder what this does” PoV).
Ted Ts’o is on enterprise real time… could be interesting as Ted’s a fun guy.
On Friday, Ted’s ext4 talk is a must see – especially from a poking him in the ribs about what would be neat from a DB PoV (and the reminder of the real numbers in a benchmark boost to performance we see with XFS versus ext3).
While wanting to be a cool kid like Rusty, Disk Encryption also sounds interesting, and Robert Collins could be talking about some interesting stuff (although the title “do it our way” isn’t giving much away).
So, I’ve pretty much just planned a week in January down to the hour. If you’re not already going – get booked for linux.conf.au 2007 now – sure to sell out quickly. Going to be totally kick-ass.
If you head over here: mysql on bkbits.net you can get a copy of the NDB team trees. This is where we push stuff before it hits the main MySQL trees so that we can get some extra testing in (also for when pulling from the main tree). So you can be relatively assured that this is going to work fairly well for NDB and have the latest bug fixes.
Of course, if anything is going to break here – it’s going to be NDB :)
This should allow you to get easy access to the latest-and-greatest NDB code.
At some point soon I’ll update my scripts that generate doxygen output (and builds) to do the -ndb trees.
enjoy!
$ svn ci Connection closed by $MAGIC_IP_ADDR svn: Commit failed (details follow): svn: Connection closed unexpectedly svn: Your commit message was left in a temporary file: svn: '/full/path/to/parent/directory/where/no/changed/files/svn-commit.tmp' $ svn ci Log message unchanged or not specified a)bort, c)ontinue, e)dit a
could have at least saved the commit message somewhere useful. like, say, somewhere so when i type “svn ci” again it reads the commit message.
grrr
Saturn’s autoweb
I’ve hacked my scripts that generate doxygen docs to also build MySQL 4.1, 5.0 and 5.1 for AMD64 (the box that it’s running on) with Cluster. This is to help my idea of running Gallery at home with NDB disk data tables in very recent MySQL builds.
How’s it going so far? Well… I’ve found some bugs and some seemingly strange behaviour here and there. However, bug reports will come, and I’m currently running a bit of an older build.
I’ll make the URL of the Gallery public at some point too
Slashdot | MySQL CEO MÃ¥rten Mickos Answers Your Questions
I’m pretty sure this is the first time I’ve been mentioned by name in the 1st commet to a /. article. Yay me!? :)
So a bunch of stuff has happenned (or happenning) that I’ve been wanting to blog about for a bit. Some stuff had to wait, others it’s just been me being slack.
Anyway, anyone who hangs closely around the MySQL circles probably now knows about MySQL Enterprise. There’s been a fair bit of talk about this internally for a little while now. When it was being talked about a bit wider within the company some of the initial communication was (in my mind) rather unclear. So I took the “what’s the worst way somebody could interpret this” viewpoint and replied with my thoughts. The idea behind this was to simulate what some of the loud-mouthed trolls of the non-shifted question mark e on a qwerty keyboard mapped to dvorak kind may do.
After a few phone calls (some at strange hours) my worst fears were not realised – we were still not being insane.
So I hope I’ve been of some use in making sure that communication has been clear and any possible fears put to rest.
There is also an increased willingness to make things saner for getting non-MySQL AB authored code into the main trees (err… now labelled Community).
We’re also getting geared up for another 5.1 release – the Cluster team has recently chased down some failures: from out-of-disk on build machine (why is it us who had to find that out?) to an “actual bug”.
Kudos goes out to Jonas who has recently found a few bugs that have been Can’t Repeat since about the year 2000 – ones that were real hard to hit, but naturally, somebody has.
I also added some new things to the Cluster Management Server (ndb_mgmd) in 5.1 that should help with debugging in the future. I basically just exposed the MgmApiSession stuff a bit, giving each session a unique id (64 bit int) that you could then check if the session had gone away or not (or list all sessions). This gives us a test case for bug 13987 which is pretty neat.
I also have geared up a change to the handler API to fix bug 19914 – and being a good boy, I’ve mailed out to the public internals list so that people are ready for the building of outside of tree storage engines to break (on 4.1 and up!). The good news is, however, that this is a real fix and that any errors on COUNT(*) will be reported back to the user (a customer was affected by this).
Also, I updated how engines fill out the INFORMATION_SCHEMA.FILES table to make it a bit nicer (Brian wants to add support for it to some of the other engines). He also pointed out a really obvious bug of mine in a recent push to that code (that probably showed up in a compiler warning come to think of it…). Paul is looking at it for PBXT too (or at least thinks it’s cool :).
Also had a bit of a ask-around the cluster team about if making the team trees (VERSION-ndb) public (up on bkbits.net etc) was good. Nobody seems to have any objections, so will (as soon as I get a minute) persue that. Basically it’ll let people get access to the latest NDB bug fixes in source-tree form (certainly not recommended for production, but could be useful in testing environments).
I’ve also been thinking about talks for the MySQL UC next year as Cluster tends to be a popular topic (had a rather full room this year).
There’s probably more to talk about too, but i’m getting sleepy.
As email is *sooo* non-“Web 2.0”, i reply in blog form….
Rusty’s Bleeding Edge Page talks about a “Writing an x86 hypervisor: all the cool kids are doing it!” session that sounds really cool (better not be on at the same time as my talk… :)
I don’t (currently) intend to be one of the cool kids though.
He also mentions a session entitled “First-timer’s Introduction to LCA”. A couple of possible suggestions (or thoughts, and stuff I’ve seen):
I wonder what should/could be written about going all fanboy/fangirl over favourite hackers? and taking/asking to get taken photos?
The last thing Rusty talks about is the “Hacking in groups” tutorial. I really liked his and Robert Love’s tutorial in Canberra (Kernel Hacking – where you wrote a PCI driver for the excellent Love Rusty 3000. A device with real specifications, coffee cup stain and all). I’ve had a bit of a mixed feeling about it from Rusty since then, but I reckon it was seriously one of the best tutorials I have ever attended. I also took the hands-on approach as great inspiration for various MySQL Cluster Tutorials I’ve given since (and people have commented on how the hands-on part is great).
I guess the thing about the kernel hacking tute was that not everybody in the room was at the same skill level (which is something you totally run the risk of with hands-on). Also, if you hadn’t done the prep material, you were probably going to be in trouble.
But anyway, the idea of having 20 talented coders with 5 people in the tute for each of them and working on some project could be interesting – although rather ambitious. I worry that people without a good enough skillset would rock up and not get much out of it. Although those with adequate skill would do well.
Picking a project that could be doable in a handful of hours (or a day) is tricky – as it’d probably be an extension to some existing project, which requires learning of it. Or, starting something from scratch can be equally as hard (to end up anywhere useful).
Some ideas for projects could include:
These have the benefit of being able to run existing good test suites against the software and see how well people did. They’d probably also help people land jobs :)
Another interesting one would be implementing a library for journaling writes to a file. i.e. instead of write to temp, sync, rename – do journaling. This would let people easily write apps that did safe updates to large files. You could then use this to implement other things (like a really simple crash-safe storage engine, FUSE file system or something).
I’m just not sure how much “cool tricks” could really happpen in that time (instead of just getting the job done). 20 coders talking about their neat tricks would probably make a good book though…
For certain evil purposes last week, I assembled the old Saturn with a hard disk I found when cleaning a little while ago (I have that kind of tech stuff – you clean up and find 40GB disks – I’m pretty sure I have an 8.4 bumming around somewhere too).
I ended up being able to do the evil I needed to, but I could tell that the room was a bit warmer due to the extra box being alive. I was also lazy and couldn’t be bothered going downstairs for the D200, so this was shot with my old and trusty Coolpix 4500.
I used the box to be able to get remote access to a customers’ test setup to do some diagnosis on a bug (that’s notoriously hard to reproduce). I think I have a fair idea of what it is now though (timing related – not fun).
Remember kids, threads are evil.
Also, an interesting thing to note is that there is, in fact, a limit to not the number of fds you can pass to the select(2) system call, but to the actual number (on my Ubuntu box here, passing a fd of, say 2000 is probably going to lead to trouble). This has nothing to do with the previously mentioned bug, but an interesting point.
SQLyog is now free software and seems to run under WINE on my Ubuntu box okay (i think i’m running stock wine from dapper….)
So those of you addicted to SQLyog (i hear there’s a couple), you can use it on linux okay too (and probably other systems that WINE supports)