Jonathan puts it in slighty different words, and doesn’t gaurantee The White House to everybody.
I do wonder when we’ll get a Drizzle or NDB using president though….
Use MySQL, get elected President of the United States
1
Author Archives: Stewart Smith
Singing in the Rain
The past 3 years, 11 months I have worked full time on NDB (MySQL Cluster). It’s been awesome. Love the product and people. In the time I’ve been on the Cluster team, we’ve gone from a small group that would easily fit in the (old old) Stockholm office to one that requires large rooms to house us all in. It’s also been all about smart people (you have to be to work on a distributed database).
With MySQL Cluster 6.4 we’re getting in a bunch of features that have been on the “wide adoption” wishlist. With each release of NDB we’ve gained a wedge of applications that can be used with it – and 6.4 is no exception.
One of the biggest things that’s been worked on is multithreaded data nodes. If you check out Jonas‘ recent posts on 500,000 reads/sec and then a massive 700,000 reads/sec.
We’ve also got a Microsoft Windows port coming up, which a number of people have asked for over the years. Mostly I think this is a “I want to try it out” thing and not a deployment thing. (can any sane person deploy a HA app on Win32?)
I’ve used “NDB$INFO” as the ultimate answer to any problem for a while now. It’s been the much-wanted monitoring interface. We have a lot of info inside NDB that currently isn’t easily user accessible (or only accessible through the magic DUMP interface or by gathering up many events in the cluster log). We have the start of NDB$INFO in 6.4 now and Martin will be continuing my work in making it truly awesome.
So go and grab the 6.4 tree and have a look – things are looking sweet.
What next for me?
Well… a while ago I started hacking on Drizzle. Why? Well… I thought we could move the database server in a new direction and make it more modular, leaner, meaner query machine.
And now, I’m starting to work on it full time.
It’s exciting, and I’ll be blogging on the first TODO which is remove the FRM file and switch to a full discovery method shortly.
UPDATE: Yes, I’m working full time on Drizzle for Sun Microsystems (in the CTO group). While not spending work time on NDB anymore, no doubt you’ll still see fun-time patches.
Microsoft Calls Today Global Anti-Piracy Day
Slashdot | Microsoft Calls Today Global Anti-Piracy Day
Happy with my software freedom thanks, news at 11.
suspend works again!
yay – latest package updates for ubuntu made suspend and resume work again for me. yet to try some of the xrandr stuff to see if its fixed yet.
on compiling with –disable-assert
It’s like removing the brakes from your car. yes, it will go faster (slightly less weight) but, dude, you just removed the brakes.
MySQL Cluster (NDB) on Win32 progress
Many things have been happenning in the land of NDB on Win32 as of late.
I’ve fixed about 700 compiler warnings (some of which were real bugs) leaving about 161 to go on Win32 (VS2003). We’re getting a few more warnings on Win64 (some of which look merely semantic, while others could be real bugs), but the main focus now is getting 32bit going really well.
I fixed a number of bugs that were around preventing lots of things from working properly:
Disk Data (i.e. CREATE TABLESPACE, CREATE LOGFILE GROUP, and CREATE TABLE… TABLESPACE ts1 STORAGE DISK) now works. The main problem here was that our filesystem abstraction layer for the NDB kernel (ndbd) once had a Win32 port… which has sorely bitrotted over the years. As new features were introduced to the file IO interface, they (of course) weren’t also added to the Win32 abstraction. In the disk data case, the OM_INIT feature, which on FSOPENREQ (open a file) allows data to be passed in for initialising the file. Previously, I fixed this to allocate the file on disk and create a file of the same size, but i didn’t add the feature that writes initial data to the file. This caused bugs as soon as you tried to use the disk data tables (the files weren’t initialised, so you hit asserts on corrupt disk data files).
Paths in the server: for whatever bizarre and stupid reason, the MySQL server can end up having paths to a table as ./database/table OR .\database\table. The latter *never* shows up on non-Win32 platforms but can *sometimes* show up on Win32. Ick ick ick ick. Anyway, we (in the NDB handler) weren’t dealing with this properly, causing problems around some metadata ops.
Our pushbuild system takes each push to a source tree, builds it on a variety of platfroms and runs the mysql-test-run.pl test suite. The Win32 hosts are actually running on vmware. In order to make tests run faster, on Linux we use /dev/shm for the data files. Microsoft Windows doesn’t have a good ram disk, so we create a file on /dev/shm on the host and map that as a drive inside Windows (and format it as NTFS). This drive is only 1GB. This is not enough disk space for running all the clusters (yes, plural) started by the test suite (and everything would die with ENOSPC). The workaround I’ve come up with is that for debug builds, we simply enable NTFS file compression on files ndbd creates.
Win64 is also working! Pushbuild builds and runs on 64bit, and the Win64 host is building with NDB and passing about the same amount of tests as the Win32 hosts!
The bad news is that the NDB with replication tests are pretty much all failing… so I’m fairly confident that cluster replication is very broken on Win32 (and 64) at the moment.
I’ve had to do a fair amount of fixing on a bunch of the test cases (mainly to do with finding where various NDB utilities are). They’ve also prompted fixes in NDB (automatically converting / to \ in ndbd on Win32 for CREATE DATAFILE/UNDOFILE).
If you want to give it a go – you can get the source from launchpad. Either in the mysql-5.1-telco-6.4 tree, or if you want a few more things fixed, always have a look at the mysql-5.1-telco-6.4-win tree. Hopefully both are synced with the latest internal trees (i.e. plain 6.4 is working on win32) by the time you read this.
Iggy and I discussed installers for NDB on Windows in Riga, and we should have something soon-ish for those of you who don’t build from source.
No opt-out of filtered Internet
Computerworld – No opt-out of filtered Internet (via Chris)
EPIC FAIL. Looks like I’m going to have to reconsider which end of the ballot paper the ALP goes at (hint, it’s now towards the bottom).
Pay no attention to the criminals behind the curtain… we’re just going to let them do their thing and make sure it’s hard for you to see them.
With a false positive rate (incorrectly blocking something) of 10,000/1,000,000 (otherwise known as 1%) we’ll no doubt see things blocked that shouldn’t be.
What happened to my free country?
evolution-data-server even worse (is that possible?)
Just caught it using 713MB of resident memory. What the fuck? I don’t even have Evolution running! There’s only the clock applet (which does pull things out of calendar i guess…).
Does Evolution win the prize for worst piece of free software yet?
new NetworkManager VPNC not better at all (in fact, much worse)
I upgraded to Ubuntu 8.10 the other day, NetworkManager promptly forgot my wireless LAN key (grr… lucky I keep a copy in a text file) as well as my VPN configuration. It’s also changed the UI for entering what specific networks to route over the VPN (’cause the last thing you want is putting all your traffic through VPN when you have a perfectly good internet connection here… or even worse, I do *not* need to go via Sydney or the US to access the machine 2ft from me thank you very much).
Generally not happy with the new NetworkManager.
getarg calls srand() ???
storage/ndb/test/src/getarg.c
Guess what? It calls srand(time(NULL)) in getarg(). Why you ask? well.. what you want to be able to when specifying a flag is have it be true, false or it could “maybe” be set.
That’s right kids… maybe.
I’m sure it’s used somewhere in our test suite to get coverage on different things.. but umm.. yeah, interesting discovery for today.
Visual Studio 2008 unreferenced local variable bug
screenshot ’cause typing is for wusses
UPDATE: not actually VS bug. Nasty macro defining strtok_r to strtok on Win32. ouch.
NDB Windows port shaping up…
It’s getting there. The tree should now pretty much always compile, and (at least mostly) doesn’t break anything on other platforms. It even works on win32… at least basic functionality. There will be monsters (bugs.. but scarier, becuase it’s win32).
Best use of the word ‘fuck’ for today
A Web OS? Are You Dense? – Ted Dziuba
All this crap about moving everything into another 5 layers of abstraction down just means we’re never going to have a computer that’s visibly any faster than an Apple // to any user sitting in front of it… Sure, it can recalculate your spreadsheet faster (as long as it’s not an online spreadsheet running in a browser), but menus take longer to appear.
4GB of RAM and swapping
Evolution: why exactly do you need 89MB when you start yet 898MB now?
Somehow I feel that 4GB of physical memory and 4GB of swap (well.. 2GB used) should be enough to run an email client, web browser, office suite, play some tunes, use a calendar, IRC and IM.
GIMPshop.com GPL fail
GIMPshop.com – A GIMP hack by Scott Moschella
“For example, the GPL does not necessarily allow access to the original source code”
umm… a FAIL that’s perhaps EPIC.
(Courtesy of Mary Gardiner’s identi.ca feed)
SetFileValidData Function (Windows) – Now with added FAIL
SetFileValidData Function (Windows)
There seems to be two options on Win32 for preallocating disk space to files.
Basically, I want a equivilent to posix_fallocate or the ever wonderful xfsctl XFS_IOC_RESVSP64 call.
The idea being to (quickly) create a large file on disk that is stored efficiently (i.e. isn’t fragmented).
From SQL, you’d do something like “CREATE LOGFILE GROUP lg1 ADD UNDOFILE ‘uf1’ INITIAL_SIZE 1G;” and expect a 1GB file on disk. One way of getting this is calling write() (or WriteFile() on Win32) repeatedly until you’ve written a 1GB file full of zeros. This means you’re generating approximately 1GB of IO.
Except it’s worse than that: every time you extend the file, you’re going to be changing the metadata (file and free space information). If you’re lucky, you won’t be using a file system that writes a new transaction to the journal for each time you do this.
If your file system allocator doesn’t like you today (even more likely when you’ve got more than one process doing IO), you may end up with rather fragmented files as well – especially if you’re doing synchronous IO. So you want some method of saying “this file will be size X, please allocate disk space to it in the most efficient way for a file size of X” as it’s not possible to infer this from everyday IO calls (I guess the Win32 CopyFile and CopyFileEx calls could though).
It probably doesn’t do it, but having a CopyFile call would be neat for copy on write file systems and saving space… although I wonder how many Win32 apps would cope with ENOSPC on a write to an existing part of a file.
On IRIX we used the magic xfsctl() with the XFS_IOC_RESVSP64 argument. On Linux (with XFS), we use the same. On ext2/ext3 the only way to get the same has been to (with the file system unmounted), parse the file system and implement it yourself. Although (and this just in) the brand new fallocate() call should help with this. The posix_fallocate() call in GNU libc has just been a wrapper around the simple method of writing 0 to a file from start to end (albeit rather efficiently).
XFS implements something called “unwritten extents”. An unwritten extent says “this range of blocks is allocated to this file. If reading from this range, return a zero page. If writing, split the unwritten extent into 3 parts: before, the newly written extent (which isn’t unwritten: i.e. now valid data), and the after extent.” Simple, rather efficient and gets really good allocation as XFS gets to search the free space btrees based on size.
So what to do on Win32 (apart from drink heavily to try and make it all go away)?
There’s SetFileValidData, but that needs special permissions and may expose previously deleted data from other users. i.e. massive security hole. FAIL
There’s SetEndOfFile which, quoting the MS docs: “If the file is extended, the contents of the file between the old end of the file and the new end of the file are not defined.” Not exactly reassuring… but introduced in W2k, so rather safe to use today. Doesn’t save you from having to fill the file with zeros as part of initialisation though.
There’s SetFileInformationByHandle, which looks like it may do exactly what I want… if you read between the lines of the documentation. But it’s only supported starting with Vista. Which you all use of course, so that’s not a problem.
Building MySQL on Windows – MySQL Forge Wiki
Building MySQL on Windows – MySQL Forge Wiki
This one covers running mysqld in the VisualStudio debugger, which can be useful.
I have no special ndb_mgmd.exe or ndbd.exe in debugger instructions or wisdom (running them from mysql-test-run.pl at least). I’ve attached debugger to already running (started by mysql-test-run.pl) ndb processes, but haven’t made any changes to mtr to make it like the mysqld of “go and enter this”.
Some thoughts on VirtualBox 2.0….
I like it.
UI looks more polished (maybe QT4 renders nicer on my Ubuntu GNOME based desktop… but it certainly looks more like the rest of my apps).
Possibly a bit faster?
64bit guests – WIN!
Shared Folders still cause the XP guest to panic if i try to do anything non-trivial (e.g. build, run cmake). I still use Samba on host to share with guest – which works okay, but isn’t super-happy-fun-time fast.
Seemless mode is a bit nicer, the Windows task bar sits above the GNOME task bar, making it a lot easier to interact. It’s not perfect yet, you can still get some real visual oddities, but it’s pretty sweet.
When just using normal, in window, it can cause X to use a bit much CPU, which goes away when the VirtualBox window is minimised – leading to faster compiles in guest.
Having a guest running (in this case Windows XP) does cause a lot more CPU wakeups – draining battery life.
I wish the snapshots showed how large they were, it’s currently kinda guesswork to work out how much disk space a snapshot is using.
MySQL Conference & Expo 2009 – CFP open
Is it that time already? MySQL Conference & Expo 2009 has opened the CFP.
Submit (well) early and often. It’s always an exciting (and exhausting) conf. Good technical, relevant content is what makes it good. Getting to talk to people who do amazing things, people who use your software, people looking to use it, people who want to chat about how you can learn off each other.
Any suggestions for what you’d like to hear from me (Cluster, Drizzle et al) are welcome – either via private mail or comments here.
OpenOffice.org 3 dev release
So after seeing Paul Fenwick rave about the presenter screen for OO 3, I decided to grab the debs and give it a go.
It still is very slow opening large presentations (i.e. mine), but it does look nicer at least… well… at least some of the widgets do.
Will report back when I’ve had a bit more time to fiddle with it.