It just it trunk – if you have HailDB installed when you build Drizzle, you will now get the HailDB plugin built. You can even run Drizzle with it (remove innobase plugin, load HailDB plugin). Previously, we had problems building both due to symbol conflicts between innobase and HailDB. We’ve fixed this thanks to the linker.
So, enjoy HailDB… welll, test it and report bugs that I can fix :)
In the current HailDB we have a couple of new API calls that you may like:
All of these are new to HailDB (and weren’t available in embedded_innodb), many in the new 2.3 development release. You can see usage examples both in the HailDB test suite and (for most of them) in the Drizzle HailDB Storage Engine.
We just released the latest Drizzle tarball (2010-10-11 milestone). There are a whole bunch of bug fixes, but there are two things that are interesting from a storage engine point of view:
Those of you following Drizzle fairly closely have probably noticed that we’ve lagged behind in InnoDB versions. I’m actively working on fixing that – both for the innobase plugin and for the HailDB library.
If building the HailDB plugin (which is planned to replace the innobase plugin), you’ll need the latest HailDB release (which as of writing is 2.3.1). We’re making good additions to the HailDB API to enable the storage engine to have the same features as the Innobase plugin.
Just in case you missed it, I’m rather thrilled that our latest tarball of Drizzle is named Beta. Specifically, we’re calling it Drizzle7. Seven is a very nice number, and it seems rather appropriate.
This release is for a stand alone database server. A lot of the infrastructure for replication is there (with testing), but the big thing we want to hammer on and get perfect here is Drizzle7 as a stand alone database server.
Can I trust it? If you trust InnoDB to store your data, then yes, you can trust Drizzle (it uses InnoDB too)
Yes, I said InnoDB+ with a plus sign at the end (also see the first comment here).
Please note that this blog post is only based on public information. It has absolutely nothing in it that I only could have learned from back when I worked at Sun or MySQL AB. Everything has links or pointers to where you can find the information out on the Internet and all thoughts are based on stringing these things together.
There was a lot of talk around the acquisition of Sun Microsystems by Oracle about MySQL (MySQL AB was bought by Sun). Some of the talk centred around Oracle and their ability to make a closed source version of MySQL with added bits that wouldn’t be released as GPL. They’ve since proved that they’re quite willing to do this to an open source project (see OpenSolaris).
Relatively recently, a bunch of history from the old InnoDB SVN trees was imported into the MySQL source tree. You can pull the revision of the SVN tree as of InnoDB Plugin 1.0.6 release by using revid:svn-v4:16c675df-0fcb-4bc9-8058-dcc011a37293:branches/zip:6263  from the MySQL repository – or just use a branch I’ve put up on launchpad for it (lp:~stewart/haildb/innodb-1.0.6-from-svn).
The first revision from the SVN tree was created on 2005-10-27, which you may remember was not too long after Oracle acquired Innobase on the 7th of October that year. The next two revisions were importing the 5.0 innodb code base, and then the 5.1 code base. Previous history can be found according to this blog post on Transactions on InnoDB.
According to Monty in the comment on the Pythian blog:
Oracle did work on a closed source version of InnoDB, codename InnoDB+, but they never released it, probably because our contract with them stopped them.
and from Eben Moglen’s letter to the EU Commission (via Baron Schwartz’s blog post):
Innobase could therefore have provided an enhanced version of InnoDB, like Oracle’s current InnoDB+, under non-GPL license
Most tellingly is a lot of references in the revision history to “branches/innodb+” as well as this commit:
revno: 0.5.148
revision-id: svn-v4:16c675df-0fcb-4bc9-8058-dcc011a37293:branches/innodb%2B:6329
parent: svn-v4:16c675df-0fcb-4bc9-8058-dcc011a37293:branches/innodb%2B:6322
committer: vasil
timestamp: Thu 2009-12-17 11:00:17 +0000
message:
branches/innodb+: change name and version
Change name from “InnoDB Plugin” to “InnoDB+” and
version from 1.0.5 to 1.0.0.
So, from the revision history I’ve managed to work out that it likely was going to have the following features:
and finally, in revid:svn-v4:16c675df-0fcb-4bc9-8058-dcc011a37293:branches/innodb%2B:6819 you can see the change from “InnoDB+” back to “InnoDB” for being the built in default for MySQL 5.5
This miniconf aims to cover many of the current methods of data storage and retrieval and attempt to bring order to the universe. We’re aiming to cover what various systems do, what the latest developments are and what you should use for various applications.
We aim for talks from developers of and developers using the software in question.
Aiming for some combination of: PostgreSQL, Drizzle, MySQL, XFS, ext[34], Swift (open source cloud storage, part of OpenStack), memcached, TokyoCabinet, TDB/CTDB, CouchDB, MongoDB, Cassandra, HBase….. and more!
On advice from one of the guys at Vanbar (the “local” – meaning still a drive away, but at least in the same city – place for photographic chemicals) I tried stand development for the roll of Efke 25 I had shot at Burning Man.
Stand development differs from normal B&W film development in that a) it takes longer and b) you don’t agitate. You just leave the film sitting in the chemicals for a while…. in this case, one hour. I used Rodinal, which is a pretty nice, easy to use and versatile developer (incidentally, it’s also the oldest photographic chemical still sold and in use – patented in 1891).
Anyway, I got some good images out of it (I think). Here are some from a roll of Lucky 100SHD I shot:
and here’s some from the Efke 25 that I shot at Burning Man:
Yesterday, I reached a happy milestone in HailDB development. All compiler warnings left in the api/ directory (the public interface to the database engine) are now either probable/possible bugs (that we need to look at closely) or are warnings due to unfinished code (that we should finish).
There’s still a bunch of compiler warnings that we’ve inherited (HailDB compiles with lots of warnings enabled) that we have to get through, but a lot will wait until after we update the core to be based on InnoDB 1.1.
There’s this neat art project that’s been at Burning Man for a number of years: Why Do You Do What You Do?
Photos of people holding up a sign with an answer to that question.
Turns out, I know people! (pleasant surprise)
Senator Kate Lundy speaking at Software Freedom Day in Melbourne.
No, really.
I have recently been constructing a 100 page document going over a whole bunch of the details for the Monorail we’re building at Burning Man this year.
Apart from randomly freezing, and then suddenly not displaying images until I had restarted it – it’s also really slow.
The last straw was when leafing through the document before getting it printed. I had inserted a bunch of pages before this last section. But now, there was this empty page in the last section of the document. The part that I hadn’t touched for days. If I tried to remove the blank page, all the images on nearby pages moved so that they were on top of each other.
I ended up just printing it. There is a blank page that I can’t get rid of.
It is a piece of software that worries me. Is this really meant to be an alternative? It has NEVER worked well for me. Basic tasks sure, but I continually find myself pining for Word 5.1a on the Mac (System 7 that is) or Nisus Writer or even ClarisWorks.
If opening Microsoft Word documents fairly accurately is your only good feature, how do you expect to survive in the free (software) world?
So, while my twitter stream may suggest desires for punning the developers in the face or their early demise through painful methods….. I really just wish that sometime in the past 10 years you had made it not shit me to tears.
Certainly another failure of Sun Microsystems and I don’t expect Oracle to do any better at all (especially considering recent actions).
… and really testing the replication code path for CREATE TABLE.
So, for a very long time now, Drizzle has been using a protobuf based structure to describe the structure of tables. The idea was to be able to have engines rather painlessly generate this structure themselves (which several now do). A secondary idea was to use this structure itself for CREATE TABLE (in progress, and embedded_innodb does in fact does only use the table message for its CREATE TABLE codepath). The third idea was to just put the table protobuf message into the replication stream instead of the CREATE TABLE statement (i.e. a SQL string). This means that you could (easily) write a replicator to a DBMS with different SQL syntax, or to a system that doesn’t speak SQL at all.
The final step, to reduce duplicated code functionality, would be to have the code that does SHOW CREATE TABLE to use a common bit of code for turning a table protobuf message back into a SQL string.
We now have that.
Just now, my branch to replace the old SHOW CREATE TABLE code (that was using TableShare and friends) with the statement_transform code (that we also use in converting a replication log to SQL commands) hit trunk.
Yay.
This is something that has never existed in the MySQL world. Nothing to help you start developing your engine.
Sure, you could start running the whole test suite against your engine…. but where it wasn’t specifically MyISAM, you’d certainly hit things that were.. well… as simple as having to change the result file so that SHOW CREATE TABLE masked out the ENGINE= part.
Also, if you were just starting out and were trying to incrementally write your engine, instead of just hacking away for 6 months on “everything” and then testing “everything” and hoping that anything at all worked – you were out of luck.
So when working on the embedded_innodb engine for Drizzle I wanted to finally fix this problem. I was not going to fix it perfectly, or completely. What I wanted was a set of simple tests, that were very short and that tested distinct bits of the engine.
So… if you look in the Drizzle tree, in plugin/embedded_innodb/tests/ you’ll find a bunch of test cases beginning with basic_ (they’re also now in plugin/pbxt/tests). These check very basic functionality and should be useful when starting to write your own storage engine.
They should also port easily to MySQL too (possibly without changes), but I haven’t tried it. Porting to PBXT was simple.. a regex to replace InnoDB with PBXT.
Enjoy.
A whole bunch of stuff got merged to the embedded_innodb engine recently. The end game is to have the embedded_innodb engine be just linking to HailDB (where we’re continuing maintenance of Embedded InnoDB).
So, what made it in? A lot of bug fixes (especially around error reporting and tables without an explicit primary key), SELECT FOR UPDATE, support for temporary tables and support of prefix indexes.
We also now, if it’s available, link to HailDB instead of Embedded InnoDB. At some point “soon” we shall just require HailDB.
I started to realise that I was liking the look of photos shot on Kodak Portra. I wanted to shoot some of it to see what I thought. I bought a pack of 5 rolls of 160VC from Glazer’s just before heading to OSCON.
Here are some of the shots I got:
(granted I didn’t press the shutter release, but I like it)
Is the one that sealed it for me. This was the “ahh… I can use this for all sorts” shot.
All of these were developed and scanned at the Walgreens down the street. I could probably do better scans of some… but this was awfully less work for me.
I am really liking the skin tones from it. The vividness of colours also comes through while retaining excellent skin tones (certainly not always the case). I may even end up shooting some at Burning Man (did buy more rolls yesterday!)
There is (of course) more being added to my Kodak Portra 160VC set on flickr.
Yazz has been making an epic over engineered shower for camp.
The bug I talked about a little while ago has now also had the fix I wrote committed to the mysql-trunk 5.5.6-m3 repository.
(Reposted from the HailDB Blog. See also the announcement on the Drizzle Blog.)
We’ve made our first HailDB release! We’ve decided to make this a very conservative release. Fixing some minor bugs, getting a lot of compiler warnings fixed and start to make the name change in the source from Embedded InnoDB to HailDB.
Migrating your software to use HailDB is really simple. In fact, for this release, it shouldn’t take more than 5 minutes.
Highlights of this release:
Check out the Launchpad page on 2.0.0 and you can download the tarball either from there or right here:
Testing how much power the rope will pull off the generators
Josh Berkus gave a great talk at linux.conf.au 2010 (the CFP for linux.conf.au 2011 is open until August 7th) entitled “How to destroy your community” (lwn coverage). It was a simple, patented, 10 step program, finely homed over time to have maximum effect. Each step is simple and we can all name a dozen companies that have done at least three of them.
Simon Phipps this past week at OSCON talked about Open Source Continuity in practice – specifically mentioning some open source software projects that were at Sun but have since been abandoned by Oracle and different strategies you can put in place to ensure your software survives, and check lists for software you use to see if it will survive.
So what can you do to not destroy your community, but ensure you never get one to begin with?
Similar to destroying your community, you can just make it hard: “#1 is to make the project depend as much as possible on difficult tools.”
#1 A Contributor License Agreement and Copyright Assignment.
If you happen to be in the unfortunate situation of being employed, this means you get to talk to lawyers. While your employer may well have an excellent Open Source Contribution Policy that lets you hack on GPL software on nights and weekends without a problem – if you’re handing over all the rights to another company – there gets to be lawyer time.
Your 1hr of contribution has now just ballooned. You’re going to use up resources of your employer (hey, lawyers are not cheap), it’s going to suck up your work time talking to them, and if you can get away from this in under several hours over a few weeks, you’re doing amazingly well – especially if you work for a large company.
If you are the kind of person with strong moral convictions, this is a non-starter. It is completely valid to not want to waste your employers’ time and money for a weekend project.
People scratching their own itch, however small is how free software gets to be so awesome.
I think we got this almost right with OpenStack. If you compare the agreement to the Apache License, there’s so much common wording it ends up pretty much saying that you agree you are able to submit things to the project under the Apache license. Â This (of course) makes the entire thing pretty redundant as if people are going to be dishonest about submitting things under the Apache licnese there’s no reason they’re not going to be dishonest and sign this too.
You could also never make it about people – just make it about your company.
#2 Make it all about the company, and never about the project
People are not going to show up, do free work for you to make your company big, huge and yourself rich.
People are self serving. They see software they want only a few patches away, they see software that serves their company only a few patches away. They see software that is an excellent starting point for something totally different.
I’m not sure why this is down at number three… it’s possibly the biggest one for danger signs that you’re going to destroy something that doesn’t even yet exist…
#3 Open Core
This pretty much automatically means that you’re not going to accept certain patches for reasons of increasing your own company’s short term profit. i.e. software is no longer judged on technical merits, but rather political ones.
There is enough politics in free software as it is, creating more is not a feature.
So when people ask me about how I think the OpenStack launch went, I really want people to know how amazing it can be to just not fuck it up to begin with. Initial damage is very, very hard to ever undo. The number of Open Source software projects originally coming out of a company that are long running, have a wide variety of contributors and survive the original company are much smaller than you think.
PostgreSQL has survived many companies coming and going around it, and is stronger than ever. MySQL only has a developer community around it almost in spite of the companies that have shepherded the project. With Drizzle I think we’ve been doing okay – I think we need to work on some things, but they’re more generic to teams of people working on software in general rather than anything to do with a company.
