I’m getting better at making prints, and starting to understand how all the bits fit together properly. I’m finding myself disappointed that I’ve shot colour sometimes :)
The light-sealing of the darkroom (also known as laundry (also known as brewery)) is not exactly pretty… but it does work:
While it’s great that MySQL 5.5 is GA with the 5.5.8 release (you can download it here), I’m rather disappointed that the bzr repositories on launchpad aren’t being kept up to date. At time of writing, it looked like this:
Yep – nothing for five weeks in the 5.5 repo – nothing since the 5.5.7 release :(
There hasn’t been zero changes either – the changelog has a decent number of fixes.
In browsing the BZR tree for lp:mysql-server, I noticed some rather exciting code had been merged into the Innobase code.
You may be aware that InnoDB will do some index dives when opening a table to get some statistics about the indexes that can help the optimiser make good query plans.
The problem being that this is many disk seeks. It means that on server restart, you have to spend a whole bunch of time seeking around the disk reading index pages.
Not any more.
There is now code merged in to store the calculated statistics in a table inside InnoDB so that these index dives don’t have to happen on startup.
Originally, this looked like it was going to make it into InnoDB+. The good news is that it’s now in a public source tree. I look forward to when it hits a stable release.
(hopefully somebody other than me can beat me to it and write a nice description of the algorithms involved… the code is pretty easy to follow, so it shouldn’t be hard)
In case you haven’t heard yet, I’ve merged in the latest InnoDB from MySQL 5.5.7 into Drizzle. The innobase plugin is now based on InnoDB 1.1.3.
This gets a lot of bug fixes and improvements from 1.1.2 (and on 1.1.1). Enjoy!
The MySQL replication system has always had the replication log (“binlog”) as a separate set of files on disk. Originally, this really didn’t matter as, well, MyISAM wasn’t transactional or crash safe so the binlog didn’t need to be either. If you crashed on a busy write workload, your replication was just going to be hosed anyway.
So then came a time where everybody used InnoDB. Transactional, crash-safe and all the goodies. Then, a bit later, came storing master rpl log position in InnoDB log and XA with the binlog. So a rather long time after MySQL first had replication, you could pull the power cord on the master with a decent amount of certainty that things would be okay when you turned it on again.
I am, of course, totally ignoring the slave state and if it’s safe to do that on slaves.
Using XA to make the binlog and InnoDB consistent does have a cost. That cost is fsync()s. You have to do a lot more of them (two phase commit here).
As you may be aware, at a (much) earlier point in Drizzle we completely ripped out the replication code. Why? A lot of it was very much still geared to support statement based replication – something we certainly didn’t want to support. We also did not really want to keep the legacy binlog format. We wanted it to be very, very pluggable.
So the initial implementation is a transaction log file. Basically, we write out the replication messages to a file. A slave reads this and applies the operations. Pretty simple and foolproof to implement.
But it’s pluggable.
What if we stored the transaction log inside innodb? Not only that, what if we wrote it as part of the transaction that was doing the changes? That way, no XA is needed – everything is consistent with a COMMIT. This would greatly reduce the number of fsync()s needed to be consistent.
Now… the first thing people will say is “arrggh! You’re writing the data *four* times now”. First being the txn data into the log, then the replication log into the log, and then both of these are written back to the data file. It turns out that this is much cheaper than doing the additional fsync()s.
In one of our tests, the file based transaction log: ~300tps. Transaction log in InnoDB: ~1200tps.
I think that’s an acceptable trade-off.
We’ve just merged the first bit of this work into Drizzle.
Props go to Joe Daly, Brian and myself for making it work.
Of course it does! We have The GIMP and Photoshop! Well…
Back in the day, when everybody shot film, things were a bit more difficult. For a lot of operations it was pretty easy: select the right film, right exposure. For control you could vary how you developed it and beyond that, you could do a million things in the darkroom when printing. However, if you wanted to do something like combine 2 images or take out part of an image or smooth a skin tone, you were in for a lot more fun.
Retouching was done by changing the negative. If you wanted to remove that pimple from a portrait? Go get some paint and pant over it. This was tricky, as for 35mm film, this was very small and fiddly.
This is why publications such as Playboy shot on larger format film. From what I’ve read, either 120 (“medium format” to you and I – bigger than 35mm, but still not huge) or 4×5 (inches – much bigger) or even 8×10. While we can all wish that we too could get hold of some 8×10 Kodachrome to play with (and presumably a lab to process it for us) – those days are long gone.
With a negative of 8×10 inches, you have a lot more to play with and it’s much easier. For one thing, a contact print is as big as most enlargements people do from 35mm!
With humans essentially painting on negatives, it became relatively easy to spot when things had been manipulated (meaning there were experts who did it). However, with the increased sophistication of digital tools, creating quite realistic (even to expert eye) manipulations wasn’t that hard.
Recently, Canon (among others) has tried to bring technology to digital that would enable you to check that the image has not been manipulated after it came out of the camera.
This technology is, of course, flawed.
From the guy who enabled blind people to read eBooks comes the breaking of this system (Boing Bong and Network World).
“Pics or it didn’t happen” just is completely not true.
Could somebody please explain what the image has to do with wine:
Really. Please. Somebody enlighten me. A google image search for “Australian wine expert” doesn’t seem to show up any photo I can easily spot as being the person illustrated here.
Okay… So I’ve blogged many times before about the Storage Engine API in Drizzle. This API is somewhat inherited from MySQL. We have very much attempted to make it a much cleaner interface. Our goals in making changes include: make it much easier to write and maintain a storage engine, make the upper layer code obviously correct and clear in what it’s doing and being able to more easily introduce optimisations.
I’ve recently added a Storage Engine that is only used in testing: storage_engine_api_tester. I’ve blogged on it producing call graphs (really state transition graphs) before both for Storage Engine and Cursor.
I’ve been expanding the test. My test engine is now a wrapper around a real engine instead of just a fake one. This lets us run real queries (and test cases) while testing what’s going on. At some point in the near future I plan to make it so that it will be able to log what calls go on to the engine and produce a graph just of those.
I added a lot more to the Storage Engine part of the wrapper. Below is what you can see is the current graph:
I’ve coded what I consider to be bugs as red and what I consider suspect as blue.
Also for the Cursor (colours mean the same):
As you can see, there’s currently some wacky possibilities. I’m investigating exactly what’s going on here – If I’m somehow missing some calls that I should be wrapping (I don’t think so) or if we are really doing some dumb-ass things in the upper layer.
Also, please do not be under any impression that any of this means that we’re going to have a stable API. We’re not. To stabilise on this would just be insane – way too much of it still makes not much sense.
Our innobase plugin in Drizzle is now based on Innobase 1.1.1 from MySQL 5.5.5-m3.
This brings all the great new improvements from the Innobase team to Drizzle.
I managed to light seal the Laundry (not pretty… but it worked) and started playing with one of the enlargers I bought recently. I had a bit of an inkling from some reading I did ages ago about what I had to do to make prints.
I didn’t really have any developer meant for prints… so I just grabbed some Rodinal and dived right in. Basically started with the lens wide open and around 0.5 to 1 seconds exposure.
Because I was just experimenting, I skipped a stop bath (did a rinse though) and then straight into some fixer.
Here are the results of my experimentation (photos taken with my phone of the drying prints)
Contrast these with the scans of the negatives:
I wonder if this comes under “Code Style” or not…
Anyway, Monty and I finished getting Drizzle ready for adding “-Wframe-larger-than=32768” as a standard compiler flag. This means that no function within the Drizzle source tree can use greater than 32kb stack – it’s a compiler warning – and with -Werror, it means that it’s a build error.
GCC is not perfect at detecting stack usage, but it’s pretty good.
Why have we done this?
Well, there is a little bit of recursion in the server… and we can craft queries to blow a small stack (not so good). On MacOS X, the default thread stack size is only 512kb. This gives not many frames if 32kb stack is a even remotely common.
I found some interesting places to throw a lot of things on the stack too – that would be rather far down on a callchain – leading to the possibility of blowing up in really strange ways.
We’d love to make it 16kb…. but that’s a fair bit more work, so something for the future.
We’ve used the Boost scoped_ptr to address a bunch of these situations as it provides pretty much minimal code change for the same effect (except that memory is dynamically allocated instead of as part of the stack frame).
My branch that updates the innobase plugin in Drizzle to be based on innodb_plugin 1.0.9 has been merged. For the next milestone, we’ll probably have 1.0.11 as well.
How’s the progress getting 1.1 and 1.2 in? Pretty good actually. We’ll have it for either this milestone or the next one.
and merging newer innodb into HailDB? It’s going well too, expect more news “soon”.
There are much better places. For example: not in my roof and especially not in my roof with no crawl space and especially especially not in the roof above my office.
I just spent some time on the Doxygen documentation for HailDB, making sure all the new APIs are documented. You can view them online over at http://www.haildb.com/doc/api/html/
Following on from my post yesterday on the various states of a Storage Engine, I said I’d have a go with the Cursor object too. A Cursor is used by the Drizzle kernel to get and set data in a table. There can be more than one cursor open at once, and more than one per thread. If your engine cannot cope with this, it is its responsibility to figure it out and return the appropriate errors.
Let’s look at a really simple operation, inserting a couple of rows and then reading them back via a full table scan.
Now, this graph is slightly incomplete as there is no doEndTableScan() call. But you can see in which order things are meant to happen. In this case, “store_lock()” means that store_lock() has been called, so when coming back from doInsertRecord() we do not call store_lock() again, rather, we’re just in a state where it has already been executed.
For MySQL handler, think ::write_row() for doInsertRecord() and ::rnd_init() for doStartTableScan().
This diagram was again auto-generated from my test engine.
I’d meant to finish writing this way back in July… but I failed at that. Now is a good time to talk about Rookie-O as my again new colleague Andrew Hutchings (Buy his and Sergei’s book on MySQL 5.1 Plugin Development!)Â just went through the same thing (but in London instead of Hong Kong) given by the same trainer (Hi Eddie!).
Rackspace is the second employer I’ve had that has some kind of new hire training (the first being Sun). I am, of course, not quite counting Salmiakki as new-hire training for MySQL (although I probably should). To quote from the Wikipedia article: “Although the rumor of the heart attack was a hoax, the drink may still cause harm. The strong flavor almost completely masks the presence of ethanol, and the drinker may not realize he is consuming a drink almost 40% alcohol by volume (80-proof), leading to possible alcohol poisoning.” A promising introduction to the company.
Monty, MÃ¥rten and Kaj with Salmiakki singing Helan GÃ¥r at the MySQL User Conference Japan in 2007
I could possibly say something about the Sun New-Hire training… but I’m just trying to find something positive to say – and I can’t. I got a bit of hacking done? Seriously.
Actually coordinating a time to attend a Rookie-O (Rookie Orientation, the Rackspace name for new hire training) was rather tricky. There was one right before the MySQL User Conference back in April (not the best of timing), one during an upcoming team meeting (again, not ideal) and one that got organised in the middle of everything for the office in Hong Kong. So, I headed to Hong Kong.
The Hong Kong office is relatively new (late 2008) and there were people there who hadn’t gone through the standard Rackspace Rookie-O (Orientation).
It was rather cool to hang out with other people who worked for the company – and in totally different areas than I do. I did get a better understanding for how the rest of the company operates and the people involved. The training itself was useful and substantially less geared towards not-my-job than Sun’s was.
The good news is that Andrew thought it was useful too. Pretty impressed so far.
Drizzle still has a number of quirks inherited from the MySQL Storage Engine API (e.g. BLOBs, row buffer, CREATE SELECT and lack of DDL transaction boundaries, key tuple format). One of the things we fixed a long time ago was to have proper methods for StorageEngines to be called for: startTransaction, startStatement, endStatement, commit and rollback.
If you’ve had to implement a transactional storage engine in MySQL you will be well aware of the pattern of “in every Storage Engine/handler call: if transaction doesn’t exist, begin.” We’ve tried to fix this in the Drizzle API for a number of reasons. I think having this obvious set of calls will make the API a lot easier to understand. I am also very interested in making things much easier to prove correct.
A while ago I spotted Bug 587772, which was the READ COMMITTED isolation level not working correctly with InnoDB. It turns out that the most basic example for READ COMMITTED failed. Hrrm… this is no good. It worked on MySQL, so this was certainly something that we broke. What was more worrying is that there wasn’t a test for this in the test suite (and at the time I couldn’t find one in the MySQL test suite either, so I think we inherited the missing test).
I recently started delving in, actually going to solve this. I noticed something worrying, endStatement wasn’t being called, which is where the innobase plugin would release the read view that it used for the statement. You’d think that it would grab a new one on startStatement, but because of the previous design of the API (remember “if txn isn’t started, start it!”) this also happened for getting the read view for the statement… so we instead got a REPEATABLE READ isolation level.
I wanted a test.
Previously, I’ve created a dummy storage engine (tableprototester) and used it to test the server code for reading the table protobuf message. I thought about doing a Storage Engine for this problem too, basically looking at the calls to the Storage Engine as transitions between states in a state machine.
A basic view of a transaction could be:
That is, a transaction starts and has zero or more statements before it commits or gets rolled back.
By coding up a data structure of allowable state transitions, a small function to assert() on invalid transitions and enough of the boilerplate to make the engine “work”, I was able to hit an assert() exactly where I’d expected it: at an invalid transition from START STATEMENT to COMMIT.
To fix the initial bug (READ COMMITTED not working), I filled in a few state transitions for the system as a whole that aren’t quite correct. From the diagram below, you can quite obviously see where the obvious bugs are (it helps that I’ve coloured them red):
There is absolutely no sense in going BEGIN -> END STATEMENT or immediately to COMMIT. These should be relatively easy to solve too, but are separate bugs.
I wish to expand this in the future to cover Cursor as well. It will also be useful to ensure that DDL can be wrapped in transactions. Not to mention the last few HTON flags that exist (and should likely go away).
To generate the diagrams, I just wrote a little utility to dump out the state transitions in dot, using it to generate the diagrams.
Way back when, for linux.conf.au coming to Melbourne in 2008, I edited together a promo video for it. IIRC the raw video was shot by Kelly on DV tape, imported in and I got a CD of some massive 400MB MPEG file of a bunch of questions. Using Cinelerra and some graphics package that I forget (very early Inkscape?), I managed to get this done in 2006. I understand things are a bit less segfaulty these days.
See it on YouTube or download the Ogg Theora video.
Amazingly enough, this is the last time I actually did any video editing.
You should also go to linux.conf.au 2011 in Brisbane this upcoming January.
A while ago (April) I bought a Pogoplug with the explicit idea of using it as a NAS device. I finally bought a new 2TB drive and plugged in the pogoplug. I pretty much instantly realised I was going to run Debian on it instead, if only because that’s what makes me comfortable: Debian, ssh and XFS.
Installing Debian was easy (google it) and incredible props for not only an attractive looking device, but an easily hackable one (the default software is probably quite good for non-experts… I just happen to want Debian).
I have to say, I’m so far rather happy.
This is meant to replace f-spot.
It just isn’t ready.
I do not have what I would consider a large photo collection. It’s about 77GB on disk, maybe 30,000 images.
Importing from f-spot is horrendously slow for what is essentially a few INSERT..SELECT statements. It does not copy your photos anywhere, yet takes about that long.
It eats memory for breakfast. No, really. I bought a digital camera around New Year 2003. In just importing the photos from 2003… It’s now using 800MB of memory… sorry, 900MB now. At final count, at the end of 2003, it’s up to 1.6GB of memory used and an additional 300MB of disk space in ~/.shotwell/thumbs. How on earth is it going to cope when it gets to where I really start shooting? Now, the Shotwell website does state that there was a memory leaking bug that is now fixed in trunk. Note where it isn’t fixed – in Ubuntu.
Ubuntu 10.10 currently ships with an unusable photo manager.
f-spot is nowhere near perfect. Relegating it to universe instead of main (i.e. it’s now “not maintained by Canonical”) is just stupid.
Meanwhile, I still love darktable – it’s simply wonderful.