well, i’m working on it.

bloody thing. all this other stuff we have to do instead of the project. It really does annoy me. I’d love to be able to get rid of coursework, assignments and these ‘intermediate’ things we have to do (which reminds me, i’ve got a heap of stuff to catch up on still) and just get on with the research. i.e. stuff that actually interests me/i care about.

i’m getting there on it – trying to work out what actually to talk about, in what order and all that. Not that we really get any help from this “subject” on these things. hmmm…..

got a site up (really basic) covering my honors stuff.

http://www.flamingspork.com/honors/

snapshot like system for in-progress transactions, when committing, search for common substrings in blocks and the offsets of these (possibly indicating an insert operation) and on a per-block basis create a “shift N spaces” entry in the delta fork of the object

Been re-reading a lot of the XFS papers that are on the SGI website (http://oss.sgi.com/projects/xfs/) and thinking more about what I want out of an object store. There are a lot of similar design goals (I think) yet some very different ways of implementing things.

Having a large B+Tree full of every object could be quite nice, kinda like inodes on conventional UNIX filesystems. On the object-store layer, we’d be able to store small objects inside the inodes. Above this, the namespace layer could find out if we can pack something into an inode, and if so, optimize for that (e.g. linear list of files for directories under 1k).

The idea of having a very layered system is increasingly appealing to me.
This means we could have some very nice optimizations for some applications. Some systems would only ever care about the object-store itself (a squid like caching system for example) and others could care a lot about a namespace system (or even be one). An example of the latter could be a database.

Expandability for the future is a given, it has to be. 64bits seems like a lot now, but no doubt somebody will be pushing it in 10 years or so.

some studies have shown that for multimedia applications, a larger block size improves throughput (e.g. 256kb blocks). For large media files, the waste of an average 128kb per file is insignificant (over several megabytes to many hundred mb or indeed GB). But, for smaller files (typically occupied by configuration files or small system binaries) a smaller block size saves more space (4k block for 100byte file).

ReiserFS goes for something slightly different allowing several files to share a single block. I am unsure if the extra effort involved in implementing this is worth it, or the slower access times that reiser reports for allowing this.

Maybe different allocation groups could have different block sizes? maybe there could be some kind of block-size migration system? or would the overhead not be worth it? Could it be one of those “maintenance” tasks that you run every month/year? How often does the average usage of a disk change that we’d need something like this?

B+Trees sorted by size and location (a-la XFS) provides:
– ability to allocate large/small objects efficiently (size)
– ability to allocate blocks near existing objects (e.g. for object expansion) by using the location B+Tree

B+Trees are good, therefor use them.

Split up into allocation groups (a-la XFS and BFS). Allows more parallel operation as different threads can work on different allocation groups (during updating).

Idea for fine grained B+Tree locking:
– each node gets an ID (which should be pretty unique… but not essential to correct operation).
– when process needs to update node, places a lock on it’s ID
– when a process accesses a node, before reading, it checks to see if a lock has been placed on that ID. IF so, we spin, waiting for the lock to be released
– when a process has finished updating a node, it releases the lock on it’s ID.
– the process waiting to read it will stop spinning and be able to read the new copy.

IF IDs overlap, there is no problem, we’ll just be spinning when a different node is being updated.

However, if we are updating a tree of nodes, then some careful locking wil have to be employed, from the BOTTOM UP! that is, update from the bottom up, locking each individual one as we go as if we only lock the root of the tree being updated, another process could have already passed it and screw us up royally.