some studies have shown that for multimedia applications, a larger block size improves throughput (e.g. 256kb blocks). For large media files, the waste of an average 128kb per file is insignificant (over several megabytes to many hundred mb or indeed GB). But, for smaller files (typically occupied by configuration files or small system binaries) a smaller block size saves more space (4k block for 100byte file).

ReiserFS goes for something slightly different allowing several files to share a single block. I am unsure if the extra effort involved in implementing this is worth it, or the slower access times that reiser reports for allowing this.

Maybe different allocation groups could have different block sizes? maybe there could be some kind of block-size migration system? or would the overhead not be worth it? Could it be one of those “maintenance” tasks that you run every month/year? How often does the average usage of a disk change that we’d need something like this?

B+Trees sorted by size and location (a-la XFS) provides:
– ability to allocate large/small objects efficiently (size)
– ability to allocate blocks near existing objects (e.g. for object expansion) by using the location B+Tree

B+Trees are good, therefor use them.

Split up into allocation groups (a-la XFS and BFS). Allows more parallel operation as different threads can work on different allocation groups (during updating).

Idea for fine grained B+Tree locking:
– each node gets an ID (which should be pretty unique… but not essential to correct operation).
– when process needs to update node, places a lock on it’s ID
– when a process accesses a node, before reading, it checks to see if a lock has been placed on that ID. IF so, we spin, waiting for the lock to be released
– when a process has finished updating a node, it releases the lock on it’s ID.
– the process waiting to read it will stop spinning and be able to read the new copy.

IF IDs overlap, there is no problem, we’ll just be spinning when a different node is being updated.

However, if we are updating a tree of nodes, then some careful locking wil have to be employed, from the BOTTOM UP! that is, update from the bottom up, locking each individual one as we go as if we only lock the root of the tree being updated, another process could have already passed it and screw us up royally.

Multivolume disks?
– uid object migration in distributed environment
– have a distributed volume (e.g. lab scenario, all lab machines have same ‘volume’ mounted and sync of primary machine)

Snapshots for all versions?
– each version would require > 1 block
i.e. 10 mods of 10 bytes to 1 block would require >= 10 extra blocks
hmm… has to be better way in this scenario….

B+Tree of unique ids (seperate from walnut object IDs).

Linux:
– transactions on UNIX filesystems
– versioning in UNIX VFS environment
– extending UNIX FS model to accommodate advanced DB like features

Walnut:
– transactions

hmmm… this is a casually tricky one…. i’ve got some ideas and thoughts down, going to mail off the PDF to Ron and Carlo and see what they think.

1/2 actual information and aims for the project
1/2 just ramblings on topics of data storage.

hmmm…..

Otherwise titled: A day of reading about Filesystems.

I’m managing to continually find stuff I haven’t already read. I think this is a good thing. I’m clearly now going to have to do some more research into how RDMBS do things.

The idea of Snapshots sound interesting, and may prove an interesting way to help avoid having to do journalling. I’ve been thinking that journalling could be a real issue if we allow large transations, so maybe a snapshot like idea could be good. Although, this could limit the number of transactions we work on concurrently if it’s implemented the WAFL way.

So looking at how RDBMS do concurrent transactions would be useful. I guess we do have the advantage of having PostgreSQL and MySQL code to peer at but I do hope there are some nice papers out there to read :)

I’ve collected a large array of useful papers to read. I’m going to try and wade through them sometime soon, but will have to do more coursework soon.

I’m thinking tomorrow might contain a bit of coursework – need to do more POVRAY/OpenGL stuff as well as study more of the Pattern Recognition stuff. Oh, and look at the Info Security assignment & exercise for this week.

I’m trying to write up what has been discussed regarding my hons project.

Got a meeting penciled in for monday morning.

Should be good.

But I have to get all these ideas that have been in my head down on paper, and all the reasoning behind them.

Finally got LaTeX on my laptop. Not from DarwinPorts though :( Got it from some i-installer program that I guy did. It works, I’m happy.

I’m thinking that I can easily do multiple FS ops in one transaction with some careful structuring of the journal.

The only problem with this is the old problem of journal size. Unlike a normal journaled FS, with user transactions, we may be dealing with a lot more data in many transactions and a static log size may be a bad thing (and severely limit the size of transactions that are possible).

I’m thinking that having an expandable log (perhaps a linked-list kind of approach) could be quite useful. In memory, we’ll probably need another data structure to have things remotely efficient (esp if we want to know where to write additional log entries).

This could also be nice performance wise as we only have to lock one of these when doing operations on the log.

I’ve also been thinking about weird things like delayed index updates. That is, you update a file (or attributes) and it doesn’t update the index pointing to it on disk until it is conveinient for it to do so.