Solaris, Linux, it is GNU folks…

Brian “Krow” Aker’s Idle Thoughts – Solaris, Linux, it is GNU folks…

Brian hits the nail on the head… The way you get a usable system is install all the GNU tools.

This is how I go from fresh Ubuntu install to building MySQL:

apt-get build-dep mysql-server

apt-get install bison

(now go and build).

(and i could do this graphically if I wasn’t so stuck in my ways)

For Solaris? umm… there was a point where I could get Solaris to apply security updates and Brian could get all the stuff needed to build a MySQL Server. Together we had the knowledge needed… but neither was as trivial as with Ubuntu and combining knowledge was too much – I just gave up and went on to more productive things.

Even on an existing Solaris system… getting your PATH right is a trip into some weird fantasy land seemingly designed to annoy you. No doubt this all made some sense back in the day… but now it just causes pain when all you want to do is compile your program, find the bug and fix it.

When I started at SGI several years ago, what’s the first thing I did? Went and installed all the GNU packages. IRIX is a lot nicer then.

Same with MacOS X – the first thing you do is go and install darwinports or fink and get a remotely usable system.

With Windows, it varies – but the shell is so outrageously shit you need cygwin just for bash, you need either emacs or VisualStudio to get an editor you don’t want to kill, Firefox for a web browser that works etc etc etc. The fact that the Windows packing system just blows chunks makes it the most painful experience of all.

So even if you’ve heard rave things about the debugger in VisualStudio – actually getting a Windows install to the state where you can run the debugger takes hours. Click click click, upgrade, yes, install, swap disks, upgrade, upgrade, wait, reboot, install manually, install manually, install manually. ick.

Project Indiana is possibly the saviour of Solaris. Default userland is gnu, default shell is bash. Starts to make it feel like home. Just as when Solaris started shipping GNOME made it feel more homely.

Solaris comes with a version of vi that is old enough to drink in bars. Project Indiana realises that a drunk editor isn’t a good idea and ships something sensible.
The BSDs get a lot of things right. Sane userland that is familiar to people. Jumping onto a FreeBSD box is remarkably easy.

The typical thing said by people is “backwards compatibility” and all that… basically so that everyone can run their apps from 1985 and not change a thing. Worthy goal. Of course, 1985 does not need to be the default environment in 2008.

There is a standard for the unixy way of things: it’s Linux with GNU tools in userland.

Just as Windows set the big standard for having a kind of usable GUI (the Mac did it better, but Windows got the numbers) – and to get people to use Linux on the Desktop we needed to get it to a stage where those people are comfortable.

If you want your UNIXy system to be used by anybody today, you need to have it be comfortable for Linux people.

On the other hand though, Ubuntu is still the best desktop I’ve ever used and am rather happy with it (no matter how much i bitch and moan about certain things being obviously broken).

(and no, I’m not switching my desktop to any Solaris variant – but wholeheartedly look forward to the days when maintaing software than runs on Solaris is a heck of a lot easier because Solaris becomes less annoying).

One more point: OSX and Solaris are the only remotely proprietary UNIXes left. Everybody else is either dead or doesn’t know it yet. Solaris is nearly all free (AFAIK there’s still just some binary only drivers around… which sucks… but these things can take time, so that’s okay) and OSX has parts which are (sometimes seemingly dependent on phase of the moon) free-ish. So really, OSX is the one last hold out of the largely proprietary UNIX world. It’s a fascinating thing to think about…. freedom wins.

(and this no doubt goes on far too long and incoherently…. but that’s because of long days and late nights because of upcoming really cool stuff which I’ll blog about later)

bzr-loom – a bzr plugin with quilt like functionality

A bzr plugin to assist in developing focused patches. in Launchpad

I use quilt a lot for development. Currently, If I had to choose between BK and quilt – I’d choose quilt.

I use bzr in other development projects like MemberDB. I use git as a frontend for SVN (it is *so* much faster than the svn client and incredibly more space efficient… A copy of the entire history of a tree stored in git is usually less than a single svn checkout). I also use darcs (and quilt) for offlineimap and just about every other revision control tool at some point.

So this is a bit of a discussion about how I work and how bzr-loom would help it… (I’ve wished for a long time that bk had stuff like this… bk collapse is just not what I want, although others use it lots).

The loom plugin to bzr looks like a fantasy world of goodness where the revision control system has some knowledge of these work in progress patches. The ability to push and pull looms around the place seems awfully nice.

What’s even more awsome is that you can push your set of patches up to a normal bzr branch and they become normal commits! i.e. you get rid of the whole “convert quilt patches into changesets” pain and just push.

Revision tracking them (so you can see what you’ve changed in your patch set) is also nice (I have thought about keeping patches/ in a bzr repo for this purpose). So I can now get a history of my patchset against various mainline versions.

One of the big advantages of quilt is speed – it’s lightning fast (basically being a diff and patch wrapper) . Hopefully bzr looms continue in this fine tradition (and I wish other systems would get something like it too)

A world of FAIL

=================================== ERROR ====================================
File holyfoot/hf@mysql.com/deer.(none)|mysql-test/r/bdb_notembedded.result|20061113160642|60022|276fa5181da9a588
is marked as gone in this repository and therefor cannot accept updates.
The fact that you are getting updates indicates that the file is not gone
in the other repository and could be restored in this repository.
if you want to "un-gone" the file(s) using the s.file from a remote
repository, try "bk repair "
takepatch: other patches left in PENDING
==============================================================================
2.85M uncompressed to 16.4M, 5.77X expansion
Pull failed: takepatch exited 1.

stewart@willster:~/MySQL/5.1/ndb$ cp ../../5.0/ndb/mysql-test/r/SCCS/s.bdb_notembedded.result mysql-test/r/SCCS/

A world of fail. Of course, the “bk repair” suggested does not fix the problem. This kind of problem should just not be possible. grr…

Getting a file size (on Windows)

The first point I’d like to make is that you’re using a Microsoft Windows API, so you have already lost. You are just not quite aware of how much you have lost.

A quick look around and you say “Ahh… GetFileSize, that’s what I want to do!” Except, of course, you’re wrong. You don’t want to use GetFileSize at all. It has the following signature:

DWORD WINAPI GetFileSize(  __in       HANDLE hFile,

__out_opt  LPDWORD lpFileSizeHigh

);

Yes, it supports larger than 4GB files! How? A pointer to the high-order doubleword is passed in! So how do you know if this errored? Return -1? WRONG! Because the high word could have been set and your file length could legitimately be 0x1ffffffff. So to find out if you actually had an error, you must call GetLastError! Instead of one call, you now have two.

The Microsoft documentation even acknowledges that this is stupid: “Because of this behavior, it is recommended that you use GetFileSizeEx instead.”

GetFileSizeEx is presumably named “Ex” as in “i broke up with my ex because their API sucked.”

You now have something that looks like this:

BOOL WINAPI GetFileSizeEx(  __in   HANDLE hFile,

__out  PLARGE_INTEGER lpFileSize

);

Which starts to look a little bit nicer. For a start, the return code of BOOL seems to indicate success or failure.

You now get to provide a pointer to a LARGE_INTEGER. Which, if you missed it, a LARGE_INTEGER is:

typedef union _LARGE_INTEGER {  struct {

DWORD LowPart;

LONG HighPart;

};

struct {

DWORD LowPart;

LONG HighPart;

} u;

LONGLONG QuadPart;

} LARGE_INTEGER,

*PLARGE_INTEGER;

Why this abomination? Well… ” If your compiler has built-in support for 64-bit integers, use the QuadPart member to store the 64-bit integer. Otherwise, use the LowPart and HighPart members to store the 64-bit integer.”

That’s right kiddies… if you’ve decided to loose from the get-go and have a compiler that doesn’t support 64-bit integers, you can still get the file size! Of course, you’re using a compiler that doesn’t have 64bit integer support… and the Microsoft documentation indicates that the GetFileSizeEx call requires Windows 2000… so it’s post y2k and you’re using a compiler without 64-bit ints? You have already lost.

Oh, but you say something about binary compatibility for apps written in the old days (handwave something like that). Well… let’s see… IRIX will give you 64bit numbers in stat (stat64) unless you build with -o32 – giving you the old ABI. I just can’t see a use for GetFileSize….. somebody please enlighten me.

Which header would you include? Any Linux/UNIX person would think of something logical – say sys/stat.h (Linux man page says sys/types.h, sys/stat.h and unistd.h). No, nothing sensible like that. It’s “Declared in WinBase.h; include Windows.h”.

So… you thought that obviously somebody went through the API and gave you all this Ex function goodness to get rid of mucking about with parts of a 64bit int? You were wrong. Let me say it with this:

DWORD WINAPI GetCompressedFileSizeTransacted(
  __in       LPCTSTR lpFileName,
  __out_opt  LPDWORD lpFileSizeHigh,
  __in       HANDLE hTransaction
);

I’ll now tell you that this was introduced in Vista/Server 2008.

Obviously, you want to be able to use Transaction NTFS on Windows Vista with a compiler that doesn’t have 64 bit ints. Oh, and you must then make another function call to see if something went wrong?

But you know what… perhaps we can get away from this complete and utter world of madness and use stat()…. or rather… perhaps _stati64().

Well… you’d be fine except for the fact that these seem to lie to you (at least on Windows Server 2003 and Vista) – it seems that even Explorer can lie to you.

But perhaps you’ve been barking up the wrong tree… you obviously don’t want to find the file size at all – what you want is to FindFirstFile! No, you don’t want FindFirstFileEx (in this case, Ex is for Extremely complicated). It’s meant to be faster too… you know, maybe.

So remember kids, smoke your crack pipe – you’re going to need it if using this thing called the Microsoft Windows File Management Functions.