Author Archives: Stewart Smith

on microblogging/twitter2blog etc…

Posted on by
3

Andrew Pollock makes the observation that a microblogging (twitter or whatever) stream added to a regular blog doesn’t add much for the reader… I’ve been debating this myself ever since I installed the WordPress Plugin that does twitter integration.

One motivation was “I don’t like everything being tied up in some proprietary system” and wanting a copy of my data somewhere. On the other hand, what I write into twitter isn’t exactly that-scottish-play and loosing it all would probably be a net gain for humanity.

So… do I turn off the Twitter plugin? I’m erring on the side of “yes, turn it off”. Thoughts?

Security question fail.

Posted on by
8

Spot the problem:

You work for company X.

  • Phone rings: “Hi, my name is Alice, I work for company X”
  • “Hi Alice, this is Bob, in order to verify that you do actually work for X, what is your employee number and phone extension, I’ll call you back when verified”.
  • “Okay Bob, it’s Alice, employee number 1234 and I’m on 555-5555”
  • You look up the employee database and sure enough, Alice is there with number 1234.

Were you talking to Alice?

Will you be talking to Alice if you dial 555-5555?

Encrypted Online Backup Preview 2 (DES and AES)

Posted on by
Reply

New preview includes:

  • 3DES support
  • AES support with 128 (default), 192 or 256 bit keysize

bzr branch lp:~stewart-flamingspork/mysql-server/stew-encrypted-backup-preview2

(you can pull this directly into the previous preview1 branch, it’s just 2 extra patches).

Examples:

  • BACKUP DATABASE test to ‘test.ba’ ENCRYPTION_ALGORITHM=3des PASSWORD=’pants’;
  • RESTORE FROM ‘test.ba’ ENCRYPTION_ALGORITHM=3des PASSWORD=’pants’;
  • BACKUP DATABASE test to ‘test128.ba’ ENCRYPTION_ALGORITHM=aes  ENCRYPTION_KEYSIZE=128 PASSWORD=’pants’;
  • RESTORE FROM ‘test128.ba’ ENCRYPTION_ALGORITHM=aes ENCRYPTION_KEYSIZE=128 PASSWORD=’pants’;

Enjoy! Feedback is very muchly appreciated.

MySQL Encrypted Online Backup Preview 1

Posted on by
4

I’ve just pushed to launchpad, a set of patches that implement AES encryption support for MySQL Online Backup. You will need to build –with-ssl to get support for encrypted online backup.

Encrypted backup files have no recognisable header – they’re just a stream of random bytes. Encryption and compression also works, and when you encrypt, you should also use compression (compress first, then encrypt).

Source tree: https://code.launchpad.net/~stewart-flamingspork/mysql-server/stew-encrypted-backup-preview1

Bzr command: bzr branch lp:~stewart-flamingspork/mysql-server/stew-encrypted-backup-preview1

Thread on internals@ with patchset: http://lists.mysql.com/internals/35759

Build instructions: be sure to ./configure –with-ssl

Example usage:

  • BACKUP DATABASE test TO ‘test.ba’ ENCRYPTION_ALGORITHM=aes PASSWORD=’pants’;
  • BACKUP DATABASE test TO ‘test1.ba’ WITH COMPRESSION COMPRESSION_ALGORITHM=gzip ENCRYPTION_ALGORITHM=aes PASSWORD=’pants’;
  • RESTORE FROM ‘test.ba’ ENCRYPTION_ALGORITHM=aes PASSWORD=’pants’;
  • RESTORE FROM ‘test1.ba’ ENCRYPTION_ALGORITHM=aes PASSWORD=’pants’;

UPDATE: preview1 is no more, preview2 is now out, contains extra stuff (see next blog post)

NDB$INFO scanning from ndb_mgm

Posted on by
Reply

In code just tested:

ndb_mgm> ndbinfo MEMUSAGE
RESOURCE_NAME    NODE_ID    PAGE_SIZE_KB    PAGES_USED    PAGES_TOTAL    BLOCK
IndexMemory,1,8192,16,160,DBACC
DataMemory,1,32768,20,640,DBTUP
IndexMemory,2,8192,16,160,DBACC
DataMemory,2,32768,20,640,DBTUP

Win!

This is the first time that we’ve been able to get this kind of info out of the cluster without using the magic “all dump 1000” (or “all report MemUsage”) which end up using events, which go to the log file, aren’t exactly reliable etc.

This performs a scan on the NDBINFO tables (in ndbd) from ndb_mgmd and returns the result to the management client. You can then use this in scripts from the command line. e.g. to find out how many pages of datamemory are used on each node:

$ ./storage/ndb/src/mgmclient/ndb_mgm -c localhost:9311 -e ‘ndbinfo MEMUSAGE’|tail -n +3|grep ‘DataMemory’|cut -d ‘,’ -f4
20
20

now, just to clean it up a bit, fix the one bug (yes, you guessed it: in metadata caching) and get a review….

but, a milestone!

NDB$INFO

Posted on by
Reply

There’s been talk over the years of better monitoring for NDB (MySQL Cluster). This has been dubiously named NDB$INFO, after some special magical naming convention for tables holding information on the insides of NDB. Otherwise known as Worklog 3363 (viewable on MySQL Forge).

The basic idea is to get a bunch of things that are already known inside NDB available through a rather standard interface (SQL is preferred).

My top examples are “How much DataMemory is used?” and “Do I need to increase MaxNoOf(Tables|Attributes|ConcurrentTransactions)?”. You can get some of this information now either through the management client (ndb_mgm -e “all report MemoryUsage”) or the MGM API using events and some other foo.

This is a rather limited interface though. It would be great if you could point all your monitoring stuff to a MySQL Server, throwing queries at it and finding out the state of your cluster.

So this year I’ve been working on implementing NDB$INFO. The big requirements (for me at least) are:

  1. Everything can be queried easly from SQL
  2. It’s easy to add a new NDB$INFO table (for a NDB developer)
  3. you can use NDB$INFO tables to diagnose problems (such as nodes not connecting)

Among the 492 things I’m currently doing, is fixing up a basic patchset for NDB$INFO and working on getting it into the tree. It’s all going to be basic scan interfaces in the current version, so things may be slow if there’s lots of rows, but they’ll get there.

What would you like to see exposed?

Encrypted Online Backup (design, thoughts, ask-the-lazyweb)

Posted on by
4

So after a ever so temporary but loud moment of insanity[1] having a decision made which I very strongly disagreed with (wanting to release online encrypted backup as closed source), we’re back in the world of freedom and the MySQL Server is (and will be) free and open source software (dual licensed, so you can buy a commercial license of the same thing).

[1] Addition (wanting to remove my use of the word): Marten (rightly) points out that although appreciating the new blog posts, he doesn’t appreciate having his decisions called insanity. He’s right. It’s the wrong way to put it. So, without wanting to censor or change history (instead preferring to illustrate my own stupidity and amazing ability to completely say the wrong thing every 6 months or so), I offer this clarification (that i have tried to express in about 3 drafts of blog posts, none of which have made the light of day as i was never really happy with them): the decision was made with all the right intentions (grow the company, end up producing more free software, making sales to enterprises easier, clearer differentiation etc) but it was one that I (and many others) rather strongly disagreed with. In the end, the dicision was made to have these parts as free software and I truly believe that this was made after more arguments were presented by myself (and others) about why having these parts as closed was a bad idea. It is quite the thing to make the decision to make modules for your free software product closed, it is about 15 steps higher to go back on it. I’ll share a phrase I used a few times when being a right nick-picker about things during employment contract negotiation this year (for MySQL Australia and then Sun): “Do I trust Marten? Absolutely. It’s the next guy. Remember, SCO was once Caldera and producing a linux distro and generally considered good.” So, that was more than I intended to write on the subject… but hopefully clarifies that I just thought the decision itself was bad, and am lucky enough to work at a place that encourages discussion when you don’t like things.

So, now I’m involved with writing up the worklog for encryption for the MySQL server native online backup. I also wrote most of the original worklog for compression of online backup (I implemented compressed backup and LCP for MySQl Cluster) as well as some proof-of-concept code (written in <5 minutes at 3am while jetlagged).

There are two main approaches to encryption: symmetric and asymmetric (public key). I think we should support both (but we’ll see what others think).

For symmetric (password based for those not up with the street lingo of crypto) we’re thinking of the following algorithms: 3DES, AES, Blowfish. Are there any others that people care about?

DES is obviously out as it’s not considered secure, and really, we should be helping users to get things right.

For public key: RSA and DSA are the obvious choices.

As for libraries implementing all of these? well….. I’m thinking about libgcrypt – it looks fairly nice and a bit similar to the kernel crypto api (which seems quite nice). Anybody got any other suggestions? Things you’d like to see? thoughts?

EDIT: Server not Service. We sell services, the server is free and open source. I fail.

eHorizons

Posted on by
2

I flew back into Sydney on Sunday morning to give a tutorial at Sun’s Expanding Horizon’s summit. It was a half day tutorial on MySQL Cluster – so a shortened version of the one I’ve given at the MySQL User Conference for the past few years. I had about 15 attendees, all of which had done their homework (It probably help that they were pestered via phone :)

The tutorial went really well. It really helps when everybody has done the homework and already have Linux and MySQL Cluster installed. Everybody got up and running (we used mysql-test-run to start a cluster, not writing the config file from scratch, which made things happen a lot faster). Also got some good feedback – yay! We may even have some people look to deploy it after attending, always a plus.

I also gave a “Scaling MySQL” talk that was well attended. I didn’t talk at all about query optimisation, mysqld configuration tuning or stuff like that – instead focusing on making the app saner, caching etc. memcached, of course, got a good mention :) It seemed to go down well, some good questions, and a rather full room.

So a rather productive two days for spreading the freedom love.

However, the conference dinner was complete FAIL on account of the venue. I don’t know which vegetarians/vegans call beef and fish vegetarian, but I’ve never met one (hint: they don’t exist). This is *after* the explanation on being vegan. Then… there was some discussion about pasta with a tomato/vegetable sauce, never came. So as others were finishing meals, again inquire – eventually, something was brought over. Undercooked rice and undercooked steamed vegetables. I don’t know who eats that for dinner (hint: nobody). Of course, after the pasta discussion, I then selected a wine that would go with it. After more of the stuff-ups, I pointed out that there was no way I was going to pay for the wine when shit like that was served (yes, in those words… perhaps I’ve been watching too much Gordon Ramsay).

It was the first time ever that I’ve left a restaurant during a function, gone down the street, gotten take away and brought it back. Novotel Brighton Beach (in Sydney) – you suck.

(there’s also a beutiful view across the bay of the runways of Sydney airport… which is fine if you can sleep through planes landing an taking off, like i can, but i know others can’t).

Will never stay at the Novotel Brighton Beach voluntarily, ever. On the plus side, the guy at the desk when checking out was very apologetic…

My Name Is…

Posted on by
4

Stewart.

With a t at the end. Not a d.

Get it wrong once, possible mistake.

When I correct you, and you do it *again* and *again*, I want to slap you in the face with a keyboard.

It’s especially bad when you’re replying to my email, as my name is spelt correctly at least THREE TIMES right in front of you (Twice in “To: Stewart Smith <stewart@…>” and again in “On X, Stewart Smith wrote:”.

(and if you’re wondering why I’ve tagged this post with “sun” and “mysql” it’s because some of you people are the worst offenders)

kthxbye

Vegan Bread Machine Bread (no preservatives)

Posted on by
2

I have purchased exactly one loaf of bread this year. The fact that it’s May is kind of amazing for this…. and it’s not as if I haven’t eaten any bread – I love the stuff. I’ve been making it.

Relatively recently, I bought a bread machine. It’s a breville and was only a couple of hundred bucks. The idea behind this was to still have bread when I was feeling too lazy to do everything by hand (or was time pressed).

I have discovered the following:

The 750g loaf is a bit small, the kneading can end up missing some ingredients in the corners of the bread machine. the 1kg loaf is fine, the 1.25 is freaking huge.

The area of the bread (as in how big a slice is) is much larger than any of the bread tins I use in the oven.

The crust on the side of the loaf is usually nicer than on the top (top isn’t as thick and crunchy).

To avoid using bread improver, instead use the juice of an orange (or I’ve also used about 1/3 cup of fresh orange juice). This gives lovely structure and texture to the bread without adding anything artificial.

Some recipes say add some milk thing, i’ve just ignored it and done fine.

Waking up to fresh bread smell is amazing.

We put on a friends bucks night last weekend, and in the afternoon we did food at a friends place. Sort of a lunch/dinner thing to keep us going with the nights drinking and general revelry. I baked 3 loaves of bread and it went down really well (10 of us finished all 3 off). Got lots of compliments for it too :)

Of course, the majority of bread from everywhere else now just tastes shit (because it is). If you’re going to buy bread – buy it from a *really* good baker. Or, take the hit – buy a bread machine and do it. Cost per loaf for yeast, water and salt is approximately nothing and bakers flour is cheap.

Join the war on bad bread, bake your own today!

Simple vegetalbe stir fry

Posted on by
Reply

Wasn’t feeling like spending a lot of effort on cooking last night, but still wanted something fresh, delicious and quick to prepare. So… I just made a nice, simple stir fry. I here try to reproduce what I did from memory.

Started by heating some oil in a wok, adding a teaspoon or two of mustard seeds. Leave until the mustard seeds start to pop (that gives the oil a nice flavour).

Add a finely chopped onion, a finely chopped chili (this is all the chili i used in this, adds nice flavour, a tiny bit of spice and it’s certainly not too much) and some crushed garlic (I used 2 cloves of some good organic garlic… may want more or less depending on the strength of your garlic).

Basically, you want the onions to start going clear… I also added a tiny amount of turmeric.

I then added some finely chopped capsicum, then some carrots. It’s about now that I added some light soy sauce. Keeps things moist and adds nice flavour.

In a mortar and pestle, I ground up some sesame seeds, cumin and cardamon (not much of each, the aim is to add subtle flavours) and then add to mix. I also added some roughly chopped swiss-brown mushrooms (button mushrooms would work too, I just felt like the swiss-brown as they have a nice flavour). I also added a bit more of the light soy, just to keep things moist.

I also added finely chopped broccoli and at the end snow peas.

Served on rice, delicious.

Was interesting for me as I usually end up using tofu and creating some sort of sauce to go over a stir fry. This was flavoursome (Michael also thought so, ended up coming over for food before we all went out), filling and there’s left overs :)

(I also used my Microwave’s wonderful “Rice” button… I love that. Obviously, put the rice on *FIRST* before you start any prep for the stir fry. Both were ready about the same time.)

timezones and when you get mail

Posted on by
1

It’s very weird having mail arrive at different times… e.g. when in the US, it’s less likely for some peolpe to mail you in the middle of the night US time.

Back home… common to get response between local time 2-5am… which is rather not 2-5am their local time.

But my habit of pulling mail in the morning after having done so late at night…. hard to break.

mysql-5.1.24-stew1 (with Maria and PBXT)

Posted on by
7

I’ve hacked around a bit to get PBXT to compile in tree, and pulled in the Maria engine. Both are latest source.

So, want to try out Maria?

Want to try out PBXT?

Just want to do ./configure and go with it, just like building a normal MySQL Server?

Grab the -stew tree. Source tarball here:

mysql-5.1.24-stew1.tar.gz

(it’s based on something close to 5.1.24… and I’m going to switch some of my systems over to it rather soon… already done some good benchmarks on one of my apps).

feedback much appreciated.

UPDATE: Got x86-64 Linux. Try my binary tarball (built from above src tarball). Built on Ubuntu Gutsy (my laptop). So it may (or may not) work. If it kills your squirrel, not my fault.

OpenLDAP, MySQL Cluster, world of awesome

Posted on by
1

Last night (okay… i’m posting thsi a bit later… so the other night), a group of us gathered around to hear about some work that had been done in getting a MySQL Cluster backend for OpenLDAP.

I’d heard a bit of rumors (where rumors is defined by somebody saying something on IRC and me being busy looking at other things) about this previously, but last night was the first time I a) saw it working and b) saw performance numbers.

Disclaimer: I am no LDAP expert.

So, what is it?

Normal LDAP can replicate asynchronously from one machine to another. You can even update on both and it has some conflict resolution. But… this costs in performance.

Normal LDAP can also replicate (asynchronously) to a remote location for read-only (e.g. make authentication go faster in Australia with the main LDAP server in the US).

The MySQL Cluster backend for OpenLDAP connects directly to MySQL Cluster, using rather optimised schema, indexes and coding (directly using everything that we’re good at – which was really awesome to hear).

So, the MySQL Cluster backed LDAP server is the 2nd fastest in the world. The fastest is OpenLDAP on a single machine. With MySQL Cluster, we’re not that much slower than a single box – but we have redundancy. So that’s totally awesome. The third fastest…. much slower than us.

This was one of the most awesome new things I’ve seen here.