Yearly Archives: 2012

Jenkins Bazaar plugin 1.19

Posted on by
Reply

I recently released a new version of the Bazaar plugin for Jenkins. This release was inspired by a problem we noticed at Percona. It is:

  • run “bzr revert” after a pull, as if you have a directory that is removed and re-added while having unknown files in said directory (e.g. build artifacts), you would end up in a very bad place (this is a BZR bug, so we work-around it with a “bzr revert”).

The update has already appeared in the Jenkins update centre, so you should already be able to upgrade to it.

New libeatmydata release!

Posted on by
Reply

I updated the web site for libeatmydata (woah!): http://flamingspork.com/projects/libeatmydata/ and the launchpad page: https://launchpad.net/libeatmydata to reflect this too.

New exciting things in the land of libeatmydata:

  • sync_file_range is now wrapped (thanks to Phillip Susi)
  • I now bundle the eatmydata helper script originally included in the debian packages
  • the autotools foo builds on MacOS X
  • I modified the eatmydata helper script to also do the right DYLD environment variables if it’s running on Darwin. i.e. the eatmydata helper script now runs on MacOS X too (well, it should – please test)
  • libeatmydata should now work just about everywhere that can LD_PRELOAD. Patches welcome.

If anyone knows how to build a non-versioned shared libray using autotools… I’d love to hear it. libeatmydata is totally not something that needs soname versioning. I guess it’s harmless though.

New York Times Curried Cauliflower Soup

Posted on by
Reply

I made the Curried Cauliflower Soup I found on the New York Times website: http://www.nytimes.com/2011/12/13/health/nutrition/curried-cauliflower-soup-recipes-for-health.html

Rather delicious I have to say. In the future, I aim to try adding some peas, maybe some parsnip or even carrot.

Damn delicious though.

Chickpea and green bean curry

Posted on by
Reply

Ingredients

oil (olive, canola, whatever you like)
2 tsp mustard seeds
1 onion (chopped)
2 cloves garlic
1 dried chilli (crushed/chopped) – or more, depending on heat preference and heat of chilli
2 teaspoon chopped ginger
2 tsp ground coriander
2 tsp ground cumin
2 good handfulls of green beans, chopped into inch long pieces
1 can chickpeas (drained, washed)
1 can diced tomatoes

Method

Put oil in saucepan, add the mustard seeds
As the mustard seeds start to pop, add the chopped onion
Cook until onion is soft, then add all the spices
Stir and cook for ~30 seconds, until aromatic
Add can tomatoes, chickpeas and beans.
stir and cook until chickpeas are done and beans are a little softer, but still nice and crunchy (you can cook for longer if you like your green beans more well done, but I like a bit of crunch)

This took not much time, and was invented this evening purely because earlier in the day I had felt the need for green beans.

Nutrition

This serves about 2-3 people (depending on how much you eat). For 3ppl, it’s about 215 calories and 9g of protein per serve.

Not a good week for Telstra and privacy

Posted on by
Reply

The Office of the Australian Information Commissioner just posted this: http://www.oaic.gov.au/news/media_releases/media_release_120629_telstra_breaches_privacy_act.html

This isn’t to do with what I’ve posted about here the past few days, but to do with an incident back in December 2011. The details of  734,000 customers were available publicly on the Internet.

Details exposed include:

  • Name
  • phone numbers
  • Services held
  • free text field (where information such as username, password, email or other information could be recorded)

The ACMA report says that up to 41,000 customers had their user names and passwords exposed.

So… who had access? I quote from the ACMA report:

Between 3 June 2011 and 8 December 2011, the Visibility Tool received 108 access requests per day from unrecognised IP addresses (IP addresses that cannot be conclusively identified as Telstra IP addresses). On the day of the media publication, this number increased to 20,498 access requests.

The information was available from 29th March 2011 through 9th December 2011 with from a date in October it being easier to access (via a google search).

Unfortunately this is yet another case of internal procedures failing and being inadequate and only when the issue was raised publicly (in Whirlpool and the media) was it swiftly fixed.

It can be hard for a person inside a company to speak up, continue to speak up and be an asshole on these issues. It’s just human nature and after all, annoying your boss isn’t what everybody wants to do all day at work. I hope that the improvements that Telstra has committed to as a result of this investigation make it easier for people to raise such problems and ensure they are resolved.

Achieving things inside large companies can be incredibly hard. I have sometimes felt I’ve had more success trying to convince a dead seal to go for a walk than to get a large company to fix something that’s obviously broken (and everybody knows it). Undoubtedly there were people inside Telstra who knew about the problem yet felt powerless to force a fix to happen. This kind of culture is poisonous and tricky to avoid in a large organisation.

Both ACMA (Australian Communications) and OAIC have full reports:

If we are extrapolate out for the latest incident (NextG and Netsweeper) we could expect:

  • Telstra Incident report in ~2 months
  • If ACMA or OAIC take action, a report in ~6months

 

Telstra has a database of your NextG web activity

Posted on by
Reply

So, in what must be my biggest blog day ever, Telstra posted this: http://exchange.telstra.com.au/2012/06/28/further-update-telstra-smart-controls-cyber-safety-tool/

What is clear from their previous post and the pickup in the media (including ABC, Crikey and news.com.au) is that people care about this, a lot.

What is also clear is that they’ve had to go and talk to the Privacy Commissioner, the Australian Communication and Media Authority, the Telecommunications Industry Ombudsman and the Australian Communications Consumer Action Network.

I’d like to thank Senator Ludlam for raising this with Telstra government affairs which without a doubt helped raise the profile of this issue.

There are a couple of issues with Telstra’s updated statement:

  1. They admit to constructing a database with your full query string and IP address
  2. They don’t address the moral issue of being involved with a company so involved in curtailing human rights (Netsweeper).
  3. Just stripping out the query string doesn’t erase all personal information

I don’t think we can ignore any of these problems, and I hope we get good responses and resolutions to them.

The significance of point 1 should not be understated. This means that some people, somewhere, have access to a decent amount of your browsing history. There is no details on who has access to this (hint: law enforcement could probably request it). There is also no explanation about why this was applied to everyone.

Update: after rereading their blog post, at best I can say it’s ambiguous on if they stored this or not. One sentence implies that they do, another implies that they don’t. Clarification would be most welcome, and given the history so far, we should not assume the best.

Personally, I’m really disappointed in Telstra for at any point thinking it’s okay to finance human rights abuses. I’m also really disappointed in world governments for permitting the sale of such software to those who use it to oppress their people. We should be in the business of exporting freedom and democracy, not exporting tyranny and oppression.

If you have a NextG handset, I strongly suggest the following:

New Jenkins Bazaar plugin release! 1.18

Posted on by
Reply

From the desk of your new Bazaar plugin for Jenkins maintainer, I give you Version 1.18.

This release has two good bug fixes:

  • UI fix for checkout option (JENKINS-12261)
  • Auto-recover from corrupt BZR branches (e.g. bzr branch/checkout killed at inopportune moment) by cleaning the workspace and trying again (this is now default behaviour, best used with the Jenkins SCM retry count feature being > 1)

We’ve been running the same code as this release at Percona for about 2 months now (the second bugfix was one I wanted to test first before submitting upstream). This is the big fix that fixed all our problems with using bazaar with Jenkins in a large deployment.

The other news? I’m now maintainer, and this is my first release.

The page on the Jenkins wiki is here:

and updates should come through the standard Jenkins channels as all the auto-foo happens.

Hacking the Jenkins BZR plugin

Posted on by
Reply

For Drizzle and for all of the projects we work on at Percona we use the Bazaar revision control system (largely because it’s what we were using at MySQL and it’s what MySQL still uses). We also use Jenkins.

We have a lot of jobs in our Jenkins. A lot. We build upstream MySQL 5.1, 5.5 and 5.6, Percona Server 5.1, Percona Server 5.5, XtraBackup 1.6, 2.0 and 2.1. For each of these we also have the normal trunk builds as well as parameterised ones that allow a developer to test out a tree before they ask for it to be merged. We also have each of these products across seven operating systems and for each of those both x86 32bit and 64bit. If we weren’t already in the hundreds of jobs, we certainly are once you multiply out between release and debug and XtraBackup being across so many MySQL and Percona Server versions.

I honestly would not be surprised if we had the most jobs of any user of the Bazaar plugin to Jenkins, and we’re probably amongst the top few of all Jenkins installations.

So, in August last year we discovered a file descriptor leak in the Bazaar plugin. Basically, garbage collection doesn’t get kicked off when you run out of file descriptors. This prevented us from even starting back up Jenkins until I found and fixed the bug. Good times.

We later hit a bug that was triggered in the parallel loading of jobs during startup. We could get stuck in an infinite loop during Jenkins starting that would just eat CPU and get nowhere. Luckily Jenkins provides a workaround: specify “-Djenkins.model.Jenkins.parallelLoad=false” as an argument and it just does it single threaded. For us, this solves that problem.

We were also hitting another problem. If you kill bzr at just the wrong time, you can leave the repository in not an entirely happy state. An initial branch can be killed at a time where it’ll think it’s a repository rather than a checkout and there’s a bunch of other weirdness (including file system corruption if you happen to use bad VM software).

The way we were solving this was to sometimes go and “clean workspace” on the jobs that needed it (annoying with matrix builds). We’d switched to just doing “clean tree” for a bunch of builds. The problem with doing a clean tree was that “bzr branch” to check out the source code could take a very long time – especially for Percona Server which is a branch of MySQL and hence has hundreds of megabytes of history.

We couldn’t use bzr shared repositories as we kept hitting concurrency bugs when more than one jenkins job was trying to do a bzr operation at the same time (common when matrix builds kick off builds for release and debug for example).

So.. I fixed that in the Jenkins bazaar plugin too (which should be in an upcoming release) and we’ve been running it on our Jenkins instance for the past ~2 months.

Basically, if we fail to check out the Bazaar tree, we wipe it clean and try again (Jenkins has a “retry count” for source checkouts). This is a really awesome form of self healing. Even if the bazaar team fixed all the bugs, we’d still have to go and get that new version of bzr on all our build machines – including ancient systems such as CentOS 5. Not as much fun as bashing your head into a vice.

After all of that, I seem to now be the maintainer of the Bazaar plugin for Jenkins as Monty pointed out I was using it a lot more than him and kept finding and fixing bugs.

Soooo… say hello to the new Jenkins Bazaar plugin maintainer, me.

Yes, I maintain Java code now. Be afraid. Be very afraid.

Tor + Firefox + Twitter + (not rooted) Android = awesome

Posted on by
31

Update: As of October 2015, you should likely install the OrFox browser which is from The Tor Project and is a port of the Tor Browser to Android. Installing OrBot and OrFox makes browsing through Tor on an Android device easy. The rest of this blog entry is left in-tact for historical record, but as of now, look at OrFox rather than this process.

This is actually pretty simple to get going once you know how. This is a short “HOWTO use Tor on Android”

Basic problem: I want to use Tor on my phone. If you’re wondering why, perhaps my previous posts on Telstra and what they do to your traffic may be a good hint.

First of all, you’re going to want to install OrBot. It’s available from the Google Play store. There is absolutely no harm in leaving this running all the time in the background. I have found it to have zero impact on battery life of my phone (the Battery thing in settings doesn’t show OrBot at all).

With OrBot running, you now have a HTTP and SOCKS proxy available on your phone. This means you can set any app that can use a HTTP or SOCKS proxy to do their Internet access through Tor instead of directly through your Wifi or cellular network.

The Twitter client wonderfully has built in support for using a HTTP proxy. You just need to go into the Twitter app’s Settings, click “Enable HTTP Proxy”, and set “Proxy Host” to localhost and “Proxy Port” to 8118. You are now done. You can test this by disabling OrBot and then trying to refresh your Twitter stream. If it doesn’t work, then Twitter is trying to use the (not running) Tor proxy. Re-enable OrBot to be able to use your Twitter client. This “just works”.

There is pretty much no excuse not to have your phone Twitter client go through Tor. We all know that Twitter gets all sorts of legal queries for information about users. We also know that they’ve been fairly good about it, and indeed hats off to Twitter for being awesome. But… guess what? We can just ensure they don’t have any information worth handing over :)

Next step… Web Browsing. The Firefox Beta is pretty awesome. It’s fast and usable (which is exactly what you want in a web browser). This may also work with the standard Firefox browser (I’m not sure when they’ve updated it to be on par with the Firefox Beta version I’ve been using).

There is no place to specify proxy settings in the normal UI (I do hope Mozilla add this). But not to worry, Firefox on Android is built on the same base as Firefox on the desktop, so it does support it (there just isn’t a good UI).

What you need to do is go to the URL bar and go to “about:config”. This shows every little thing you can tweak in Firefox (a lot). Luckily, there’s a search bar. Search for “proxy” and modify the following settings to the following values (the = sign means “click modify and enter the value after the =”):

  • network.proxy.http = 127.0.0.1
  • network.proxy.http_port = 8118
  • network.proxy.socks = 127.0.0.1
  • network.proxy.socks_port = 9050
  • network.proxy.ssl = 127.0.0.1
  • network.proxy.ssl_port = 8118
  • network.proxy.type = 1
  • UPDATED: network.proxy.socks_remote_dns to “true” (click “toggle”)

Then head to http://check.torproject.org to check that it’s working!

This doesn’t provide you with all the features and benefits of using the TorButton in the desktop firefox, but it will stop your mobile phone provider spying on all the web sites you visit (unless they break into your phone itself).

Luckily, Android is fairly awesome and whenever you try to open a URL it can ask you what program you want to use to do that with. Guess what? Just select the Firefox you configured with Tor to open it and you’re browsing through Tor. Brilliant and easy with no need to go and “root your phone” or anything else that may turn people off from doing so.

Update: Thanks should also go to François Marier for his site that helped me get this right: http://feeding.cloud.geek.nz/2012/06/browsing-privacy-and-ad-blocking-on.html

Update: Added setting of socks_remote_dns

Telstra stops tracking, still supporting Netsweeper

Posted on by
1

http://www.zdnet.com.au/telstra-halts-customer-tracking-339340404.htm

The big news:

“We are stopping all collection of website addresses for the development of this new product,” Telstra said in a statement.

This does not change their association (and presumed financial support) of Netsweeper, helping make its technology affordable to its government customers who use it to suppress free speech and access to information.

See also:

Telstra funding censorship in Middle East

Posted on by
2

This post inspired by https://twitter.com/BernardKeane/status/217535549731389440

So, we know that Netsweeper is used by Telstra - http://www.zdnet.com.au/telstra-logs-customer-history-for-new-filter-339340337.htm

We know that Netsweeper is used in Qatar, the UAE and Yemen ( http://en.wikipedia.org/wiki/Internet_censorship – see also http://www.guelphmercury.com/news/local/article/577673–aiding-repression-or-just-doing-business ) and these states use it to suppress free speech and access to information.

The majority of countries that implement suppression of free speech on the internet could not afford the high cost of developing such software. The only thing that makes it possible is the subsidies from companies in the free world. With Telstra using Netsweeper, they directly contribute to the development costs of this software.

In years gone past free speech was suppressed by members of secret police and guns. Now you can do a lot of that with software. Software that is made affordable because the development costs are shared with companies such as Telstra.

See also my last two posts on the topic:

An update on Telstra’s surveillance of what you do online

Posted on by
5

http://www.scmagazine.com.au/News/306441,telstra-tracks-users-to-build-web-filter.aspx

I’d suggest going and reading: http://arstechnica.com/tech-policy/2009/09/your-secrets-live-online-in-databases-of-ruin/ to learn a bit about anonymization failures.

What we know:

  1. Telstra has the ability to monitor every URL you visit on a NextG connection
  2. Telstra is, in fact, monitoring every URL you visit through your NextG connection and piping that to some computer system that then takes action on it.
  3. None of this was disclosed to customers.
  4. Telstra is building a system for censorship.

What we don’t know:

  1. If this is a violation of any Australian privacy law (I’m not a lawyer)
  2. Who else has access to this “anonymised” data (hellooo US legal system)
  3. What universal surveillance infrastructure they have running

Update: this is a followup from yesterday’s post: http://www.flamingspork.com/blog/2012/06/25/on-telstra-tracking-nextg-http-requests/

On Telstra tracking NextG HTTP requests

Posted on by
15

http://lists.ausnog.net/pipermail/ausnog/2012-June/013833.html and http://www.scmagazine.com.au/News/305928,telstra-says-its-not-spying-on-users.aspx were recently published saying that Telstra NextG users were seeing some interesting things. (Yes, there’s a Whirlpool post too, but since they block requests from Tor I’m not going to link to them)

Basically, on their servers they were seeing HTTP requests to the same URL as they had just visited with their phone, but from an IP address that certainly wasn’t their phone.

I started to investigate.

I put up a simple HTML page on a standard HTTP server and then got a NextG device to query it. I saw a log that came from a TELSTRA owned block of IPs. I didn’t see any suspicious second request though. Sadness.

Turns out you have to request the URL twice to get this other request. It is after this second request that you get a query from a Rackspace/Slicehost IP (cloud provider, so it is unlikely Rackspace itself is involved any more than as a Cloud provider) with the same URL (although via HTTP/1.0 instead of 1.1). On a subsequent request, I didn’t see a corresponding one from this IP. Also, when accessing this URL from a different NextG device, I did not see a request from the Rackspace/Slicehost IP block.

If I change the content of the file and try to fetch again, it doesn’t download it anew. This suggests that there is not inspection of the content of what’s coming back from the HTTP server.

The User Agent pretends to be Firefox running on Windows. I have not yet found out anything specific about it.

What can we learn from this?

  1. If you think that putting a URL up and only telling 1 person about it is private you are very, very, very much mistaken
  2. Telstra is quite possibly spying on you, from servers in the USA, which is under a different set of laws than if it was done in Australia.
  3. Telstra is sending what websites you visit on your NextG connection to the USA. If you are at all involved in anything that may make the US government unhappy (e.g. disagreeing with it) this may have interesting implications. Further research is needed as to what exactly
  4. Telstra keeps a record of all URLs as otherwise it could not implement “on the second request”
  5. The iPhone needs Tor more than ever and it needs it on a system level.

Update: I have been pointed to http://v3.mike.tig.as/onionbrowser/ which is an Open Source Web Browser that uses Tor on iOS.

Update: http://www.flamingspork.com/blog/2012/06/26/an-update-on-telstras-surveillance-of-what-you-do-online/

Loveday’s Ginger Beer

Posted on by
Reply

In my search for the best alcoholic ginger beer I’ve been mostly disappointed. I’m pretty sure Bundaberg takes the cake for non-alcoholic ginger beer and is unlikely to ever be beaten in both quality and availability. I had Loveday’s Ginger beer a couple of weeks ago and it was okay, but I think the Matso‘s still takes the cake for alcoholic ginger beer.

image

My first Perry

Posted on by
Reply

I’m pretty sure this was the first time I’ve ever had Perry. I’ve had plenty of beer and cider over the years, but never Perry. I’d like to try more of them, I can’t really relate this to anything else, except to say that it’s nice ,not overwhelming and not too sweet. At 7.3% it packs a decent amount of alcohol content too.

This one is a light yellow colour and in front of the bottle you see the little plush dolphin that we got around the time Sun acquired MySQL. Typically, you’d see photos of it next to Salmiakki and not something as low alcohol content as this.

image