I needed a 32bit hash generation function and there appeared to be no such obvious hash in ruby. With a bit of help I found murmurhash 2.0 which appeared to fit the job. Below is the code for a pure ruby version of the endian-neutral version:

module Digest
  def self.murmur_hash2( string, seed )
    # seed _must_ be an integer, but I do try to enforce that.
    # m and r are mixing constants generated offline.
    # They are not really magic, they just happen to work well.

    raise "seed isn't an integer, and I can't convert it either." unless 
      seed.is_a?( Integer ) or seed.respond_to?( 'to_i' )

    seed = seed.to_i unless seed.is_a?( Integer )

    m = 0x5bd1e995
    r = 24
    len = string.length

    h = ( seed ^ len )

    while len >= 4
      string.scan( /..../ ) do |data|
        k = data[0]
        k |= data[1] << 8
        k |= data[2] << 16
        k |= data[3] << 24

        k = ( k * m ) % 0x100000000
        k ^= k >> r
        k = ( k * m ) % 0x100000000

        h = ( h * m ) % 0x100000000
        h ^= k

        len -= 4
      end
    end

    if len == 3 then
      h ^= string[-1] << 16
      h ^= string[-2] << 8
      h ^= string[-3]
    end
    if len == 2 then
      h ^= string[-1] << 8
      h ^= string[-2]
    end
    if len == 1 then
      h ^= string[-1]
    end

    h = ( h * m ) % 0x100000000
    h ^= h >> 13
    h = ( h * m ) % 0x100000000
    h ^= h >> 15

    return h
  end
end

To use it copy the above into a separate .rb file (say murmurhash2.rb) and:

require 'murmerhash2.rb'

string = "the string to be hashed"
seed = an integer
result = Digest::murmur_hash2( string, seed )

I note that MurmurHash 3.0 is currently in beta so I shall have a go at coding that up once it becomes stable. I also would quite like to get this into a gem but before I do if anyone has any comments on the above code please let me know!

I’m currently writing some code that fetches a file from an sftp server. Using ruby 1.8.7 / net-sftp 2.0.4 the following code triggers an exception:

Net::SFTP.start( sftpHost, sftpUser, :password => sftpPass ) do |sftp|
  $stderr.puts "Downloading: #{sftpPath}#{sftpFile}"
  xml = sftp.download!( "#{sftpPath}#{sftpFile}" )
end

The exception is:

/usr/lib64/ruby/gems/1.8/gems/net-sftp-2.0.4/lib/net/sftp/session.rb:123:in `download!': uninitialized constant Net::SFTP::Session::StringIO (NameError)

The solution is to add:

require 'stringio'

to the top of the session.rb file mentioned in the exception message.

Update: 2010-08-19 16:00 I’ve just heard back from the developer having emailed him this blog entry. He is hopefully pushing a fix this evening. Now that’s service!

Update: 2010-08-20 Last night version 2.0.5 was released which has the fix in it.

I have a query against an Oracle database that returns a decimal value for a price. Ruby’s Oracle connector returns that as a BigDecimal object.

Once I have got all the items I have to aggregate them. As part of the aggregation I run this code:

@hash[id].price = ( ( @hash[id].price * @hash[id].fill_volume ) +
                    ( item.price * item.fill_volume ) ) /
                  ( @hash[id].fill_volume + item.fill_volume )

Each time it runs this code it gets slower. After about 250 loops of this code it gets really slow. Here’s the time it takes to do the 269th loop (the times are UNIX time with microseconds):

269
1281710074.568992
1281710075.574573
270

…just over a second! This seems to happen more if the id is the same over many loops.

A very simple solution is to make the price be a Float rather than a BigDecimal. When I do that, I get the following times:

269
1281710378.886902
1281710378.886920
270

Significantly faster (and given I have 10000s of lines to parse this is a big thing) worth the potential loss of accuracy. What I’d like to know is why BigDecimal behaves like this (at least in ruby 1.8.6). Any ideas?

UPDATE 20101221: Matt Patterson has had a look into it and it looks like BigDecimal is O(n²) (or worse) as the size of the BigDecimal gets larger. It looks like I’m going to have to have a look at BigDecimal#limit and BigDecimal#round.

Many others have written and created more than I can right now so here’s a collection of the posts that say what I want to say but better:

• Skeptobot: The Digital Economy Bill passed: The internet watched live as a handful of MPs ignored democracy in their attempts to control that which they don’t understand.
• Adam Liversage, Head of the BPI caught bang to rights abusing copyright
• Steve Lawson: My letter to the Musicians Union About the Digital Economy Bill
• Fiona MacTaggart MP: a barnstorming speech
• They Work for the BPI
• A comic sharing my reaction on twitter
• Lets thank the 32 MPs who voted against the bill
• William Heath’s first thoughts on the passing of the bill
• Reading Type: Apathy and ignorance from dying UK Parliament
• BPI Boycott: List of Shame. They also have a list of record companies to boycott and those that are safe if you are that way inclined.
• panGloss: Last night: the DEB passes in the Commons
• The Telegraph / Emma Barnett: Digital Economy Bill: ‘a bad day for democracy’
• The Metro: Digital Economy Bill passes as critics warn of ‘catastrophic disaster’
• Left Foot Forward: Digital Economy Act puts UK on endangered list
• TechCrunch: Doublethink – The Digital Economy Bill against the digital economy
• The Next Web: UK’s Digital Economy Bill approved. Democracy fails, social media wins
• Future Identity: Putting DE Bill in Perspective
• out-law.com: Digital Economy Bill passed by House of Commons
• GeekLawyer: The Digital Enterprise Bill: the next Dangerous Dogs Act
• The Times / Struan Robertson: The legislative farce of the Digital Economy Act
• Comment Is Free / Felix Cohen: Copyright law must be simplified
• Guardian / John Naughton: Shame Peter Mandelson didn’t download some common sense
• Cameron Neylon: A letter to my MP
• Guardian / Dan Sabbagh: Digital Economy Act likely to increase households targeted for piracy
• ComputerworldUK / Glyn Moody: Digital Economy Act: Built on Sand
• Guardian / Cory Doctorow: Digital Economy Act: This means war

I’m sure there will be more as the day goes on.

Finally, comments from me: suffice to say, I am very disappointed with the result and disgusted with Parliament. In particular I’d like to single out Ed Vaisey and his colleague Adam Afriyie for using the debate to score political points and not actually debate the bill. It was an example of the ‘change’ and ‘hope’ the Conservative Party offers. It was repellent. The bill was opposed from all sides of the house and then forced through by a three line whip and a compliant front bench from both the major parties. The stench that has dogged this last parliament just got more foul and does not bode well for the next. We need to seriously examine how our parliament works and how we can improve it. Whipping should be one of the first things we look at.

Huge thanks to all those such as Tom Watson who risked their future by breaking the whip and voted against the bill and actually represented their constituents rather than the interests of large lobbying organisations. It’s hard for MPs to break a three line whip in particular so they should be praised through the roof. Thanks should also go to all of the Liberal Democrats who voted against for whatever reason. The way the ‘game’ is played meant that it looked like they wouldn’t at some points so it was a big relief to see it happen. They have all given us a base that means that…

…the fight continues.

It’s been quiet around here I know, but here for the record is the contents of the letter than I sent to my MP, Jim Fitzpatrick today:

Dear Jim,

I apologise up front if this email comes over as sounding frustrated but last night spoke for itself.

The Digital Economy Bill Debate last night was, frankly, embarrassing for Parliament. This is an issue of confidence in our democratic process for a lot of people. Some stats from http://debillitated.heroku.com/

Out of 646 MPs only 40 turned up and only 10 stayed for the whole debate.

In contrast there were 5251 individuals discussing it on twitter, with 16180 tweets published. At peak it was one of the top 10 topics globally and has 20x more tweets/minute than any tweets to do with the UK election. People do care about this and to see the scorn that parliament is treating this issue is damning.

Some quotes from the debate last night to save you looking it up in Hansard:

Austin Mitchell said:

“What would a delay of another three months or so in order to discuss the Bill properly and give it full consideration do? There would be no danger in taking such an approach. A delay of three months would not produce the collapse of the creative industries, which has been held up as the threat hanging over us. This needs to be settled and discussed by the Commons through its full procedures; this should not be a rushed Bill, carried on the basis of the half attention of a discredited, dying and distracted Parliament.”

John Redwood said:

“It is a disgrace that the House is not treated with courtesy by the Government. It is quite wrong that a Bill of such importance and magnitude was not tabled earlier. It is quite wrong that there is an attempt to rush through all parts of the Bill without proper scrutiny and debate.”

John Grogan observed:

“I looked at all the Bills that received a Second Reading after a Prime Minister had gone to the palace-I went back as far as 1987… I managed to find the Diplomatic and Consular Premises Bill in 1987, the Architects Bill in 1997, the Lieutenancies Bill in 1997 and the International Organisations Bill in 2005. What they all shared in common was that there was no dispute between hon. Members in respect of them. It really is shameful that we are proceeding in this way.”

Tom Watson said:

“It is a very great pity that the Bill cannot be tested in Committee in the light of some of those questions. As a twice ex-Whip, I am rather embarrassed by the fact that the Bill is being railroaded through in the wash-up. Frankly, there has been a squalid deal between the three Front Benches, and they should be ashamed of themselves. The people who care about this Bill-and there are many out there-will see that for what it is.”

Please, take the time to vote against this horrendously badly draughted bill, or at least push to have it delayed for full scrutiny in the next Parliament. If you feel you can’t do that then please talk to the whips about it to explain that there is a significant groundswell of opinion about this and that it very much is a voting issue.

What happened to the Court of Public Opinion that Harriet Harman relied on so much over Fred Goodwin? Doesn’t it matter now? Does the lobbying and discredited statistics of the BPI matter more?

I have also signed the open letter here which you may find a worthwhile read: http://nevali.net/post/501647501/an-open-letter-to-sion-simon-pete-wishart-david

I will be publishing this letter on my blog at http://toblog.bryans.org/.

Many thanks once again for taking the time to read this, I do appreciate it.

Best wishes,

Toby

UPDATE 2010-04-07 11:20: Jim Fitzpatrick’s reply:

Like many MPs Toby, I was watching much of the debate in my office here at Westminster until the close. The debate resumes today and I will update you with the outcome. It will be interesting to see what changes are made today but there is a lot of consensus that much of the Bill is needed as I read in Hansard from some of the speeches I missed. I’ll let you know the outcome,

Best Wishes, Jim

UPDATE 2010-04-07 11:40: My reply to Jim Fitzpatrick:

Jim,

Thank you for your reply as always.

Unfortunately the statistics that the pro consensus are based on are incorrect or based on a flawed basis:

The entertainment industry as a whole has seen record revenues. The period from January until October 2009 saw record singles sales within the UK, according to the British Phonographic Industry. So much so that the number of singles sold in that period was nearly three times the number sold for the whole of 2002. The statistics presented to Parliament are a simplistic measure of losses, based upon estimates of file-sharing supplied by the British Phonographic Industry (themselves extrapolated from a survey of a small group of people, and was widely debunked in June 2009). Indeed, the sectors of industry which the referenced report claims could suffer three hundred thousand job losses in the UK employs only marginally more than that to begin with. Is it really the case that legitimate sales will cease altogether, despite rising steadily for the past ten years?

I should also note that the consensus on the pro side consisted of the same number of the consensus against in the debate.

There are an awful lot of creative people who do not want this bill. It is only the large companies that have not moved quickly with technology that want it. As always increased regulation will only serve to shore up the market incumbants and make it more difficult for independents and new startups to enter the industry. I consider this a bad thing. Do you? This will also apply to the ISPs that the burden of cost will fall on for the (technological unfeasable) measures the bill forces on them.

Please, at the least, ask your whips to drop clause 11 to 18 which are by far the most damaging clauses in this bill.

Many thanks again for reading,

Best wishes,

Toby

UPDATE 2010-04-07 14:28: Jim Fitzpatrick’s reply:

Thanks Toby, I’ll let you know what happens,
Jim

UPDATE 2010-04-08 12:16: Jim Fitzpatrick was one of the Ayes.

Since the blog has been very quiet of late, here is the transcript of the twitterview I have just finished where I discussed social media, technology trends in the trading world and football with the CME…

CMEGroup: @tobybryans Good afternoon and thank you for joining us. Let’s start with you describing your role at Schneider Trading #exchtalk

tobybryans: @cmegroup It’s my pleasure! I have two roles: Technical Exchange Liaison and Strategic Technologist. #exchtalk

tobybryans: @cmegroup I make sure comms between Exchanges and ISVs & our technical departments run smooth & both sides are always up to date #exchtalk

tobybryans: @cmegroup I also keep an eye on current technology trends to see if there is anything of interest to our business #exchtalk

CMEGroup: @tobybryans speaking of staying up to date – how do technology trends affect your business? #exchtalk

tobybryans: @cmegroup Technology immensely important for our edge. We’re always looking for better and faster solutions for all our systems #exchtalk

tobybryans: @cmegroup We talk to exchanges too about how they can help improve their solutions for us (just ask Mark!) #exchtalk

CMEGroup: @tobybryans We won’t ask which one you like working with the most ;) #exchtalk

tobybryans: @cmegroup I couldn’t possibly comment on that ;-) #exchtalk

CMEGroup: @tobybryans You are a busy man for sure. So when did you join Twitter and why? #exchtalk

tobybryans: @cmegroup I was at a BBC Backstage party when I first saw it in 2006 and I had no idea what it was but it intrigued me so I joined #exchtalk

tobybryans: @cmegroup Obviously #twitter has exploded recently and has become a far more useful tool than just keeping in touch with friends! #exchtalk

CMEGroup: @tobybryans So how do you use Twitter for work/Schneider? #exchtalk

tobybryans: @cmegroup Keeping up to date with tech, following the latest news, keeping in touch with and building new relationships. #exchtalk

CMEGroup: @tobybryans What do you think traders can get out of #Twitter? #exchtalk

tobybryans: @cmegroup Up to date and immediate discussion of latest news and market events from around the world often before the main stream #exchtalk

CMEGroup: @tobybryans You mentioned technology trends – are there any you want to say that Schneider is looking at? #exchtalk

tobybryans: @cmegroup two big areas we’re looking at are virtualization and latency measurement with #correlix and #corvil #exchtalk

CMEGroup: @tobybryans Two big trends for sure #exchtalk

tobybryans: @cmegroup absolutely, we’ve always been very good on latency but it’s got to the point where the old tools aren’t good enough #exchtalk

CMEGroup: @tobybryans London is one of the world’s soccer/football capitals – care to admit who you support? #exchtalk

tobybryans: @cmegroup HA! I was born in Wembley and so I saw close up how some football fans behave. Put me off for life… #exchtalk

CMEGroup: @tobybryans Last question: do you want to make a prediction for how traders will use Social Media a year from now? #exchtalk

tobybryans: @cmegroup It will bring new diversity to how they currently communicate and for brokers expanding and messaging their client base. #exchtalk

tobybryans: @cmegroup Given how pervasive access is becoming I think it’s going to be interesting for the regulators! #exchtalk

CMEGroup: @tobybryans Great point re: regulators and community growth. We shall see #exchtalk

CMEGroup: @tobybryans Well you are a busy man & sure you have other exchanges to tend to. We appreciate your time and thank you again #exchtalk

tobybryans: @cmegroup Thank you! It’s been fun to be your second vict… er… interviewee! #exchtalk

What is it with the wireless on Apple laptops and Netgear wireless routers? I have been aware of wireless problems (time outs on connection, random disconnects) with the DG834PN in the past but i fixed that by upgrading to the DG834N.

Recently, though, this now no longer works for both my clients and myself. There have been no changes on the router side but suddenly MacBooks have been unable to connect. I can only surmise that there has been an update to the wireless drivers on the Apple side that has caused this problem to surface on this router.

I have solved it by installing the latest DGTeam firmware (version 850), setting both the region and channel to AUTO (you also really need to do a complete router reset after installing this firmware). This is not particularly satisfactory though and Apple really needs to sort out the incompatibility with the entire family of Netgear routers. Anyone else experienced this and have a better way of fixing it?

I’ve recently had to upgrade from XenCenter 4 to XenCenter 5 and after the upgrade the console tab was not working: it was just a blank grey window.

After a bit of googling around archived forum posts I found out that the solution is:

1. Uninstall the upgraded XenCenter
2. Rename (for backup purposes) your home directory_\Application Data\_anything to do with Citrix or Xen
3. Reinstall XenCenter

I’ve bought the blogroll back since it disappeared during the upgrade. It’s now generated from my google reader opml feed which I export and run though some ruby which means it’s going to be up-to-date more frequently.

Here is the code that takes the opml and outputs the html fragment:

#!/usr/bin/ruby

require 'rexml/document'
require 'pp.rb'

class FeedData
  attr_accessor :name, :url, :category

  def initialize( name, url, category )
    ( @name, @url, @category ) = name, url, category
  end
end

class FeedDataList
  def initialize()
    @array = Array::new
  end

  def add( name, url, category )
    tmpFeed = FeedData::new( name, url, category )
    @array.push( tmpFeed )
  end

  def eachCategory
    catArray = Array::new
    @array.each do |feed|
      if !catArray.member?( feed.category ) then
        catArray.push( feed.category )
      end
    end

    catArray.sort.each do |result|
      yield result
    end
  end

  def eachItemInCategory( category )
    @array.each do |feed|
      if feed.category == category then
        yield feed
      end
    end
  end
end

def parse_opml( opml_node, feeds, parents_names=[] )
  opml_node.elements.each('outline') do |element|
    if element.elements.size != 0 then
      feeds = parse_opml( element, feeds, parents_names + [ element.attributes[ 'text' ] ] )
    end
    if element.attributes['xmlUrl'] then
      feeds.add( element.attributes['title'], element.attributes['htmlUrl'], parents_names.last )
    end
  end

  return feeds
end

opml = REXML::Document.new( STDIN )
feeds = FeedDataList::new
feeds = parse_opml( opml.elements['opml/body'], feeds )

feeds.eachCategory do |category|
  print "<br /><strong>#{category}</strong> "
  feeds.eachItemInCategory( category ) do |item|
    print "<a href=\"#{item.url}\">#{item.name}</a> "
  end
  print "\n"
end

To make it work:

$ ./opml2html.rb < <opml file>

I got the inspiration for the recursive opml parsing code from the Dekstop blog, so thanks are due to them!

Well, that upgrade seems to have been completed OK with a couple of issues: I had a couple of glitches one of which was to do with the rails backup of my database – fixed by turning off the backup stage in the rails installer. There was also an issue with my tags: I’ve known that my tags caused problems in the past with my use of the full stop. In the new typo it causes major problems so I wrote a quick script to replace full stops with underscores. Hopefully all is well now!