Clearwater and Juju

Posted on Sun 21 June 2015

I wrote a short piece a while ago on Juju, the software orchestration tool written by Canonical (the company behind Ubuntu Linux), and specifically on orchestrating Project Clearwater with Juju. You can find that at or

Concurrent-ruby and Minitest

Posted on Sun 07 December 2014

I spent some time earlier today debugging an interesting interaction in Ruby between minitest/autorun and concurrent-ruby, and since none of my searches on it turned up quite the right answer, figured I'd write up my notes.

I'm currently writing a Diameter stack in Ruby, partly because I need a good Diameter test library for testing/fixing FHoSS issues, but also so that I can think about good library/API design. One of the features I've just added is to have the message-sending function return a Concurrent::Promise object that gets fulfilled with the answer - striking a nice middle ground between callbacks (which are hellishly complicated if you have a long scenario where each message depends on what came before it) and just returning the result (which means lots of blocking and inefficiency). But when I came to UT it, this test failed with a deadlock:

Stack 2::A client DiameterStack with an established connection to 'bob'#test_0002_fulfils the promise when an answer is delivered:
fatal: No live threads left. Deadlock?
    /usr/share/ruby/thread.rb:72:in `sleep'
    /usr/share/ruby/thread.rb:72:in `block (2 levels) in wait'
    /usr/share/ruby/thread.rb:68:in `handle_interrupt'
    /usr/share/ruby/thread.rb:68:in `block in wait'
    /usr/share/ruby/thread.rb:66:in `handle_interrupt'
    /usr/share/ruby/thread.rb:66:in `wait'
    /home/rkd/.gem/ruby/gems/concurrent-ruby-0.7.1-x86_64-linux/lib/concurrent/atomic/condition.rb:43:in `wait'
    /home/rkd/.gem/ruby/gems/concurrent-ruby-0.7.1-x86_64-linux/lib/concurrent/atomic/event.rb:89:in `wait'
    /home/rkd/.gem/ruby/gems/concurrent-ruby-0.7.1-x86_64-linux/lib/concurrent/obligation.rb:55:in `wait'
    /home/rkd/rb_diameter/test/test_stack.rb:105:in `block (2 levels) in <top (required)c>'

Debugging this eventually led me to the conclusion that the code in my Promise wasn't getting run at all. I narrowed it down to this completely minimal testcase that failed:

require 'minitest/autorun'
require 'concurrent'

describe 'promises' do
  it 'works' do
    p = Concurrent::Promise.execute{ puts "Hi" }

and this one, identical but for the minitest infrastructure, that passed:

require 'concurrent'

p = Concurrent::Promise.execute{ puts "Hi" }

How bizarre. I could tell that there was some kind of interaction between concurrent-ruby and minitest, but I had no idea what and all my hypotheses seemed implausible (some code in concurrent-ruby that spots when you're in a test and spawns fewer background threads? some code in minitest that forces everything to happen on a single thread?). I couldn't find the immediate answer on the web (searching "minitest concurrent-ruby" or variations on that tended to come up with "how to make minitest run your tests concurrently"-style tips), but did eventually find the two bits of information I needed.

The first was, which says:

"But here is the question: How can minitest run our test if the test is defined after we require minitest? You probably already know the answer: it uses at_exit hook to trigger test running"

The other was the concurrent-ruby's API docs, particularly

Instance Attribute Summary

    - (Object) auto_terminate
    defines if executors should be auto-terminated in at_exit callback.

So that means:

  • minitest/autorun works by running an at_exit callback that runs all the tests you've defined
  • concurrent-ruby has an option that termminates all the background thread pools at exit (and it turns out that defaults to true)
  • ...and I'm requiring minitest/autorun before concurrent-ruby, so minitest/autorun's at_exit callback triggers last
  • by the time my tests run and try to execute a Promise, the worker threads that would run that Promise code have been destroyed and it silently fails

One simple line added to my minitest_helper.rb fixed that up:

# Compatibility with minitest/autorun
Concurrent.configuration.auto_terminate = false

Hopefully this is useful to anyone who hits similar issues in the future! I've also spotted (which wouldn't have fixed this, but would have meant I got an exception and thus a head start on debugging), and submitted a pull request to fix that.

New PPA for OpenIMSCore FHoSS

Posted on Sun 06 July 2014 • Tagged with ims, hss, packaging

A while back, I got in touch with the Fraunhofer FOKUS team behind OpenIMSCore, with some fixes I wanted to make to their HSS, and ended up being granted commit access to their repository. I've now made those fixes and wanted to distribute them more widely, so I've set up a PPA and made the Fraunhofer Home Subscriber Server (FHoSS) available as a Ubuntu 12.04 LTS package. (It should be trivial to build and install it for other Ubuntu systems, too, but 12.04 seemed like the best one to start with).

The PPA is at - sudo add-apt-repository ppa:rkd-u/fhoss && sudo apt-get update && sudo apt-get install openimscore-fhoss should install the package.

By default, it asks a series of configuration questions during the install, such as the MySQL database password. I'll be writing a post on debconf automation shortly, for anyone who wants unattended install of FHoSS servers with non-default settings.

I'm now subscribed to the openimscore-hssdev mailing list, so should be able to answer questions there or in the comments.

Four cool things from KamailioWorld 2014

Posted on Tue 15 April 2014

A week ago I got back from KamailioWorld in Berlin, where I was giving a talk on performance testing. I really enjoyed myself - it's a nicely-run conference, and it's useful to talk to the broader open-source community and find out about new trends (WebRTC, stronger cryptography) and tools.

I've decided to write up (partly for my own reference) four cool tools I learnt about at KamailioWorld and hope to get some use out of in the near future.

The Twinkle softphone

I'd been vaguely aware of the Twinkle softphone, but because it's Linux-only and I use Windows 7 at work, hadn't paid much attention to it - I mostly use cross-platform phones like Jitsi or Blink. However, I noticed the Twinkle configuration window during another presentation, and noticed that it has support for IMS-AKA authentication - this is something hardly any phones have, but which is really imnportant if you'redoing any IMS development or testing. I'll definitely check Twinkle out next time I'm working on AKA.

sipgrep v2

This is an updated rewrite of the old sipgrep tool, moving from Perl to C. I've never used the new or old version, but it has a lot of features that make checking SIP traffic easy compared to just tcpdump - such as coloured output and the ability to match on specific headers. (It may be that it's not much more useful than taking a capture file and opening it in Wireshark - but it's worth a look, and if nothing else it'll be quicker on remote systems.)


I was discussing Seagull, the Diameter test tool, and someone pointed me at MTS as an alternative. I'd never heard of it before - and it doesn't obviously come up in search results for a Diameter test tool - but it looks reasonably fully-featured and usable. I'll give it a try next time I need to reproduce a FHoSS bug - I had been building a test suite in JRuby that wrapped the jDiameter stack, but maybe MTS will be a good alternative.

Jitsi Videobridge

This is basically a free-software version of Google Hangouts (which I use a lot for team meetings) - video conferencing over WebRTC. The live demos were pretty impressive, and I'm hoping to set it up one evening and have a play with it for myself. (Unfortunately Firefox's WebRTC support isn't good enough for it yet, which is sad.)

There was a lot of other cool stuff on display - sip:provider ce, CGRateS and OpenEPC, just to name a few - but I've tried to limit this post to just the ones I expect to use myself in the near future. AS you can tell, I learnt a lot from KamailioWorld - here's to next year!

Notes from the December 2013 Clojure eXchange

Posted on Tue 07 January 2014 • Tagged with clojure

Last month, I went to the Clojure eXchange in London (run by SkillsMatter). I've been meaning since then to write up my notes from the most interesting talks, so here goes. You can see all these talks (and others I've not mentioned) at

Speech Acts (@gigasquid)

This was basically a talk about speech acts and John McCarthy's Elephant 2000 paper , with a Parrot AR drone thrown in for a bit of robotic overlord interest.

The idea is that you have three types of speech acts: - Assertions, which tell the listener that something is true - Persuasion, which creates beliefs, which draw inferences from assertions - Requests, which (if accepted) create a commitment and will be acted upon at a future time

In software terms: - assertions set program state (in contrast to traditional 'assertions' which make a program fail if it's not true) - persuasion sets conditional state - the program will believe something is true when something else is true - requests create deferred actions - the program will do something when something else is true

The example given was something like this:

(assert sunny false)
(convince #nice-day "It's a nice day" #(= sunny true)) ; #nice-day is now true when sunny is true
(request *open-window when #nice-day #(println "Opened the window")) ; this commits the program to execute this println function when #nice-day is true
(assert sunny true) ; this sets sunny to true, which causes #nice-day to be true, which causes..
; => "Opened the window"

As a concept for structuring programs, I think it's really interesting - I can see it making program logic a lot more explicit. A lot of programs can be conceived of taking some external facts (assertions in this context, but input more generally), deriving state from them, and matching that state against its programmed commitments to decide what to do in a given case - expressing the derived state as "beliefs" in a human-readable framework could be a real aid to debugging.

The Babar language is a Clojure-like language based around this concept of speech acts - though I think it's an interesting enough way of structuring programs that I'd use a Clojure/Python/Ruby library implementing these ideas. (That link also contains videos of babar-programmed drones, if you're interested.)

One final note - I like the idea of # and * as sigils to denote different types of variable. Perl was my first language, and the idea of encoding that sort of information in variable names has always stuck with me.

Scala vs. Clojure (@dpp)

I have to admit that I'm not that interested in Scala - when I want a strongly typed FP language I'm going to learn Haskell - but this was a thought-provoking talk just because it made me think about what makes programming languages (and, by extension, other developer-oriented tools) good or bad.

One idea I liked was about being opinionated - that languages and frameworks become successful by making people follow successful patterns. Ruby on Rails is a good example here (because it forced people to write web apps in an object-oriented MVC style), and Clojure is more opinionated than Scala (in enforcing immutability and making the functional style much more natural than the object-oriented one). To some extent there's a risk there for designers - a language which isn't opinionated is more likely to be successful than one that's opinionated about the wrong things - but it feels like that's a typical tradeoff of risk versus reward.

The other thing to come out of this was a play on Greenspun's Tenth Law - "any sufficiently complicated Lisp program contains an ad hoc, informally-specified, bug-ridden, slow type system". In the light of libraries like Prismatic/schema (which I use and have contributed to) and core.typed (which I haven't), it's a pretty plausible rule.

Conversely, the fact that the Clojure community are gathering around a couple of core ways to do this (rather than just an ad-hoc implementation in every program) is promising - like how having specific libraries like GNU Guile to implement an embedded Lisp works around the original rule.

Concurrency (@thattommyhall)

I found this talk really useful because it talked about the core.reducers library, which I'd heard about but not really come to grips with.

The main discussion was of the fold function, which is essentially a parallel reduce - it partitions the collection into 512 pieces, reduces each piece, and then combines the reductions with a combiner function. The combiner function has two rules: - it must be associative (in the mathematical sense) - calling it on its own must provide an identity

So + and * are both combiner functions:

user=> (+)
user=> (*)

There were two tips given for designing combiner functions. Firstly, you need to check that your combiner is associative. This kind of property-based testing is a good fit for generative testing (QuickCheck-style testing), and so clojure.test.generative is a useful sanity-check here.

The other is that reducers has a monoid function, which simplifies the requirement that your combiner function must provide the identity when called without arguments. monoid takes a function and an identity constructor, and returns a combiner, which returns the given identity when called with no arguments, and the result of the given function otherwise. So if the + function didn't already return 0 when called with no arguments, you'd just need to call (monoid + 0).

The programming language as a musical instrument (@samaaron)

I found this talk interesting because it asked the question "what if programming was not considered a sub-discipline of engineering"? I'm finding that question particularly interesting at the moment because I'm reading Recoding Gender, about the early computer movement, women's participation in it, and how the different conceptualisations of programming affected that - so I'm going to talk about this train of thought more once I finish the book.