Dan Poltawski's Blog

Rate Limiting on Three UK’s ‘Feel at Home’ Roaming Service


  • Three’s “Feel at home” service severely restricts the speed of roaming to much more than just streaming or file-sharing sites
  • Some services such as Google, Facebook, Wikipedia and WhatsApp respond at a reasonable speed and have a pleasant experience, where as others not in the whitelist are slow and deliver extremely poor experience
  • I’m curious to know the technical rationale/explanation/details of whats going on behind the scenes

Three’s roaming offering

As a frequent traveller and geeky internet type, connectivity when travelling is very useful to me, so when I joined Three UK’s network, their “Feel at home” inclusive roaming service was quite an attraction. Sadly, despite the promise of their branding, in many cases the experience does not match at all the “Feel at home” name. Webpages can load excruciatingly slow, feeling slower than GPRS rather than 20Mb/s+ 4G I get at home.

To be fair to Three, the small print does make a hint about not-like-home experience:

  • We restrict your ability to use your data allowance for file-sharing, like peer to peer download services, and using certain sites to download or share large files. This is so that we can manage demand from overseas and ensure that everyone can use the service.
  • You should be able to stream and connect to VPNs, but you’ll find it’s slower and the quality won’t be as good as it is in the UK. For example, we don’t recommend streaming TV programmes and films. If you’re connecting more than one device to Mobile Wi-Fi, your streaming experience will be even poorer.

While I’d love to truly roam like home, I recognise the price competitive nature of Three’s offering means that such restrictions are necessary and I would be content them…if it was just file sharing/streaming and affected…

Streaming restricted, or all webpages crippled?

My experience roaming with three is that even the most basic webpages are severely speed restricted. I’ve taken a number of trips toto feel at home destinations in the last year or so and I began to pay more attention to the performance and notice a discrepancy between sites; Google maps, search results, wikipedia and WhatsApp would all load promptly, whereas as other sites like the BBC, Washington Post and various tech blogs would take an age to load. The discrepancy between sites was most noticeable when I was catching up on my twitter feed — trying to quickly scroll through my tweets I became frustrated that every image thumbnail would take about 20 seconds to load, somewhat diminishing the fast glance nature of twitter. In contrast, my Facebook feed would load instantly.

fbcdn.net vs twimg.com

In September on a trip to Spain the twitter image problem was frustrating me and I started a query with Three support to see if they could improve it. At first I was told that this was out of the control of Three and the fault of their roaming partner (in this case Telefonica), but after some persistence the support team agreed to pass on my query to their second line technical support. At that time I was traveling with a laptop and a three data sim in addition to my usual mobile sim so I decided to do a bit of a technical experiment to back up my perceived experience of performance.

  • I uploaded the same image to both Facebook (fbcdn.net) and twitter (twimg.com)
  • I wrote a script to download the image multiple times from both sites to record the speeds
  • I compared the averages

The results were fairly conclusive, on average the image loaded 10x faster on Facebook than Twitter.

The support response came back:

After them looking into this they have advised that the reasoning behind this is that we have a direct cache link with Facebook, meaning that your speeds will be quicker with Facebook than Twitter.

About what I expected – I suggested if such a ‘cache link’ could be established with twimg.com it would give myself and other customers a better experience. To my pleasant surprise, this appears to have happened since then and i’ve notice twitter images load at an acceptable speed whilst roaming from Spain this month.

What is going on behind the scenes?

If I wanted to do something about this, writing this blog post isn’t going to achieve it — the solution is to switch away.1 But I’ve got enough networking knowledge from my previous job and geek chats with friends in the field that I’m intrigued to understand what is actually going on.

The message from three support mentioned a ‘direct cache link’ which in normal circumstances I’d speculate is some sort of peering arrangement, but that does not make sense to me, the performance differential is too significant, it’d also seem strange to not use the same infrastructure as the local network provider.

My theory is:

  1. Three severely limit the speed to all networks when roaming
  2. They whitelist typical sites which customers will use whilst roaming (e.g. Facebook, google maps, WhatsApp etc) so they perform acceptably (and keep bandwidth-heavy things like Youtube/Spotify etc restricted)
  3. This comes at the cost of any ‘unidentified’ web services, which generally perform poorly
  4. This is all about reducing overall bandwidth consumption (which presumably reduces their costs from their commercial arrangements with roaming partners)

AMP, Cloudflare sites and other sites in the whitelist

Some other things I’ve noticed in day to day web browsing:

  • If I search for something and there is a Accelerated Mobile Pages (AMP) result, it’s a massive speed boost. But I’m pretty sure that this is not the result of all the fancy optimisations AMP does, rather it’s simply they are being served from Google’s servers which appear to be in the whitelist
  • I’ve noticed that Cloudflare fronted sites (e.g. the Moodle tracker) appear to be in the whitelist and perform well. In fact I tested this theory by remotely downloading a podcast hosted at traffic.libsyn.com (which performs terribly) and placing the mp3 on one of my sites which is fronted by cloudflare. The file could download at 200KB/s through cloudflare, vs <1KB/s from libsyn.
  • It’s curious that three.co.uk seems not in the whitelist and BBC is also missing — perhaps to prevent iPlayer use? 2
  • It’s also curious to me that webpages are restricted so dramatically, a little bit of extra speed so that I could lookup a local attraction without waiting minutes for the webpage to load would make the service dramatically better.

If anyone has knowledge of the mobile industry, how data roaming works in practice or any insight into this, I’d love to hear from you — please leave a comment, send me an email or tweet me! It will also be interesting to see if this approach gets followed by other mobile networks when EU roaming rules enforce fee-free data roaming later this year.

  1. EE now offer inclusive EU-roaming on some plans and suggest they offer full speed access, which I am strongly considering switching to, or wait it out to see what offerings there are when EU roaming charges are abolished in June

  2. This traceroute to BBC.co.uk seems crazy

Testing Email Features With MailCatcher

Email is one of those areas of software which is a tricky to get a good test setup and thus rarely gets tested well. In Moodle we have quite a few flags which can be used for development but I’ve never quite felt happy with it. In a previous job we ended up setting up Exim to intercept all messages and write to a file which could be read from a browser. It was pretty complex, a pain to setup and you ended up just looking at the message source.

So I was delighted this week when I came across across MailCatcher via One Thing Well. As it says on their site: MailCatcher runs a super simple SMTP server which catches any message sent to it to display in a web interface.

The instructions are simple:

  1. gem install mailcatcher
  2. mailcatcher
  3. Go to http://localhost:1080/
  4. Send mail through smtp://localhost:1025

So in my Moodle config.php I set: $CFG->smtphosts = 'localhost:1025'; then start doing some email-generating operations and voilà:

Everything you need is availble in one place, source view, html view, plain text view – right in your browser. Great job Samuel Cochran!

Excellent Contributing Experience With MediaWiki

Moodle Docs is built upon MediaWiki and this week I’ve been working on upgrading Moodle Docs to the latest MediaWiki release (1.23.0).

I use open source software all the time, but Mediawiki has strong parallels to Moodle1 and it gave me an opportunity for reflection whilst I was doing the sort of tasks Moodle administrators do; reading release notes, testing customisations and discovering bugs. I became involved in Moodle coming from exactly this situation but it’s been quite a few years since I was far enough removed to consider the experience objectively.

I’ve long held the view that making the process clear and responses prompt & encouraging is the key to the successfully converting first-time contributors into long time contributors.2 Mediawiki have nailed this experience in the last few hours:

Great job MediaWiki! I saw the emails this morning (Saturday) and was excited to respond to your feedback.

  1. Being a PHP based web application (though clearly the site profiles and functionality is dramatically different to Moodle)

  2. It was exactly that experience which lead me to stick around in Moodle.

  3. I have used gerrit with Mahara before, so may have had a head start..

Notifying the End of Phpunit Executions on OSX

Whilst integrating issues into Moodle it’s a multiple times per day occurrence to run phpunit tests on the whole Moodle codebase. Each day we add more and more tests and so the time taken to run these tests increases gradually. I’m not one for sword fighting on wheely chairs, so whilst waiting for phpunit tests to run I’ll generally move onto another task, then frequently i’d forget completely that the phpunit run was going on in the background and not know what the phpunit output was showing me, need to run the tests again and fall into the same trap!

In order to resolve this situation I was inspired by Brett Terpstra’s post about using the ruby gem terminal-notifier to send messages to Mountain Lion’s notification center and decided to use this to remind me of phpunit executions.

After installing terminal-notifier and using Bret’s finished.rb script I added a bash function for phpunit:

function phpunit() {
    vendor/bin/phpunit $@
    if [[ $? == 0 ]]; then
        finished "phpunit OK"
        finished "phpunit FAILED"

In order that I don’t miss a phpunit run I’ve configured the notifications to be ‘Alert style notifications’ (configured in OS X Preferences > Notifications > terminal-notifier) which works great, the only remaining issue is there does not seem to be a way to dismiss alerts using keyboard shortcuts.