Tracking my Internet with a Raspberry Pi

I've been having Internet troubles. Why yes, I am in Australia! Funny you ask!

Update: I wrote a little script to do a regular HTTP GET and ran the tests again over a weekend. See Part 2 for more!

In summary, I've been getting really short drop-outs semi-regularly. Only a handful of times per day, and usually only for a few minutes at a time. Just enough to stop me in my tracks and force me onto a tethered mobile phone connection. I think Aussie Broadband (my provider) calls them "flaps", and according to their app, I've had 150 of them in the last 8 days.

The trouble is, I'm told NBN (Australia's National Broadband Network) doesn't see them as serious enough to trigger a response. An "outage" is something longer, and it's really only outages that matter.

So what can I do?

After a swathe of other troubleshooting attempts, one of the representatives at Aussie Broadband (and I want to be clear - I love Aussie) suggested I keep my own records. They can track outages, but there's only so much they can see from their end. They know NBN won't do anything without proof. This suggestion was echoed by a number of friends and colleagues.

My solution

Suggestions ranged from just keeping a ping running on a laptop, to setting up an end-to-end tracing solution using half a dozen Azure resources and client applications.

Instead, I implemented something in the middle - albeit a bit closer to the former.

I wanted something pretty simple, so I just set up a couple of cron jobs on a Raspberry Pi - using the Ookla Speedtest CLI, and a standard ping.

First, I needed the Speedtest CLI. On my Pi running Rasbian, I installed that fairly simply with easy_install:

sudo easy_install speedtest-cli

Once installed, if you run speedtest from your commandline, you get a nice result... but it's not very parsable if you want to keep track over a period of time.

Thankfully, the CLI has a couple of arguments that will help populate a CSV, and you can use standard shell features to append to a file.

First, I created a new CSV file with headers using the --csv-header argument and piping to a new file:

speedtest-cli --csv-header > speedtest-track.csv

Then, to test it out, I ran a speedtest with the --csv argument, appending to that file:

speedtest-cli --csv >> speedtest-track.csv

Now to set up a cron job to do it every 15 minutes. You can set this up with the crontab -e command. One sticking point I encountered (thanks Vaughan Knight for helping diagnose) - you'll need the full path to both the CLI and the output file in your cron job.

# run this every 15min
0,15,30,45 * * * * /usr/local/bin/speedtest --csv >> /home/pi/speedtest-track.csv

The result (after running it for a while) is a nicely parseable csv file!

A screenshot of a spreadsheet showing details from an Ookla speed test every 15min

I mentioned, however, that the dropouts were only for a few minutes at a time, so there's a good chance I'll miss some if I'm only testing every 15min. I could do a speed test more often (and I might if I'm not getting the details I expect), but I figured a ping every minute would be enough to at least see those additional drops.

Again, a straight ping will give me plenty of details, but they're not terribly parseable if appended to a file.

A bit of google-binging led me to this solution:

# large ping every minute
* * * * * ping -c 1 -s 65507 | head -n 2 | tail -n 1 | perl -nli 'print scalar(localtime), ": ", $_' >> /home/pi/pingtest.txt

In short, I'm running a single ping of maximum size against I'm pulling out the second line of the output (the important one), prepending the current time, and appending it to the pingtest.txt file. Simples!

This is the result, and you can already see some outages in the preview strip in the right-hand side in Visual Studio Code.

A screenshot of a text file showing the results of a ping every minute. The preview bar on the right shows gaps where no ping response was received.

The results

First the speed test. I let this run for just under 24 hours and charted the result:

Chart showing download speeds usually around 93Mbps but drops down to around 55Mbps about 17 times on a regular basis. Uploads are consistently at 38Mbps with a few drops to around 22Mbps and one to 0Mbps towards the end of the graph.

It's not horrible, but you can see it's very inconsistent. There's even a drop with no upload at all for half an hour or so.

What about the ping? This one has run for about 21 hours so far, and you can see the results below.

Chart showing a consistent low ping with only 4 exceptions, and two periods of Internet outage. One for two minutes, and one with a number of drops over 25 minutes, both in the first quarter of the graph.

You can see a couple of legitimate outages, one for only a couple of minutes, and one on and off for about 25 minutes.

What's interesting is that these traces are over approximately the same time period. If you overlay those graphs, the outages don't line up. At all. So sometimes the ping shows a failure, and sometimes the speedtest shows a failure.

If you add them up, that's at least 3 periods of serious instability over about 21 hours. Not insignificant.

My plan is to keep this running over the next couple of days, then present the results to Aussie Broadband with the hope that there'll be an escalation. Fingers crossed!

Damian Brady

I'm an Australian developer, speaker, and author specialising in DevOps, developer process, and software architecture. I love Octopus Deploy, Visual Studio Team Services, and reducing process waste.