Blog of Random Thoughts and Pictures

Bridging MQTT brokers and using security certs from Let’s Encrypt

October 14th, 2019

This is an item that came up while working on a project within the TSSG and so might be worth sharing.

Have you ever tried to use a MQTT broker ? Message Queuing Telemetry Transport (MQTT) is a machine-to-machine (M2M), Internet of Things data protocol, which is in line with other data protocols such as XMPP, CoAP, AMQP, and Websockets. Invented in 1999, MQTT is now an OASIS (Organization for the Advancement of Structured Information Standards) standard, and ISO standard (ISO/IEC PRF 20922).

MQTT is extensively used in Amazon Web Services, Microsoft Azure IoT Hub, IBM WebSphere MQ, and is a publish/subscribe message exchange pattern, that can support persistent message storage on the broker and supports security in the form of authentication using user name and password, and encryption using SSL/TLS.

For something like the Eclipse Mosquitto broker, MQTT it has a really small code footprint, the libmosquitto (client library) is about 1.3 MB and is ideal if processor or memory resources are limited and also ideal if bandwidth is low or network is unreliable. Classic problems in the IoT space.

In the case of using MQTT for the smart grid, scale and security are top priorities. To achieve scale I’ve looked at bridging MQTT brokers in a hub and spoke model, where a very light MQTT broker is at the edge of the network (at the end of the spoke) and there’s a large MQTT broker at the hub which can aggregate all the data.

However the purpose of this post is to highlight the security aspects within MQTT and in particular the application of encryption (SSL/TLS) when using Let’s Encrypt certificates. Applying a certificate to an MQTT broker is not too hard, there’s a nice guide here on Mosquitto SSL Configuration for MQTT TLS Security and here too on SSL/TLS Client Certs to Secure MQTT however in the vast majority of cases the examples use self-signed certs and not certs as provided by Let’s Encrypt.

By the way if you don’t know Let’s Encrypt is a non-profit certificate authority run by the Internet Security Research Group that provides X.509 certificates for Transport Layer Security encryption at no charge. The certificate is valid for 90 days and the renewal process is quite simple.

Now bridging two MQTT brokers can be relatively straight forward too however getting the certs right when you want that bridge to be encrypted can be a little tricky look at how much you have to do to bridge a Mosquitto MQTT Broker to AWS IoT.

In my case I wanted to bridge two Mosquitto MQTT Brokers, each with encryption enabled by a Let’s Encrypt cert. Firstly I created a [special Docker container]() that could pick up the Let’s Encrypt cert, and having followed all the guides I kept getting the following error in the logs

OpenSSL Error: error:14037418:SSL routines:ACCEPT_SR_KEY_EXCH:tlsv1 alert unknown ca

OpenSSL Error: error:140370E5:SSL routines:ACCEPT_SR_KEY_EXCH:ssl handshake failure

Socket error on client , disconnecting.

I tried verifying the certs, by installing openssl

openssl verify cert.pem

But all was fine.

I thought I had to download the trusted root CA certificates for Let’s Encrypt and place it somewhere in the Alpine linux system (the base OS of the broker), but I must admit this “somewhere” was not so clear me.

The problem is that the MQTT broker does not know how to verify its own CA before starting the ssl exchange with any client. This is because the CA signing the Let’s Encrypt cert is not yet distributed and bundled by default in to the Alpine Linux system and therefore has to be added manually.

In the Mosquitto MQTT broker configuration, instead of just pointing directly at the chain.pem file I decided to point at the default place where all ca certs should be.

#cafile /mosquitto/config/certs/chain.pem
capath /etc/ssl/certs

And this write up on installing certificates in an Alpine Image to establish Secured Communication (SSL/TLS) really got to the heart of the matter, the cert needs to be copied to a special directory /usr/local/share/ca-certificates/and then you need to run the program update-ca-certificates so it gets placed in the right way into the folder /etc/ssl/certs.

After much head scratching, it all comes down to 2 command lines

cp /mosquitto/config/certs/chain.pem /usr/local/share/ca-certificates/chain.pem

update-ca-certificates

Once done (via a docker-entrypoint.sh command) the container is able to handle the CA issue, and bridging 2 Mosquitto MQTT brokers that are using Let’s Encrypt certificates can be achieved.

Splash World 10KM

April 12th, 2015

I did this race a few years ago and thought hey why not try it again and I’m glad I did, a new course which was super flat and nice to run. Grabbed a time of 42.12, which has me in a good mood! Also nice event afterwards for the kids to enjoy and play around Tramore.

Run Mount Juliet 10km

February 14th, 2015

What is it with me and hilly races. First of the year and I have to say this Mount Juliet race is a great one to start off with. There’s a steady incline for the first 1.5 kms then a down hill section and then the WALL …….. the hill that just seems to go on for ever.

Once I got over that the race was normal enough and grabbed a time of 43.17, which is another few seconds scrapped off from last year (2014 was 43.49). 

Rathgormack 5 Mile Race

July 11th, 2014

Okay I was warned that this would be hilly …. but boy was that race hilly. Enjoyable though and in a time of 33:54 I’m pretty happy with that.

Deadmans 5 Mile

June 6th, 2014

Carrick, Deadman …. hills ahhh I was just not feeling right for this one, dare I say had a slight injury …. excuses anyway in a time of 33:24, which should have been better.

Tom Jordan 5 Miles

May 9th, 2014

The runs are coming thick and fast these days and up today was the Portlaw 5 mile run. I must admit I’m not too sure about these shorter than 10km runs but then again I should be running for pace so 5 miles it is.

This year the Portlaw run was on a new course and we had to walk a mile outside of the village to get to the top of a hill for the start line. It felt like I was being wound up, getting up the hill just to let it all out going back down again and sure enough once the horn went for the start I was off down that hill in a shot just like everyone else.

I controlled the enthusiasm though I’ve had these quick starts before and it never works out in the long run. Down through Portlaw was fun and really only mile 3 caught me out a little.

Hitting the 4 mile marker was funny as someone called for the time I yelled 26 mins and it reminded me that I was on track for my goal time so I went for it hell for leather in the last mile. That was fun as I left 4 people behind although they nearly all caught me on the line was maybe I went too early again!

All was well at the end and I grabbed a time of 32:42 which was 1:16 quicker than I did l last year. Now I was super happy with that result.

Bluewall Waterford to Tramore 7.5 mile

May 3rd, 2014

What a challenge, a run from Waterford to Tramore I just couldn’t pass off this opportunity to participate in such a long standing famous run in Waterford.

This was a wet and windy day and certainty made the warm up session an important part of the race preparation. The Bolton Street car park was the perfect spot for such a warm up and I wasn’t the only one thinking along those lines as the car park was packed of runners in their pre-race routine.

In fact I may overdone my time in the warm up area as I made my way to the start line on The Mall I was way back amongst the large crowd and it was tough going under starters orders as it was hard to gain a full race stride at the start.

2 miles into the run and the pace had settled down and I was in a good group of runners some of whom I recognised from previous runs so I knew I was on track.

What surprised me most about the run out to Tramore was the up hill nature of the road I felt as though I was running up hills the whole time (which I hate). I pushed hard at the 5 mile mark but on reflection it was too early because the crew I was with caught and passed me too easily at the 6.5 mile mark and it wasn’t until I got into the last 0.5 mile that some streng came back for the finish line.

In fact I was finishing even stronger when I heard Jonathan Brazil give a huge roar of encouragement near the end of the run.

In the last stretch and hurting

I had set a goal of 7 minute miles and I caught a time of 50:53 for the 7.5 miles which was 1 minute & 37 secs inside my goal time of 52:30 mins. I was hugely proud of achieving that goal.

Wexford 10km

April 26th, 2014

At last really feel back on track and this run in Wexford town has come at a great time in my schedule. It was an early start to get to Wexford but it was nice to be able to relax once on location and to grab a fantastic parking spot close to the finish line.

There was a great buzz about the Talbot Hotel which is near the starting area and I was a little surprised at the number of familiar faces at the start line.

I managed to grab a nice position at the start avoiding the big crowds of people and once off I got into a steady rhythm very early into the race and really just took it from there.

Taking a stroll in Wexford town

In fact the majority of the race was quite lonely I had carved out a bubble between those ahead and those behind and it wasn’t until the last 2km that a chap really pushed me, which was great really.

There was plenty of encouragement from the crowd in the last 500 metres and I brought it home in a time of 42:13 which was two minutes quicker than what I did in 2012 (44:12), which I have to say I’m super happy with. Also place 23rd out of 515 runners, which isn’t too shabby either.

SplashWorld 10km

April 13th, 2014

What a disastrous couple of weeks since the 1st run of the sesson in Mount Juliet. I now know I really shouldn’t have done that run, the calf injury just was too much.

Now it’s been weeks of frustration as my training plan had to go out the window. It also took weeks for my head to get around this injury so I entered this SplashWorld event with trepidation.

It’s been a few years since I did this race and I hadn’t realised that the course had changed to a more flatter one. 2 laps around the main sea front area of Tramore.

At the start I managed to get a good starting position and away we went at high speed. Although I was determined to stay on a predefined split time as I didn’t want to over do the come back race. Things were good to the half way point then the energy drained a little and the last 2 kms were tough going but end3d with a time of 41:42 which is just 8 secs from my PB and a whole 4 mins 07 secs faster than the time I did here 2 years ago.

So finally I feel back on track.

Run Mount Juliet

February 15th, 2014

Back to picturesque Mount Juliet, Kilkenny for the first run of the season. I’ve been looking forward to this all winter long but disappointment has struck in as I’ve picked up an injury breaking in my new runners.

I know they say break in new foot wear slowly and I thought I had. I placed at least 20km in them but on a rare fast leg of a training run just one week before this race, pop went the left calf muscle.

As I had registered for this event ages ago I was determined to do it and so had some intensive physio on coming into the event.

Now it has been constantly raining for the past few weeks and a big storm has just passed so there was also a wonder as to whether this race would be run at all. Turns out on the morning registration desk the 10km was fine but the half marathon had to be cancelled, the course was flooded.

And sure enough as I made my way down to the start line we had to walk across a section of wooden pallets over a flooded road within the Mount Juliet grounds. Those that had come for the half were super disappointed and the consolation of running the 10km didn’t appear to appease a large section of the crowd.

Finally under starters orders and who’s there on the start line only Sonia O’Sullivan, Irish European/World/Olympic medallist and it was super cool to be able to run with Sonia for 2 kms and then that dreaded hill came into sight and sure enough it defeated me. I tried I really tried but it was just so steep.

The rest of the run was normal enough and I didn’t feel the calf muscle that much and closed the event with a time of 43:49 which isn’t too shabby and 3 minutes within Sonia’s time (which I’m sure was just a training stroll for her).

When I look back at my previous run here I had a time of 43:55 so a few seconds clipped off.

I leave Mount Juliet limping and not feeling too good I hope this isn’t a bad sign for future runs.