This is an item that came up while working on a project within the TSSG and so might be worth sharing.
Have you ever tried to use a MQTT broker ? Message Queuing Telemetry Transport (MQTT) is a machine-to-machine (M2M), Internet of Things data protocol, which is in line with other data protocols such as XMPP, CoAP, AMQP, and Websockets. Invented in 1999, MQTT is now an OASIS (Organization for the Advancement of Structured Information Standards) standard, and ISO standard (ISO/IEC PRF 20922).
MQTT is extensively used in Amazon Web Services, Microsoft Azure IoT Hub, IBM WebSphere MQ, and is a publish/subscribe message exchange pattern, that can support persistent message storage on the broker and supports security in the form of authentication using user name and password, and encryption using SSL/TLS.
For something like the Eclipse Mosquitto broker, MQTT it has a really small code footprint, the libmosquitto (client library) is about 1.3 MB and is ideal if processor or memory resources are limited and also ideal if bandwidth is low or network is unreliable. Classic problems in the IoT space.
In the case of using MQTT for the smart grid, scale and security are top priorities. To achieve scale I’ve looked at bridging MQTT brokers in a hub and spoke model, where a very light MQTT broker is at the edge of the network (at the end of the spoke) and there’s a large MQTT broker at the hub which can aggregate all the data.
However the purpose of this post is to highlight the security aspects within MQTT and in particular the application of encryption (SSL/TLS) when using Let’s Encrypt certificates. Applying a certificate to an MQTT broker is not too hard, there’s a nice guide here on Mosquitto SSL Configuration for MQTT TLS Security and here too on SSL/TLS Client Certs to Secure MQTT however in the vast majority of cases the examples use self-signed certs and not certs as provided by Let’s Encrypt.
By the way if you don’t know Let’s Encrypt is a non-profit certificate authority run by the Internet Security Research Group that provides X.509 certificates for Transport Layer Security encryption at no charge. The certificate is valid for 90 days and the renewal process is quite simple.
Now bridging two MQTT brokers can be relatively straight forward too however getting the certs right when you want that bridge to be encrypted can be a little tricky look at how much you have to do to bridge a Mosquitto MQTT Broker to AWS IoT.
In my case I wanted to bridge two Mosquitto MQTT Brokers, each with encryption enabled by a Let’s Encrypt cert. Firstly I created a [special Docker container]() that could pick up the Let’s Encrypt cert, and having followed all the guides I kept getting the following error in the logs
OpenSSL Error: error:14037418:SSL routines:ACCEPT_SR_KEY_EXCH:tlsv1 alert unknown ca
OpenSSL Error: error:140370E5:SSL routines:ACCEPT_SR_KEY_EXCH:ssl handshake failure
Socket error on client , disconnecting.
I tried verifying the certs, by installing openssl
openssl verify cert.pem
But all was fine.
I thought I had to download the trusted root CA certificates for Let’s Encrypt and place it somewhere in the Alpine linux system (the base OS of the broker), but I must admit this “somewhere” was not so clear me.
The problem is that the MQTT broker does not know how to verify its own CA before starting the ssl exchange with any client. This is because the CA signing the Let’s Encrypt cert is not yet distributed and bundled by default in to the Alpine Linux system and therefore has to be added manually.
In the Mosquitto MQTT broker configuration, instead of just pointing directly at the chain.pem file I decided to point at the default place where all ca certs should be.
#cafile /mosquitto/config/certs/chain.pem
capath /etc/ssl/certsAnd this write up on installing certificates in an Alpine Image to establish Secured Communication (SSL/TLS) really got to the heart of the matter, the cert needs to be copied to a special directory /usr/local/share/ca-certificates/and then you need to run the program update-ca-certificates so it gets placed in the right way into the folder /etc/ssl/certs.
After much head scratching, it all comes down to 2 command lines
cp /mosquitto/config/certs/chain.pem /usr/local/share/ca-certificates/chain.pem
update-ca-certificates
Once done (via a docker-entrypoint.sh command) the container is able to handle the CA issue, and bridging 2 Mosquitto MQTT brokers that are using Let’s Encrypt certificates can be achieved.