The Long-Winded Guide to setting up Amlen: TLS

This is a section of a multi-part guide to setting up the Amlen Message Broker. If you want to see the other parts check out the Introduction/Contents.

It would be unusual (but not unheard of) for MQTT connections to not be encrypted, so the next step is to set up our Amlen server with TLS certificates.

The most common way TLS is used involves having the server present a certificate signed by a certificate authority to prove to the client that it is genuinely the server. The client verifies the signature and can then send a username and password securely to the server in order to prove the identity of the client.

Another way TLS is used is that both the client and the server have certificates signed by a certificate authority. These certicates can then be checked by both parties and authenticate is therefore acheived without need of passwords being sent. This method is called mutual authentication.

Amlen supports both TLS where only the server has a signed certificate and mutual authentication where TLS can be used to authenticate the client.

In essence, TLS is based on having a public and private key pair. The public key is signed by a trusted third party (a certificate authority – CA) – this signed public key is combined with some metadata into a certificate. The client then needs to have the certificate of the CA (not the server it is connecting to) in its truststore so that during a TLS handshake it can check that the certificate the server sends is a validate certificate.

In order to make sure that we’ve got certificates we can use, we will set up a toy CA (using openssl). If you’ve already got a certificate from a well-known CA and want to put it into Amlen then skip down this page to the “Adding the Server Certificate to the Server” section.

Setting up our toy CA

We’re going to very quickly set up a toy CA following this guide so we can easily get some certificates to play with. If you plan to deploy certificates from your toy CA in production, there are a lot of issues (e.g. intermediate certificates, expiry, choice of algorithms and revocation) that are outside the scope of this article but we will bravely press on for the moment. We’ll need openssl which is usully installed/easily available in the package manager for Linux, but it also available for Win/Mac.

We’ll start by making a directory for all our certificates and creating the key for the root CA.

mkdir ~/toyCA
cd ~/toyCA
openssl genrsa -des3 -out myCA.key 2048
(note down the password you use – you’ll need it later)

Next we generate the root certficate for our toy CA (this is the certificate our clients will need to trust)

openssl req -x509 -new -nodes -key myCA.key -sha256 -days 1825 -out myCA.pem

(this will ask you a series of questions, the answers to which aren’t important apart from identifying the cert)

Hurrah, now we have a private key and a public certificate – we re in business as a toy CA. Now we are a CA, it’s time to generate the key and certificate for our Amlen server:

First we create the key for the server:
openssl genrsa -out test.server.key 2048

Then we need to generate a certificate signing request (CSR) which is what we’d send away to a real CA to ask them to sign us a certificate. Again the questions only help you identify the certificate later. One of the questions is a challenge password – this is a shared secret between the person creating the CSR and the CA, it is not used to encrypt any keys etc – as we are the CA we can leave it blank so create the CSR:
openssl req -new -key test.server.key -out test.server.csr

We need to create a text file called test.server.ext using a text editor. The contents need to look like:

The alt_names section is key. TLS clients often check that the server they are connecting to is listed as one of these names (or the common name) of the certificate) so it is important it matches the DNS record (or hosts file) record we use as the host the client connects to – if the system you have the server on, has a DNS record put that in this section. If the system only has an IP address, we’ll create a local hosts entry for testserver.getting.started later.

Now we use this file to create the certificate:

openssl x509 -req -in test.server.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out test.server.crt -days 825 -sha256 -extfile test.server.ext

We now have all the file we need to create our first TLS connection:

• myCA.pem – The certificate of our toy CA. We need this in the truststore of our client.
• test.server.key – The private key for our Amlen server, we’ll install this on the server shortly.
• test.server.cert – Certificate signed by our toy CA for the Amlen test server (again we’ll install this on the server shortly).

Adding the Server Certificate to the Server

Now we need to log back into the WebUI for our server that we set up in the previous section.

The first thing we’ll set up is a certificate profile:

1. In the WebUI go to Server > Security Settings
2. Click on the green ‘+’ above the Certificate Profiles table
3. Set a name for our new certificate profile e.g. “TestServer-from-ToyCA”
4. Click the browse button by certificate and find the test.server.cert file
5. Similarly add the test.server.key file to the private key field
6. Save the new certificate profile.

In step 4 above we uploaded the certificate (which is in PEM format). For a really CA, there are usually one or more intermediate certificates. If that were they case, we’d upload a file that had the server certificate followed by the intermediate cert(s). Sometimes people include the root CA in this list of certificates to be transmitted – in many cases it is unnecessary (as the client will have the root CA cert in its trust store) but some clients use a Trust-On-First-Use (TOFU) security model and then it is useful for the server to send the complete chain so the device trusts the root CA and isn’t affected by certificate renewals.

Now we have a certificate profile, we need a Security Profile that uses it. This is at the bottom of the page in the WebUI (Server>Security Settings) that we just used to add a certificate profile. So let’s create the Security Profile:

1. Click the green ‘+’ above the Security Profile Table.
2. Give the new profile a name (e.g. TestServerSecProfile)
3. Ensure ‘Use TLS’ is ticked
4. Ensure ‘Use Client Certificates’ is unticked (we’ll come back to that in the mutual TLS auth section below)
5. Choose the certificate profile we just created (suggested name was “TestServer-from-ToyCA”)
6. Ensure “Use Password Authentication” is unticked – we’ll come back to that in another section of the guide (LDAP)
7. Ensure we don’t have an LDAP or OAuth profile set up
8. Save our new Security Profile

The final bit of setting up the server to use these certificates is to configure it to listen on a port and use the security profile we just created.

1. In the WebUI, from the bar at the top, choose: Messaging>Message Hubs
2. Click on ‘DemoHub’ and then the pencil icon above the table.
3. Switch to the “Endpoints” tab
4. Click the green ‘+’ above the Endpoints Table.
5. Enter a Name for the Endpoint e.g. TLSTestPort
6. Ensure “Enabled” is ticked
7. Choose a port (MQTT with TLS is 8883 by default but pick say 8889)
8. Choose the security profile we created above
9. Add the DemoConnectionPolicy (we’ll talk about policies in a future installment – for the moment we let anyone connect and pub/sub on any topic).
10. Add the DemoTopicPolicy and DemoSubscriptionPolicy
11. Save the policy

Now we have our server set up we need to connect a client to it. Like in the last (non-TLS) installment we’ll talk about two: an in browser client and a python command line one – choose whichever is most convenient.

Setting up a host entry to point to the server

Whichever client you are using, you want to be able to connect to the DNS entry listed in the common name/Subject Alternate Name of the server cert. If your system has a DNS entry listed in the server cert, you can skip this section, if you had testserver.getting.started in the certificate we need to config the system the client will run on to route that to the server.
On Linux/Mac/Windows you can do this routing by editing a hosts file.
For example if the client is going to run on the same (Linux) system as the server then we would edit /etc/hosts on that system and add:

#For testing TLS certs on Amlen server
127.0.0.1 testserver.getting.started

For Windows and Mac, the process is very similar:

• Adding a hosts entry on Windows
• Adding a hosts entry on MacOS

Once you’ve added the host entry check that pinging the new hosts entry is pinging the right server:

ping testserver.getting.started
PING testserver.getting.started (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.095 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.070 ms

Assuming that the ping works, you’re ready to get started

Connecting a client in a webpage

The browser must be able to reach the webpage with an MQTT client in and the server, if there are firewalls etc stopping that, use the python client below set up on a machine inside the server firewall (e.g on the server system).

1. Add the root CA (myCA.pem if you are using the toy CA) as a trusted certificate in your browser.
   In Firefox Settings > Privacy & Security > Certificates > View Certificates > Authorities Tab > Import
   In Chrome Settings> Privacy & Security > Security > Manage certificates > Authorities > Import
2. Visit https://amlen.org/samples/MQTTClient/ (note we can use https this time as our MQTT connect will be TLS as well)
3. Enter the server name (e.g. testserver.getting.started and port 8889)
4. Ensure SSL is “On”
5. Click connect

Did it work? If so great, you can do the same in another browser windows and should be able to pub/sub between the windows.

Connecting the Python Paho client over TLS

A command line the following will connect and subscribe to our server:
python3 client_sub_opts.py -H testserver.getting.started -P 8889 -t topic1/# -s --cacerts myCA.pem

and:
python3 client_pub_opts.py -H testserver.getting.started -P 8889 -t topic1/subtopic2 -s --cacerts myCA.pem -N 5
Will send a message to the subscriber

Debugging

Assuming that you’ve been able to connect to your server then you don’t need to debug anything – but it’s still useful to run through a few tools that are often useful when debugging TLS/connection issues (after that we’ll talk about using mutual TLS to authenticate clients).

• ping – If you can run run ping on the client try ping <server>. If it can’t, it’s possible that the server is configured not to respond to pings or that there is a firewall (which potentially only blocks pings). So it’s not 100% reliable but a useful tool that often helps show whether the client can send data to the server.
• sudo netstat -tanp | grep -i listen
  Running netstat on the system with the server shows whether the server process (imaserver) is listening on the port that you want it to be. If not, there’s a server configuration issue – have you enabled your endpoint?
• telnet
  Telnetting from the client to port on the server you are trying to connect to shows whether the client can connect to that port. If it can connect, you won’t be able to do anything but if not it points to a firewall or network issue between client and server
• openssl s_client -connect <host>:<port> -servername <host> -showcerts
  openssl s_client (and s_server) are incredibly useful if the connection is TLS based – this shows what certificates are received from the server so if the client is rejected them it helps in determining whether the problem is that the server is sending the wrong certs or the client is not correctly configured to trust them.
• openssl x509 -noout -in <certificate file> -text
  This shows the details about the first certificate in a file, things like the common name and the expiry date which can be figuring out whether a file has the certificate you think it does.

Mutual TLS: Using TLS for client authentication

Now we have the client connecting to and trusting the server, we can if we choose issue each client a certificate and use that certificate to authenticate the client to the server. If your client is a human in a webpage that would be an unusual thing to do, humans tend not to want to fiddle with certs and normally use username+password. For IoT devices, having a certificate onboard is quite common (but make sure there is a way to update the firmware including the cert). If we want to have client certificates, it’s very similar to what we did above.

First we create a certificate for a client with our toy CA:

cd ~/toyCA
openssl genrsa -out test.client1.key 2048

Then create the CSR as we did for the server. When answering the questions, the key question is the common name which is roughly equivalent to a username and may be used in the policies (see a future installment):

openssl req -new -key test.client1.key -out test.client1.csr

The create the certificate:

openssl x509 -req -in test.client1.csr -CA myCA.pem -CAkey myCA.key -CAcreateserial -out test.client1.crt -days 825 -sha256

Now we have the key for the client test.client1.key and the certificate for it test.client1.crt

We’ll make a new security profile that checks client certificates (so we can still have ports that don’t):

1. In the Web UI head to Server > Security Settings
2. Add a new seccurity profile with the green ‘+’ above the security profile table
3. Choose a name e.g. TestServerMutualAuthSecProfile
4. Ensure ‘Use TLS’ is ticked
5. Pick the same Certificate profile as before (suggested name was “TestServer-from-ToyCA”)
6. Ensure “Use Client Certificates” is ticked
7. Ensure “Use password authentication” is unticked
8. Save our new profile (Note the wrning that we have enabled clients certs but we haven’t configured our trust store)
9. Select the TestServerMutualAuthSecProfile row in the table and choose ‘Other Actions>Trusted Certificates
10. Upload the ‘myCA.pem’ root certificate from our toy CA

Now the server has a profile that will require a client certificate but no port is using that profile yet. So we’ll add a port listening for clint certificates:

1. In the WebUI go to Messaging > Message Hubs
2. Select the DemoHub row and click the Pencil icon above the table
3. Switch to the Endpoints Tab
4. Click the green ‘+’ to add a new endpoint:
5. Pick a name e.g. ‘ClientCertsTLS’
6. Ensure ‘Enabled’ is ticked
7. Pick a port e.g. 8890
8. Set the Security profile to be TestServerMutualAuthSecProfile
9. Add the DemoConnectionPolicy (we’ll talk about policies in a future installment – for the moment we let anyone connect and pub/sub on any topic).
10. Add the DemoTopicPolicy and DemoSubscriptionPolicy
11. Save the policy

Now we have our server set up and listening on 8890 for connections with valid certificates.

Using our python client we can connect to our new port using a valid client key+cert:
client_sub_opts.py -H testserver.getting.started -P 8890 -t topic1/# -s --cacerts myCA.pem -D -K test.client1.key -C test.client1.crt
But if we don’t send a valid key then we can’t connect e.g. client_sub_opts.py -H testserver.getting.started -P 8890 -t topic1/# -s --cacerts myCA.pem -D
will fail to connect.

At this point we’ve shown how to authenticate clients with certificates, but that’s not the only way to do authentication in Amlen – in the next installment we’ll look at setting up an LDAP server and using that for authentication.