Microcontroller to Raspberry Pi WiFi MQTT communication

Microcontroller to Raspberry Pi WiFi MQTT communication

In this tutorial we create a WiFi communication between an ESP8266 or ESP32 microcontroller and a Raspberry Pi.

With this article you learn

  • the basics of MQTT communication
  • how to publish data to a MQTT broker
  • how to setup an MQTT broker
  • validate the published data as subscriber
NodeMCU Raspberry Pi MQTT WiFi

In my last article I wrote a tutorial how to send data from an Arduino to a Raspberry Pi via the serial USB communication. But the better solution to create a smart home is of course to send the data via WiFi communication.

If we want to send data via WiFi we need a transportation protocol. The most popular transportation protocol I know is MQTT which stands for Message Queuing Telemetry Transport and is suitable for microcontrollers like the Arduino, NodeMCU or Raspberry Pi.

Table of Contents

What is MQTT?

In this section I want to give a summary how MQTT works. If you are interested in more details about how MQTT works please visit my detailed tutorial about MQTT.

In the MQTT message chain are 3 parties involved:

  1. Publisher who generate and send data to the MQTT broker. In case of a smart home the publisher could be a weather station which sends temperature and humidity every 5 minutes to the broker.
  2. Broker which is like a server to collect the messages from all publishers, save the data and distribute the data messages to the right subscribers.
  3. Subscriber is a component which subscribes to a certain kind of messages coming from publishers. A subscriber in the smart home use case could be a monitor or panel which shows the different temperatures and/or humidity of the home and outside.
Arduino 2 Raspberry Pi WiFi Communication

If we want to send data via MQTT we have to define some variables and setting which are linked between the publisher, the broker and the subscriber. The following picture gives a perfect overview about how defines the setting and to which parties are the settings distributed.

For our example we use an ESP8266 NodeMCU as publisher. Of course you can use an Arduino microcontroller or an other EPS8266 based model as well as an ESP32 NodeMCU. The MQTT Broker will be a Raspberry Pi and we choose Mosquitto as MQTT software for the broker. The subscriber is the same Raspberry Pi as the broker. That the broker and subscriber is on the same device makes no problem and is common practice.

The first setting is the IP of the MQTT broker. Of course the IP of the MQTT broker is the same IP address that the Raspberry Pi has in your network. This IP is a variable in the publisher and subscriber script because they have to connect to the broker.

The second setting is the MQTT topic which is defined by the publisher. Which data you put in a topic it totally up to you. For me if I had several indoor weather stations in different room in the house I would give each room a topic. But in the end every topic has to be unique. Therefor topics are build like a folder structure. For examples you can have the following topics:

  • home/bathroom/temperature
  • home/bathroom/humidity
  • home/livingroom/temperature

The advantage is that one subscriber can have the subscription to the bathroom with all data following down the folder structure and a second subscriber can be only interested in the temperature of the bathroom.

The MQTT username and password is defined in the configuration file of the MQTT broker (mosquitto.conf). This configuration has to be exactly the same in the publisher and subscriber script to get access to the MQTT broker.

Only the publisher gets a MQTT client ID. With this client is the publisher is identified. This prevents the broker for disallowed data from unknown publishers and you know which publisher is sending what data. You can think for the client ID like a name.

The last setting is also only for the publisher. It is the SSID and password for your home network. Without this information the microcontroller can not send data via the local network.

After we defined all settings and which device has to define what setting or variable we can dive into your example. In this example we want to send the temperature and humidity from a DHT11 sensor or a DHT22 sensor module to the Raspberry Pi. The sensor is connected to an ESP8266 or ESP32 NodeMCU microcontroller. The Raspberry Pi as subscriber should print the last sent temperature and humidity to the terminal.

Because we will need all these variables and settings in the following part of this tutorial the table below shows the specific variables and settings. You can copy them or define your one. This is up to you.

MQTT Server

192.168.0.8

MQTT Topic

home/livingroom/temperature

home/livingroom/humidity

MQTT username

cdavid

MQTT password

cdavid

MQTT client ID

client_livingroom

WiFi SSID

XXXXX (your personal SSID of you home network)

WiFi Password

XXXXX (your personal password of you home network)

The following table gives you an overview of all components and parts that I used for this tutorial. If you want to support my work, you can buy something from the following links and I will earn a small commission. This does not affect the price you pay for the products.

If you are interested in components and parts that I used in other tutorials, visit the components and parts page.

 ESP8266 NodeMCUAmazonBanggoodAliExpress
ORESP32 NodeMCUAmazonBanggoodAliExpress
ANDRaspberry Pi 4 Model B KitAmazon AliExpress
ORRaspberry Pi 4 Model BAmazonBanggoodAliExpress
ORRaspberry Pi 3 B+ KitAmazon AliExpress
ORRaspberry Pi 3+AmazonBanggoodAliExpress
ANDDHT11 Module (blue)AmazonBanggoodAliExpress
ORDHT22 Module (white)AmazonBanggoodAliExpress

Publisher Setup Part 1 (ESP8266 / ESP32 NodeMCU)

The first step is to set up the publisher so that we can be sure that the temperature and humidity is read correctly and published to the network via MQTT.

In the following picture you see the wiring between the ESP8266 NodeMCU and the DHT11 sensor.

NodeMCU_Steckplatine

The program sketch is easy. First we have to import the DHT library, define the pin the DHT is connected to the NodeMCU and define the DHT type. In my case I use the DHT11 temperature and humidity sensor. Of course you can also use the DHT22 sensor. The wiring is the same between the DHT11 and DHT22.

In the setup function we set the baud rate to 9600 and initialize the sensor. The temperature and humidity are each saved in one variable and the sensor values are printed to the serial monitor. At the end of the script we pause for 10 seconds.

#include "DHT.h"
#define DHTPIN D2        // what digital pin the DHT11 is connected to
#define DHTTYPE DHT11   // there are multiple kinds of DHT sensors
DHT dht(DHTPIN, DHTTYPE);

void setup() {
  Serial.begin(9600);
  dht.begin();
}

void loop() {
  float h = dht.readHumidity();
  float t = dht.readTemperature();
  
  Serial.print("Humidity: ");
  Serial.print(h);
  Serial.println(" %");
  Serial.print("Temperature: ");
  Serial.print(t);
  Serial.println(" *C");

  delay(1000*10);       // print new values every 10 seconds
}

If you did everything right you should see the sensor values in the serial monitor.

NodeMCU temperature and humidity

This was part one of the publisher setup. Now we have to setup the MQTT broker and come back to the publisher to send the temperature and humidity with MQTT to the broker.

MQTT Broker Setup (Raspberry Pi)

First you have to setup the Raspberry Pi. If you do not know how to setup your Raspberry Pi, here you find the tutorial.
The first thing is that we have to connect to the Raspberry Pi via SSH to install the necessary packages and libraries. You should see the terminal in front of you.

Pi_Terminal

The first thing is to install the MQTT broker Mosquitto on the Raspberry Pi with the command line clients for debugging in case of error. Use the commands in the following table for the installation.

Install the MQTT broker/server

sudo apt-get install mosquitto

Install command line clients in case for debugging

sudo apt-get install mosquitto-clients -y

In case the packages can not be found run sudo apt-get update to update what packages are in general available for your Raspberry Pi.

After the installation of the MQTT broker Mosquitto we have to change some configurations. There are two different configurations to control Mosquitto:

  1. /etc/mosquitto/mosquitto.conf: Default configuration
  2. /etc/mosquitto/conf.d: Extension to the default configuration.

In our example we only want to use the default configuration. Therefor open the configuration file with the text editor nano and the following command:

Open the Mosquitto MQTT broker configuration

sudo nano /etc/mosquitto/mosquitto.conf

You see the default configuration after the installation of Mosquitto.

Raspberry Pi Mosquitto Configuration 2

There are 4 things we want to change apart of the default configuration:

  1. The broker should only include the default settings. That is done be comment the line where the conf.d file is included.
  2. We do not want anonymous users to connected to the MQTT broker: allow_anonymous = false
  3. We want to save the passwords in a separate file: password_file /etc/mosquitto/pwfile
  4. The MQTT broker should be accessible on port 1883

After all changes your Mosquitto configuration file should look like the following.

Raspberry Pi Mosquitto Configuration 3

Click Ctrl + X, then Y to confirm to save and hit the enter button to save to the existing file.
Now we have to set the username and the password that publishers and subscribers get access to the MQTT broker. Use the following command to create a new user and password for this user.

Create new user with username and password

sudo mosquitto_passwd -c /etc/mosquitto/pwfile username

sudo mosquitto_passwd -c /etc/mosquitto/pwfile cdavid

Raspberry Pi Mosquitto User

In my case I use the following username and password:

  • Username: cdavid
  • Password: cdavid

Of course this password is bad regarding the security. Feel free to create a stronger password. This username and password is also used in the publisher and subscriber script.

If you want to delete an existing user, you can use the following command.

Remove existing user

sudo mosquitto_passwd -d /etc/mosquitto/pwfile username

At this point we created the MQTT broker as well as every setting we use in this tutorial. Now we want to check is Mosquitto is already running. You can test the current status of the MQTT broker with:

See current status of MQTT broker

sudo systemctl status mosquitto

Raspberry Pi Mosquitto Status

My broker is running. The following commands start, stop and restart and start the MQTT broker at boot of the Raspberry Pi. Use the following commands to control the broker:

Start MQTT broker

sudo systemctl start mosquitto

Stop MQTT broker

sudo systemctl stop mosquitto

Restart MQTT broker

sudo systemctl restart mosquitto

Start MQTT broker at boot

sudo systemctl enable mosquitto

Now after the MQTT broker is ready to receive the data via MQTT we have to make sure that the publisher is sending the data via MQTT to the broker.

Publisher Setup Part 2 (ESP8266 / ESP32 NodeMCU)

The publisher part 1 finished that the temperature and humidity are stored in a variable and printed to the serial. In this part 2 we establish the WiFi connection to the home network and send the data as payload via the MQTT protocol to the broker.

Now we go part by part over the Arduino script for the ESP8266 or ESP32 NodeMCU. At the end of this chapter you find the complete script for copy and paste.

#include "DHT.h"
#include "ESP8266WiFi.h"  // Enables the ESP8266 to connect to the local network (via WiFi)
#include "PubSubClient.h" // Connect and publish to the MQTT broker

#define DHTPIN D2        // what digital pin the DHT11 is conected to
#define DHTTYPE DHT11   // there are multiple kinds of DHT sensors
DHT dht(DHTPIN, DHTTYPE);

At the beginning of the script we have to include two more libraries. The ESP8266WiFi library to connect the NodeMCU with the local network and the PubSubClient to enable the microcontroller to be a MQTT publisher. If you do not know how to install Arduino libraries, here you find a step by step tutorial. The rest stays the same compared to the setup in part 1.

// WiFi
const char* ssid = "KabelBox-XXX";                 // Your personal network SSID
const char* wifi_password = "9434763855XXXX"; // Your personal network password

// MQTT
const char* mqtt_server = "192.168.0.8";  // IP of the MQTT broker
const char* humidity_topic = "home/livingroom/humidity";
const char* temperature_topic = "home/livingroom/temperature";
const char* mqtt_username = "cdavid"; // MQTT username
const char* mqtt_password = "cdavid"; // MQTT password
const char* clientID = "client_livingroom"; // MQTT client ID

// Initialise the WiFi and MQTT Client objects
WiFiClient wifiClient;
// 1883 is the listener port for the Broker
PubSubClient client(mqtt_server, 1883, wifiClient); 

In the next part we define a lot of variables. Of course you have to fill in your personal data here. The following table gives you an overview of the variables. If you have to modify the variable to your local setting, the row is colored in orange.

Variable

Value

Description

ssid

KabelBox-XXX

You personal network name

wifi_password

9434763855XXXX

The password of your home network

mqtt_server

192.168.0.8

IP address of the Raspberry Pi where Mosquitto is installed

humidity_topic

home/livingroom/humidity

Name of the MQTT topics that have to match the logic mainlocation/sublocation/measurement

temperature_topic

home/livingroom/temperature

mqtt_username

cdavid

Username of the Mosquitto broker

mqtt_password

cdavid

Password of the Mosquitto user

clientID

client_livingroom

Name of the client

If you followed the settings and variables I used you can copy them. If for example you want to use another username, make sure you also change the username in the Mosquitto configuration file.

After all variables are defined we initialize the WiFi as well as the MQTT client objects. Make sure you use the identical listener port (1883) compared to the Mosquitto configuration file.

// Custom function to connet to the MQTT broker via WiFi
void connect_MQTT(){
  Serial.print("Connecting to ");
  Serial.println(ssid);

  // Connect to the WiFi
  WiFi.begin(ssid, wifi_password);

  // Wait until the connection has been confirmed before continuing
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  // Debugging - Output the IP Address of the ESP8266
  Serial.println("WiFi connected");
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());

  // Connect to MQTT Broker
  // client.connect returns a boolean value to let us know if the connection was successful.
  // If the connection is failing, make sure you are using the correct MQTT Username and Password (Setup Earlier in the Instructable)
  if (client.connect(clientID, mqtt_username, mqtt_password)) {
    Serial.println("Connected to MQTT Broker!");
  }
  else {
    Serial.println("Connection to MQTT Broker failed...");
  }

}

The next part is a custom function to connect to the MQTT broker via the wireless connection. First the SSID and password is printed. Then the script waits until the wireless connection is established and prints the IP of the microcontroller. After the WiFi is setup we connect to the MQTT broker with the client ID, the MQTT username and MQTT password. We print in the serial if the connection is successful or failed.

void setup() {
  Serial.begin(9600);
  dht.begin();
}

The setup function is the same as in part 1

void loop() {

  connect_MQTT();
  Serial.setTimeout(2000);
  
  float h = dht.readHumidity();
  float t = dht.readTemperature();
  
  Serial.print("Humidity: ");
  Serial.print(h);
  Serial.println(" %");
  Serial.print("Temperature: ");
  Serial.print(t);
  Serial.println(" *C");

  // MQTT can only transmit strings
  String hs="Hum: "+String((float)h)+" % ";
  String ts="Temp: "+String((float)t)+" C ";

The loop function starts with executing the MQTT connection function we discussed before. Then we read the temperature and the humidity from the sensor like in part 1 and print the values to the serial. New is that the temperature and humidity are also stored as strings because MQTT can only transfer strings and no other data types.

  // PUBLISH to the MQTT Broker (topic = Temperature, defined at the beginning)
  if (client.publish(temperature_topic, String(t).c_str())) {
    Serial.println("Temperature sent!");
  }
  // Again, client.publish will return a boolean value depending on whether it succeeded or not.
  // If the message failed to send, we will try again, as the connection may have broken.
  else {
    Serial.println("Temperature failed to send. Reconnecting to MQTT Broker and trying again");
    client.connect(clientID, mqtt_username, mqtt_password);
    delay(10); // This delay ensures that client.publish doesn't clash with the client.connect call
    client.publish(temperature_topic, String(t).c_str());
  }

  // PUBLISH to the MQTT Broker (topic = Humidity, defined at the beginning)
  if (client.publish(humidity_topic, String(h).c_str())) {
    Serial.println("Humidity sent!");
  }
  // Again, client.publish will return a boolean value depending on whether it succeeded or not.
  // If the message failed to send, we will try again, as the connection may have broken.
  else {
    Serial.println("Humidity failed to send. Reconnecting to MQTT Broker and trying again");
    client.connect(clientID, mqtt_username, mqtt_password);
    delay(10); // This delay ensures that client.publish doesn't clash with the client.connect call
    client.publish(humidity_topic, String(h).c_str());
  }

  client.disconnect();  // disconnect from the MQTT broker
  delay(1000*60);       // print new values every 1 Minute
}

The last step is to send the temperature and humidity as strings to the MQTT broker via the established connection. Therefore we use the function client.publish of the PubSubClient library. If the message is not sent successful we try to establish the connection to the broker again and try to send the payload a second time. This is done with the temperature and the humidity.

After the temperature and the humidity is sent to the MQTT broker, we disconnect the publisher from the broker and add a delay at the end of the script of 1 minute.

If you want to copy the whole script for your microcontroller, use the following button to download a zip file with the Arduino script for your microcontroller:

Let us quickly summarize the steps in part 2 of the publisher setup:

  1. We establish a WiFi and MQTT connection.
  2. The temperature and humidity are stored as strings.
  3. The strings are send via MQTT to the broker.

If everything runs correctly you should see the following output in your serial monitor.

NodeMCU send MQTT

Check if MQTT data is received by Mosquitto

Before we continue, we want to make sure that the MQTT messages from the NodeMCU is received by the Mosquitto MQTT broker. Therefore we start the Mosquitto console on the Raspberry Pi to see the connecting publishers. Enter the following command in the console of the Raspberry Pi to start the Mosquitto console:

Start Mosquitto console

sudo mosquitto

You should see that the publisher connects to the broker with the defined publisher ID and also that there is a clean disconnect. (The client ID differs in my case, because is updated the article with a new publisher)

Mosquitto see connected publisher

Subscriber Setup (Raspberry Pi)

At this point the setup of the publisher and the broker are finished. The last step is to setup the subscriber. In our case the subscriber is the same Raspberry Pi. There is no need, that a subscriber is another device like the broker. We want to use the script language Python to program a script that is listen to the MQTT topics of the publisher. Therefor we have to install a python library for MQTT. Use the following command in the Raspberry Pi terminal:

Install Python MQTT language bindings

sudo pip install paho-mqtt

Python MQTT

So we want to create a python script. To create the file we use the text editor nano again:

Create a python file for subscribing to MQTT data

sudo nano get_MQTT_data.py

Like in the part 2 of the publisher setup we go step by step through the following python script.

import paho.mqtt.client as mqtt

MQTT_ADDRESS = '192.168.0.8'
MQTT_USER = 'cdavid'
MQTT_PASSWORD = 'cdavid'
MQTT_TOPIC = 'home/+/+'


def on_connect(client, userdata, flags, rc):
    """ The callback for when the client receives a CONNACK response from the server."""
    print('Connected with result code ' + str(rc))
    client.subscribe(MQTT_TOPIC)


def on_message(client, userdata, msg):
    """The callback for when a PUBLISH message is received from the server."""
    print(msg.topic + ' ' + str(msg.payload))


def main():
    mqtt_client = mqtt.Client()
    mqtt_client.username_pw_set(MQTT_USER, MQTT_PASSWORD)
    mqtt_client.on_connect = on_connect
    mqtt_client.on_message = on_message

    mqtt_client.connect(MQTT_ADDRESS, 1883)
    mqtt_client.loop_forever()


if __name__ == '__main__':
    print('MQTT to InfluxDB bridge')
    main()

The first thing in the script is to import the MQTT library we installed before. After the library is imported we define all necessary variables. Make sure that the MQTT broker IP (IP of your Raspberry Pi) is correctly as well as the MQTT username and password if you changed them during this tutorial. The MQTT topic is defined with wildcards to create the script as general as possible for reusability.

The on_connect function handle what happens when the MQTT client connects to the broker. If our clients connects to the broker we want to subscribe to the temperature and humidity topics and print the message that the connection was established. The variable rc holds and error code if the connection is not successful so the debugging is easier:

  • 0: Connection successful
  • 1: Connection refused – incorrect protocol version
  • 2: Connection refused – invalid client identifier
  • 3: Connection refused – server unavailable
  • 4: Connection refused – bad username or password
  • 5: Connection refused – not authorized
  • 6-255: Currently unused.

The on_message function is called every time the topic is published to. In this case the topic and the message are printed to the terminal. We expect that on this point we will get the current temperature and humidity printed to the terminal.

The main function holds the main part of the python script where all defined function are executed. First we create a client object and set the username and password for the MQTT client. Then we tell the client which function is called when we want to connect to the MQTT broker and receive messages.

Once everything has been set up, we can connect to the broker with the broker IP and the broker port. Once we have told the client to connect, the client object runs in the loop function forever.

Click Ctrl + X, then Y to confirm to save and hit the enter button to save to the existing file.

Now we test if everything is running correctly. Therefore run the python code with the following statement in the Raspberry Pi console and watch the terminal if you receive data.

Run the MQTT subscriber

python get_MQTT_data.py

You should see the temperature and humidity in the terminal like the following picture.

It can take up to 1 minute before the NodeMCU is sending new data. And of course make sure that the NodeMCU is running.

NodeMCU Raspberry Pi MQTT

Conclusion

During this tutorial we build a full MQTT pipeline to send sensor data from a micro-controller like the NodeMCU to a MQTT broker on a Raspberry Pi. Moreover we created a subscriber who reads the temperature and humidity data published to the broker. It is also possible to change the payload to the status of a light if it is on or off. Also the status of a switch can be send via MQTT. In my apartment I have in total 3 indoor weather stations and 1 outdoor weather station measuring the temperature, humidity and light. All 4 devices send the data to 1 MQTT broker and I can easily add more sensors and devices to the system.

If you struggle at some point in this tutorial or if you have questions regarding the project or MQTT in general, fell free to use the comment section below to ask questions. I will answer them as quickly as possible.

9 Responses

Leave A Comment