A simple connected object with NodeMCU and MQTT

TL,DR; How to create a simple(physical !) object that can connect to a WiFi and that uses MQTT to deliver and receive messages very fast.

1. Connectivity in real life

When it comes to creating connected object, the connection part is always the trickiest since power, network availability and form factor generally limit the possibilities.

A fairly new enabler in this market came around in 2014 : Espressif, a Chinese microchip brand that creates the ESP8266, a microcontroller that includes Wi-Fi capability.

Not only this chip is quite good, it's also small, is more robust than its counterparts like the CC3100 from Texas Instruments, and it's ... cheap as hell. I mean, $6.95 on Sparkfun, come on !

Many brands such as Sparkfun, Olimex and Adafruit created their dev board based on this chip : basically, just a simple breakout board that exposes all the pins the chip has to offer, and adds in a USB chip to facilitate firmware upload and programming. These are great little tools to develop for the so-called "Internet of Things".

The NodeMCU dev kit

Another interesting implementation is the NodeMCU firmware and dev kit. While I'm still wondering why they have a picture of a bunch of sheep on their homepage (see here), their solution is pretty neat : it's an ESP8266 on a custom dev board with a firmware written in MicroPython that incorporates GPIO, PWM, IIC, 1-Wire and ADC functions, along with a Lua environment and built-in libraries to use Wifi, MQTT, telnet, etc ...

On this kit, connecting to an access point can't really get much simpler than this :

wifi.setmode(wifi.STATION)
wifi.sta.config("SSID","password")
print(wifi.sta.getip())
--192.168.18.110

The development toolchain

It took me a bit of time to make it work flawlessly on the NodeMCU, and once I got it to work on the first version of the dev kit pictured below, I had to rethink it for the second version (named "Amica R2").

Drivers

First of all, you need the good driver for the actual version of your board;

The first kit needs the CH341 USB-TTL driver that you can find here for different platforms.

For the Revision 2 kit, the "Amica", the bridge has been replaced by a more broad CP210x USB to UART driver that you may need to install (depending on your system). Silicon Labs have you covered here.

Flashing the firmware

esptool from themadinventor is the solution of choice. It's a simple python script available here on Github.

Once you downloaded it and python setup.py install, you're ready to go, you just need a binary image of the firmware.

There are different firmware flavors out there :

... and some variations.

You can find the different links here too : http://forum.sh-hackspace.org.uk/t/list-of-esp8266-firmware/98

We'll settle for the Lua firmware. You can either download a release from the github repo of nodemcu but it's much more convenient to get a custom build. Marcel Stoer created a nifty tool for that :

http://frightanic.com/nodemcu-custom-build/

You can choose the modules that you include and the branch you're building against, input your email address, wait a few minutes and voilà !

Back to flashing

Now that you have your firmware.bin somewhere, time to flash it !

For Rev. 1 boards, this will boil down to something similar to :

python esptool.py --port /dev/tty.wchusbserial1420 write_flash 0x00000 firmware.bin

For Rev. 2 (Amica) boards, it'll be a little different :

python esptool.py --baud 115200 --port /dev/tty.SLAB_USBtoUART write_flash -fm dio -fs 32m 0x00000 firmware.bin

Of course you might want to check if your port name is the same, it can vary from machine to machine.

If everything goes fine, you should see the following output (flashing takes about 2 minutes) :

Connecting...
Erasing flash...
Writing at 0x00061400... (100 %)
Leaving...

In this case, unplug and replug your nodeMCU dev kit, and you're good to go !

Uploading code : ESPlorer

The Russians at http://esp8266.ru/ have put together a nice tool to facilitate code upload on the board, compatible with most firmwares : ESPlorer.

You can download it here : http://esp8266.ru/esplorer/
It needs JAVA SE 7 and that's about it.

2. And now for the object !

We want to create a simple object that will connect to a pre-configured network, and will receive (and/or send) messages via MQTT.

First things first, the requirements

We'll suppose you have a running instance of a MQTT broker somewhere, such as the excellent mosquitto.

We'll take a very standard configuration : broker.example.com:1883, no user/password.

The Lua project
The structure

Since the project is not going to take up much space and could be coded in a single file, there is little use having a complex structure for the code. We'll settle with a very simple best practice :

  • a config.lua file that will hold the configuration, and editable variables
  • a setup.lua file that will take care of setuping the connectivity
  • an application.lua file that will hold our app code
  • the init.lua file that will not be compiled (you can't)

This separation has the advantage of putting different code blocks in different files, making it much easier to recompile only the bits we need.

To compile a file :

 node.compile('file.lua');
The config file

Let's have a look at the configuration file first — we'll make it into a module for convenience :

-- file : config.lua
local module = {}

module.SSID = {}
module.SSID["myWifi"] = "12345679ABCDEF"

module.HOST = "broker.example.com"
module.PORT = 1884
module.ID = node.chipid()

module.ENDPOINT = "nodemcu/"
return module

Apart from the standard lua boilerplate, a few things to note here :

  • We declare an array of WiFi AP : this comes in handy when you have to add more networks; we'll see in the setup that we can iterate through all these and connect to the first one available

  • each NodeMCU dev kit has a handy unique identifier that is the node.chipid() that you can use to differentiate objects. It's very nice to have that.

  • I declare a standard endpoint prefix so I don't have any name collisions on my broker (that happens)

The Wifi setup

The WiFi setup is pretty straightforward we're going to iterate over the available APs, and if there is one that we have the password for, we'll try to connect :

-- file: setup.lua
local module = {}

local function wifi_wait_ip()
  if wifi.sta.getip()== nil then
    print("IP unavailable, Waiting...")
  else
    tmr.stop(1)
    print("\n====================================")
    print("ESP8266 mode is: " .. wifi.getmode())
    print("MAC address is: " .. wifi.ap.getmac())
    print("IP is "..wifi.sta.getip())
    print("====================================")
    app.start()
  end
end

local function wifi_start(list_aps)
    if list_aps then
        for key,value in pairs(list_aps) do
            if config.SSID and config.SSID[key] then
                wifi.setmode(wifi.STATION);
                wifi.sta.config(key,config.SSID[key])
                wifi.sta.connect()
                print("Connecting to " .. key .. " ...")
                --config.SSID = nil  -- can save memory
                tmr.alarm(1, 2500, 1, wifi_wait_ip)
            end
        end
    else
        print("Error getting AP list")
    end
end

function module.start()
  print("Configuring Wifi ...")
  wifi.setmode(wifi.STATION);
  wifi.sta.getap(wifi_start)
end

return module

We're using the timer functions tmr.alarm() and tmr.stop(), it's the recommended way to run code at designated intervals since it's non-blocking. More info here.

We're using app.start() here eventhough it's not declared : we'll declare it later in init.lua

The init.lua file

The init.lua file is pretty special on the NodeMCU since it's automatically played on startup. This means that everything we put in there will be executed the second the NodeMCU is starting.

While it's a pretty important feature, it could be a pain to use when developping: if you ever put code that can segfault in the init.lua file or in its imports, then your NodeMCU will reboot in cycles and you won't be able to do anything except reflashing it.

Life hack : only use init.lua at the end, when you're sure your code is rock solid. In the meantime, create a test.lua file that has the same content, and run it when you need it from the command line of ESPlorer :

dofile('test.lua');

It'll save you time, and some hair...

Onto the content

-- file : init.lua
app = require("application")
config = require("config")
setup = require("setup")

setup.start()

Remember : the init file is not a module, and is not compiled.

Our application code

Finally, our main code. The process will be :

  • Creating a callback function for when we receive a message
  • Connecting to the broker
  • Subscribing to the nodemcu/_mynodeid_ endpoint to receive targeted messages
  • Setting a recurring ping function that will send the node ID to nodemcu/ping so that the broker (and the application consuming the messages) will know that we're alive and well
-- file : application.lua
local module = {}
m = nil

-- Sends a simple ping to the broker
local function send_ping()
    m:publish(config.ENDPOINT .. "ping","id=" .. config.ID,0,0)
end

-- Sends my id to the broker for registration
local function register_myself()
    m:subscribe(config.ENDPOINT .. config.ID,0,function(conn)
        print("Successfully subscribed to data endpoint")
    end)
end

local function mqtt_start()
    m = mqtt.Client(config.ID, 120)
    -- register message callback beforehand
    m:on("message", function(conn, topic, data) 
      if data ~= nil then
        print(topic .. ": " .. data)
        -- do something, we have received a message
      end
    end)
    -- Connect to broker
    m:connect(config.HOST, config.PORT, 0, 1, function(con) 
        register_myself()
        -- And then pings each 1000 milliseconds
        tmr.stop(6)
        tmr.alarm(6, 1000, 1, send_ping)
    end) 

end

function module.start()
  mqtt_start()
end

return module

Again we use the timer functions to send the pings at regular intervals. The MQTT API is on the Github wiki as well.

Uploading and compiling

Create all these files locally, and then send them to the NodeMCU kit with ESPlorer. Once it's done, compile them all except for the init.lua one, and unplug / plug your NodeMCU back. It should start fresh and execute your code.

Testing with a desktop MQTT client

Now that we have to test the behaviour. A very nice MQTT client that can be used to test without having to write a second app is MQTTSpy. You can download it here (it's a Java application).

Let's start our NodeMCU and connect it to ESPlorer to see the output in the console. Let's then use MQQTSpy and connect to broker.example.com:1883, and subscribe to nodemcu/ping :

We see that messages arrive with a payload indicating the id of our NodeMCU : 16892456

So we can now send some data to our object by using the nodemcu/16892456 endpoint .. and we'll see the message pop up in ESPlorer console :

Tadam ! We have a two-way communication with our object over MQTT.

What next

Well, now, you can imagine all the possibilities to develop the backend interface to this object (these objects !). MQTT libraries exist in almost every language :

Why MQTT

In a nutshell : because it's fast ! And you can have QoS as well (not covered here), which means you can make sure messages get delivered once and once only to the recipient. When you create an end-user object that must be resilient and deterministic, this is key.