A Practical Review of RabbitMQ as a Message Broker for an Internet of Things Application

Transporting and storing data is a critical part of any remote sensor system. For Internet of Things (IoT) based system, handling large quantities of messages in an efficient and reliable way is critical for the success of the system. Our system, a network of IoT based, long distance environment sensors, is no exception. Message brokers (such as RabbitMQ) enable two parts of a computer to communicate, with a focus on reliability and speed,  and are necessary to pull data from the sensors to the database.

More powerful message brokers are capable of sorting messages based on their properties. Different messages may need to be delivered to entirely different devices on a network. RabbitMQ allows for the sorting of different types of messages based on their properties, ensuring they go straight to the correct delivery point.

 

 

There are many ways in which RabbitMQ can be instructed to distribute messages. For example, in a “fanout” exchange every message received can be sent to every known consumer. It is capable of sorting messages based on their content.

In our deployment, RabbitMQ was to be used to take messages from The Things Network gatweays and store them. MQTT (Message Queue Telemetry Transport) is a protocol designed for machine-to-machine (Internet of Things) applications, and was the protocol used with RabbitMQ. It may be necessary to enable the MQTT plugin:

RABBITMQ-PLUGINS ENABLE RABBITMQ_MQTT

RabbitMQ can be interacted with using a multitude of different programming languages; we used Python. Messages from the gateways were correctly received by RabbitMQ, however it was not possible to process them further. The complexity of RabbitMQ made it difficult to perform any processes more complex than sending and receiving messages from the localhost. After many hours of tweaking, it was still not possible to obtain data via RabbitMQ, and an alternative solution was necessary.

Mosquitto is an alternative message broker, developed exclusively for use with MQTT. It is a much simpler, lightweight alternative to RabbitMQ. Due to its reduced feature set, performing complex routing arrangements is less practical. RabbitMQ also features a slick, built in user interface, viewable through a web browser, a luxury not available on the Mosquitto broker. This makes it less easy understand the status of the message broker and the data. This simplicity was an advantage however; shortly after installing and switching to Mosquitto, data was correctly received. Furthermore, it is often possible to process and sort the messages in Python (or another language), negating an advantage of RabbitMQ

When choosing a message broker, it is important to evaluate whether the extra features of RabbitMQ are really necessary, and justify the extra complexity. In our use case the goal was simply to transfer data from The Things Network to be stored in MongoDB. It was far quicker and more efficient to use MQTT than RabbitMQ, and there was no loss in functionality in doing so.

 

Pika (RabbitMQ-Python Interface)/

Paho MQTT (Python MQTT client)

import paho.mqtt.client as mqtt  # library for communicating with mqtt
from pymongo import MongoClient  # mongoDB library
import json  # for (de)serialising javascript objects


def TTN_add():
    try:
        print("TTN_Add running")
        mqclient = mqtt.Client()
        mqclient.on_connect = on_connect  # on connection runs "on_connect function"
        mqclient.on_message = on_message  # on message...
        mqclient.connect("localhost", 1883, 60)  # 60 indicates keep-alive time
        mqclient.loop_forever()  # keeps it going forever
    except KeyboardInterrupt:  
        mqclient.loop_stop()
        print("stopped")


def mongo_insert(Gateway_ID, payload):  # function to add stuff to the database
    collection = db.test_collection  # this creates the collection
    print ("Gateway_ID " + Gateway_ID)
    payload = json.loads(payload)  # payload to json version
    collection.insert_one(payload)
    return


def on_connect(mqclient, userdata, flags, rc):
    print("Connected with result code "+str(rc))
    mqclient.subscribe("gateway/#")  # only gateway topics


def on_message(mqclient, userdata, msg):
    print("on_message")
    print (msg.topic, msg.payload)
    # print (type(msg.payload))
    mongo_insert(msg.topic, msg.payload)


moclient = MongoClient('localhost', 27017) 
db = moclient.ttn_database
TTN_add() #starts event handlers