Sending Telemetry to Azure IoT Hub

As I promised in my previous article, I will show you how to connect a temperature sensor to a Raspberry Pi, read real telemetry data, and send it to the Azure IoT Hub:

I am using a Raspberry Pi 4, a DHT11 temperature sensor, and a GPIO Extension Board to connect the sensor to the Raspberry. We’ll be using Azure IoT Python SDK enabling connection to the Azure IoT Hub. And we need DHT11 Python library for reading temperature and humidity from DHT11 sensor on Raspberry Pi.

First, we need to create an IoT Hub and to set up a Raspberry Pi. Repeat the steps described in my article “Connect Raspberry Pi 4 to Azure IoT Hub“:

Connect the sensor to Raspberry Pi

The DHT11 sensor can collect temperature and humidity data. Use the following wiring to connect the sensor to GPIO pins:

DHT11 Sensor pinsGPIO pins
Vcc (+)5V (pin 2)
Ground (-)GND (pin 6)
DataGPIO17 (pin 11)

Turn on your Raspberry Pi and connect it to your network.

Run the application

When it is done, clone the azure-iot-blog GitHub repository into a folder in your Raspberry Pi:

git clone https://github.com/jevgenij-p/azure-iot-blog.git

Find the Python application send_sensor_data.py in the raspberry-to-iot-hub/send-telemetry folder:

import asyncio
import time
import board
import RPi.GPIO as GPIO
import dht11
from azure.iot.device import Message
from azure.iot.device.aio import IoTHubDeviceClient

CONNECTION_STRING = ""

DELAY = 5
TEMPERATURE = 20.0
HUMIDITY = 60
PAYLOAD = '{{"temperature": {temperature}, "humidity": {humidity}}}'

async def main():

    try:
        # Create instance of the device client
        client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)

        # Initialize GPIO
        GPIO.setwarnings(False)
        GPIO.setmode(GPIO.BCM)
        GPIO.cleanup()

        # Read data using pin GPIO17
        dhtDevice = dht11.DHT11(pin=17)

        print("Simulated device started. Press Ctrl-C to exit")
        while True:

            try:
                result = dhtDevice.read()
                if result.is_valid():
                    temperature = result.temperature
                    humidity = result.humidity

                    data = PAYLOAD.format(temperature=temperature, humidity=humidity)
                    message = Message(data)

                    # Send a message to the IoT hub
                    print(f"Sending message: {message}")
                    await client.send_message(message)
                    print("Message successfully sent")
                else:
                    # print("Error: %d" % result.error_code)
                    continue

                await asyncio.sleep(DELAY)

            except KeyboardInterrupt:
                print("Simulated device stopped")
                GPIO.cleanup()
                break

    except Exception as error:
        print(error.args[0])

if __name__ == '__main__':
    asyncio.run(main())

IMPORTANT

Make sure you copy-paste the Primary Connection String you saved when you created your IoT device, into the quotes in the Line 9.

Now, open your Raspberry Pi terminal and install Python packages:

pip3 install azure-iot-device
pip3 install asyncio
pip2 install dht11

Run the application:

python3 send_sensor_data.py

If everything is configured and connected correctly, the program will be sending sensor data every 5 seconds to your IoT hub:

Open Azure Portal, find your IoT hub, and select Overview in the left navigation menu. You should see number of messages received by your IoT hub from your device:

Stay tuned. I’ll continue the “Azure IoT” series.

Connect Raspberry Pi 4 to Azure IoT Hub

I have a Raspberry Pi 4 running Raspbian and I wanted to implement the simplest IoT Architectural pattern of direct connecting an IoT device to an Azure IoT Hub:

Create IoT Hub

First, we need to create an Azure IoT Hub. Login to Microsoft Azure Portal, find an “IoT Hub” in the Azure Marketplace, and create it:

Then, we need to specify as existing Resource Group, or create a new one, specify a Region closest to you, and a globally unique IoT hub name:

On the Management tab, select F1: Free tier and press Review + create button to create the hub:

When the hub is created, we need to create a device identity in the identity registry in your IoT hub. In the left navigation menu, open IoT devices, and then press New to add a device:

In Create a device panel, provide a name for your device (Device ID), and press Save:

Soon, you will see your device in IoT devices:

Click on it to open its properties. You need to copy its Primary Connection String and save it somewhere in a text file. We will need this string a bit later to establish connection between your device and the IoT hub.

Set up your Raspberry Pi

Before we continue, you need to prepare you Raspberry Pi. First, make sure SSH and SPI interfaces are enabled in your Raspberry Pi Configuration.

You can use PuTTY to connect to your Raspberry. Copy the IP address of your Raspberry Pi into the Host name and select SSH as the connection type:

Then, open your terminal and log in to your device:

Run the application

I created a simple Python application, sending simulated data to the Azure IoT hub. The code is in the azure-iot-blog GitHub repository. You can clone it into a folder in your Raspberry Pi:

git clone https://github.com/jevgenij-p/azure-iot-blog.git

Or, you can clone it to a folder on your computer, and copy only the send_simulated_messages.py file to the Raspberry Pi using WinSCP (if you are working on a machine with Windows).

The Python application is send_simulated_messages.py in the raspberry-to-iot-hub/send-telemetry folder:

import asyncio
import random
from azure.iot.device import Message
from azure.iot.device.aio import IoTHubDeviceClient

CONNECTION_STRING = ""

TEMPERATURE = 20.0
HUMIDITY = 60
PAYLOAD = '{{"temperature": {temperature}, "humidity": {humidity}}}'

async def main():

    try:
        # Create instance of the device client
        client = IoTHubDeviceClient.create_from_connection_string(CONNECTION_STRING)

        print("Simulated device started. Press Ctrl-C to exit")
        while True:

            temperature = round(TEMPERATURE + (random.random() * 15), 2)
            humidity = round(HUMIDITY + (random.random() * 20), 2)
            data = PAYLOAD.format(temperature=temperature, humidity=humidity)
            message = Message(data)

            # Send a message to the IoT hub
            print(f"Sending message: {message}")
            await client.send_message(message)
            print("Message successfully sent")

            await asyncio.sleep(5)

    except KeyboardInterrupt:
        print("Simulated device stopped")

if __name__ == '__main__':
    asyncio.run(main())

IMPORTANT

Make sure you copy-paste the Primary Connection String you saved before, into the quotes in the Line 6.

Next, open your Raspberry Pi terminal and install some Python packages:

pip3 install azure-iot-device
pip3 install asyncio

Azure IoT Python SDK is a library, providing functionality for communicating between IoT devices and the Azure IoT Hub.

Now, you can run the application:

python3 send_simulated_messages.py

The program is sending data every 5 seconds to your IoT hub:

Let it work a few minutes and open Microsoft Azure Portal. Find your IoT hub, and select Overview in the left navigation menu to see your IoT Hub Usage:

Now you know how to send telemetry data (simulated in this case) to your Azure IoT hub. Next time I will show you how to send real data.

Camera calibration using OpenCV

Being not ideal, all cameras distort their images. Usually, it’s not so important and can be even funny. But if your image analysis requires quite precise images, you should take care about calibrating your camera to remove distortions.

Luckily, OpenCV provides everything we need for that. I don’t want to repeat the theory and description of that process – you can easily find it on the OpenCV website. I just created a simple Python application calibrationlab, calibrating cameras:

calibration-screen1

The application was written on Python 3.6 using opencv v3.4.1 and wxpython v4.0.3. So you’ll need to install them first, using pip or conda. Also, you will need to print out a chessboard.docx.

Start the application running calibrationlab.py. If you have a few cameras connected to your computer, you can select one using the “Camera” menu. Place the chessboard image in front of your camera and press the “Calibrate” button. Make a few tries to get minimal Mean Error. Resulting calibration parameters can be saved in a file. They contain camera matrix and distortion coefficients. Later you can use them loading from the file without the need to recalibrate your camera again. As an example, I included such a file (camera.json) into the source repository.

When the calibration process is done, you can check the”Undistort” checkbox to apply calibration parameters. The undistorted image is a bit smaller than the original one. I intentionally left a black border around the image. But you can crop the image, using roi (region of image) parameter, returned by the cv2.getOptimalNewCameraMatrix() method.

If you are using a good camera you hardly spot the difference between original and undistorted images. Otherwise, you will notice that straight lines in the real world become straight in the image.

The source code: GitHub download

Who is Data Science Engineer?

Who is Data Science Engineer, and is there any difference between them and Data Scientists? Your answer to this question is an indicator of success or failure of your Data Science project.

Software development is a risky business, but the risks rocket up when you give in to the persuasion of your client “to add some AI to the system”. Your next move is to hire Data Scientists and delegate this task to them.

“What are you doing?” – Scrum masters are wondering. “Building models” – they say. After a few months, you are starting to realize you cannot control this process. Why?

Because Data Science Process IS NOT Software Development Process! Any attempts to ignore this fact quickly bring the project to epic fail.

Data Science Process has its own set of inputs, outputs, roles, deliverables and process flow. Look at some them: TDSP, CRISP-DM, KDD, SEMMA. I hope your Project Managers are aware of them. But even if they are – epic fail is still your main option.

Because there is a gap between these two processes: Data Science and Software Development.

If your developers know everything about programming, they hardly know something about Statistics, Machine Learning, Data Science (excepting popular articles). Just as Data Scientists know nothing about building Line-of-Business Applications, and truly speaking are not too strong in programming. How are you going to cope with it?

ds-engineer

Project Manager staring at a Data Science project.

Data Science Engineer is the answer. You need a person

  • with strong programming skills
  • with basic Statistical skills
  • with Data Science Process understanding and ability to participate on each stage of the process
  • knowing specialized DS languages and tools like R, Tensorflow and so on.

Data Scientists and developers live in different worlds. Data Science Engineer lives in both. It is a magic adhesive tape without which your Data Science project will fall apart.

Visualize missing values in R

Data quality is extremely important for analysis, modeling, and predictions. Microsoft data science utilities for Team Data Science Process contain a script (IDEAR), which can visualize missing values. You can specify the number of segments to split your data, in order to calculate average missing value rate for each segment, and visualize it with the levelplot.

I refactored this function a bit and replaced the levelplot with the ggplot. My plot_missing() function requires two packages: ggplot2 and reshape. The code of the MissingValues.R script can be found on the GitHub download

If you pass a data.frame with some missing values (NA) to the function, you will get a visual distribution of densities of the missing values:

plot_missing(data)

missing-plot1

The leftmost column “All” shows average missing value rates for the whole data set by variables.

You can change the number of segments and the color palette. For instance, you can use a palette from the RColorBrewer package:

if (!require(RColorBrewer))
   install.packages("RColorBrewer")

library(RColorBrewer)
plot_missing(data, 5, col = brewer.pal(n = 9, name = "Blues"))

missing-plot2

If you do not remember palette names, you can display them:

display.brewer.all()

Here is another nice palette:

plot_missing(data, col = brewer.pal(n = 9, name = "YlOrRd"))

missing-plot3

The source code: GitHub download