The Internet of Things has opened vast possibilities undreamed of until recently. We firmly believe that ZeroMQ could be the glue that holds IoT.

This blog presents the fastest and easiest way to make applications that need to communicate. This blog presents code that can scale across cores, processes and boxes. The heart of this is ZeroMQ, a proven rock-solid toolkit for messaging.

This blog presents code that will allow noobies to create their own applications using the ZeroMQ toolbox by tutorials that detail python code.

We feel that the code explained and detailed herein could unleash a tidal wave of creativity for hobbyists, developers and small companies with a need to intelligently send messages and communicate amongst multiple computers.

Opportunities abound for the inexpensive but powerful single board computers such as the Raspberry Pi, Arduino and others. Applications only dreamed about earlier can now be realized. Future progress will empower even more astonishing applications.

I certainly hope that anyone reading this blog already has similar excitement.

We cover python coding of a powerful library, ZeroMQ, with tutorials, testing, tools, maintenance and advanced applications with many working code examples. We explain how to use modern test driven development paradigms to ensure highest quality code.

NOTE: Please download from PyIoTMessaging. The code discussed in this blog is in the ch01 directory.

Who is this blog for?

Managing multiple unconnected Raspberry Pis can become a daunting task if each system maintains its own independent realm. Connecting with messaging can significantly enhance your systems usefulness.

We assume:

• You use one or more Raspberry Pis, Arduino or similar single board computers.
• You have at least a beginner’s level of Python. We will use Python 2.7 or higher for this blog.
• While we will use Linux, only an elementary knowledge of Linux is necessary. When we offer tools using Linux commands, we will explain each tool. Creating directories, copying files, using an editor effectively, accessing github.com, installing packages covers most Linux skills.

We will develop on Linux systems and message with Raspberry Pis. Because Raspberry Pis use Debian as the default OS. Most other Linux OSs will should work as well. The term “Raspberry Pi” provides a generic term for many small board computers running a Linux system including Arduinos, NUCs, etc. This software should work well with little or no modifications.

Your Systems as a Messaging Network

Using two or more of these boards suggests they could communicate. One board becomes the “master” in the sense that the board collects all the information from the other boards to some summary location on the master.

Does this make sense for your system? Would you prefer to have your sensor readings collects in one place? Do you need to control a switch or valve on a remote computer?

Using two or more of these boards with a master implies some method of communications must exist. They may be hard-wired with an ethernet cable, or they may use WiFi, or they may have any other combination.

How about using a desktop system with all the board communication to the desktop as master?

Home Desktop Network

This diagram displays the connectivity of four systems: one master desktop system and three client systems. Assume the systems collect information in the front yard, the back yard and within the house.

Each remote client system sends messages to the master system. Depending on the application, the master system may send control messages to clients to start or stop events.

The important pieces of our connectivity uses the Views code to display elements of each of the client systems. While these views get clarified later, assume they are, at a minimum, summaries of logs and perhaps graphs.

Notice “Logs” get attached to the desktop and perform a vital role in this networked system. The logs may consist of a simple text file to a more complete database. More on this important subject later in a logging chapter.

Our Toolbox

This series discusses the code and philosophy of connecting the client systems to a master system. Specifically this toolbox provides:

1. A solid messaging system for fail-safe communications. Much more about this later
2. Debugging tools and advice.
3. Command line utilities. These utilities focus on our theme of connected systems with debugging, testing and field maintenance. The UNIX philosophy of small programs connecting together in combinations yields remarkable synergy.
4. Comprehensive, usable documentation: “If you can’t explain your product to a customer, you don’t have a product”.
5. Great tutorials. This comes with the great documentation.
6. Intermediate and advanced examples. Most sites provide simple examples and the rest gets left as “an exercise for the student”. In making industrial strength applications the changes from simple tutorial examples need explanation. Your household network deserves industrial strength code.
7. Coded in python. Most users of Raspberry Pi and similar systems code in python. Python offers a huge library of applications. ZeroMQ provides a communications library par excellence and this series of chapters builds upon ZeroMQ.

The Messaging Backbone: ZeroMQ

ZeroMQ Distributed Messaging solves our messaging problems. This popular toolkit offers connectivity to code in over 30 languages on multiple OSs.

Perhaps you have heard of or even tried programming with sockets, locks, semaphores and hand-crafted buffering algorithms? Did you encounter problems? Did you figure out how to debug your code? Were your tools in this area adequate? How did you discover and fix deadlocks?

I can almost guarantee you had problems that took awhile to solve. How many books did you read attempting to find an “easy” solution to this quandary? Can you strongly defend your code and its resilience or lack thereof? While you may have taken lots of time and care in testing, how solid do you think you solution stands?

ZeroMQ, also written as “0MQ” and “zmq”, has proven to provide a solid, well proven solution to networking issues. Rather than sing its praises here, please refer to You Aren’t a One Man Army: Introducing 0MQ

With ZeroMQ, all the locks, semaphores, buffering problems disappear.

OMQ is perhaps the nicest way ever to write multithreaded applications. Whereas 0MQ sockets require some readjustment if you are used to traditional sockets, 0MQ multithreading will take everything you know about writing multithreaded applications, throw it into a heap in the yard, pour gasoline over it, and set it alight. It’s a rare book that deserves burning, but most books on concurrent programming do.

To make utterly perfect multithreaded programs, we don’t need mutexes, locks or any other form of inter-thread communications except messages send across 0MQ sockets.

ZeroMQ – Messaging for Many Applications, p. 63

Debugging Tools and Advice

Specific debugging tools implement ways to ensure testing can proceed to produce solid, reliable code. Most tutorials and books paint the happy path. However, challenges almost arise when attempting to stay on this happy path while learning. Recovery can be problematic. Expect the unexpected when exploring and expanding you horizons.

Students need insights for the inevitable problems and guidance for recovery. While tutorials may yield few problems, more advanced work guarantees problems. This series will highlight recovery strategies and explanations of the possible deeper meaning.

Command Line Utilities

Command line utilities support the development and debugging effort. These utilities arise as needed. They prove their worth in life-cycle development and maintenance.

Examples

Working tutorials form a basis for future growth. While simple tutorials may appear trivial due to their bare-bones structure, they can provide quick examples of the potential of an approach. By actively engaging with the tutorials, skill emerges.

Simple copy/paste might be OK for a quick test of workability, but a serious developer needs to take a tutorial, expand on it, and stress it.

When I learn something new, I start with working tutorials with example code and expand upon the ideas. I don’t feel I’ve learned something until I make the same mistake in a different context 10 times over. Your experience may be smarter than mine.

The engaged reader definitely needs to modify and extend tutorials.

The laboratory of experiments provides the bedrock of learning.

This series develops applications typical of the needs of master/client messaging systems commonly found in Raspberry Pi networking. Sensors and devices attached to a Raspberry Pi offer monitor and control functions using these techniques.

What You Need for This Series

Development Considerations

If you like, you can develop this course on a single desktop or laptop. This allows exploration of using ZeroMQ within a single process and between processes. Of course, you could develop and test entirely a stand-alone Raspberry Pi system.

Developing on a single desktop/laptop can illustrate the power of ZeroMQ in multithreading. Messaging between multiple processes can be easier. Later, when connecting to remote systems, the messaging between local and remote processes is easy to configure.

The lack of messy systems configuration makes initial development on a single computer attractive. Later the discussion of a local DNS server and its interfacing with DHCP will add a level of messiness to any robust network configurations.

Serious developers using this series on Raspberry Pi or similar boards connect one to a primary development computer. I assume this is a desktop, laptop or even another Raspberry Pi.

Python

Python implements almost all the code for this series. Python comes with the Debian OS default installation on Raspberry Pis.

This series will use Python 2.7. While the code may or may not work for python 3.X, we choose not to confuse code variants.

ZeroMQ

On your development system, install the python hooks to ZeroMQ as described in: OSDevLab:

    sudo apt-get install python-dev python-pip
    sudo pip install pyzmq

The first line installs the development packages for python and for pip. Pip is a python utility to ease package installments.

Answer Y to any prompts and the system will crank along for a few minutes for each command.

At this point the zmq package should be available. Test this by starting python and importing the zmq package:

    Python 2.7.5 (default, Aug  4 2017, 00:39:18) 
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>>import zmq
    >>> 

If the import succeeds, that is, no error messages, then python knows about ZeroMQ and we are ready for more fun things.

If, for some reason, the install was not successful and the test appears similar to the output below, please refer to Building and Installing PyZMQ.

    Python 2.7.5 (default, Aug  4 2017, 00:39:18) 
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>>import zmq
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    ImportError: No module named zmq
    >>> 

ZeroMQ Examples

The github ZeroMQ code provides lots of examples used as a basis for explanation and extension. The python code can execute directly in the Python directory. This code directory provides a great reference.

Github.com provides such an assortment of useful software that most people have a directory dedicated to this code. Create a “my-github-dir” if you don’t have one. Of course, “my-github-dir” can be whatever name you like.

Create a github directory on you local machine and download the ZeroMQ example files:

    cd my-github-dir
    git clone https://github.com/booksbyus/zguide.git
    ... some chatter ...

Now cd to the Python files just downloaded:

    cd zguide/examples/Python

Smoke Test Your New Installation

Phew! That wasn’t too bad, was it?

We now have a python development package and a ZeroMQ package. We have the ZeroMQ packages essential for our new communications software.

Let’s test this new ZeroMQ to ensure that something, anything, works.

Under the my-github-dir directory, in the zguide/examples/Python subdirectory, find the files hwserver.py and hwclient.py. The “hw” prefix refers to the ubiquitous “Hello World”.

This smoke test requires two terminals: one for the server and one for the client. For this test, in each terminal, cd my-github-dir;zguide/examples/Python to get both terminals in the zguide python examples.

In one terminal start the client with python hwclient.py and in the other terminal start the server with python hwserver.py. The order does not matter and you should try this test both ways.

The server output should display a series of 10 messages at about one message per second:

    python hwserver.py
    Received request: Hello
    Received request: Hello
    ... additional output ignored ...

The client output displays a corresponding series of messages:

    python hwclient.py
    Connecting to hello world server...
    Sending request 0 ...
    Received reply 0 [ World ]
    Sending request 1 ...
    Received reply 1 [ World ]
    ... additional output ignored ...

Kill the hwserver.py when it pauses by a Ctrl-C. The client will exit when it finished. We will explain this necessity in a coming episode.

Congratulations!

Some really interesting work lies ahead!

We will delve into exactly what happens between hwclient.py and hwserver.py and enhance the code a bit.

We will introduce some utilities to assist in our explorations.

We will unveil potentials for development and testing that will serve us well as we deepen our explorations.

Future Episodes

This list provides a general direction as we progress in our messaging system. These points are certainly subject to change. Your input will drive enhancements to this blog!

• Get your hands dirty with our first code demo and demonstrate timing of ZeroMQ messaging.
• Setup a directory structure for further development. A logical consistency always helps.
• Break down the smoke test above into an overview of the ZeroMQ library from a high level viewpoint.
• Provide suggestions for modifying and extending the smoke test: encapsulate the smoke test into a single process, use local and remote computers, etc.
• Test Driven Development, TDD, forms the core of our development cycles.
• Several messaging patterns require explanation. Use of these patterns greatly simplifies our code. Patterns get introduced as needed.
• Create a logging application. Logging will become the extended application to develop into an advanced presentation of concepts. We believe logging provides an excellent and useful application to create and expand.
• The logging application becomes multithreaded using ZeroMQ concepts. A simple exposition of multithreading demonstrates the basics. The ideas then jell into the logging application.
• A distributed application using the logging application and database logging and analysis again demonstrates the power of ZeroMQ and the ease of implementation.
• Outlines for possible applications get presented. These include file backup, a DNS, a file server, and others.
• As sessions progress, tools, concepts and utilities support our code development. Some use standard Linux such as bash while others relate to python such as test setup, etc.
• Security. Do you want your neighbors, drive-bys of anyone else breaking into your home network? Unless you secure it, anyone could snoop about or invade your home network for whatever reason.
• A chat application. While not sophisticated, the ease of implementation should provide ease of mind in starting similar projects.

Links

All the code discussed: PyIoTMessaging in github in the PyIoTMessaging/ch04 directory.

0MQ Distributed Messaging The contact for all things ZeroMQ. This site contains the complete documentation with tutorials, community forums, etc. While purchasing the book “ZeroMQ – Messaging for Many Applications” by Peter Hintjens is highly recommended, this site contains the print book contents plus much more.

OSDevLab and similar sites for installing python ZeroMQ.

You Aren’t a One Man Army: Introducing 0MQ. A quick summary for reasons to use ZeroMQ as a basis for your messaging issues.