Overview

This blog will talk about system scalability with relation to microservices and how a monolithic application would be less desired in this example.

GitHub repositories

All of the code relating to this blog post can be found at the following links and is released under the GPL-3.0 license, with the exception of the official NASA api which is released under the Apache-2.0 license.

  • Official NASA APOD API - The official NASA API for getting the Astronomy Picture of the Day.
  • APOD - The API wrapper we use to query the NASA api.
  • Apod-to-MQTT - This service uses the api wrapper above to actually do the query.
  • Apod-MQTT-to-Discord - This service reads off an MQTT subscription and publishes the messages to Discord.

System Architecture

Given the simplicity of this system I haven’t created an actual diagram and you should be able to get the general gist of the system from the above section, but in short Apod-to-MQTT runs on a cronjob to query the official NASA API using the APOD wrapper we wrote. Once it gets the daily picture it sends it to an MQTT broker, we then have Apod-MQTT-to-Discord that is subscribed to the MQTT topic and sends the daily picture to Discord.

System Scalability

When architecting this system the goal was for highly scalable consumers, by that I mean I wanted to be able to create any number of backends to post the Astronomy Picture of the Day to many different places.

This system setup should support any number of backends and scale nearly indefinitely. The service that queries the NASA API for the picture and sends it to MQTT will work perfectly regardless of the number of backends we have, any backend we have will function independently of eachother so no issues there either.

The only system we have to worry about is the MQTT broker but given how lightweight some of the MQTT servers out there are we should have no issue even if we scale to thousands of backends, however unlikely that is.

Differences from a Monolithic Application

This service could have written as a single application that provided all the functionality without needing MQTT. Eg: An application either constantly running or on a cronjob that takes the Astronomy Picture of the Day and sends it directly to Discord.

The downside of a monolithic application would be the difficulty in scalability for this kind of a setup. You could easily add functionality to send the picture of the day to a few more places, but every time you make edits you’ll need to test to make sure nothing broke and then update the service in production. The more you add, the more you have to test and ensure is not broke prior to updating in production.

This is in contrast to the Microservices setup where once you have a working backend to handle the image you never need to test that service when adding new ones, the only thing that needs to be tested is the service your currently working on implementing and once it’s finished, you’re all set.

We also have to consider system resources. While handling a couple of backends a monolithic application would be of similar resource requirements to our microservices setup but as the number of backends grow our monolithic application would need to scale vertically to meet the requirements. Our microservice setup on the other hand can either scale vertically or horizontally depending on how we’re adding resources.

Conclusion

While definitely not necessary for such a small service, this was a fun way to get practical hands on experience with MQTT and gave reason to write this simple blog post on Microservices.

At the time of writing this the service isn’t 100% complete, we’re still lacking authentication for the MQTT brokers pub/sub functions, but for what I wanted to learn and write about this was enough. I will at some point finish out the setup proper but it wont be right off.

Pull requests are welcome from anyone that wants to write in authentication before I get around to it!

Other fun idea with MQTT

When coming up with the idea for this service I had also thought about some other sciencey services I could write to use MQTT, one of which was an earthquake monitoring system. General architecture would be the same but we’d use multiple topics for the severity, eg: earthquake/sev_1, earthquake/sev_2 … earthquake/sev_9. The thought being that backends could listen to the severity of earthquakes they wanted and process them accordingly, for example a twitter bot might want to post anything 2 and higher but an email alert system might only want 6 and higher.