Why you might need a messaging queue and why RabbitMQ is a winner!
One of the hard parts of development is using high-level language to describe a problem so that you can find existing solutions rather than re-inventing the wheel.
A low level discussion might involve the fact that we want an event-based system that allows lots of subscribers to act on events in their own way without killing a database in the process. We also need some fault-tolerance since some of these events might e.g. cause a push of data into Salesforce and might fail and need to retry, possibly significantly later.
The problem with the low level discussion is we are possibly already assuming we might use C# events, a SQL database table with some consumer application that reads the table and acts on the information.
A high level discussion might instead be, We need an asynchronous publish/subscribe message system with a buffer facility and visibility of messages that could not be processed in a timely fashion or at all.
At a high level, we are talking about a message queue and one does already exist: RabbitMQ. Of course, others are available including cloud service based products but RabbitMQ is open source with paid support (my favourite model for onboarding - who wants to agree rental payments before you really know if it works?).
I have had a play and it is FAST and also pretty easy to setup with the right instructions.
Firstly, I used a docker image - nice and quick and easy although the official instructions don't make it clear that you need to map the docker imager ports to the host ports in order to access the management interface from your host browser. The correct command is:
docker run -d --hostname my_hostname --name node_name -p 15672:15672 -p 5672:5672 rabbitmq:3-management
This was literally all I had to do to get it working and access the instance with the default credentials (guest:guest).
After this, I found a simple tutorial for .net (http://www.andyfrench.info/2017/08/quick-rabbitmq-using-docker.html) which explains that you can either create the exchange from code (it is implicitly created when you send a message if it doesn't exist) or you can create it via the management interface.
The only thing to be aware of is that there are different topologies for Rabbit systems (https://www.rabbitmq.com/getstarted.html) and they require slightly different setup in the client. For example, a simple work queue can use a named queue whereas a pub/sub system generally does not need to care about names and can use auto-generated ones instead. Also, some queues will be deleted automatically if there is no longer a consumer on the end of it.
It's fairly easy to build something based on similar code to Andy French's tutorial and demonstrate the different behvaiour of topologies.
I wrote a windows forms app for a demo and it was so fast that as soon as you pressed the button to send a message, the receivers instantly updated their text panels as if the text was inserted directly. I was running on a local machine with a docker image but it was still impressive.
I also played around with durable queues which means that if the consumer dies and/or the agent is restarted, the messages are not lost. Connecting the consumer again causes the buffered messages to be downloaded.
The last thing I want to try is how to retrieve a message without deleting it from the queue and only mark it as completed once the other work has been completed e.g. once we have pushed the message contents into Salesforce or whatever.
A low level discussion might involve the fact that we want an event-based system that allows lots of subscribers to act on events in their own way without killing a database in the process. We also need some fault-tolerance since some of these events might e.g. cause a push of data into Salesforce and might fail and need to retry, possibly significantly later.
The problem with the low level discussion is we are possibly already assuming we might use C# events, a SQL database table with some consumer application that reads the table and acts on the information.
A high level discussion might instead be, We need an asynchronous publish/subscribe message system with a buffer facility and visibility of messages that could not be processed in a timely fashion or at all.
At a high level, we are talking about a message queue and one does already exist: RabbitMQ. Of course, others are available including cloud service based products but RabbitMQ is open source with paid support (my favourite model for onboarding - who wants to agree rental payments before you really know if it works?).
I have had a play and it is FAST and also pretty easy to setup with the right instructions.
Firstly, I used a docker image - nice and quick and easy although the official instructions don't make it clear that you need to map the docker imager ports to the host ports in order to access the management interface from your host browser. The correct command is:
docker run -d --hostname my_hostname --name node_name -p 15672:15672 -p 5672:5672 rabbitmq:3-management
This was literally all I had to do to get it working and access the instance with the default credentials (guest:guest).
After this, I found a simple tutorial for .net (http://www.andyfrench.info/2017/08/quick-rabbitmq-using-docker.html) which explains that you can either create the exchange from code (it is implicitly created when you send a message if it doesn't exist) or you can create it via the management interface.
The only thing to be aware of is that there are different topologies for Rabbit systems (https://www.rabbitmq.com/getstarted.html) and they require slightly different setup in the client. For example, a simple work queue can use a named queue whereas a pub/sub system generally does not need to care about names and can use auto-generated ones instead. Also, some queues will be deleted automatically if there is no longer a consumer on the end of it.
It's fairly easy to build something based on similar code to Andy French's tutorial and demonstrate the different behvaiour of topologies.
I wrote a windows forms app for a demo and it was so fast that as soon as you pressed the button to send a message, the receivers instantly updated their text panels as if the text was inserted directly. I was running on a local machine with a docker image but it was still impressive.
I also played around with durable queues which means that if the consumer dies and/or the agent is restarted, the messages are not lost. Connecting the consumer again causes the buffered messages to be downloaded.
The last thing I want to try is how to retrieve a message without deleting it from the queue and only mark it as completed once the other work has been completed e.g. once we have pushed the message contents into Salesforce or whatever.