DB is good solution for a low traffic application:
If you are building a simple application with low traffic, there is
something to be said about keeping another component out of your system. It
is very likely not using a message bus is the right answer for you.
However, I would suggest building your system in a way you could swap out
the database based queue system for middleware solution. I agree with the
article. A database is not the right tool for queue based system, but it
may be good enough for you.
Advantages of Queue based systems:
Queue based system like RabbitMq are built around massive scale on moderate
hardware. Their architecture is able to achieve this by avoiding processes
which make ACID compliant database system slow by their nature. Since a
Message bus -
only needs to ensure a message is stored and successfully processed,
it doesn’t need to bother with locking and writing transaction logs.
Both of these concepts are absolutely required for an ACID system but are
often a cause of contention.
Issue with DB:
Performance wise it comes down to, you have a SQL table. Lots of reads and
lots of writes. Both require some sort of locking to update rows, pages and
indexes. Your polling mechanisms is constantly locking an index to do look
ups on it. This prevents writes from happening, at best they are queued.
The code doing the processing is also locking to update the status on the
queue as they complete or fail. Yes, you can do query optimization after
optimization to get this to work, or you can use a system specifically
designed for the work load you are asking.
A RabbitMq eats up this type of workload without even breaking a sweat.
On top of that you get to save your database from the workload giving it
more room to scale doing other things.
Polling:
Another thing to consider is most queue system typically do not use a
polling technique (some allow for HTTP, but recommend to avoid using for
the receive side). RabbitMq uses network protocols specifically designed
for message buses such as AMPQ.
References:
http://mikehadlow.blogspot.fr/2012/04/database-as-queue-anti-pattern.html
brianfeucht in stackoverflow -
http://programmers.stackexchange.com/questions/231410/why-database-as-queue-so-bad
If you are building a simple application with low traffic, there is
something to be said about keeping another component out of your system. It
is very likely not using a message bus is the right answer for you.
However, I would suggest building your system in a way you could swap out
the database based queue system for middleware solution. I agree with the
article. A database is not the right tool for queue based system, but it
may be good enough for you.
Advantages of Queue based systems:
Queue based system like RabbitMq are built around massive scale on moderate
hardware. Their architecture is able to achieve this by avoiding processes
which make ACID compliant database system slow by their nature. Since a
Message bus -
only needs to ensure a message is stored and successfully processed,
it doesn’t need to bother with locking and writing transaction logs.
Both of these concepts are absolutely required for an ACID system but are
often a cause of contention.
Issue with DB:
Performance wise it comes down to, you have a SQL table. Lots of reads and
lots of writes. Both require some sort of locking to update rows, pages and
indexes. Your polling mechanisms is constantly locking an index to do look
ups on it. This prevents writes from happening, at best they are queued.
The code doing the processing is also locking to update the status on the
queue as they complete or fail. Yes, you can do query optimization after
optimization to get this to work, or you can use a system specifically
designed for the work load you are asking.
A RabbitMq eats up this type of workload without even breaking a sweat.
On top of that you get to save your database from the workload giving it
more room to scale doing other things.
Polling:
Another thing to consider is most queue system typically do not use a
polling technique (some allow for HTTP, but recommend to avoid using for
the receive side). RabbitMq uses network protocols specifically designed
for message buses such as AMPQ.
References:
http://mikehadlow.blogspot.fr/2012/04/database-as-queue-anti-pattern.html
brianfeucht in stackoverflow -
http://programmers.stackexchange.com/questions/231410/why-database-as-queue-so-bad
