> latency characteristics are incompatible with the producer's runtime
> handle work which doesn't need to be done transactionally and which has a high probability of needing to connect to resources whose uptime you don't control (queues make retries easy
I agree. I think you do need to break the request/response pipe for this use case.
I traditionally have treated this as a state in a database. I know there are limitations and I can see MQ being a solution. I want to try the email idea sometime.
I do think queues are a poor technology for dealing with performance problems in a web stack.
> This also isn't unique to MQs.
Agreed. My ranting about async and callbacks is an ongoing project.
I think pipes solve producer/consumer problems in a way most engineers don’t appreciate.
> While-blocking-read plays poorly with timeouts, restarts, and multiplexing/wait-any.
The primary problem solved by not using blocking primitives is to try to free up OS resources from threads (green threading). Why would an internal queue have that problem? It’s not accepting arbitrary connections.
Likewise; thanks for engaging constructively as well.
> I traditionally have treated this as a state in a database.
Which is a super appropriate tool many (most) times! I've been using "MQ" in this context refers to the conceptual capability to externalize/distribute/persist work with a push/pop API. That can be provided by a database, a UNIX pipe, or a more traditional message broker--the reasons-to-adopt and costs/benefits are largely the same.
> I want to try the email idea sometime.
Email (and the print spool) are some of the oldest message-queue primitives on UNIX systems, I think? I'm not sure if POSIX/shmem MQs predate them or not, but they're all quite venerable proofs that the pattern has its uses. I'm sure you can (maybe others already have?) use SMTP and the mailq to manage application-internal queue communication. I'm not sure how low-overhead or performant it'll be given that it's highly optimized for one/very few consumers and delivery attempts >1 being attempted on a pretty large time scale compared to application traffic, but it'd be an interesting experiment to be sure!
> latency characteristics are incompatible with the producer's runtime
> handle work which doesn't need to be done transactionally and which has a high probability of needing to connect to resources whose uptime you don't control (queues make retries easy
I agree. I think you do need to break the request/response pipe for this use case.
I traditionally have treated this as a state in a database. I know there are limitations and I can see MQ being a solution. I want to try the email idea sometime.
I do think queues are a poor technology for dealing with performance problems in a web stack.
> This also isn't unique to MQs.
Agreed. My ranting about async and callbacks is an ongoing project.
I think pipes solve producer/consumer problems in a way most engineers don’t appreciate.
> While-blocking-read plays poorly with timeouts, restarts, and multiplexing/wait-any.
The primary problem solved by not using blocking primitives is to try to free up OS resources from threads (green threading). Why would an internal queue have that problem? It’s not accepting arbitrary connections.