[erlang-questions] design pattern question for messaging

[Youngkin, Rich]

Hi,

An application I'm working on has a similar requirement regarding message
persistence.  We elected to use RabbitMQ.  One notable difference between
this app and the one you're describing is we're not moving a message
between states as in a work-flow model. But as Dmitry said below, "if the
message persistency is a MUST then you have to look into a reliable message
broker..." or some other way to manage persistence.

I may have missed it, but in Jesper's reply I didn't see the persistence
requirement addressed.  Jesper has a nice write-up "On Erlang, State and
Crashes that you might find useful (
http://jlouisramblings.blogspot.com/2010/11/on-erlang-state-and-crashes.html
).

Cheers,
Rich

Date: Tue, 22 Jul 2014 22:58:08 +0300
From: Dmitry Kolesnikov <dmkolesnikov@REDACTED>
To: Miles Fidelman <mfidelman@REDACTED>
Cc: erlang-questions Questions <erlang-questions@REDACTED>
Subject: Re: [erlang-questions] design pattern question for messaging
        system
Message-ID: <B61EC9A8-DC4B-48F1-84A2-D8FCE65C3994@REDACTED>
Content-Type: text/plain; charset=windows-1252

Hello,

"Except, that model kind of falls down because Erlang message are
unreliable by design, and don't persist in the event of a process crash
(much less a node crash).?

This is a trade off you have to accept. If the message persistency is MUST
then you have to look into reliable message broker solution then.

I think you already articulate the basic pattern, which is used in many
applications

 ?[ mq ]?[ worker ]?[ mq ]-[ worker ]?

You are right if you keep messages within mailbox then crash of process
destroys mailbox. Thus, you need an intermediate process to hold messages.
The intermediate process does not do any work except enqueue / dequeue
operation. Therefore, the number of failures is limited. The worker is pool
of processes Jesper Louis gave you list pool libraries, I can give one more
if you need ;-)

I am using a similar pattern at one of my application. The biggest problem
is node crash due to OOM or external factors. I am trying to solve it by
duplicating the processing path into N-distinct node (my app uses
last-write wins but you might use other conflict resolution technique).

You can extend the pattern by having persistent mq to check-point
intermediate results to survive node crash but I found it complicated.

?[ p-mq ]?[ worker ]?[ mq ]?[ worker ]?[ mq ]?[ worker ]?[ p-mq ]-

Best Regards,
Dmitry

On 22 Jul 2014, at 14:53, Miles Fidelman <mfidelman@REDACTED> wrote:

> Hi Folks,
>
> So far, I've mostly been experimenting w/ Erlang, and using Erlang-based
technology (notably CouchDB).  As I'm thinking about a new application, I'm
having trouble getting my hands around an appropriate design pattern.  I
wonder if anybody might be able to point me in the right direction.
>
> The application is message handling (back to that in a minute).  I
realize that I have a pretty good idea how to handle some kinds of
applications in a highly concurrent fashion, such as:
> - modeling/simulation (obviously, each entity - such as a vehicle - is a
process) - this is what led me to Erlang in the first place
> - protocol engines as state machines - e.g., spawn a process for each tcp
connection
> - transaction systems - spawn a process for each transaction
> - transaction oriented
>
> But I'm looking at a work flow application that maps onto a
paper-forms-based model.  It's a classic queuing system - work elements
move from queue to queue as they're worked on.  The obvious first thought
is:
> - a process for each queue
> - a worker process for each work step
> - a message for each piece of work-in-process -- moving from queue to
queue via the worker processes
>
> Except, that model kind of falls down because Erlang message are
unreliable by design, and don't persist in the event of a process crash
(much less a node crash).
>
> My first two thoughts are:
> - spawn a process for each queue entry, pass around the PIDs
> - use Mnesia to hold the queues
>
> But neither of those feels quite right.  This must be a solved problem,
but I'm hitting a blind spot.  So... what is the design pattern for queuing
systems and/or reliable message passing in Erlang?
>
> Any good examples to look at?  Good presentation slides or reference
materials to review?
>
> Thanks very much,
>
> Miles Fidelman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://erlang.org/pipermail/erlang-questions/attachments/20140723/56986fb2/attachment.htm>