Since the announcement of Broadway a couple of months ago, we’ve received a very positive response from the community. We were able to gather some great feedback regarding the architecture, suggestions for extending the API and requests for new connectors to allow attaching other technologies to Broadway pipelines. One of the most requested additions was a RabbitMQ connector that would allow Broadway pipelines to consume messages from RabbitMQ.

Today we’re glad to announce the first release of BroadwayRabbitMQ, a RabbitMQ connector for Broadway by Plataformatec.

Features

The BroadwayRabbitMQ connector ships with a custom Broadway producer that is responsible for:

• Consuming messages from a RabbitMQ queue
• Automatically acknowledges/rejects messages
• Handles connection outages using backoff for retries

Usage

In order to consume messages from RabbitMQ, you need to configure your Broadway pipeline with one or more BroadwayRabbitMQ producers:

defmodule MyBroadway do
   use Broadway

   def start_link(_opts) do
     Broadway.start_link(__MODULE__,
       name: __MODULE__,
       producers: [
         default: [
           module: {BroadwayRabbitMQ.Producer,
             queue: "my_queue"
           },
           stages: 2
         ]
       ],
       processors: [
         default: [
           stages: 50
         ]
       ]
     )
   end

   def handle_message(_, message, _) do
     IO.inspect(message.data, label: "Got message")
     message
   end
 end

If you’re consuming data from an existing broker that requires authorization, you’ll need to provide your credentials using the :connection option:

...
producers: [
  default: [
    module: {BroadwayRabbitMQ.Producer,
      queue: "my_queue",
      connection: [
        username: "user",
        password: "password",
      ]
      ...
    }
  ]
]
...

The producer also exposes other options to customize :connection and :qos configurations. For a full list of options, please see the BroadwayRabbitMQ.Producer documentation.

Next steps

If you want to give BroadwayRabbitMQ a try, we strongly recommend you to take a look at our RabbitMQ Guide. There you’ll find more details on usage and configuration, as well as valuable information about some of the design choices behind the implementation.

We’ll continue to collect feedback from the community in order to improve the tools around Broadway. We also encourage you to contribute to this project or any other in the Broadway ecosystem, either by reporting bugs or suggesting improvements. If you’re working with data ingestion/processing pipelines and you’d like to discuss GenStage, Broadway or anything Elixir related, reach out and we’ll be glad to talk.

Happy coding!