Outboxer

A transactional outbox for Elixir and Ecto.

Write a message to the outbox in the same database transaction as your business data. Outboxer relays it to Kafka, SQS, Oban or your own sink, in order, retrying until it gets through.

Repo.transaction(fn ->
  order = Repo.insert!(changeset)
  {:ok, _} = Outboxer.publish(%OrderPlaced{order_id: order.id, total: order.total})
  order
end)

If the transaction commits, the message will be delivered. If it rolls back, the message never existed. That is the guarantee "save to the database, then publish to the broker" cannot give: a crash between the two steps either loses the message or publishes one for data that was never saved.

Features

• Postgres and SQLite storage, behind a behaviour.
• Ordered groups. Messages sharing a group key are delivered in order. Ordering is optional per message and per outbox, and a whole outbox can be strictly ordered.
• Strict by default. A failing message blocks its own group (and nothing else) until it goes through. Opt into dead-lettering after N attempts when you prefer progress over order.
• Relays for Kafka (brod), SQS (ex_aws) and Oban, each compiled only when you have its dependency, plus a function relay and a small behaviour for writing your own.
• Small footprint in your database. A delivered message costs one INSERT and one DELETE. No claim/lease updates, and indexes that cover only the backlog.
• No global configuration and no application. Everything is passed to {Outboxer, opts} in your supervision tree, and only the processes that configuration needs are started.
• Multi-node safe on Postgres, with no leader election and no leases to expire.
• Benchmarked. mix bench runs against a real database; see Performance.

Installation

def deps do
  [
    {:outboxer, "~> 0.1"},

    # the driver your Repo already uses
    {:postgrex, ">= 0.0.0"},        # or {:ecto_sqlite3, ">= 0.0.0"}

    # add only the relay you need
    {:brod, "~> 4.0"},               # Outboxer.Relay.Kafka
    {:ex_aws_sqs, "~> 3.4"},         # Outboxer.Relay.SQS
    {:oban, "~> 2.18"}               # Outboxer.Relay.Oban
  ]
end

A relay module exists only if its dependency is in your project when Outboxer is compiled. If you add one later, run mix deps.compile outboxer --force.

Requires Elixir 1.18 or later.

Quick start

1. Create the table.

defmodule MyApp.Repo.Migrations.AddOutboxer do
  use Ecto.Migration

  def up, do: Outboxer.Migration.up()
  def down, do: Outboxer.Migration.down()
end

2. Define a message.

defmodule MyApp.Events.OrderPlaced do
  use Outboxer.Message, outbox: :orders

  defstruct [:order_id, :total]

  # Optional. Messages with the same group key are delivered in order.
  @impl true
  def group_key(%__MODULE__{order_id: id}), do: "order:#{id}"
end

3. Add Outboxer to your supervision tree, after the repo.

children = [
  MyApp.Repo,
  {Outboxer,
   repo: MyApp.Repo,
   outboxes: [
     orders: [
       relay: {Outboxer.Relay.Kafka, client: :my_brod_client, topic: "orders"},
       partitions: 4
     ]
   ]}
]

4. Publish inside your transaction, with Outboxer.publish/2 as above or in an Ecto.Multi:

Ecto.Multi.new()
|> Ecto.Multi.insert(:order, changeset)
|> Outboxer.publish_multi(:event, fn %{order: order} ->
  %OrderPlaced{order_id: order.id, total: order.total}
end)
|> Repo.transaction()

Two common layouts

One table per outbox. Each outbox can have its own table; without table they share outboxer_messages. Create each with Outboxer.Migration.up(table: "order_outbox").

outboxes: [
  orders:   [table: "order_outbox",   relay: {Outboxer.Relay.Kafka, client: :my_brod_client, topic: "orders"}],
  invoices: [table: "invoice_outbox", relay: {Outboxer.Relay.SQS, queue_url: "..."}]
]

Publish in one pod, relay in another. Start the same outboxes everywhere and turn delivery off where it is not wanted. A relay: false node starts no processes and needs no broker settings or dependencies.

{Outboxer,
 repo: MyApp.Repo,
 relay: System.get_env("OUTBOXER_RELAY", "true") == "true",
 outboxes: [...]}

See Storage and Operations for the details.

Try it

A demo that needs nothing but this repository (SQLite, no broker):

mix deps.get
mix run examples/demo.exs

It publishes order events in transactions, rolls one back, and relays through a sink that fails one time in four, so you can see ordering and retries.

What it is good at, and what it is not

Good at

• Never losing a message that belongs to a committed transaction, and never emitting one for a transaction that rolled back.
• Keeping per-entity order (all events of one order, one account, one aggregate) end to end.
• Staying out of the way: the table stays close to empty, and the library is rarely the bottleneck. See Performance.

Not good at / not for

• Exactly-once delivery. Delivery is at-least-once. Consumers must tolerate a duplicate; every message carries a stable id for that.
• Very high throughput streaming. Every message is a row written and deleted in your primary database. For sustained high volume, log-based change data capture (Debezium, WalEx) adds no writes and scales further.
• Large payloads. The payload lives in your database until it is relayed. Store a reference.
• Low-latency request/response. Latency is "next poll" (default up to 1 s), or a few milliseconds with notify: true on Postgres.
• Delayed or scheduled messages, priorities, fan-out to several relays from one outbox. It is an outbox, not a job queue. Use Oban for those (possibly fed by Outboxer.Relay.Oban).
• Databases other than the one your business data is in. The guarantee comes from sharing a transaction. See the Storage guide for why Redis does not qualify.

The guides go into each of these:

• Overview: the pattern, the guarantees and the trade-offs
• Choosing a setup: outboxes, tables, partitions, notify and where the relay runs
• Ordering and groups
• Storage: Postgres, SQLite, and why not Redis
• Relays: Kafka, SQS, Oban, writing your own
• Operations: sizing, table bloat, monitoring, stuck and dead messages
• Testing

Performance

Measured with mix bench, 200-byte payloads, and a relay that only counts, so these are the outbox's own costs:

The run was on this setup:

A poll costs the same at any backlog depth, and relay speed stays flat from the first message of a backlog to the last. Real throughput is set by your database and the destination, so run it on your own hardware:

docker compose up -d postgres
mix bench                    # Postgres
BENCH_REPO=sqlite mix bench  # SQLite

The benchmark is bench/outbox_bench.exs. The full results, including the slow spots, are in Operations.

Development

mix setup
docker compose up -d postgres      # Postgres for the test suite
mix test                           # SQLite + Postgres, relays against fake clients
mix test --exclude postgres        # SQLite only, no Docker
docker compose up -d               # + Kafka and MiniStack (SQS)
mix test.integration               # everything, against the real services
mix precommit                      # compile (warnings as errors), format, credo --strict, test

Building brod's dependencies needs cmake (brew install cmake).

License

MIT. See LICENSE.