Kafka relay

Outboxer.Relay.Kafka produces to Kafka with brod. It is compiled only when :brod is among your dependencies.

{:brod, "~> 4.0"}

brod's dependencies include a NIF that needs cmake at build time.

Setup

Use a brod client you already start:

orders: [relay: {Outboxer.Relay.Kafka, client: :my_brod_client, topic: "orders"}]

or let Outboxer start and supervise one, by giving it endpoints:

orders: [
  relay:
    {Outboxer.Relay.Kafka,
     client: :outboxer_orders,
     endpoints: [{"kafka-1", 9092}, {"kafka-2", 9092}],
     client_config: [ssl: true, sasl: {:plain, "user", "secret"}],
     producer_config: [required_acks: -1, compression: :snappy],
     topic: "orders"}
]

The client is a child of that outbox's supervisor and is started before its workers. A topic's producer is started the first time the topic is used, with :producer_config.

Route by message with a function or MFA:

topic: fn envelope -> "events." <> envelope.type end
topic: {MyApp.Routing, :topic_for, []}

A group must always resolve to the same topic, or its order is only kept per topic.

Mapping

Good to know

• Order. Same group key, same Kafka partition, sent with one synchronous request per topic-partition and acknowledged before the rows are removed. Kafka then keeps that order for consumers.
• Keep required_acks: -1 (the default here). With 1 or 0 a broker failover can lose messages Outboxer already deleted.
• Duplicates. A batch can be produced twice after a failure between "acked" and "deleted". Deduplicate on the outboxer-id header if you need to.
• Partitioning is done by Outboxer, with :erlang.phash2/2, not by brod and not with Kafka's default murmur2. Other producers writing the same keys to the same topic will not agree on the partition. Pass a :partitioner function if they must.
• Adding partitions to a topic moves keys, as with any hash partitioner. Drain the outbox first if order across that change matters.
• Outbox partitions and Kafka partitions are unrelated. The first is how many workers deliver in parallel, the second is the topic's layout.
• Message size. A batch that exceeds the broker's limits fails as a whole and is retried as a whole. Lower batch_size if your payloads are large.

Not a fit when

• You need Kafka transactions or exactly-once semantics across topics.
• You produce at a volume where an extra row per message strains the database: it, not Kafka, becomes the limit. Consider CDC.