Quick tips for distributed event-based systems

There comes a time in every startup’s growth when you think “Wow we do be stuffing a lot of side-effects in that endpoint”. Usually placing an order.

You want an email to go out, things saved to the database, a few 3rd party systems to get notified, kick off any processing, the list keeps growing. At Tia we got up to ~20 promises called after you make an appointment, it was our most brittle endpoint. At Plasmidsaurus we’re juuuust starting to go “Where do we put this? Oh yeah, right after you place an order” 😅

Stuffing a bunch of side-effects into an endpoint makes your code slow and brittle.

Each new side-effect has, let’s say, a 0.5% chance of erroring out. Hopefully less. But lots can go wrong when calling a 3rd party or internal API. Most of it transient in nature.

Plus it’s slow. Say a great API has a 30ms response time. We’re seeing more like 150ms for our own things. Flask and Python can get pretty slow for IO-bound server workloads. But they’re great for compute.

That means stuffing 10 side-effects into an endpoint gives you a 5% error rate with a 300ms lower bound response time. You can improve the time by parallelizing those calls, but the error rate stays. Assuming you need all effects always to run.

You don’t want 5% of your Place Order calls to fail do you [name|]? That hits you right in the revenue. How many users retry after a failure? Depends what you’re selling.

The typical solution is to move to an events-based system.

1. An order was placed,
2. an event flies into the aether,
3. systems react and do their thing

Your first implementation of this can be simple:

async function placeOrder() {
	// stuff
	await Promise.allSettled([
		schedule(effect1),
		schedule(effect2),
		schedule(effect3),
		...
	])
}

Loop through your side-effects and schedule a background task on the queue for each thing that needs to happen. Can be 1 queue or many. Each effect would have its own consumer.

Putting work on the queue has 1 major benefit [name|]: Your tasks are stored.

The queue acts as persistent storage. Your queue machinery ensures tasks don’t get lost. Don’t build your own – use Celery, Kafka, or any of the popular queueing systems.

Stored tasks let you retry on error. API down? Responding with issues? That’s okay. Let your task wait on the queue and try again later.

Eventually your task will succeed and all will be well.

Have a system in place to detect poison pills – tasks that never succeed because there’s a bug. Add alerting to notify engineers if a task failed more than X number of times. Having it on the queue makes this easy to investigate.

Whole system went down for a few hours?

That’s okay. Your queues will restart processing when everything’s back. Careful of the stampede! All your queues starting back all at once is a common cause of a 2nd outage 🙂

Put as little code as possible in your scheduling. Keep your task packets small. Make your task function do all the work.

Anemic scheduler

function scheduleTask(task) {
  logger.log("Scheduling task", task)
  queue.push(task)
}

Small task

const task = {
  event: "order_placed",
  order_id: 123,
}

Smart task function

// called by queue system
function doTheTask(task) {
  logger.log("Checking task", task)
  const order = db.query(`select * from orders where id=${task.order_id}`)
  if (order.task_not_done_yet) {
    logger.log("Doing task", task)
    // do the work
    // THROW on error

    logger.log("Finished task", task)
  } else {
    logger.log("Skipping task", task)
  }
}

function rebuildQueue() {
	const orders = db.query(
		`select id from orders where task_not_done_yet`
	)
	for (const order in orders) {
		await scheduleTask(order)
	}
}