Reverse the arrow

"How long does the whole run take?"

That was the first question I asked about the design. Nobody had timed the whole run, because at eight machines nobody needs to. The answer changed the design.

The setup

The project is an R&D pilot. Eight machines on one private network run automation tasks all day, from about 10:00 to 17:00. Each machine writes two small summary files for the day: one for cost and one for anomalies. The team wants those summaries in a Slack channel every day at 17:30 Bangladesh time (UTC+6), through a webhook that already exists.

I'll call the machines workers, and the machine that collects from them the coordinator.

Version 0: every worker posts

The first version was the simplest one. At 17:30, every worker sent its own two messages to Slack.

Version 0. No coordinator: every arrow is two more messages in the channel.

With n workers, that's 2n messages a day. At eight workers it's 16, and nobody reads 16 messages to find out whether one machine had a bad day. I asked for two messages in total: one cost summary and one anomaly summary, each covering every worker.

That meant something had to collect the files first.

Version 1: the coordinator pulls

The team came back with a clear design. One machine becomes the coordinator. It connects to worker 1 over SSH (the secure shell every Linux machine on the network already accepts), reads that worker's two summary files, saves them locally and closes the connection. Then it does the same for worker 2, then worker 3. One connection at a time, never more. When it has all of them, it builds the two global summaries and sends them to Slack.

Version 1. Look at where the arrows start: every connection begins at the coordinator.

It's a good first design. It never opens more than one connection, and it never reads the full logs, because the workers already wrote the summaries. At eight workers it works. That's what makes pilot scale tricky: a design that will break at a hundred machines passes every test at eight. The task asked for two messages from eight machines, and the design delivered exactly that. Nothing in the pilot gave anyone a reason to time the run at a hundred.

The questions

When the design came to review, I first asked the room whether anyone saw an improvement. Nobody spoke. That says more about the question than about the room: "any improvements?" gives nobody a place to start. I had already drawn a revised version, and I didn't show it, or point at the problems directly. I wanted to give the team room to find them. So I asked questions instead:

1. "How long does the whole run take, from first worker to Slack?"
2. "When does the coordinator have to start, and does that change when we add a worker?"
3. "What happens when one worker doesn't answer?"
4. "What does the run cost the coordinator: CPU, memory, network?"
5. "So what would you change?"

The first two questions are the same question seen from two sides.

What breaks

Reading a small file takes almost no time. Saving it takes almost none. The time goes into the connection: opening it, the SSH handshake, authenticating, and closing it again. Our estimate was about 10 seconds per worker, with 5% of that spent on the file and 95% on SSH and the network. That's an estimate, not a measurement, and the first thing to do before building anything is to measure it. But the shape holds either way. The cost is per connection, not per byte.

At 10 seconds per worker, one at a time, the run takes 10n seconds. Eight workers take 80 seconds. Nobody notices 80 seconds.

The workers finish at about 17:00 and the summary goes out at 17:30, so the whole job has a 30-minute window: 1,800 seconds. At 10 seconds each, a one-at-a-time pull fills that window at 180 workers. Long before that, the coordinator's start time has a problem. To send at 17:30, it has to start 10n seconds earlier, and that moment slides earlier with every worker you add. Start too early and you read a worker that hasn't finished its day yet. So the start time is a setting someone has to remember to change whenever the fleet grows.

Then the third question. If worker 6 doesn't answer, version 1 has no retry. The coordinator either waits, which pushes everyone after worker 6 later, or skips it, and the summary is quietly incomplete.

The obvious fix is parallel pulls: connect to p workers at a time, and the run takes 10n/p seconds. That shortens it, but every parallel session costs the coordinator CPU, memory and bandwidth, and they add up. Push p too high and the coordinator slows down and has to be throttled. Parallel pulls make a missed deadline less likely. They don't rule it out, because a large enough n still doesn't fit in 30 minutes.

The reversal

After the questions, a teammate asked the one that mattered: what if we switch the direction?

Version 2. Same machines, same SSH. The arrows now start at the workers.

In version 2, each worker sends its own two files to the coordinator with SCP (secure copy, file transfer over SSH) when it finishes its day. The coordinator keeps a folder per worker, such as .../summaries/worker_6/, with one cost file and one anomaly file per date. Workers send in batches, sized to how many SSH connections the coordinator can take at once. If that's ten, they go ten at a time. (Ten is an example, not a measured limit.) If a send fails, the worker retries, until 17:20.

This changes who has to know what. In version 1, the coordinator had to know when every worker would be done and how long all of them would take. In version 2 it knows nothing about time except 17:20 and 17:30. Each worker knows when its own day is finished, so each worker decides when to send. Adding a worker no longer moves anyone's start time. A worker with a flaky connection retries on its own, without holding up anyone else.

The fallback

Worker 6 has been failing since 17:00. At 17:20 it stops trying. The coordinator checks which folders are missing today's files, and for those workers only, it does what version 1 did: connects over SSH and copies the files itself.

The fallback. Version 1 survives, but only for the one worker that needs it.

At 17:30 the coordinator reads the files in its own folders, builds the two global summaries and sends two messages to Slack.

The old design is still there, as the exception path. Most days it never runs, and when it does, it runs for one or two workers instead of all of them.

Why SCP and not an API

A small HTTPS endpoint on the coordinator would also work: each worker POSTs its two files. We chose SCP for one reason. SSH was already open between every machine on that network, for Ansible and for two other automation applications that run on each node. SCP adds nothing new to run, secure or keep private. An API would be one more service, with its own authentication and its own way to fail, for a few small files a day.

I'd revisit that if the coordinator ever had to accept files from outside the private network.

What we turned down

Option	Why not	When we'd switch
Every worker posts to Slack (version 0)	2n messages a day, nobody reads them	Never, for a daily summary
Coordinator pulls one at a time (version 1)	Start time depends on n, no retry, misses 17:30 at scale	Kept only as the fallback after 17:20
Coordinator pulls in parallel	Load on the coordinator grows with p, and a large n still misses the window	If the coordinator is a big machine and workers can't initiate connections
HTTPS API on the coordinator	One more service to run and secure; SSH was already there	If workers move outside the private network
A queue or object storage	More moving parts than eight, or even a hundred, workers need	If the fleet grows past about 100 workers, which we don't expect for now

What's still open

The proposal is agreed, and it isn't built yet. One question doesn't have a final answer: what happens when a worker is still missing at 17:30, after the fallback pull also failed?

My current answer is two messages. At 17:30, send the summary with what's there, and say so: "7 of 8 workers reported." Then, once every worker has submitted, send the complete report again. The channel gets one extra message on a bad day, and nobody mistakes a partial summary for a full one.

One thing to check before building: every worker decides on its own when 17:20 has passed. The machines are on one network and all set to UTC+6, but the same time zone isn't the same clock. If they don't sync time (NTP, the network time protocol), one worker's 17:20 can be another's 17:23.

What to take from it

The time in a fan-in job (many machines reporting to one) is almost never in the data. It's in the connections. And whoever starts the connections has to know when to start them. In version 1 that was one machine guessing about eight others. In version 2 each machine only has to know about itself.

The other thing I'd keep is how the fix arrived. When I asked the room for improvements, nobody spoke. After the specific questions, a teammate proposed the reversal before I showed anything.

Version 1 wasn't one person's miss. It was a sound answer to the task as it was given, and a pilot of eight machines had no way to show where it would break. The gap was in what the review asked. Once the questions were about the run at a larger fleet, the team found the fix on its own.

Three questions

1. The fleet grows to 60 workers, batches stay at ten, and on Monday three workers in the same batch all fail to push. What does the coordinator do at 17:20, and what does that cost it?
2. A worker finishes at 16:50 instead of 17:00 and pushes early. A worker in the batch after it is still running at 17:25. What does the 17:30 summary show for each?
3. Someone proposes making the coordinator pull everyone at 17:20, "to keep it simple". What would you ask them?

Answers

1. It finds three workers with no files for today and pulls those three over SSH, the version 1 way. Three pulls at the estimated 10 seconds each is about 30 seconds, well inside the 10 minutes before 17:30. The fallback stays cheap as long as only a few workers need it, which is why it's the exception path and not the plan.
2. The early worker's files are in its folder and are included. The late worker hasn't pushed, so the 17:20 fallback pulls whatever it has, which may be an unfinished day, or finds nothing. Either way the 17:30 message should say which workers were incomplete, then the complete report follows once that worker submits.
3. "How long does pulling everyone take at n workers, and does it fit in ten minutes?" At 10n seconds, ten minutes is 60 workers. That's version 1 with less time to do it in.

---

There's a 4-minute video of this review, with the diagrams moving as it's told: youtube.com/@nocturops.