Strive for clarity

In 2025, I was working on Grafana Mimir 3.0, a new major version of our time series database at Grafana Labs. One of its biggest changes was a new architecture that decoupled ingestion and query paths, putting Kafka in between.

When paired with a Kafka-compatible backend running on top of object storage – like Warpstream – it suddenly became possible to run Mimir across multiple availability zones (AZs) without paying cross-AZ data transfer for the data plane. And the data plane is where most of the bytes are transferred. At around $0.02/GB in major cloud providers, cross-AZ transfer can easily become the single largest cost of running Mimir in a multi-AZ setup.

But even if the data plane stays in-zone, the control plane still needs to sync across AZs, and that part isn’t free. Mimir uses a gossip-based protocol for the control plane. Gossip is beautifully resilient to node failures, but it can be hilariously inefficient from a networking perspective.

A gossip update doesn’t spread once; it fans out multiple times. To transfer a 1 MB change set, you might end up pushing 10x, even 100x, that amount across the cluster, depending on node count and desired propagation speed. On a medium-sized Mimir cluster, you can easily hit a few tens of MB/s of cross-AZ transfer just for gossip.

Now scale that out. Suppose you run Grafana Cloud and you have 100 Mimir clusters, each doing 10 MB/s of cross-AZ data transfer. At $0.02/GB, that’s roughly $50K per month, or $600K per year — money that could fund a team of three or four people. Stakes are high.

We had an idea to fix it: split a multi-AZ Mimir cluster into per-AZ clusters, keep gossip isolated within each AZ, and then use a more efficient synchronization method between AZs that avoided write amplification. Conceptually simple — but implementation-wise? Fragile, complex, and one bug away from split-brain. Doing it robustly would take weeks.

The idea worked on paper, but my gut hated it. Too many moving parts. Too fragile. Too much that could go wrong. But after brainstorming with the team, it was the only working idea we had.

That’s when I forced myself to slow down. Instead of jumping into code and heroically “just building it”, I closed my laptop, got on my bike, and went out for a ride.

At first, I stopped thinking about it. I just enjoyed the sun, the nature, and the silence. Then, as it often happens, the problem came back to me — but with space this time, not pressure.

And suddenly it hit me: this wasn’t a cluster-splitting problem. It was a networking routing problem. We didn’t need per-AZ clusters. We didn’t need a second synchronization mechanism. We didn’t need to redesign the control plane. We just needed gossip to be zone-aware.

All we needed was to control how gossip messages propagate: pick a few nodes responsible for cross-AZ synchronization, let them push updates across zones, and let regular gossip handle everything inside each AZ. No changes to the protocol — just a smarter strategy for choosing how updates flow.

Once that clicked, the whole thing became trivial. I got home, opened my laptop, and wrote the patch, which was just a few dozen lines of code. Four working hours later, it was deployed to a development cluster.

The result was impressive: a 95% reduction in cross-AZ data transfer, with no significant impact on propagation latency. The solution wasn’t a multi-week engineering effort. The solution was an hour of coding, but I couldn’t see it earlier because I didn’t have clarity.

This story is just one example, but in hindsight, I’ve solved many hard problems the same way: stop, let the idea digest, and let clarity form before touching the keyboard. And I’ve seen the same pattern in many impactful engineers I’ve worked with.

Thinking before acting doesn’t mean moving slower. It doesn’t mean endless analysis or getting stuck in your head. It means refusing to remove thinking from the process. Good engineering isn’t just about execution — it’s about clarity before execution.

Now, I know what you might be thinking: “If I had to wait for clarity every time, I’d never get anything done.” I hear you. And you’re right.

Sometimes you have to act without perfect clarity, ship something, and let understanding catch up later. That’s fine. But that should be the exception, not the default. Strive for clarity first. Only fall back to building without it when you truly must.

So, what techniques can you adopt? Here are a few that have proven powerful over time.

Write down the problem statement, the current context, the options you’ve considered, and what you think about each. Even if you start from an existing specification or a GitHub issue, you should still write down how you understand the problem and the context.

Not to produce documentation, but to produce clarity.

The moment you try to articulate a problem precisely, with your own words, you discover all the parts you don’t actually understand. Thoughts that felt “clear enough” in your head suddenly look vague, contradictory, or incomplete when written down. Writing forces your brain to stop bluffing.

This isn’t just a personal preference. Richard Feynman famously said that “if you can’t explain something simply, you don’t really understand it”. Writing is how you force that simplicity. It exposes fuzzy reasoning and missing assumptions instantly.

Daniel Kahneman, in Thinking, Fast and Slow, explains that our brain loves to take shortcuts — jumping to conclusions, smoothing over gaps, and confidently believing half-formed thoughts. Writing breaks that spell. It switches you from fast, intuitive thinking into slow, deliberate thinking, where you’re forced to confront the actual logic of what you’re saying.

Derek Sivers captured it perfectly in his blog post How to ask your mentors for help:

This is exactly what happens when you write: you become your own mentor. You force yourself to think clearly, to be specific, to justify your reasoning. And very often, the answer quietly appears while you’re writing.

This doesn’t mean you should write a multi-page essay for every three-line bug fix. Writing has a cost too, so keep it proportional. For small problems, a few bullet points in a scratch file are enough. For big, ambiguous, or expensive problems, go deeper. The goal isn’t pretty prose — it’s to invest a little time to get clarity now so you don’t waste a lot more time later.

Your brain is a magnificent machine — fast, intuitive, creative — but your subconscious can also steer it in subtle, predictable ways. These are cognitive biases.

A cognitive bias is a systematic error in the way your brain interprets information. It’s your mind taking a shortcut — fast, effortless, but potentially wrong. These shortcuts can become traps. They distort your reasoning, hide better options, and nudge you toward the wrong conclusion with total confidence.

You can’t eliminate biases — nobody can. But you can put guardrails in place: learn to spot when they show up, pause for a moment, and adjust your thinking before they lead you astray.

What are the worst offenders? These aren’t the only cognitive biases, but they are the ones I’ve seen derail engineering decisions most often:

Cognitive biases often hide behind feelings. If you notice any of these signals, treat them as red flags:

If you notice any of these, pause. You’re likely thinking with your fast brain, not your clear brain.

So, how can you avoid these traps? A few simple techniques:

Biases won’t disappear — they’re part of being human. But once you learn to spot them, you take back control.

Clear thinking isn’t about being perfect. It’s about catching the trap before it catches you — and making decisions based on reality, not instinct.

You’re smart, but you’re also biased, limited, and occasionally blind to the obvious. Welcome to being human.

One of the most impactful ways to gain clarity is to talk through your ideas with someone else. A peer, a teammate, someone from a different team, someone with more experience, someone with less experience — it doesn’t matter. What matters is a different perspective.

Diversity of thinking, background, and experience is the whole reason teams exist. If everyone thought and acted exactly like you, a team wouldn’t have more value than a single engineer; it would just be more expensive. The magic of a team is that the value becomes greater than the sum of the parts. Different brains see different things. Someone spots an assumption you didn’t know you were making. Someone else sees a simpler path because they solved a similar problem before. Someone new to the team asks a “stupid” question that reveals complexity you’ve stopped noticing.

Brainstorming works best in small groups. Two to four people is usually ideal. In big groups, people talk less, hold back their weird-but-useful ideas, and focus more on sounding smart than thinking clearly.

This isn’t just feel-good team culture — it’s well studied. Pixar built a whole creative process around this idea with their Braintrust meetings: bright, diverse minds challenging a story from different angles, to reveal blind spots. Scott Page, in The Difference, goes even further: diverse groups often outperform groups made of “the best” individuals because they explore more of the solution space. They literally see more.

You don’t brainstorm with others to get the answer. You brainstorm to expand the problem space, uncover blind spots, and challenge the narrative forming in your head.

And this applies not just to humans.

I’ve found brainstorming with AI surprisingly useful — not because the AI magically produces perfect solutions, but because it forces me to explore the problem from angles I wouldn’t naturally consider. Some of those angles may be wrong. Some may be naive. Some may reveal misunderstandings. But wrong angles still spark new ideas.

AI is like an endlessly patient early-career engineer with infinite stamina: it throws ideas at you — some good, some bad, many forgettable — but enough interesting ones to help refine your thinking. You’re not outsourcing the problem; you’re outsourcing the angle-hunting. And sometimes that’s exactly what you need.

You don’t need a full brainstorming meeting for every trivial issue. A five-minute chat or a quick back-and-forth with an AI is often enough.

What matters is widening the angle, not chasing every idea. Talking through an idea — with a teammate, with an AI, or both — often reveals the clarity you couldn’t find alone.

If writing things down and brainstorming with others still doesn’t make the problem or the solution space clear, resist the urge to start coding right away. Pause, and step away.

Disconnect. Take a shower. Ride your bike. Run in the park. Walk your dog. Call a friend for an apéritif. Or simply switch to a different task or project — just stop actively thinking about your problem and let your subconscious do the work.

Two things happen when you do this:

Sometimes the solution clicks after a few minutes; other times it takes longer. But the result is almost always the same: when clarity comes, the implementation feels effortless. You just know what to do.

I know what you’re thinking: “My project is late! I can’t afford a break!” But hear me out. When you’re stuck, you’re not working — you’re spinning in circles. The code won’t fix itself, and neither will you by rerunning the same thought process ten times.

It feels counterintuitive. It feels like a waste of time to “wait for inspiration” while sipping coffee or working on another project. But you will spend that incubation time anyway. If you rush into building without a clear understanding, you’ll spend days rewriting, debugging, and second-guessing. That’s like a forced incubation.

So when you can, give your ideas room to breathe. Step away. Let them ferment.

Breaks aren’t laziness — they’re strategy.

Clear thinking isn’t a luxury. It’s not something you do when you “have time”. It’s the foundation of impactful engineering in a world where writing code is easier than ever, but choosing the right code to write is harder than ever.

The techniques in this chapter — writing things down, spotting cognitive traps, brainstorming with others, letting ideas incubate — are not rituals. They’re safeguards. They’re how you stay aligned with reality. They’re how you keep your work pointed in the right direction instead of sprinting into a wall at full speed.

Clear thinking prevents that.

You don’t need perfect clarity — you need enough clarity for the size of the bet you’re making. A good rule of thumb: if you can explain the problem, the constraints, your chosen option, and its main risks in a few clear sentences (or a five-minute conversation), you’re ready to act. If you can’t, you’re probably still guessing.

So before your next big decision — before you open a large pull request, approve a design doc, start a rewrite, or ship a risky change — pause for a moment. Take one clear step before you take ten fast ones. Run your idea through the habits in this chapter. Sharpen your axe before you swing it.

Because in the long run, the engineers who make the biggest impact are the ones who think clearly — and then act with a fast feedback loop.