Posts tagged "engineering-management" - nolan caudill's internet house

Writing a Simulator

Tue, 17 Feb 2026 13:46:32 -0800

To give myself a slightly-bigger project, I came up with the idea of writing an simulator game where you’re the CTO of a small startup with a goal of helping your company get to an IPO.

I just started it around dinnertime last night, writing up a few paragraphs with how things like projects, budgets, hiring, incidents, and so on work. I let Claude and Codex rip on a planning (“here’s some words. now ask me questions.”) and implementation loop. After every loop, I’d play around with it, think of a thing that’d make it better, and repeat.

This post isn’t about writing software with agents (though I am amazed and am learning a lot about what they are good at and not so capable of) but a more general thing I’m learning when you’re writing a simulator.

Writing this simulator is as much encoding how I think the job of engineering manager works than anything else. If someone said, “How would you do the mechanics of the job?,” this would be an interactive answer to that question which is neat in a meta way.

When thinking through the elements of the game and how they work and interact, I started to use (my many) years of seeing these things up close to model how I think they should behave.

The first part of writing the simulator was mostly coming up with the nouns, the physical objects: projects, engineers, budgets, incidents and then, what sort of attributes they’d have. Projects have a timeline, whether or not they are started or not, which team they are assigned to and other things that seemed relevant to the flow of the game.

But, thinking harder, I started to think of things that “real” projects have. They have a distribution curve of when they’ll actually finish. The individuals that start on a project might not be the same set throughout. And, when you assign a project to an already busy team, each of project takes a little longer (though you as the CTO/manager might not know that until too late).

I realized that writing a simulator about something you know well, you quickly see where the simulator isn’t correct and you add that in. And, before long, I realized I haven’t written a generic “you’re the CTO” game but instead an interactive way to explore how I specifically see the job. I believe someone else given this prompt would pick different things to emphasize, objects interacting differently, and likely even different success criteria.

(Also, I don’t think the game is “fun” yet but I’m not sure a simulation of a fairly bureaucratic job would be fun? If it does up being fun, or at least interesting in my eyes, I’ll likely open source it but I’m not there yet.)

The other part of this exercise that is interesting is that the game does have some emergent properties. For example, new engineers have some amount of ramp before they are fully productive. New engineers on a project might take longer or make more mistakes (which is natural!). Newer engineers add some risk to projects and missed projects hurt the CTO’s standing and lowers the morale of the team. Each of these consequences follow naturally based on some simple rules, but when interacting together, every game is slightly different and somewhat surprising.

Three Levels of Developer Productivity

Fri, 11 Feb 2022 22:28:31 +0000

If you work on developer productivity, you sit in an unique spot. You are often given a broad mandate — “make the engineers more productive”-and typically little guidance on what the word ‘productive’ means. (Spoiler: There’s no one right answer here.)

To start to give some shape to the job, in my experience, there are two important categories of problems you are solving. There are other categories, of course, but these two feature prominently and will drive much of the work.

The first problem is one of engineering morale. The tools engineers use for their work factor greatly into the satisfaction they get from their jobs. If you place too high of a hurdle between an engineer making a change and them seeing the impact of that change, or if you make the risk of making those changes too great, you will sap people’s satisfaction. People do quit over slow build times and people do quit over dangerous deploys.

There’s also a breadth of research that show that teams that have fast feedback loops are higher performing. While this research may speak to many, what I’ve found to be universally resonant is the ethos of “don’t frustrate engineers with bad tools and processes.” Nearly everyone can get behind that.

The second problem, in coarse terms, is engineering return on investment. There is a good chance that one of your company’s biggest expenditures is engineering headcount. And every new feature shipped and every bug fixed is generally gated by an engineer doing something. Therefore, engineering cycles are one of the most debated, negotiated and protected resource in any tech company and the bulk of processes have their roots growing out from the perceived scarcity of those engineering cycles. There will always be a natural pressure, from all angles of the business, to get more from less, to speed engineering up, to help engineering do “more” and whether said directly or not, part of your job is to understand this and make improvements here.

If you’ve been around larger organizations, you’ll quickly see that some of these productivity problems can be solved with engineering elbow grease, but it’s not hard to see that engineering as a whole is often slowed down by the sheer coordination overhead that comes naturally to complex organizations running complex technical systems.

The challenge is that improving build times will make a small dent if any on coordination costs (unless you are starting on one extreme and moving it another extreme), leaving the “engineering is the bottleneck” problem largely unaddressed. But, when you start to focus on the second problem, the work isn’t as easily understandable to others.

To make the work more understandable, a framework I’ve had some success with is to explicitly put developer productivity efforts under 1 of 3 different banners: individual productivity, team productivityandteam-of-teams productivity.

Individual productivity: In my experience, this category of work is what is typically seen as the core duty of developer productivity teams. This work includes efforts like speeding up build times, reducing test flakiness, increasing deploy cadence, making debugging easier and so on. The primary focus is on the individual developer and moving the code they author through the pipeline faster and more reliably.

There is tremendous value in this category. If you can take a routine process that every engineer is involved in multiple times per day and speed it up by 10%, that’s a huge number of valuable engineering cycles that are freed up, making a scarce resource that much less scarce, speeding up valuable feedback loops and making engineers enjoy their jobs a bit more.

And, at some point, some of these processes hit a phase change where because something has become so cheap to do, it encourages a whole lot more of that thing to happen. Think of the difference between a full build taking a day and that same build taking 10 minutes, or a weekly deploy becoming on-demand. The entire economics of your engineering processes change and it has knock-on impact across the business.

**Team productivity:**This category focuses on improving production by focusing on the flow of work between engineers, and usually engineers within the same team.

This is the realm of team processes: how does code get reviewed, how many projects can one team undertake, how does a backlog get managed and so on. This is where words like “agile” and “scrum” get mentioned.

In my experience, individual teams often have a lot of leeway in how they manage themselves. Where bad tales arise is often from a top-down mandate to “become agile.” There is also value here but it is riskier and efforts here will falter unless the most senior leadership is actively involved, you have a network of coaches throughout the org and you are willing to take a short term (eg, 3 to 6 months) dip in productivity while teams learn to operate differently and tweak their processes. Lining up and sustaining these preconditions is tough and there is no one weird trick to pursue here.

One approach to this level of productivity is to focus on the handful of teams that are recognized as a shared resource (eg, SRE, centralized tools teams, platform teams) and are on the critical path for some key projects and then help improve how they take in, prioritize and execute their work. You’ll likely get a large percentage of the value of “becoming agile” mandate while not actually having to spend the time and effort to make every team implement Scrum, as an example. Your org likely has a few critical teams that are the bottlenecks or dependencies on many efforts and there is high leverage in improving those specific teams’ processes.

**Team-of-teams productivity:**Problems here possess the delightfully confounding combination of being harder to see, requiring a delicate touch, are usually seen as “not your job” and are notoriously hard to measure. Yet, in my opinion and experience, making headway on these problems is where real leverage lies, not just for the engineers but for the whole business.

This level is the realm of quarterly/yearly planning processes, org design, platform migrations and other processes that are expensive, hard to change and slow. Simply because these things are expensive, hard to change and slow make any improvements in efficiency here incredibly valuable.

One example: Migrations are a fact of life of larger and older orgs. At any point, there are likely many migrations any individual team could be involved in, or that the organization would like to be doing. This presents a tradeoff: is it better to focus on one migration at a time, or to spread your efforts across many migrations? In general, migrations have little value mid-migration — you have the overhead of living in two worlds simultaneously plus the overhead of the migration itself. So reducing in-progress time should be a priority. This leads naturally to focusing teams on a single migration, which reduces overhead, reduces technical complexity, reduces status checks and reduces context switches, which are all good things.

So in that example, a team-of-teams productivity improvement would be to cap the number of in-flight migrations occurring at any one time.

The ways to make improvements at this level is by having relationships with pivotal engineers and managers, getting involved at the right time (eg, quarterly or yearly planning) and making a case for a productivity-improvement constraint when stakes are fairly high and involve influential people with differing goals and motivations. Nudges and tweaks to existing processes typically work much better than large campaigns.

When the “CI/CD person” shows up to suggest improvements to quarterly planning processes, using the framing of improving team-of-teams productivity, with a focus on overall program and organizational effectiveness to help the whole organization deliver more value sooner, helps explain the motivation behind the suggestions.

— — -

Working on developer productivity can be hugely rewarding and there is a lot of value that comes from working on it at different levels, even outside the typical domains of automation and CI/CD.

Hello, Slack

Sat, 28 Jun 2014 07:00:00 +0000

Two weeks ago, I started at Tiny Speck as their engineering manager, working on Slack. Slack is getting bigger in almost every way that matters and I’m excited about getting to take part in it.

So what does taking this particular role at this particular company mean to me? It means I’m working for the same people that built the company that hired me that moved me from the east coast to San Francisco. It means I’m managing at a company that shaped most of my thoughts about software development and how to build products that people love. It also means I get to take those experiences and principles and help build frameworks where we keep doing those good things but at a different scale than we are all used to.

So, is this a little scary? You betcha. Am I excited? Oh yeah.

Shipping Software

Wed, 15 Jul 2009 07:00:00 +0000

Shipping ordinary software on time is damned hard. Shipping great software in any time frame is extraordinary. Shipping great software on time is the rarest of earthly delights. – Jim McCarthy, Dynamics of Software Development

I received my copy of Dynamics of Software Development by Jim McCarthy in the mail today, and this quote was just sitting there on the first page.

I’ve shipped great software on time and I’ve shipped ordinary software late. This inconsistency to a logic-driven mind is maddening. I think I’m fairly decent at writing code, but getting the whole process to a point of repeatable success from a business standpoint is what I’m in the dark on.

Though my computer science education was fantastic in teaching me about logic and theory, the nuts and bolts of software engineering is something I am lacking. There was a software engineering course offered but I opted for the programming language design course, instead. I don’t regret that decision–I love programming languages–but in the world of waiting customers and deadlines that come much too quickly, I’m having to play catch up.

So for now, it’s time to hit the books.

Truth about Software

Thu, 25 Jun 2009 07:00:00 +0000

I spend a lot of my time thinking about software develoment, keeping up with what works for other people and what works for me. I’ve recently read Peopleware and Code Complete, and I’ve formulated some thoughts about software development and technical management that I’ve been meaning to write about. Hopefully, this copy-and-paste will get me going with that.

I found this printed out on Hacker News, and I think there is a lot of truth in it about the perceptions of software and its development.

The problem with IT, generally, is lack of respect. Because it’s something that looks easy and looks like something anyone could do, people don’t respect experience and talent.

For some reason this just doesn’t work in other fields. While everyone feels they can put a sticking plaster on a cut, they go to a doctor for more serious stuff. Any fool can build a garden wall, but if they want a tower block one hires an architect and a construction company and lets them do decisions.

Whereas, it’s quite common in IT for micromanagers to overrule the professionals.

It doesn’t happen in building. Managers simply aren’t allowed to say things like “Oh, I don’t think we should RSJs in those load bearing walls”. If the architect says they’re needed, then they go in. We also don’t hire doctors by picking the tools they’ll use. When you want a doctor to take out an appendix, you want a qualified general surgeon. You don’t ask them whether they prefer a #3 or a #4 handle on their scalpels.

And yet it’s pretty common to see projects which have picked a technology first (“Oh we’re going to use J2EE for this”) and THEN hire the team around that (“Wanted, tech arch, 5 years exp with J2EE, C , C++, JAVA, Perforce, Apache, Perl, Python.”) and only at that point produce the spec. And then if the job isn’t one the technology is suited for, that ends up being the developers fault and they’re the ones who work overtime to put that square peg in the round hole.

It’s scarcely surprising that a) almost nothing works properly, b) there’s constant chaos and that therefore c) almost no-one finds their work rewarding in ways other than money.

I freely admit that a lot of my time, I feel like some sort of thief. I sit in projects which are doomed. They’re more doomed than a plane that’s lost both wings and is on fire. I see the people around the project running about setting more of it on fire as it hurtles towards the ocean and removing more control surfaces and making it worse. And I watch developers switch in and out of seats as we trail wreckage and smoke down. And it’s always been like this. The project is a free-fall disaster before I join and after I leave and there’s just nothing, nothing, nothing I can do to rescue it at that point in time. And yet I’m being paid to be there.

And I feel bad about that. I feel that I ought to be being paid to achieve something. But usually I’m just being paid to sit in a seat in a vehicle performing a ballistic trajectory straight into its crash site. And even if I quit then the next thing will be the same and whoever takes my place in this seat will be in the same situation.

And this is not how I wanted to work. I wanted to build things that people wanted. Not turn up at an aircrash and fill a seat in it for a while. And while it’s financial fulfilling, it’s not very emotionally fulfilling.

Most bridges don’t fall down. Most patients don’t die. Most IT projects are a failure in one way or another. It’s like being a surgeon back in the days before anaesthetics or antibiotics. Most of our patients die… and that’s got to be bad for morale.

And I can’t help but think that this really has its roots in the fact that people think software is simple enough that they can exercise control over it at a level they ought not to be and that they also don’t want to pay what it actually costs. Architects generally don’t compete on price and you don’t get to tell architects that their idea of how strong structural steel is is an underestimate you can feel you can ignore in this project. But people time and time and time again pick the cheapest option for software, design it like they’d design a tin shack and then act surprised when the end results turns out to be flimsy tin shack instead of a tower block.

Originally from http://discuss.joelonsoftware.com/default.asp?joel.3.762361.7, it appears.