Effective Engineering Productivity

Often, people who work on engineering productivity either come into conflict with the developers they are attempting to help, or spend a long time working on some project that ends up not mattering because nobody actually cares about it.

This comes about because the problem that you see that a development team has is not necessarily the problem that they know exists. For example, you could come into the team and see that they have hopelessly complex code and so they can’t write good tests or maintain the system easily. However, the developers aren’t really that aware that they have complex code or that this complexity is causing the trouble that they are having. What they are aware of is something like, “we can only release once a month and the whole team has to stay at work until 10:00 PM to get the release out on the day that we release.”

When engineering productivity workers encounter this situation, some of them just try to ignore the developers’ complaints and just go start refactoring code. This doesn’t really work, for several reasons. The first is that both management and some other developers will resist you, making it more difficult than it needs to be to get the job done. But if just simple resistance was the problem, you could overcome it. The real problem is that you will become unreal and irrelevant to the company, even if you’re doing the best job that anybody’s ever seen. Your management will try to dissuade you from doing your job, or even try to get rid of you. When you’re already tackling technical complexity, you don’t need to also be tackling a whole company that’s opposed to you.

In time, many engineering productivity workers develop an adversarial attitude toward the developers that they are working with. They feel that if the engineers would “just use the tool that I wrote” then surely all would be well. But the developers aren’t using the tool that you wrote, so why does your tool even matter? The problem here is that when you start off ignoring developer complaints (or don’t even find out what problems developers think they have) that’s already inherently adversarial. That is, it’s not that everything started off great and then somehow became this big conflict. It actually started off with a conflict by you thinking that there was one problem and the developers thinking there was a different problem.

And it’s not just that the company will be resistive—this situation is also highly demoralizing to the individual engineering productivity worker. In general, people like to get things done. They like for their work to have some result, to have some effect. If you do a bunch of refactoring but nobody maintains the code’s simplicity, or you write some tool/framework that nobody uses, then ultimately you’re not really doing anything, and that’s disheartening.

So what should you do? Well, we’ve established that if you simply disagree with (or don’t know) the problem that developers think they have, then you’ll most likely end up frustrated, demoralized, and possibly even out of a job. So what’s the solution? Should you just do whatever the developers tell you to do? After all, that would probably make them happy and keep you employed and all that.

Well, yes, you will accomplish that (keeping your job and making some people happy)…well, maybe for a little while. You see, this approach is actually very shortsighted. If the developers you are working with knew exactly how to resolve the situation they are in, it’s probable that they would never have gotten themselves into it in the first place. That isn’t always true—sometimes you’re working with a new group of people who have taken over an old codebase, but in that case then usually this new group is the “productivity worker” that I’m talking about, or maybe you are one of these new developers. Or some other situation. But even then, if you only provide the solutions that are suggested to you, you’ll end up with the same problems that I describe in Users Have Problems, Developers Have Solutions. That is, when you work in developer productivity, the developers are your users. You can’t just accept any suggestion they have for how you should implement your solutions. It might make some people happy for a little while, but you end up with a system that’s not only hard to maintain, it also only represents the needs of the loudest users—who are probably not the majority of your users. So then you have a poorly-designed system that doesn’t even have the features its actual users want, which once again leads to you not getting promoted, being frustrated, etc.

Also, there’s a particular problem that happens in this space with developer productivity. If you only provide the solutions that developers specify, you usually never get around to resolving the actual underlying problems. For example, if the developers think the release of their 10-million-lines-of-code monolithic binary is taking too long, and you just spend all your time making the release tools faster, you’re never going to get to a good state. You might get to a better state (somewhat faster releases) but you’ll never resolve the real problem, which is that the binary is just too damn large.

So what, then? Not doing what they say means failing, and doing what they say means only mediocre success. Where’s the middle ground here?

The correct solution is very similar to Users Have Problems, Developers Have Solutions, but it has a few extra pieces. Using this method, I have not only solved significant underlying problems in vast codebases, I have actually changed the development culture of significant engineering organizations. So it works pretty well, when done correctly.

The first thing to do is to find out what problems the developers think they have. Don’t make any judgments. Go around and talk to people. Don’t just ask the managers or the senior executives. They usually say something completely different from what the real software engineers say. Go around and talk to a lot of people who work directly on the codebase. If you can’t get everybody, get the technical lead from each team. And then yes, also do talk to the management, because they also have problems that you want to address and you should understand what those are. But if you want to solve developer problems, you have to find out what those problems are from developers.

There’s a trick that I use here during this phase. In general, developers aren’t very good at saying where code complexity lies if you just ask them directly. Like, if you just ask, “What is too complex?” or “What do you find difficult?” they will think for a bit and may or may not come up with anything. But if you ask most developers for an emotional reaction to the code that they work on or work with, they will almost always have something. I ask questions like, “Is there some part of your job that you find really annoying?” “Is there some piece of code that’s always been frustrating to work with?” “Is there some part of the codebase that you’re afraid to touch because you think you’ll break it?” And to managers, “Is there some part of the codebase that developers are always complaining about?” You can adjust these questions to your situation, and remember that you want to be having a real conversation with developers—not just robotically reading off a list of questions. They are going to say things that you’re going to want more specifics on. You’ll probably want to take notes, and so forth.

After a while of doing this, you’ll start to get the idea that there is a common theme (or a few common themes) between the complaints. If you’ve read my book or if you’ve worked in engineering productivity for a while, you’ll usually realize that the real underlying cause of the problems is some sort of code complexity. But that’s not purely the theme we’re looking for—we could have figured that out without even talking to anybody. We’re looking for something a bit higher level, like “building the binary is slow.” There might be several themes that come up.

Now, you’ll have a bunch of data, and there are a few things you can do with it. Usually engineering management will be interested in some of this information that you’ve collected, and presenting it to them will make you real to the managers and hopefully foster some agreement that something needs to be done about the problem. That’s not necessary to do as part of this solution, but sometimes you’ll want to do it, based on your own judgment of the situation.

The first thing you should do with the data is find some problem that developers know they have, that you know you can do something about in a short period of time (like a month or two) and deliver that solution. This doesn’t have to be life-changing or completely alter the way that everybody works. In fact, it really should not do that. Because the point of this change is to make your work credible.

When you work in engineering productivity, you live or die by your personal credibility.

You see, at some point you need to be able to get down to the real problem. And the only way that you’re going to be able to do that is if the developers find you credible enough to believe you and trust you when you want to make some change. So you need to do something at first to become credible to the team. It’s not some huge, all-out change. It’s something that you know you can do, even if it’s a bit difficult. It helps if it’s something that other people have tried to do and failed, because then you also demonstrate that in fact something can be done about this mess that other people perhaps failed to handle (and then everybody felt hopeless about the whole thing and just decided they’d have to live with the mess forever, and it can’t be fixed and blah blah blah so on and so on).

Once you’ve established your basic credibility by handling this initial problem, then you can start to look at what problem the developers have and what you think the best solution to that would be. Now, often, this is not something you can implement all at once. And this is another important point—you can’t change everything about a team’s culture or development process all at once. You have to do it incrementally, deal with the “fallout” of the change (people getting mad because you changed something, or because it’s all different now, or because your first iteration of the change doesn’t work well) and wait for that to calm down before moving on to the next step. If you tried to change everything all at once, you’d essentially have a rebellion on your hands—a rebellion that would result in the end of your credibility and the failure of all your efforts. You’d be right back in the same pit that the other two, non-working solutions from above end you up in—being demoralized or ineffective. So you have to work in steps. Some teams can accept larger steps, and some can only accept smaller ones. Usually, the larger the team, the more slowly you have to go.

Now, sometimes at this point you run into somebody who is such a curmudgeon that you just can’t seem to make forward progress. Sometimes there is some person who is very senior who is either very set in their ways or just kind of crazy. (You can usually tell the latter because the crazy ones are frequently insulting or rude.) How much progress you can make in this case depends partly on your communication skills, partly on your willingness to persist, but also partly in how you go about resolving this situation. In general, what you want to do is find your allies and create a core support group for the efforts you are making. Almost always, the majority of developers want sanity to prevail, even if they aren’t saying anything.

Just being publicly encouraging when somebody says they want to improve something goes a long way. Don’t demand that everybody make the perfect change—you’re gathering your “team” and validating the idea that code cleanup, productivity improvements, etc. are valuable. And you have something like a volunteer culture or an open-source project—you have to be very encouraging and kind in order to foster its growth. That doesn’t mean you should accept bad changes, but if somebody wants to make things better, then you should at least acknowledge them and say that’s great.

Sometimes 9 out of 10 people all want to do the right thing, but they are being overruled by the one loud person who they feel they must bow down to or respect beyond rationality, for some reason. So you basically do what you can with the group of people who do support you, and make the progress that you can make that way. Usually, it’s actually even possible to ignore the one loud person and just get on with making things better anyway.

If you ultimately get totally stopped by some senior person, then either (a) you didn’t go about this the right way (meaning that you didn’t follow my recommendations above, there’s some communication difficulty, you’re genuinely trying to do something that would be bad for developers, etc.) or (b) the person stopping you is outright insane, no matter how “normal” they seem.

If you’re blocked because you’re doing the wrong thing, then figure out what would help developers the most and do that instead. Sometimes this is as simple as doing a better job of communicating with the person who’s blocking you. Like, for example, stop being adversarial or argumentative, but listen to what they person has to say and see if you can work with them. Being kind, interested, and helpful goes a long way. But if it’s not that, and you’re being stopped by a crazy person, and you can’t make any progress even with your supporters, then you should probably find another team to work with. It’s not worth your sanity and happiness to go up against somebody who will never listen to reason and who is dead set on stopping you at all costs. Go somewhere where you can make a difference in the world rather than hitting your head up against a brick wall forever.

That’s not everything there is to know about handling that sort of situation with a person who’s blocking your work, but it gives you the basics. Persist, be kind, form a group of your supporters, don’t do things that would cause you to lose credibility, and find the things that you can do to help. Usually the resistance will crumble slowly over time, or the people who don’t like things getting better will leave.

So let’s say that you are making progress improving productivity by incremental steps, and you are in some control over any situations that might stop you. Where do you go from there? Well, make sure that you’re moving towards the fundamental problem with your incremental steps. At some point, you need to start changing the way that people write software in order to solve the problem. There is a lot to know about this, which I’ve either written up before or I’ll write up later. But at some point you’re going to need to get down to simplifying code. When do you get to do that? Usually, when you’ve incrementally gotten to the point where there is a problem that you can credibly indicate refactoring as part of the solution to. Don’t promise the world, and don’t say that you’re going to start making a graph of improved developer productivity from the refactoring work that you are going to do. Managers (and some developers) will want various things from you, sometimes unreasonable demands born out of a lack of understanding of what you do (or sometimes from the outright desire to block you by placing unreasonable requirements on your work). No, you have to have some problem where you can say, “Hey, it would be nice to refactor this piece of code so that we can write feature X more easily,” or something like that.

From there, you keep proposing refactorings where you can. This doesn’t mean that you stop working on tooling, testing, process, etc. But your persistence on refactoring is what changes the culture the most. What you want is for people to think “we always clean up code when we work on things,” or “code quality is important,” or whatever it takes to get the culture that you want.

Once you have a culture where things are getting better rather than getting worse, the problem will tend to eventually fix itself over time, even if you don’t work on it anymore. This doesn’t mean you should stop at this point, but at the worst, once everybody cares about code quality, testing, productivity, etc. you’ll see things start to resolve themselves without you having to be actively involved.

Remember, this whole process isn’t about “building consensus.” You’re not going for total agreement from everybody in the group about how you should do your job. It’s about finding out what people know is broken and giving them solutions to that, solutions that they can accept and which improve your credibility with the team, but also solutions which incrementally work toward resolving the real underlying problems of the codebase, not just pandering to whatever developer need happens to be the loudest at the moment. If you had to keep only one thing in mind, it’s:

Solve the problems that people know they have, not the problems you think they have.

One last thing that I’ll point out, is that I’ve talked a lot about this as though you were personally responsible for the engineering productivity of a whole company or a whole team. That’s not always the case—in fact, it’s probably not the case for most people who work in engineering productivity. Some people work on a smaller part of a tool, a framework, a sub-team, etc. This point about solving the problems that are real still applies. Actually, probably most of what I wrote above can be adapted to this particular case, but the most important thing is that you not go off and solve the problem that you think developers have, but that instead you solve a problem that (a) you can prove exists and (b) that the developers know exists. Many of the engineering productivity teams that I’ve worked with have violated this so badly that they have spent years writing tools or frameworks that developers didn’t want, never used, and which the developers actually worked to delete when the person who designed them was gone. What a pointless waste of time!

So don’t waste your time. Be effective. And change the world.

-Max