Dysfunction junction

You know Conway’s Law, of course. I’ll recap it here, just in case:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure. — Melvin E. Conway

This law is a deep one, because communication drives everything that humans do. We are deeply social; we cooperate with each other on large projects by communicating with each other. Communication is a reflection of our social structures because it defines our social structures! Of course the things we make reflect those structures. (Put like that it sounds obvious, but Conway’s statement is so pithy.)

Conway’s Law has some fun expressions, like:

  • If two teams are communicating well with leaders who get along, their software will work well together. Conversely, if two managers don’t get along, the software written by their teams won’t work together well.
  • Bugs form along organizational fracture lines.
  • Organizational silos become software silos.
  • The engineers in the org right now don’t like the software written by the engineering team two generations ago. 1
  • Broken organizations produce broken software.

Broken human systems make broken software systems

The darkest expression of Conway’s Law is that organizational dysfunction is expressed as software dysfunction. If the leader of an organization is failing, the software it builds starts to fail as well. If you as a technical leader are tasked with fixing a broken software system, you might need to start by fixing the broken organization that produced it.

Is the site falling down on the regular? Is shipping new features impossible? Is the tech debt at meme levels? Look for:

  • A toxic culture.
  • Teams that don’t communicate or cooperate.
  • A rapid feature-shipping pace that devalues software quality.
  • Managers who aren’t caring for their reports.
  • Managers who don’t accept organizational priorities.
  • Leadership that incentivizes bad behavior or unsustainable behavior.

And by “leadership”, I mean all the way up to the CEO.

I’ve been struck by the influence of company leadership on overall work quality over and over again. Each company has a culture that it promotes, consciously or unconsciously, suffusing its personality through everybody there. The culture starts with the CEO and cascades downward. Intelligent, thoughtful CEOs inspire the people around them to be the same. Bullying CEOs drive good people away. Outright stupid CEOs (and I’ve seen several of these in recent years) make the teams around them stupid and inspire bad work.

The management that’s right next to your engineering team has even more direct effect than an exec team. Management can turn an engineering team into a 0.1x team. I’ve seen leadership that inspired what people told me was the worst work of their careers. This leadership actively made their teams worse. Depressing, yes, but flip it: leadership can turn a team into a 10x team through inspiration, support, good incentives, emotional safety, and all the other things that good management can do. You can provide this leadership.2

I maintain that you must if you want to be effective as a technical leader.

Systems are self-reinforcing

I resisted coming to the conclusion that I had to think at the organizational level to fix technical problems for a long time. Surely, I thought, surely the organization will welcome solid, sensible suggestions aimed at fixing the trouble everybody agrees we’re in. The reality was far messier than this. Sensible suggestions might be nodded at as sensible, but their implementation will be resisted overtly and covertly.

The reasons why are squishy and human. People respond to incentives, and they do more of what the organization around them rewards them for doing. They get practice at whatever that is. They unconsciously build structure around themselves to support whatever they’re being rewarded for doing. The human organization that produces the software is a system all on its own, with feedback loops and incentive structures. When the org is healthy, those feedback loops promote a virtuous cycle. When the org is unhealthy, the feedback loops promote a doom spiral.

If the organization remains dysfunctional and you try to fix its output, it will resist your technical fixes, subvert improvement projects, and continue to drift back to the status quo. Managers will refuse to accept org-wide priorities, refuse to move staff to critical projects, refuse to cooperate with each other. Teams will attempt to maintain their silos. People will dig in rather than change their practices. Even more toxic things can happen in extremely broken organizations.

Systems are self-sustaining and self-reinforcing. To repair a software system, you have to repair the human system that built it, and you do that by breaking those self-reinforcing loops3.

People will tell you what’s wrong

The first step, therefore is to discover the incentive loops. Investigate and diagnose organization dysfunction methodically, the way you’d diagnose software.

One exhausting but effective way to figure out what the worst problems are is to sit down in a one-on-one with every single member of the engineering team and with adjacent teams. Your team is smart and they’ll tell you what’s wrong if they feel safe with you. Adjacent teams will have insights that people directly in the mess might not have, so include them too.

Ask everybody the same questions. I let everybody know in advance that I am doing this and to expect a calendar invite. I give everybody the questions in advance, and message people with personal reminders that they aren’t in trouble and I am meeting with everybody. Not everybody will need that reminder, but the ones who do will appreciate it keenly.

What questions should you ask? It depends on the circumstances, but you might try asking people if they feel they can do their jobs effectively, and then ask them to expand on their answer. Keep the questions open-ended and focused on the experiences of the person you’re talking to.

Take notes as you talk with people. (I ask for permission as I do this.) Themes will emerge. After talking to everyone in your organization, you’ll know what’s not working and what needs to be repaired.

Be kind to people

If you’re like me, you might find yourself angry about some of the things you learn in this exploration of the organization. Anger is a important response to learning that your values have been violated. It’s a signal you should pay attention to. That doesn’t mean you vent it at anybody else. Instead, use it to identify urgent work.

Remember also that you’re not the only person who has figured out that things are broken. People in the org know it and have responded in their own ways. People will work very hard to fix things in their own specific corners, with what control they have. These efforts aren’t going to be coordinated and might end up at cross-purposes with each other, unintentionally making things worse. Honor the intent. You are here to coordinate.

You must exercise power

You have diagnosed the problem. Now you want to fix it, because you are a software engineer and fixing things is what you do.

If you have reporting authority and can start making organizational changes, this is the time to use that authority. If you’re a technical leader without that authority, it’s harder. You will need an ally who does have it, and you will need to have the respect and trust of that person. Perhaps this person is an exec who agrees that the company isn’t getting what it needs from engineering. Maybe this person is a new leader brought in at your request.

Spend time with this ally getting aligned with them on the values you’re bringing to the problem. What matters most? What does a healthy organization look like?

You will also need to invest time to gain the respect and trust of the team around you. How do you gain trust? By behaving in a trustworthy way. Demonstrate that you know your trade. Show leadership in small ways. Handle incidents well (there might be lots of opportunity to do this). You might not have hard power, but you definitely have soft power. Influence. Persuasive skills. Moral authority. The ability to inspire people. Learn to use these as best you can.

If all else fails, get that promotion to a pure leadership position you’ve been avoiding so you can stay technical. You can always escape it later (I say, as a person who escaped it).

Change is about alignment

Let’s assume the happy path: you have an ally with organizational power, and you’re aligned with that person on values and goals. What happens next is alignment and reinvention for the broader organization. Now you make sure everyone in the org, starting with line managers, shares leadership’s values and understands the end goal. Now you clean up or change how the org accepts work and executes on it, giving all your processes a good shakeout.

Entire books could be written about this part and have been, of course, because what you’re doing is setting up a healthy engineering organization.

This is what happens next:

  • The company around you has to understand that the engineering org is not going to ship features for a bit.
  • Managers need to get into alignment first, or leave.
  • If managers have burned their relationships with their reports too badly, they might have to leave anyway.
  • People stuck in toxic patterns must leave.4
  • You need to design a new way for the org to accept work and execute on it.
  • Leadership must rebuild trust by offering trust.
  • Leadership must rebuild respect by offering respect.

Now give the team a project they can succeed with that will improve something noticeably. Give everybody a win. Reduce that technical mess just a tiny bit. The team will be behind you, helping. (Finally! You get to fix some of the technical problems you’ve been itching to fix!)

Remember that lots of the people in that org want change too. They’re good engineers who are doing bad work, and they know it. They’ll be relieved to execute on a plan to make things better. They will shine if you let them.

Looking back at doing this

Change is difficult to coax into being. You know the saying about how people have to want to change? This is true of organizations as well, as an collective decision from everyone in that org. The organization needs to feel that the cost of changing is less than the cost of continuing to operate as it has. Change is costly and risky; local maxima are attractive even if they’re not very maximal. Also note that moving from a local maximum to a higher spot requires going downhill first. That’s something that looks very dangerous to an embattled exec who’s already under pressure because their org is underperforming.

Doing this at an organization was the hardest thing I’ve done in my career. I feel I was only partially sucessful. I have notes from early in the process about things that needed to change that were still applicable two years later. You might not be willing to take this work on, which is okay. If you can’t make traction, it’s okay to leave. Organizations deserve to fail sometimes.

It was incredibly satisfying to hear from my colleagues that the changes made their work lives better, though, so the rewards are deep.

Acknowlegements

Thanks to Chris Dickinson who read early drafts of this post. Read his blog!

I keep thinking about that insight @isntitvacant had about Conway’s Law being true over time as well, and the software that felt good in the hands of a past time being uncomfortable to a team that has entirely turned over. — Ceej “oh well” Silverio (@ceejbot) February 8, 2021


  1. This is Conway’s Law over time. Teams are immutable: adding or removing a person to a team produces a different team. After enough change, the team is different enough that it no longer recognizes itself in the software system it produces. The result is people being vaguely unhappy about software that might be working perfectly well. This probably deserves its own short blog post. ↩︎

  2. It’s always a leadership problem. ↩︎

  3. My thinking here is influenced by Donella Meadows’s Thinking in Systems. You should read the book, but this blog post has a great summary if you’d like a teaser. ↩︎

  4. Orgs in failure modes can burn people so badly that they can’t get past that emotionally. Burned people will often be perfectly fine and healthy if they get to press a big reset button and go somewhere else. ↩︎

Ceejbot

This internet thing seems to have taken off.


By C J Silverio, 2022-05-10