Transparency and Organizational Design in Scaled Agile Environments

Agile execution with a single team is tough as it is, but scaling Agile in the enterprise with dozens or hundreds of teams is an order of magnitude tougher. I've been thinking a lot recently on what makes scaling Agile so difficult and have come to the conclusion that it all boils down to transparency and intentional organizational design.

Transparency

Transparency lays the foundation of any adaptive, iterative environment. It's one of the key "Pillars of Scrum". If you can't see what you're doing or where you're headed, you're likely to crash and burn. Transparency means rapid feedback, observability, and an ability to see the big picture while also being able to delve into small details. Analysis is impossible and changes are risky when the organization and processes lacks transparency.

Within a small, individual Scrum team, transparency comes cheap and easy. Daily Standups keep the team in sync. A Scrum or Kanban board to help track at most a few dozen stories in a Sprint backlog. Review and Planning ceremonies to inspect the increment and adapt for the next.

In some ways, transparency is the natural outcome of a small team of people working closely, trusting and collaborating with one another. (A prime reason that Agile methods like Scrum mandate small teams!) But as soon as we move from one team to two Scrum teams, we immediately run into problems. Transparency is no-longer cheap and easy, it requires forethought and planning overhead to ensure teams are coordinated and able to inspect and adapt together.

From One to Two

With only two teams, we are still faced with challenging and critical organizational design decisions. We know from Conway's Law that our organizational design decisions will have significant impact the architecture and design of the product. Therefore, system architects should be consulted in every organizational design decision, splitting an organization alters the communications structure and inevitably the design of the product.

Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations. - Melvin Conway

The architecture of the product should also inform the communications structure of the organization. For example, if you have an API driven micro-service architecture, you might choose to formalize the the communication structure between the micro-service developers and the consumer application developers. If you are building something more monolithic in nature, you might want to encourage more flexibility and informality in the communication structure. In the end (per Conway's Law) whatever organizational choices you make will ultimately be reflected in the resulting product architecture, so choose with care.

Whatever organizational decisions you make, you also need to consider how to maintain transparency into the flow of work, process and progress. This means careful consideration of shared language, aka naming things.

"There are only two hard things in Computer Science: cache invalidation, and naming things" - Phil Karlton

When you have more than one team working on a project, you must be conscious to have consistent terminology and definitions, otherwise the isolated conversations within the separated teams will result in semantic drift. Communication must be structured in a thoughtful way so that, despite the inherent separation of teams, the communicated signs and symbols carry the same meaning.

In practical terms, this means developing a systemic approach to produce specific labels for product features and other objects relevant to the product (aka taxonomy). APIs are versioned in a particular and predictable way, major features are referred to by the same meaningful name, ambiguity is stamped out when it crops up. As Phil Karlton notes above, naming this is hard and should be taken seriously. Communication gaps and misunderstandings must be avoided to maintain transparency as we scale up beyond a handful of teams.

Organizational Design: From Two to Many

The larger your program & organization becomes, the more difficult it is to maintain transparency into your processes, workflows and progress. Therefore, it is of critical importance to pay close attention to both the organizational design and the related communication support structures needed to ensure the work is transparent.

Organizational Design is, by it's very nature, a design of communication structures and protocols. When you move teams around under one leadership structure or another, those teams become disconnected from the previous structure and must adapt to the rules of the new organization. Inevitably these internal communication structures will impact how product decisions are carried out in software and for the user.

Unlike small isolated teams, large organizations require a well defined communications structure to operate efficiently at scale and provide the process transparency we need to inspect and adapt. Designing and implementing organizational communication structures is a major challenge requiring input from Engineering, Product and Program management, as well as Executives.

Event Storming

We know the importance of Organizational Design and communication structures, but in enormously complex enterprise systems, it seems an impossible task to even begin to make sense of the complexity enough that we can begin to shape it.

This is where the Event Storming event comes to the rescue. Event Storming originated as a mapping process to discover Domains and Bounded Contexts within Domain Driven Design (DDD), but it's usefulness as a method to explore and understand complex business processes transcends that specific development approach.

Event Storming is a method for modelling business processes and arriving collaboratively at a sensible organization of the component organizational and business contexts that comprise the whole. It begins by bringing the right people together, Leadership, Engineering, Product and walking through processes end-to-end in several phases.

• Phase 1. - Events - Start by mapping every Event that occurs end-to-end in our process.
• Phase 2. - Commands - Map each command (and commander or actor) who initiates Events. Identify actors as internal (within the domain) or external (3rd party partners, etc).
• Phase 3 - Aggregates - Now we look for groups of Events and Commands that cluster together and identify these logical domains. Aggregates can be further coalesced around Business Units and other organizational entities.

From here, all parties to the Event Storming process gain valuable insight (transparency!) into the natural and logical structure of the business and how best to structure the organization to support the flow of value.

Conclusion

I've attempted here to tie a number of threads together, from the foundational principles of Agile and the difficulties in scaling them, to Conway's Law highlighting the impact of organizational divisions on software architecture, to Event Storming as a method to make visible the hidden organizational contexts of our business logic.

Gaining and maintaining transparency into software development and business processes at scale is an enormously important and complex task. It takes a significant and focused effort to make and sustain this visibility, but without it we're left in the dark unable to perform any meaningful inspection or adaptation to the rapid pace of change in the business landscape.