Backfires of Agile Coaching

I was recently listening to episode 39 of Agile for Humans, which is all about how to become an agile coach, featuring several guests who are professional agile coaches. During the episode, one of the guests indicated that while consulting as an agile coach with various teams, he noticed that some team members picked up on the agile ideas very quickly while some did not. Likewise, he saw that in some engagements he succeeded at transforming the organization to agile, and in other engagements he didn’t produce a lasting organizational change. After the attempt, in those organizations that didn’t succeed at the transformation, all of the programmers who took strongly to the agile lessons quit. The other coaches on the episode all had similar experiences of their own, where once a transformation backslides, people would leave.

I had never put this idea into words, but it makes sense intuitively. Once you understand agile ideas, working using other methodologies is less satisfying. I know I have left a job due to a backslide in the project management practices at the organization. The backslide was the difference between how the team was operating and how the organization was operating. The team was operating in an agile way but the management of the team moved to a new executive who wasn’t with the transition and it created undue friction in the organization.

An agile team does their work in small iterations and produces workable changes. An agile organization must therefore be capable of accepting changes from the team at the same rate. An agile organization can make decisions about what it wants in order to keep up with the team. It understands that building comprehensive documentation doesn’t benefit you if there is no working software. An agile organization does status reporting to match the structure and cadence of the team, whether that is sprint-based or continuous flow.

Considering the specific example raised by the podcast, on one hand, the coach succeeded at influencing someone in a way that moved them to action and that is a great success. On the other hand, the organization that brought in the coach lost valuable talent because they were exposed to ideas that the organization wasn’t ready to implement fully. If a coach does right by the individual they may do wrong by the organization in the process.

In my experience, an agile transition starts at the team level with the team learning the agile processes without much focus on the theory . Sometimes the theory is explained, but since the team doesn’t actually need to understand the “why” for the transformation to continue, it’s not a point of emphasis. Therefore, the change to the team processes happens pretty quickly, but the individuals may or may not understand the rationale behind the situation fully even though the processes often resonates on their own with some team members. Meanwhile, the greater organization has its own transition where each part of the organization adapts related processes to the respond to the faster movement driven by the agile processes.

But, the theory matters much more to the organization, since the same processes that resonate with individual agile team members are a much more significant procedural change at higher levels, wherein it often feels like control is being lost. If the broader organization doesn’t really respond to the agile ideals, it creates an impedance mismatch between the happy agile teams and the rest of the organization. Most people aren’t ready to handle being of a different mindset than the organization, or inclined to personally be change agents to pull the organization forward. They prefer that the organization match their level of understanding of how things “should” work; if they have not responded to the agile ideal then they will resist it. This friction is what happens when you have an agile transition at the level of how the work is done but budgeting and planning still happen at a radically different cadence. The desire to resolve this friction leads into ideas like Beyond Budgeting and open allocation which try to push the agile ideals out into the rest of the organization.

I’ve been places where individual teams had a strong agile tradition but the higher levels of the organization and other departments did not embrace the concepts. Being on those teams was harder than being at an organization where everyone was working from a similar mindset – even when those organizations weren’t using agile. Just being on the same page is a huge benefit to morale because you don’t feel like you’re swimming against the current. At my current workplace, all of the development teams across the organization are working in an agile fashion but the executive level hasn’t fully changed their reporting processes to be aligned with the team’s agile structures. The friction this causes is mostly absorbed by mid-level management and kept away from the developers, but if we were more strongly aligned at all levels of the organization, there would be further efficiencies gained.

Ultimately, a half-hearted agile transition is just like any situations where you are uncomfortable with how things are being done – you want to resolve that discomfort, and the simplest way to do that is often to leave. If a particular person transitions to the new way of thinking you’ve started a clock on when they will be unable to put up with the difference and will, if it doesn’t seem like real change is being made, leave. At my current organization the agile coaches we have are working with the teams but we don’t have any coaches for the executive level, to meet its different needs. In my opinion, the transition as a whole would be well served if there were coaches at that level as well to ensure things continue to evolve toward a more agile situation. I’m sure that’s true for other organizations as well.

Book Chat: The Phoenix Project

The Phoenix Project is a novelization of an agile transformation at an autoparts company. It is written to show how the transformation helps the whole company and not just the IT department. There is also a resource guide to provide additional details for those inspired to implement agile after having read the book. The novel itself is about Bill, an IT manager who gets promoted up the chain during a series of crises.

Over the course of the novel, Bill learns – through a series of interactions with a guru who had previously helped the company resolve issues with their production lines – how to deal with a failing software development project, a complex system outage, and mundane day-to-day IT tasks like replacing a dead laptop, and he thereby gains control over his work life. Being a novel rather than a normal technical or business book allows it to introduce concepts and show how they are applied. This is beneficial too in breaking a major concept into digestible pieces, since if you look at the entire set of agile practices being suggested, some of them are designed to solve problems that the reader hasn’t necessarily seen yet. The book goes through the learning and rollout processes in a company that was very close to having a complete IT meltdown.

Showing the concepts in action seems to be a powerful way to assist in visualizing for those who have been exposed to the concepts but not necessarily put together what they mean in practice. I found the depiction of the work in progress lesson particularly intriguing since it is always somewhat difficult to understand in the abstract. The work in progress situation was shown through Bill finding that there was one person who was involved in almost everything going on even though most of those things don’t seem related to what the team was doing. Most of the examples are made in a physical production analogy rather than directly discussing the ideas in the software context. It seems to me that the structure was to keep the book from being a single monologue from the guru who is mentoring Bill, but I also think it makes the story more interesting by letting Bill discover the steps on his own. The author shows a bit of self-aware humor, as Bill gripes about the guru’s oblique descriptions in the text.

The target audience seems to be upper management, which makes the story more about leading the change and telling people to do it rather than how the change works for the actual developers. This makes sense to a certain degree, since a bottom-up agile transition is more difficult to do. The management perspective is especially enlightening in the section discussing change control; the company is looking at all of the scheduled changes that aren’t getting done. Bill then asks the insightful question: if we are failing to make hundreds of supposedly important changes a day, why isn’t it causing a problem? It helped them recognize there was a lot of work being considered that, while useful, was not resolving their major issues.

If you are already up on agile stuff from an individual contributor role I would probably pass on the book in favor of something else more immediately relevant. As an agile coach, scrum master, or executive it seems like a useful perspective to add to your portfolio. As a standalone novel it is somewhat dry, since other than Bill and the CEO the characters lack depth and there isn’t much dramatic tension.

Book Chat: Thinking Fast and Slow

Thinking Fast and Slow by Daniel Kahneman is a psychological profile of the two systems of human thought – a topic on which he is one of the pioneering researchers, and for which he shared a Nobel Prize. System 1 is the fast intuitive system that allows you to make snap judgments. System 2 is the slow methodical system that allows you to really dig into the data and do math. The book is mostly about the weaknesses of the two systems and how System 1 can be tricked and System 2 gets tired. The material is presented as a series of experiments mostly paired with an anecdote of how the idea came up. It tries to outline which problems engage which system and why. None of this material is directly applicable to computer programming; however, the understanding of human cognition is useful to understand how you are thinking about problems and hopefully be able to recognize when you should be trying to engage System 2 and when taking the quick answer from System 1 is acceptable.

One of the most interesting experiments to me was one that described how patients rated their pain over time during a procedure and how they rated the experience as a whole afterwards. The goal was to find the relationship between the two of them. The peak level of pain suffered was an obvious component to the whole, the experience towards the end of the procedure also had a strong component. However the total duration of the procedure did not impact the total rating of the experience. This implies that the mind doesn’t remember things based on the duration of the experience. Kahneman suggests that therefore,  if you were trying to structure an experience where there will be discomfort, extending the session to make the last part nicer would improve the memory of the whole experience. I’m not sure if this would apply to pulling off bandaids or not.

There was another anecdote that caught my attention, where Kahneman had spent his time in the Israeli Defense Force working on figuring out how to assign incoming recruits to the various branches. The previous method had been a sort of free-form interview relying on the judgement of the interviewers, which had not been overly successful. He designed a new system to try and remove the individual biases of the interviewers from the process. The system he put together had six categories and a fixed set of factual questions for each category. Based on the answers to those questions each category was assigned a rating between one and five then the results were summed to produce an overall evaluation. This produced a significant increase in successful placements over the old system. He also suggests that this theory could be applied to hiring. I’m not certain if that would be a good choice since this was to sort recruits to different types of units, rather than to pick one person from many.

Overall there are thirty-some of these different little experiences that are all drawn together here from Kahneman’s 40+ years of research and quite entertaining. Despite the somewhat dense material, it reads easily and is quite accessible without any sort of former exposure to these sorts of topics. If this sort of thing is interesting to you, he also references other similar books talking in more detail about some of the specific ideas discussed.

Book Chat: The Leprechauns of Software Engineering

The Leprechauns of Software Engineering is a deconstruction of different claims of ‘fact’ in software engineering that are shown to have dubious origins. The three main claims discussed are: the origins of the waterfall process, 10x programmer productivity, and the defect cost curve. If you are a fan of meta analysis this book is definitely something you should check out. Even if you aren’t into that sort of thing the ‘facts’ discussed are commonly being cited in other places that are accepted without much thought.

The 10x study was especially affecting, because while many of us have anecdotal experience that says that this is true; however, the research that supposedly shows it was so haphazardly done that it shouldn’t be trusted. For instance, some of the supporting studies had people debugging a piece of code who weren’t familiar with the programming language it was written in, but were included in the study anyway. Of course the people with greater familiarity with the programming language would be greatly superior at debugging. In other studies, the researchers measured lines of code per hour as a measure of skill, but more lines is not necessarily the mark of greater productivity. Some of the studies had different participants using different programming languages and tools; some measured whole projects. Altogether, it wasn’t information that feels like it shows the claim with the certainty it was normally made with.

The investigation of the defect cost curve was more about the definition of the defect, the definitions of the various phases, the numbers attached to the curve, and how somewhat reasonable claims got distorted into the commonly seen form. The evidence does not show a rigid cone with phases and multipliers. However, to me, it shows that the cone does exist, but it shows the best you can expect to do, not how well you will do.

Overall, the book is an impressively researched and cited effort that digs through years of history to try and uncover the truth. It calls into question some of the foundational ‘facts’ of software engineering that may seem true, but we haven’t bothered to do the rigorous research to prove it. I want to help in getting the word out there on these misconceptions, and hope you join me. The sooner we acknowledge the basic research isn’t there the sooner we can build that solid foundation on the truth.

Book Chat: Managing Humans

Managing Humans by Michael Lopp, of Rands in Repose, is a humorous take on a management book. It is built as short chapters describing an event or a kind of person and how to deal with or work with them. If you are familiar with his blog there is a lot of overlap between that and the book. However, it fills in the blanks and creates a complete thought about how to deal with common management issues. There is a strong connection to the differences between doing technical work and other kinds of work.

One of the big takeaways I got from the book was the way he described how to give annual reviews. It consists of two parts: separation of the feedback from the outcome (i.e., raises and promotions), and a way to help determine what the employee needs. When I first experienced the separation on the receiving end of the review, I wasn’t a fan, but I can see now that the lack of discussion of outcomes let us focus on a productive discussion. The second part Lopp suggests is to determine what an employee needs using a two-by-two matrix:  if the employee has the skills to do the job, and if they have the desire. High skill low desire individuals need new challenges and responsibilities to re-energize. Low skill high desire individuals need training to get their skills in alignment with their ambitions.

There was another chapter that gave a look at a situation that was similar to something I had experienced recently – a communication breakdown between myself and someone on the team. Lopp describes a way to rebuild communication by going back to a very verbose way of communicating that takes a lot of the implied context out of it, until you get back to a fully trusting arrangement. This was what we ended up doing but it took us a long time to find our way back there. Makes me wish I’d read Lopp’s suggestion a while ago, it would have saved us some time.

Overall, I enjoyed the book. Check out the blog; if you get there and decide you want more of the same check out the book for yourself. He’s also got another book on finding career path as a software engineer that I may check out as well since I enjoyed this one.