Book Chat: Designing Data-Intensive Applications

I start most of these book chats by describing the book and end by describing the reader who it seems like it would benefit. This time I’m going to describe the reader first.

If you write software that interacts with another computer you should read this. To me, the title is somewhat misleading since it focuses on the data aspect, however the intensity comes from the size of the data and implies a distributed system. You also get an amazing survey of different database implementations as a side benefit. The only complaint that I have is that the first part of the book starts very much at the beginning of data-intensive systems (e.g.,  data locality and how it is organized on disk), which is important if you are building your own database system but isn’t as applicable to the average reader.

There are sections on a vast number of different topics. It covers both SQL-style ACID databases as well as BASE-style NOSQL databases. It even gets into things like graph databases that don’t fit neatly in either box. The authors cover topics as varied as locking and commit strategies, the levels of consistency available in a database and what they really mean, distributed consensus, replication, and streaming.

The majority of the text is written in a technology-agnostic way but it will reference specific implementations that demonstrate a concept. There is also a deep academic rooting with a well-referenced selection of footnotes to satisfy any further curiosities you may have on the topics. It seems like it should be fairly accessible as a whole text to a relative beginner since it introduces concepts in a way that doesn’t require much prior knowledge. I don’t think a beginner could jump into a chapter in the middle and be able to follow along, but given the complexity of the topic I don’t think that’s an unreasonable thing.

Even if you aren’t building a database, deeply understanding the tradeoffs of the database you are using will make your application more correct. The difficulty of testing into a lot of the concurrent failure scenarios makes understanding the system at a logical level the only way to attempt to handle all failure cases. I do think all software engineers working on the web would benefit from the material here. It won’t make your day-to-day much better but it will help keep you out of the really bad places where the system is intermittently failing.

Advertisements

Book Chat: Manage It!

Manage It! is an overview of modern project management techniques. Most of it was accessible to me as a person who has never formally run a project, but has been involved in many. The less accessible material was concentrated towards the end which I dutifully worked through thinking there might be more immediately relevant portions after. It heartily embraced three different practices: strong meeting facilitation, rolling wave planning, and avoiding schedule games.

The meeting facilitation advice started out fairly straightforward – have an agenda, stick to the topic at hand, and hold one-on-ones with the team. It then goes on to discuss some more radical advice, like don’t go to meetings that aren’t about solving problems, question why you had the meeting if it ends and nobody has any action items, and avoid serial status meetings. If your project has a problem, getting the relevant people into a room and coming out with a solution is a great way to break the impasse. Other sorts of meetings can impact the progress of a project, but to me that doesn’t make them immediately a bad idea. From the perspective of the project manager I can see that other sorts of long-term work or out of band activity can impact the potential of the project, but it seems necessary for the functioning of a healthy engineering organization. I agree that the lack of action items coming out of a meeting seems like a warning sign it wasn’t a good usage of time. If a decision was made generally one or more people would leave the room and do something because of it. Serial status meetings are more complex; if you are holding a meeting where people tell you about progress being made on initiatives where the others in the room aren’t involved or impacted, it may be a good use of your time but it’s a bad use of everyone else’s time. If you are being invited to meetings to provide status, the book advises to send status via email and skip the meeting, the idea being that if the organization doesn’t accept that behavior then it isn’t the place you should be. Daily standups are not impacted from this practice because they’re about the impediments, not just the status. Overall it seems like a good package of advice as to how to interact with meetings.

Rolling wave planning is an implementation of the idea that your plan will fail, but that the exercise of planning is valuable regardless. Your short term plan should be pretty solid but the further out, the more vague the plan gets. So, you don’t worry as much about the long term, and as you acquire more information you update the plan. This works both for changes from outside the project and things you learn from executing the project itself. The one experience I have had with explicit rolling wave planning did not go well, but I feel that was because we were trying to keep the near-term solid plan too far into the future and engaging in some devotion to the schedule.

The schedule games section was the part that felt the most real to me, it listed out 16 ways a project plan can go awry and ways to cope with each of them. I felt like I had seen the vast majority of the pitfalls out in the wild. This previous visibility involved me more in the rest of the material since I felt the authenticity of this compared to a lot of books which don’t give practical guidance on  how to get from poor practices to good ones. The section on “schedule chicken” felt particularly familiar to me after having been involved in a high stakes version many years ago. We even made progress in getting out of the situation using one of the techniques described.

I would recommend this book to someone who has already led a team or project for a little bit and is interested in doing more of it. If you’ve been doing that sort of work as your primary role it probably still has some bits of interest. If you haven’t run a team or project yet you may get something from it but I feel that for a lot of the information you need to have seen the problems in action some before you can appreciate the importance of  avoiding it.

Book Chat: Extreme Programming Explained

Extreme programming (XP) is an alternative software development methodology that would be described as an agile methodology. It’s a competitor to scrum, but more focused on the developer experience, less prescriptive of specific organizational practices, and more prescriptive of technical practices. I was familiar with the concepts of XP and recently picked up the second edition of Extreme Programming Explained. This new edition refined some of the technical practices about deployment since tools now exist for even more rapid deployment than what was initially conceived.

The build time practice is interesting, the idea being that a continuous integration build/test cycle should take ten minutes. While you could make the build faster than 10 minutes, keeping it a bit longer generates a decent mental break to allow someone to get a cup of coffee or get up and stretch. Whereas, if it’s slower than that, there is a tendency to move onto a different task and you can lose context on the old task and the new task. It matches with my experience; although I hadn’t been able to articulate the solution, I had seen the problem.

The overall methodology seems solid, however it doesn’t market itself to the whole business the way scrum does which seems to have impacted the adoption of the methodology as a whole. The practices suggested are all pretty straight forward:

  • colocate the team,
  • construct a team with all necessary skills on the team,
  • have visible progress locators,
  • work when you can really concentrate on it,
  • pair program,
  • user stories,
  • a weekly cycle,
  • a larger quarterly cycle,
  • slack,
  • the above build time practice,
  • continuous integration,
  • test first programming, and
  • incremental design.

Most modern software teams would be in favor of most, if not all, of these practices. Some of the practices are outside the control of the team and would need significant management support, but most are things the team can control.

I don’t think that the differences between this and other agile project management methodologies are that significant. The biggest difference with scrum I can see would be that scrum has fixed reflection periods whereas XP has continuous reflection with impromptu kaizen events. I think that this difference between XP and scrum would allow you to differentiate yourself from all of the scrum implementations that are out there but never finished. I don’t think that the book adds much to my understanding of software engineering, however it’s an excellent selection of software engineering practices. If you’re looking for a different perspective on agile methodologies this would be an interesting read.

Strike Teams Part 2

This is a follow up to the original strike team post I did a while back. I’m writing this as the strike team is wrapping up. It’s been an interesting experience. We hit most of our broad goals but did it in a significantly different way than anticipated.

The first big change came early on when we were looking into the proposed changes to an existing piece of software that was going to be a consumer of what we were building. We realized that accommodating it was probably weeks of work by itself. The initial assumption of how much work was needed had been minimal, but apparently that particular solution was untenable to the people who supported the consuming software in the long term. This ended up with them pulling support from the strike team and becoming uninvolved. At the time, the scope and resourcing change was challenging since it threw everything into an uncertain state. I think it ended up being a good thing; we were able to focus on building out the functionality desired without spending a lot of time making changes to one consumer of the service. It does make the follow up more complex since we do need to make sure that work gets done otherwise the whole system will end up in a more complex state.

There was one portion of what we were going to deliver that had a complex deployment step. There were other changes that had to go either before or after this particular piece so we tried to move that deployment as far forward on the project plan as possible. The intention was to deliver it at the end of the first sprint. We encountered some significant scope changes that all unfortunately all had to go before that step. This ended up pushing the actual deployment out until the middle of the third sprint. At this point we had about ten tickets that we did the work for all sitting in this feature branch just waiting for the eventual opportunity to merge the whole thing back to master with nearly 5000 lines of changes. We ended up performing a series of messy merges bringing changes from master back to the feature branch. The final merge from the feature branch back to master was ugly since we wanted to preserve some but not all of the history of that branch. In retrospect we should have seen the merge issues coming and been more proactive about doing more small merges, but lesson learned for later.

Every team member lost some time to dealing with issues brought to them from the team they were on loan from. We had anticipated this during the first sprint but didn’t expect it to continue through the remainder of the strike team. I know I personally got sucked into some production oddities that were occurring and that after the first sprint without me my team missed my presence in code review to the point that they got me back to reviewing basically everything. Both of the other members of the team got pulled back to the their original team for issues that happened. We didn’t fight any of these needs figuring that since they seemed like critical issues for those teams, having their people available was the best overall usage of time, even if it hurt our overall ability to get things done on the strike team.

The team as a whole never really jelled to any significant degree. Everyone kept working the way their team did, which created some minor conflicts. One engineer’s team has a very lax overall structure and he would sort of disappear for days then show up with large amounts of excellent code, but it was frustrating for him to essentially go off the grid, without forewarning. The other came from a team with experience in a different technical stack and vacillated between asking not enough questions and spending too much time stuck and asking too many questions. The three of us were also in three different locations which made it difficult to really all get on the same page and get working together.

Overall the strike team model worked for this, since having representatives from some of the teams that were going to consume this service working on it made sure that we didn’t run into any issues with a misunderstanding of the domain. There were problems setting up a new team, that we should have attacked proactively since setting up any new team needs to initially get organized and form their own norms. The transient nature of the strike team prohibits a lot of identity building which, in my opinion is key to building a good team. Overall based on this experience I think that the strike team model can deal with software projects crossing multiple software team boundaries, but there may be other better ways out there to be found still.

Agile Architecture

I’ve been involved in a set of architecture discussions recently and it had me thinking about the role of architecture in an agile development team. The specific discussion mostly was around which services owned what data and which services were confederates using that data. My team owned all of the services in question and felt that the data should be centralized and the edge pieces should query the central store. There were some other interested parties that felt that the data should be distributed across the edge pieces which would coordinate amongst themselves and had a central proxy for outside services to query through. Both options had some pros and had some cons.

I had proposed some changes to the initial centralization plan to account for some issues raised by the other parties. Then something unexpected happened – they said that’s great but you need to do it our way anyway. This had me stunned at first. This didn’t change the way that they would use the resulting system so why was it their decision to make? Sure, they had higher titles but that isn’t a license to make architectural decisions universally. I hadn’t been in a situation like this before and wasn’t quite sure what to do with it to convince everyone involved that my proposal was the best option for the organization as a whole..

Fortunately the exchange was over email so I took a while to regroup and collect some opinions about what to do. I asked a few different people and got a couple of specific pieces of advice. First, that I should set a meeting with the all of the sets of stakeholders, so they would remember that there were more people involved. Second, that I should prep a full written description of the various options in advance. The goal of setting a meeting was to get a concentrated block of attention from the other parties rather than a series of ad hoc email exchanges. The written description  was to make it clear what the full state was, rather than just a series of deltas from the original plan. That would make it clearer what was going on. Bringing together all of the stakeholders meant that those opposed to it for reasonable but team-specific reasons would see the other stakeholders and hopefully see their perspective.

All of this went down over two sprints where we had set aside time to sort out this architecture for our next big project. Defining these tasks was tricky; the first sprint had tickets to handle figuring out all of the requirements and speccing several different options. The requirements weren’t put together by product since this was not an end user-facing feature. The second sprint was to gain consensus on the various options, which brings us back to the anecdote above.

The two points came together and once we had their concentrated attention and the complete description of the plan they came around to our centralization plan, with modifications.

Theories of Technical Debt

There are a couple of different major causes of technical debt even on the best run projects.

  1. Schedule pressure
  2. Changed requirements
  3. Lack of understanding of the domain

You can choose to take on debt strategically to accommodate an aggressive schedule, you can accumulate debt from having things change, or you can collect debt from doing what appeared to be the right thing but turned out not to be once you learned more about the underlying situation. There are plenty of other reasons that technical debt can be acquired, but most of those can be avoided. Changing requirements can’t really be avoided; things can change, that’s the nature of life. Understanding of the domain is a tricky issue, since you can spend more time upfront to better understand the domain but you will still uncover new aspects as you go.

Schedule pressure is the most demanding of the three. You can consciously say that technical debt should be taken on by doing something the expedient way. You can also have implicit schedule pressure that pervades the decision making process. This sort of pervasive pressure causes people to value different things. If leadership discusses the schedule day in, day out, but doesn’t mention quality, it ends up lacking.

Technical debt is fundamentally a lack of quality; not in the defect sense but in the lack of craftsmanship sense. All of those 5,000 line classes got written by other engineers who were doing the best they could within the constraints of the environment at the time. But some engineers look at that and don’t want to touch it afraid of what it is and how hard it is to change. Some other engineers look at it and see a mountain to be climbed, or a wilderness to be civilized. The problem code is something to be taken and broken to your will. Each kind of engineer has a place in the development lifecycle.

If a company needs to hit a product window and is only full of the kind of engineers who see debt as a challenge to be dealt with they might not be able to make that tradeoff. If you only have the engineers who are concerned with maximum velocity but leave behind chaos in their wake you will have trouble as the codebase matures. Every engineer seems to have a range on the spectrum where they are most comfortable. Where in that range they land day to day seems to be controlled by the environment around them.

If you are surrounded by one side or the other you might lean towards that side of the range. The messages management sends out about the quality of software relative to the messages they send out about schedule of the project is another important factor. If they keep hammering home to get more done, people will find corners they think can get cut. What those corners are will differ between people, but they will take on debt in various corners of the codebase. This sort of debt is really insidious since it was never consciously decided on. If you decide that you will defer good error messages, or avoid building out an abstraction now, but you do it explicitly because of the schedule is the right business choice, then since the team discussed and decided to do it, they know as a whole that’s not the best technical solution but is the best business solution, whereas if someone just does it everyone else may not even be aware of the other options.

Modern Agile

I heard some references to Modern Agile and went to investigate further. To me, it’s just a reimagining of values espoused in the Agile Manifesto in a response to how the original manifesto was misused. The four values of the modern agile movement are:

  • Make People Awesome
    • The “people”: here is those making, using, or otherwise impacted by software
  • Make Safety a Prerequisite
    • Mostly psychological safety, but includes physical safety too if that is a concern
  • Experiment and Learn Rapidly
    • Try to find ways to do whatever they do better
  • Deliver Value Continuously
    • This, to me, means the technical practices around continuous delivery. But could mean other things outside of the web application space.
    • This is also breaking work into smaller individually valuable pieces.

At first I wasn’t that enthusiastic about the idea, and it seemed to be splintering the agile community. Then I started thinking about the recent resignations at the Scrum Alliance, and consider that maybe the community was already splintering and this is just people writing down what had already happened. The people for whom orthodox scrum is agile had already disregarded “individuals and interactions over process and tools.” Both LeSS and SAFe give up on “responding to change over following a plan” to varying degrees.  

The “focus on people” is, to me, the only right way to build systems and products, but the diffusion of who you are supposed to be making “awesome” makes this focus less impactful. If you were trying to empower the makers to do awesome you would do things differently than a single-minded focus on the customer, and some of those things are mutually exclusive. Modern agile examples of focus on the customer from Amazon and Apple come with horror stories on the engineering side of the late nights and insane demands – can you really make all the parties named “awesome” at the same time?.

“Safety as a prerequisite” is the opposing force to the focus on the customer and the negatives it can cause. This can be protecting work-life balance and everything else that makes people able to engage with the work. Another aspect of this safety is an openness to discuss problems in a candid way. I don’t know if I’ve ever seen the sort of openness to admitting problems that this is designed to inspire. I’ve been places where small mistakes were admitted to openly, but it’s unclear what would happen with larger mistakes since as far as I know none happened while I was there. I don’t think I’ve ever seen the sort of transparency from above which would make this level of openness either which makes it difficult to reciprocate.

I think experimentation is the key insight of modern agile. The original manifesto implied that if external stimuli did not change, the team did not need to try to change. This implication resulted in a challenge-response feedback loop and sought a steady state where if things around you didn’t change much you didn’t change. Even then there are a lot of organizations where the right thing to do is whatever is specified by the scrum guide and anything else is wrong. This experimentation adds an additional focus on continuously trying to find a way to do better regardless of how well you are already doing.

I see “delivering value continuously” as a continuation of working software over comprehensive documentation. Working software, from the agile manifesto, is the expression of what is most valuable,” so you could say it described this concept in a narrower sense. The focus on “value” rather than “software” is helpful since sometimes software isn’t the right solution – for instance, if you have an internal API that works but nobody uses because they are having trouble with the API, activities other than writing more software would be more valuable.

If we as practitioners wanted to stop and reconsider how we go about doing work, this seems like a good place to start that conversation. I would have liked to have seen it as a more open discussion rather than appearing fully formed from the head of a consultancy. These ideas seem reasonable, but it’s not clear what the next step is in taking up this mantle. I would always try to be on a team that was doing all of these things, but I don’t know if this alone is enough to make a team that I want to work on.

ChatOps and ChatZero

Inbox Zero was always a concept I never fully understood the need for before I moved to a big company. You get some email every day and you read it, what’s the problem? Now that I’ve been in a big organization for a while, I think I understand the problem better: you eventually end up on email lists that are about things that are only tangential to what you do. I’m on one particular email list about communications from partners we integrate with; I’m not involved in those integrations at all, but can’t unsubscribe from the list. Inbox Zero both helps you stay focused on doing real work and makes sure you don’t miss anything important.

Chat and ChatOps has been replacing email usage across the industry, since it is an easier way to collaborate in real time and the chat histories make it easier to share information asynchronously. As ChatOps has picked up in my office I will get to the office and see there are hundreds of new messages spread across the roughly 20 channels I’m in. I used to read all of them every morning, but some of the channels had been getting chatty, which made it harder to keep up. I first dealt with this by muting or leaving channels that I was less interested in, though apparently leaving channels in Slack is considered bad etiquette in some circles. This hasn’t helped, since I keep getting involved in more and more channels, no matter how many I drop.

I had started skimming some of the channels I am in rather than really reading all of it. This helped keep up with the increased flow, but deciding which channels to skim was hard. I recently listened to Arrested DevOps talk about ChatOps, which was an interesting because they came from multiple different backgrounds and chat usage styles. They made a couple of suggestions, one of which was counterintuitive but has been working great for me since I’ve implemented it – make more channels, especially short-term channels for intense conversation. More channels helps keep the content of each channel more focused so instead of having four semi-related conversations in one channel, you have four channels each with one conversation. With each of those breakout conversations being linked as it moves out you can either go read it or skip it freely. One of the other great but more obvious suggestions was the use of dedicated update channels for notifications from automated systems.

The switch to chat from an email based system has been a great experience overall in my opinion, but it will take a while for all of the productive patterns of how to use chat to get established. The great part is that the Chat system is open and pluggable and not centrally controlled. This ability to innovate on how the platform works brings about hubot and all of the other bots and integrations that make ChatOps an amazing tool.

Agile Signalling

It seems like agile has become a signal in software development, particularly Scrum. Everyone feels the need to say that they’re up on how “things are done,” and being allegedly “agile” seems to be how they’ve decided to show it. From an executive perspective, to remain competitive in the hiring market, you must be “agile.” And indeed, the teams may be working in sprints and doing all of the prescribed activities, but they aren’t actual building software any differently than they were before they called themselves an “agile” organization.

I don’t mean here that an organization has truly implemented waterscrumfall where you have an agile cycle in the middle of a waterfall process, which I’ve never seen in person. Instead, you’ve taken how you worked before and rephrased everything in agile terminology without embracing agile theory. You don’t have releasable software at the end of an iteration; you still have a two-year roadmap where dates are promises inside and outside of the organization. You are still producing reams of documentation about software that may never exist. For every principle on the agile manifesto there are a myriad of ways that principle can be half-assed.

Scrum has a bad reputation in relation to half-assed implementations. I think part of it is the ubiquity of scrum offerings and commercially-available training where the sales pitch overpromises the results. Scrum is more accessible since, at first glance, it can just be interpreted as doing the same things you are doing now, just in smaller pieces. If the scrum implementation isn’t achieving it shows up in the retrospective. The team identifies their impediments but if they can’t deal with them it becomes an exercise in futility. If the organization doesn’t want to help resolve the impediments then the team’s morale can easily collapse.

Among the twelve principles of agile, I think there is one in particular that, if you follow it, best represents doing the right things (and can’t be half-assed cheaply). I think that the principle “Business people and developers must work together daily throughout the project” is the definitive signal of being truly agile in spirit. The daily interaction is a specific measurable objective that can be monitored.

From the developer’s point of view, this gets you answers to questions when you need to clarify intent. This helps the developers to understand why the product should be doing something and produce additional insights into what the product could do. This insight can be invaluable to the business people since it produces ideas of how the software can better meet objectives  through solutions that are not apparent to those looking at the problem from the outside. Even this benefit can be dwarfed from the morale benefit of everyone trusting everyone else to do their best.

Without the organization being wholly aligned in embracing agile theory, the business stakeholders would never have the incentive to put in the time investment to provide this level of involvement. The business stakeholders wouldn’t be inclined to put that sort of investment into the process practice if they don’t see the value. If they see the value of this one principle, they will hopefully understand the rest of the intertwined processes of the agile manifesto.

The signal that the strong businessperson-developer interaction sends is that the whole of agile practices are valued. Compare that to the signal of someone stating “we’re agile,” which just indicates that the speaker thinks you want to hear them say that. Usually those that say those things are doing some agile things, but the difference between a half-assed agile implementation and a  true one is the difference between night and day when you are working in it. By looking for the strong signal that the business stakeholder involvement represents, hopefully you can find those that are truly agile and weed out those who are just posing. It is true here that actions signal louder than words.

Backfires of Agile Coaching

I was recently listening to episode 39 of Agile for Humans, which is all about how to become an agile coach, featuring several guests who are professional agile coaches. During the episode, one of the guests indicated that while consulting as an agile coach with various teams, he noticed that some team members picked up on the agile ideas very quickly while some did not. Likewise, he saw that in some engagements he succeeded at transforming the organization to agile, and in other engagements he didn’t produce a lasting organizational change. After the attempt, in those organizations that didn’t succeed at the transformation, all of the programmers who took strongly to the agile lessons quit. The other coaches on the episode all had similar experiences of their own, where once a transformation backslides, people would leave.

I had never put this idea into words, but it makes sense intuitively. Once you understand agile ideas, working using other methodologies is less satisfying. I know I have left a job due to a backslide in the project management practices at the organization. The backslide was the difference between how the team was operating and how the organization was operating. The team was operating in an agile way but the management of the team moved to a new executive who wasn’t with the transition and it created undue friction in the organization.

An agile team does their work in small iterations and produces workable changes. An agile organization must therefore be capable of accepting changes from the team at the same rate. An agile organization can make decisions about what it wants in order to keep up with the team. It understands that building comprehensive documentation doesn’t benefit you if there is no working software. An agile organization does status reporting to match the structure and cadence of the team, whether that is sprint-based or continuous flow.

Considering the specific example raised by the podcast, on one hand, the coach succeeded at influencing someone in a way that moved them to action and that is a great success. On the other hand, the organization that brought in the coach lost valuable talent because they were exposed to ideas that the organization wasn’t ready to implement fully. If a coach does right by the individual they may do wrong by the organization in the process.

In my experience, an agile transition starts at the team level with the team learning the agile processes without much focus on the theory . Sometimes the theory is explained, but since the team doesn’t actually need to understand the “why” for the transformation to continue, it’s not a point of emphasis. Therefore, the change to the team processes happens pretty quickly, but the individuals may or may not understand the rationale behind the situation fully even though the processes often resonates on their own with some team members. Meanwhile, the greater organization has its own transition where each part of the organization adapts related processes to the respond to the faster movement driven by the agile processes.

But, the theory matters much more to the organization, since the same processes that resonate with individual agile team members are a much more significant procedural change at higher levels, wherein it often feels like control is being lost. If the broader organization doesn’t really respond to the agile ideals, it creates an impedance mismatch between the happy agile teams and the rest of the organization. Most people aren’t ready to handle being of a different mindset than the organization, or inclined to personally be change agents to pull the organization forward. They prefer that the organization match their level of understanding of how things “should” work; if they have not responded to the agile ideal then they will resist it. This friction is what happens when you have an agile transition at the level of how the work is done but budgeting and planning still happen at a radically different cadence. The desire to resolve this friction leads into ideas like Beyond Budgeting and open allocation which try to push the agile ideals out into the rest of the organization.

I’ve been places where individual teams had a strong agile tradition but the higher levels of the organization and other departments did not embrace the concepts. Being on those teams was harder than being at an organization where everyone was working from a similar mindset – even when those organizations weren’t using agile. Just being on the same page is a huge benefit to morale because you don’t feel like you’re swimming against the current. At my current workplace, all of the development teams across the organization are working in an agile fashion but the executive level hasn’t fully changed their reporting processes to be aligned with the team’s agile structures. The friction this causes is mostly absorbed by mid-level management and kept away from the developers, but if we were more strongly aligned at all levels of the organization, there would be further efficiencies gained.

Ultimately, a half-hearted agile transition is just like any situations where you are uncomfortable with how things are being done – you want to resolve that discomfort, and the simplest way to do that is often to leave. If a particular person transitions to the new way of thinking you’ve started a clock on when they will be unable to put up with the difference and will, if it doesn’t seem like real change is being made, leave. At my current organization the agile coaches we have are working with the teams but we don’t have any coaches for the executive level, to meet its different needs. In my opinion, the transition as a whole would be well served if there were coaches at that level as well to ensure things continue to evolve toward a more agile situation. I’m sure that’s true for other organizations as well.