Starting Scala

I’m starting a new job soon which is going to be entirely Scala programming. I’m trying to dig in a little bit and get started with a leg up. I’m going to outline my basic plan here and post updates as time goes on. I’ve done a tiny bit of Scala before when learning some about Spark. I’ve done some Java work in the past as well so the ecosystem is a little bit familiar. So I’m not coming in from nothing.

The place I’m going to be working uses Macs so getting off of Windows seems like the right way to get started. I had previously used Eclipse when working on JVM based languages. The last time I had used Eclipse though it seemed like more had changed than was the same as I remembered. So I’m going to take a different tack this time and try IntelliJ. So the overall development environment is going to be an Ubuntu VM and use IntlliJ on it.

I plan to start with Scala School to make sure I’ve got all the basics down. Once I feel I’ve got that put together I’ll try Scala Exercises for some more guided exercises. Then I’m going to do some of the Project Euler problems I had done in F# since I understand the problems and can focus on learning the idiomatic Scala way of doing things.

I figure this should get me up to speed on Scala pretty quickly. We’ll see how well it goes.

ChatOps and ChatZero

Inbox Zero was always a concept I never fully understood the need for before I moved to a big company. You get some email every day and you read it, what’s the problem? Now that I’ve been in a big organization for a while, I think I understand the problem better: you eventually end up on email lists that are about things that are only tangential to what you do. I’m on one particular email list about communications from partners we integrate with; I’m not involved in those integrations at all, but can’t unsubscribe from the list. Inbox Zero both helps you stay focused on doing real work and makes sure you don’t miss anything important.

Chat and ChatOps has been replacing email usage across the industry, since it is an easier way to collaborate in real time and the chat histories make it easier to share information asynchronously. As ChatOps has picked up in my office I will get to the office and see there are hundreds of new messages spread across the roughly 20 channels I’m in. I used to read all of them every morning, but some of the channels had been getting chatty, which made it harder to keep up. I first dealt with this by muting or leaving channels that I was less interested in, though apparently leaving channels in Slack is considered bad etiquette in some circles. This hasn’t helped, since I keep getting involved in more and more channels, no matter how many I drop.

I had started skimming some of the channels I am in rather than really reading all of it. This helped keep up with the increased flow, but deciding which channels to skim was hard. I recently listened to Arrested DevOps talk about ChatOps, which was an interesting because they came from multiple different backgrounds and chat usage styles. They made a couple of suggestions, one of which was counterintuitive but has been working great for me since I’ve implemented it – make more channels, especially short-term channels for intense conversation. More channels helps keep the content of each channel more focused so instead of having four semi-related conversations in one channel, you have four channels each with one conversation. With each of those breakout conversations being linked as it moves out you can either go read it or skip it freely. One of the other great but more obvious suggestions was the use of dedicated update channels for notifications from automated systems.

The switch to chat from an email based system has been a great experience overall in my opinion, but it will take a while for all of the productive patterns of how to use chat to get established. The great part is that the Chat system is open and pluggable and not centrally controlled. This ability to innovate on how the platform works brings about hubot and all of the other bots and integrations that make ChatOps an amazing tool.

Agile Signalling

It seems like agile has become a signal in software development, particularly Scrum. Everyone feels the need to say that they’re up on how “things are done,” and being allegedly “agile” seems to be how they’ve decided to show it. From an executive perspective, to remain competitive in the hiring market, you must be “agile.” And indeed, the teams may be working in sprints and doing all of the prescribed activities, but they aren’t actual building software any differently than they were before they called themselves an “agile” organization.

I don’t mean here that an organization has truly implemented waterscrumfall where you have an agile cycle in the middle of a waterfall process, which I’ve never seen in person. Instead, you’ve taken how you worked before and rephrased everything in agile terminology without embracing agile theory. You don’t have releasable software at the end of an iteration; you still have a two-year roadmap where dates are promises inside and outside of the organization. You are still producing reams of documentation about software that may never exist. For every principle on the agile manifesto there are a myriad of ways that principle can be half-assed.

Scrum has a bad reputation in relation to half-assed implementations. I think part of it is the ubiquity of scrum offerings and commercially-available training where the sales pitch overpromises the results. Scrum is more accessible since, at first glance, it can just be interpreted as doing the same things you are doing now, just in smaller pieces. If the scrum implementation isn’t achieving it shows up in the retrospective. The team identifies their impediments but if they can’t deal with them it becomes an exercise in futility. If the organization doesn’t want to help resolve the impediments then the team’s morale can easily collapse.

Among the twelve principles of agile, I think there is one in particular that, if you follow it, best represents doing the right things (and can’t be half-assed cheaply). I think that the principle “Business people and developers must work together daily throughout the project” is the definitive signal of being truly agile in spirit. The daily interaction is a specific measurable objective that can be monitored.

From the developer’s point of view, this gets you answers to questions when you need to clarify intent. This helps the developers to understand why the product should be doing something and produce additional insights into what the product could do. This insight can be invaluable to the business people since it produces ideas of how the software can better meet objectives  through solutions that are not apparent to those looking at the problem from the outside. Even this benefit can be dwarfed from the morale benefit of everyone trusting everyone else to do their best.

Without the organization being wholly aligned in embracing agile theory, the business stakeholders would never have the incentive to put in the time investment to provide this level of involvement. The business stakeholders wouldn’t be inclined to put that sort of investment into the process practice if they don’t see the value. If they see the value of this one principle, they will hopefully understand the rest of the intertwined processes of the agile manifesto.

The signal that the strong businessperson-developer interaction sends is that the whole of agile practices are valued. Compare that to the signal of someone stating “we’re agile,” which just indicates that the speaker thinks you want to hear them say that. Usually those that say those things are doing some agile things, but the difference between a half-assed agile implementation and a  true one is the difference between night and day when you are working in it. By looking for the strong signal that the business stakeholder involvement represents, hopefully you can find those that are truly agile and weed out those who are just posing. It is true here that actions signal louder than words.

Backfires of Agile Coaching

I was recently listening to episode 39 of Agile for Humans, which is all about how to become an agile coach, featuring several guests who are professional agile coaches. During the episode, one of the guests indicated that while consulting as an agile coach with various teams, he noticed that some team members picked up on the agile ideas very quickly while some did not. Likewise, he saw that in some engagements he succeeded at transforming the organization to agile, and in other engagements he didn’t produce a lasting organizational change. After the attempt, in those organizations that didn’t succeed at the transformation, all of the programmers who took strongly to the agile lessons quit. The other coaches on the episode all had similar experiences of their own, where once a transformation backslides, people would leave.

I had never put this idea into words, but it makes sense intuitively. Once you understand agile ideas, working using other methodologies is less satisfying. I know I have left a job due to a backslide in the project management practices at the organization. The backslide was the difference between how the team was operating and how the organization was operating. The team was operating in an agile way but the management of the team moved to a new executive who wasn’t with the transition and it created undue friction in the organization.

An agile team does their work in small iterations and produces workable changes. An agile organization must therefore be capable of accepting changes from the team at the same rate. An agile organization can make decisions about what it wants in order to keep up with the team. It understands that building comprehensive documentation doesn’t benefit you if there is no working software. An agile organization does status reporting to match the structure and cadence of the team, whether that is sprint-based or continuous flow.

Considering the specific example raised by the podcast, on one hand, the coach succeeded at influencing someone in a way that moved them to action and that is a great success. On the other hand, the organization that brought in the coach lost valuable talent because they were exposed to ideas that the organization wasn’t ready to implement fully. If a coach does right by the individual they may do wrong by the organization in the process.

In my experience, an agile transition starts at the team level with the team learning the agile processes without much focus on the theory . Sometimes the theory is explained, but since the team doesn’t actually need to understand the “why” for the transformation to continue, it’s not a point of emphasis. Therefore, the change to the team processes happens pretty quickly, but the individuals may or may not understand the rationale behind the situation fully even though the processes often resonates on their own with some team members. Meanwhile, the greater organization has its own transition where each part of the organization adapts related processes to the respond to the faster movement driven by the agile processes.

But, the theory matters much more to the organization, since the same processes that resonate with individual agile team members are a much more significant procedural change at higher levels, wherein it often feels like control is being lost. If the broader organization doesn’t really respond to the agile ideals, it creates an impedance mismatch between the happy agile teams and the rest of the organization. Most people aren’t ready to handle being of a different mindset than the organization, or inclined to personally be change agents to pull the organization forward. They prefer that the organization match their level of understanding of how things “should” work; if they have not responded to the agile ideal then they will resist it. This friction is what happens when you have an agile transition at the level of how the work is done but budgeting and planning still happen at a radically different cadence. The desire to resolve this friction leads into ideas like Beyond Budgeting and open allocation which try to push the agile ideals out into the rest of the organization.

I’ve been places where individual teams had a strong agile tradition but the higher levels of the organization and other departments did not embrace the concepts. Being on those teams was harder than being at an organization where everyone was working from a similar mindset – even when those organizations weren’t using agile. Just being on the same page is a huge benefit to morale because you don’t feel like you’re swimming against the current. At my current workplace, all of the development teams across the organization are working in an agile fashion but the executive level hasn’t fully changed their reporting processes to be aligned with the team’s agile structures. The friction this causes is mostly absorbed by mid-level management and kept away from the developers, but if we were more strongly aligned at all levels of the organization, there would be further efficiencies gained.

Ultimately, a half-hearted agile transition is just like any situations where you are uncomfortable with how things are being done – you want to resolve that discomfort, and the simplest way to do that is often to leave. If a particular person transitions to the new way of thinking you’ve started a clock on when they will be unable to put up with the difference and will, if it doesn’t seem like real change is being made, leave. At my current organization the agile coaches we have are working with the teams but we don’t have any coaches for the executive level, to meet its different needs. In my opinion, the transition as a whole would be well served if there were coaches at that level as well to ensure things continue to evolve toward a more agile situation. I’m sure that’s true for other organizations as well.

Book Chat: The Phoenix Project

The Phoenix Project is a novelization of an agile transformation at an autoparts company. It is written to show how the transformation helps the whole company and not just the IT department. There is also a resource guide to provide additional details for those inspired to implement agile after having read the book. The novel itself is about Bill, an IT manager who gets promoted up the chain during a series of crises.

Over the course of the novel, Bill learns – through a series of interactions with a guru who had previously helped the company resolve issues with their production lines – how to deal with a failing software development project, a complex system outage, and mundane day-to-day IT tasks like replacing a dead laptop, and he thereby gains control over his work life. Being a novel rather than a normal technical or business book allows it to introduce concepts and show how they are applied. This is beneficial too in breaking a major concept into digestible pieces, since if you look at the entire set of agile practices being suggested, some of them are designed to solve problems that the reader hasn’t necessarily seen yet. The book goes through the learning and rollout processes in a company that was very close to having a complete IT meltdown.

Showing the concepts in action seems to be a powerful way to assist in visualizing for those who have been exposed to the concepts but not necessarily put together what they mean in practice. I found the depiction of the work in progress lesson particularly intriguing since it is always somewhat difficult to understand in the abstract. The work in progress situation was shown through Bill finding that there was one person who was involved in almost everything going on even though most of those things don’t seem related to what the team was doing. Most of the examples are made in a physical production analogy rather than directly discussing the ideas in the software context. It seems to me that the structure was to keep the book from being a single monologue from the guru who is mentoring Bill, but I also think it makes the story more interesting by letting Bill discover the steps on his own. The author shows a bit of self-aware humor, as Bill gripes about the guru’s oblique descriptions in the text.

The target audience seems to be upper management, which makes the story more about leading the change and telling people to do it rather than how the change works for the actual developers. This makes sense to a certain degree, since a bottom-up agile transition is more difficult to do. The management perspective is especially enlightening in the section discussing change control; the company is looking at all of the scheduled changes that aren’t getting done. Bill then asks the insightful question: if we are failing to make hundreds of supposedly important changes a day, why isn’t it causing a problem? It helped them recognize there was a lot of work being considered that, while useful, was not resolving their major issues.

If you are already up on agile stuff from an individual contributor role I would probably pass on the book in favor of something else more immediately relevant. As an agile coach, scrum master, or executive it seems like a useful perspective to add to your portfolio. As a standalone novel it is somewhat dry, since other than Bill and the CEO the characters lack depth and there isn’t much dramatic tension.

Book Chat: Thinking Fast and Slow

Thinking Fast and Slow by Daniel Kahneman is a psychological profile of the two systems of human thought – a topic on which he is one of the pioneering researchers, and for which he shared a Nobel Prize. System 1 is the fast intuitive system that allows you to make snap judgments. System 2 is the slow methodical system that allows you to really dig into the data and do math. The book is mostly about the weaknesses of the two systems and how System 1 can be tricked and System 2 gets tired. The material is presented as a series of experiments mostly paired with an anecdote of how the idea came up. It tries to outline which problems engage which system and why. None of this material is directly applicable to computer programming; however, the understanding of human cognition is useful to understand how you are thinking about problems and hopefully be able to recognize when you should be trying to engage System 2 and when taking the quick answer from System 1 is acceptable.

One of the most interesting experiments to me was one that described how patients rated their pain over time during a procedure and how they rated the experience as a whole afterwards. The goal was to find the relationship between the two of them. The peak level of pain suffered was an obvious component to the whole, the experience towards the end of the procedure also had a strong component. However the total duration of the procedure did not impact the total rating of the experience. This implies that the mind doesn’t remember things based on the duration of the experience. Kahneman suggests that therefore,  if you were trying to structure an experience where there will be discomfort, extending the session to make the last part nicer would improve the memory of the whole experience. I’m not sure if this would apply to pulling off bandaids or not.

There was another anecdote that caught my attention, where Kahneman had spent his time in the Israeli Defense Force working on figuring out how to assign incoming recruits to the various branches. The previous method had been a sort of free-form interview relying on the judgement of the interviewers, which had not been overly successful. He designed a new system to try and remove the individual biases of the interviewers from the process. The system he put together had six categories and a fixed set of factual questions for each category. Based on the answers to those questions each category was assigned a rating between one and five then the results were summed to produce an overall evaluation. This produced a significant increase in successful placements over the old system. He also suggests that this theory could be applied to hiring. I’m not certain if that would be a good choice since this was to sort recruits to different types of units, rather than to pick one person from many.

Overall there are thirty-some of these different little experiences that are all drawn together here from Kahneman’s 40+ years of research and quite entertaining. Despite the somewhat dense material, it reads easily and is quite accessible without any sort of former exposure to these sorts of topics. If this sort of thing is interesting to you, he also references other similar books talking in more detail about some of the specific ideas discussed.

Book Chat: The Leprechauns of Software Engineering

The Leprechauns of Software Engineering is a deconstruction of different claims of ‘fact’ in software engineering that are shown to have dubious origins. The three main claims discussed are: the origins of the waterfall process, 10x programmer productivity, and the defect cost curve. If you are a fan of meta analysis this book is definitely something you should check out. Even if you aren’t into that sort of thing the ‘facts’ discussed are commonly being cited in other places that are accepted without much thought.

The 10x study was especially affecting, because while many of us have anecdotal experience that says that this is true; however, the research that supposedly shows it was so haphazardly done that it shouldn’t be trusted. For instance, some of the supporting studies had people debugging a piece of code who weren’t familiar with the programming language it was written in, but were included in the study anyway. Of course the people with greater familiarity with the programming language would be greatly superior at debugging. In other studies, the researchers measured lines of code per hour as a measure of skill, but more lines is not necessarily the mark of greater productivity. Some of the studies had different participants using different programming languages and tools; some measured whole projects. Altogether, it wasn’t information that feels like it shows the claim with the certainty it was normally made with.

The investigation of the defect cost curve was more about the definition of the defect, the definitions of the various phases, the numbers attached to the curve, and how somewhat reasonable claims got distorted into the commonly seen form. The evidence does not show a rigid cone with phases and multipliers. However, to me, it shows that the cone does exist, but it shows the best you can expect to do, not how well you will do.

Overall, the book is an impressively researched and cited effort that digs through years of history to try and uncover the truth. It calls into question some of the foundational ‘facts’ of software engineering that may seem true, but we haven’t bothered to do the rigorous research to prove it. I want to help in getting the word out there on these misconceptions, and hope you join me. The sooner we acknowledge the basic research isn’t there the sooner we can build that solid foundation on the truth.

Book Chat: Managing Humans

Managing Humans by Michael Lopp, of Rands in Repose, is a humorous take on a management book. It is built as short chapters describing an event or a kind of person and how to deal with or work with them. If you are familiar with his blog there is a lot of overlap between that and the book. However, it fills in the blanks and creates a complete thought about how to deal with common management issues. There is a strong connection to the differences between doing technical work and other kinds of work.

One of the big takeaways I got from the book was the way he described how to give annual reviews. It consists of two parts: separation of the feedback from the outcome (i.e., raises and promotions), and a way to help determine what the employee needs. When I first experienced the separation on the receiving end of the review, I wasn’t a fan, but I can see now that the lack of discussion of outcomes let us focus on a productive discussion. The second part Lopp suggests is to determine what an employee needs using a two-by-two matrix:  if the employee has the skills to do the job, and if they have the desire. High skill low desire individuals need new challenges and responsibilities to re-energize. Low skill high desire individuals need training to get their skills in alignment with their ambitions.

There was another chapter that gave a look at a situation that was similar to something I had experienced recently – a communication breakdown between myself and someone on the team. Lopp describes a way to rebuild communication by going back to a very verbose way of communicating that takes a lot of the implied context out of it, until you get back to a fully trusting arrangement. This was what we ended up doing but it took us a long time to find our way back there. Makes me wish I’d read Lopp’s suggestion a while ago, it would have saved us some time.

Overall, I enjoyed the book. Check out the blog; if you get there and decide you want more of the same check out the book for yourself. He’s also got another book on finding career path as a software engineer that I may check out as well since I enjoyed this one.

Software Ownership vs Caretaking

Recently, I was trying to describe the relationship of my team to a particular piece of software within our company. I ended up settling on the term “caretaking.” We don’t have ownership of this particular piece of software in the traditional sense, wherein we are responsible for the content of the software and whether it is correct. Yet, we were in the terrible position of being the first people called whenever it ran into problems.

The software itself was in weird but decent shape; a lot of old battle-tested code that could benefit from some modernization and better automated testing (it had a couple of high-level integration tests but they were finicky and would only run in one particular QA environment). It is in that zombie state where we don’t make any changes to it since it meets the requirements, but nobody would look at it and say that’s some nice code. Nobody on the team had any particular experience with this code, but someone who had been on the team years ago had worked on it before joining this team. The code just sort of followed along with the person, and nobody ever really pushed back on it, since it didn’t run into problems. This particular code is even slated to be replaced, as a new team – we’ll call them “Triangle” – is centralizing all of responsibilities in this application’s domain into a couple of different services. We were caretaking – being nominally responsible for now but not paying too much attention to it.

This was all fine, right up until a serious security bug was found in this code. Our security team was rightfully saying this needed to get resolved ASAP. We wanted Triangle to just replace it now since that was going to happen anyway and we were already booked on other time-critical activities. Triangle wasn’t inclined to get pulled into this and commit to something on a deadline, which is a completely fair reaction on their part.

We took a look at our options for trying to resolve the problem and realized we had three. The first was a short-term fix, which would prevent the security exploit but would likely cause other problems in the medium term. The second was to gut the code and rebuild it without the exploit in place; this was pretty quickly rejected because it seemed wasteful to gut it for this then replace it again in the short term, especially given that the other team was planning to replace it in a different tech stack than the current one. The third was a nuanced tweak to the existing behavior that looked like it would fix the security issues but it was unclear as to if it would have any other negative side effects. We decided to go with the third approach since the first approach would likely end up coming back and costing us more time in the long term.

We decided at this point that we were also going to actively take ownership of the piece of code in the medium term. We didn’t gut the existing solution, but I applied some iterative refactoring to the situation to safely create the seams we needed to test the software to our satisfaction. The existing code was one big method, about 1500 lines. I started by extracting out some methods from that, then converting those methods into a couple of classes. I did it this way so that the available tools handled most of the refactoring so that I could be confident in the correctness of the refactored code, even with the limited integration tests available. We felt that this little bit of unit testing would enable us to make the change we needed to with confidence that we didn’t break any of the broader test cases.

We ended up fixing the problem and one little corner of the codebase. Applying some TLC to this portion of the codebase was rewarding, since every time we had looked at it before when it did something odd it was a negative experience everyone wanted to avoid. Overall, I only spent about a day cleaning it out and making the basic workflow conceptually clean. The software got much better and it just took the will to make the changes. The effort was lower than what would have been expected, to try and fix this and it built a good bit of confidence if anything else in this area. It was a rewarding little win.

The pride of ownership and fully tackling a problem to make the system better,  rather than continually avoiding, has lots of benefits. You actually make things better, which can be motivating to continue to tackle more problems. Making something a little better, then a little better, then a little better again, and all of a sudden you’ve changed the tenor of the system. In this instance, tackling the problem head-on proved to be easier than we thought. We are still caretaking a bunch of other code, a good bit of which is slated to be replaced as well, but if push comes to shove we’ll hopefully go ahead take full ownership of that code too.

Book Chat: Don’t Make Me Think Revisited

Don’t Make Me Think Revisited by Steve Krug is a usability and user experience book. It’s a series of well-illustrated rules for getting the basics of usability into a web site. There is also an excellent chapter on how to run your own usability test on the cheap. The revisited edition includes some information about working on mobile apps, and mobile web sites as well. The rules of creating a good navigation system was a great way to codify things you know but may have struggled to express.

Krug’s guidance on running your own cheap usability study comes down to using fewer users and putting together a simpler reporting plan. Instead of having 20 users and generating a giant report, you get two or three users and select a couple of important things that can be actively fixed. You don’t need a report with hundreds of issues, since once you fix a couple of issues the rest of it might not be useful anymore. Once you run one study and fix those issues, iterate and try again. It’s almost like A/B testing except it works on smaller scales.

The information on mobile testing goes beyond the usual advice of “make the buttons bigger” and “put less on each page.” Krug’s advice on deep linking on mobile is obvious, that it needs to work when you navigate to a full site url on a mobile device needs to work, but nobody seems to get it right. And, even though this is minor, the number one thing I got from the mobile chapter was a way to combine a webcam and a clip on light, and build a mount that lets you see what the user’s hands are doing while they’re using the app. That’s way more useful than  conventional video recording equipment where you would mount the phone but then they don’t hold the phone normally. Plus it only takes an hour to put together and costs about $30.

Krug’s rules for a navigation system basically boil down to the obvious. Keep the navigation consistent page to page. Make it clear where you are in the application. Put the things most people want in easy to find places. Ensure that every page has a name and it is in a consistent place, which maybe isn’t as immediately intuitive as the other rules. However, when I went and checked some sites and most seem to do this. Putting all of these rules together comes out as an explanation of the tabbed design and why it became so common.

Overall design is not one of my strong interests, but this is a good primer on the topic to help keep you from doing anything too silly. I certainly feel like I can put Krug’s advice into practice. There are also lots of references to some other books if you want to try and dig in deeper.