Goals For Senior Engineers

It’s annual review time around my office. Looking back at my personal goals for the last year and trying to come up with some for next year is a good opportunity for reflection. First up are the ones from last year, summarized.

  1. Make creating a new microservice simpler
  2. Lead a major project with a company wide impact
  3. Get the team into a better place regarding technical debt
  4. Mentor some Junior Engineers on the team
  5. Improve on call documentation/processes

I didn’t succeed at the first one, largely because I never really got a chance to try. It’s unfortunate because it would be great for the company in the long term. However, it’s an engineering-centric efficiency improvement for which the benefits are difficult to quantify, so it was never urgent. Not getting this done was disappointing for me personally. I had identified a way to help lots of engineering teams through an idea that’s obvious in hindsight, but I wasn’t able to find the time to get it implemented.

The second one was a pretty big success. The design I had put together got built out and the resulting architecture worked out great. Despite running into some issues with some of the prerequisites to operationalizing it we were able to get the whole thing into production ahead of schedule. On top of that we beat our load, response time, and resource usage targets handily. Overall I’m amazingly pleased with how this one came out.

The third one is an odd one, since by the metrics I set out for myself we nailed it, but by other metrics you could use to measure it we failed. I led some process changes to help us stop accruing technical debt, and start making more time to deal with the debt. This resulted in us closing an amazing number of tickets, three times the target my boss said was ‘too ambitious’ when I set it. He had suggested tracking the number of open technical debt tickets, but I stuck to the number of tickets dealt with since I felt as people saw these tickets got resolved they would be more inclined to file more of them. I was right on that, so while our metrics got worse, our position really got better.

I spent significant time mentoring our two new grads and two other new hires to help them get up to speed, not just on Scala but also on our systems and software engineering in general. The plan was to set up pair programming time and do accomplish the mentoring during that time. Since I’m on the East coast and a morning person and the new grads are on the West coast and not morning people it made things a bit complex at first in terms of scheduling time. Eventually we found a rhythm of me pairing with each of them an hour a week. This ended up helping them out a bunch. They got solutions to their problems and I taught them a lot of little things like IDE shortcuts and how to use some problem solving resources like symbol hound (a search engine that will search for special characters).

The last goal was meant to resolve a problem that had existed from the time I joined the team. Then, the on call rotation was two people trading off with no backup. The reason was that there wasn’t any documentation for how to deal with some of the ‘done’ systems the team supported, i.e., the systems that are feature complete and don’t have any active development on them. Nobody learned anything about them since they never touched them, but it also meant that the new engineers, myself included, couldn’t really go on call since if the system broke they didn’t have any idea what to do. I ended up doing a deep dive into each of the systems and producing docs about how they worked/logged so that if you got an error you had somewhere to start and an idea of what to do next. We opened up the pager rotation and everyone on the team gets the opportunity to be primary. It’s unclear if the documentation is any good because the systems haven’t had any problems.

This all brings me to goals for next year. There is some reorganization going on so I may end up on a new team. With that unknown right now, I’m not sure how I’ll figure out my goals for the coming year. This complicates the issues I have with this topic every year. We do scrum, so the team is supposed to be working on tickets selected by the product owner, so while you can influence them, you don’t have a ton of control over what you build. You can pick goals around how you work, like the mentoring and technical debt goals I had last year, which you can apply to whatever you end up doing. The more project based goals were based on large work items I knew were coming down the pipeline and probably wouldn’t be moved.

If I stay on my current team I have a good idea of what sort of work I’ll be doing and some process items to try to get to. If I move to a new team I’m not sure what exactly I’ll be doing. I also don’t know if what team I end up on will be finalized before I need to submit my official goals. I could write some goals around community involvement, or doing internal training/presentations. Maybe finding a way to automate some new metrics would be worth doing. I’ve always had difficulty finding good individual goals, since I see what we do as a team effort.

Individual goals are difficult to align with the team-based metrics generally used when doing scrum. This creates a sort of tension between individual goals that the team isn’t aware of and the team metrics and goals which are secondary to everyone’s individual performance. I see the goals for Senior Engineers as needing to be more team focused than those of lower lever engineers since they are more big picture oriented. If you are a Senior Engineer, you’ve reached the point where you are trusted to tackle issues independently, which should creates space to focus not just on merely doing, but on doing better.

Advertisements

Long Term Open Source Success

I was having a discussion with some of my coworkers at our engineering book club, talking about the first few chapters of Clean Architecture. We were discussing whether anyone had worked at a place that took a really long-term view of the architecture of a system. Most people didn’t think they had, I thought I might have once but it was hard to say if the company’s leadership had that view or that they got lucky with some key engineers just making it happen. Unfortunately the company ran into some business issues, with the very large competitors in the industry picking off their largest customers. From there I posited a different question to the group: is the architectural difference between open source software and commercial software one of the reasons for its long term success?

There ended up being a palpable pause and the idea at once made sense to everyone, but nobody was sure of how to try to confirm or deny the thought. We all agreed open source, at least initially, is built by software devotees for personal reasons. Whether it is for personal usage, to learn something, or to prove out an idea, people take time out of their day and build a piece of software and put it out there for the world to see. It’s finished when it’s finished and if you aren’t enjoying working on it then it’s no big loss to just set it aside and do something else. There was some discussion around the open source code with paid support model and whether that had more to do with the development of large chunks of open source software. There was a discussion about a resurgence in popularity for postgres because it was feature packed and solid, and whether its continuation in flexibility was because of an underlying architectural quality difference, given that it does not have corporate backing like other common relational databases.

From an ideological point of view, the idea that software succeeds in the long term because of better architecture is greatly pleasing. I would love to see data bearing that idea out, but I don’t know how you would get access to an appropriately large selection of equivalent projects. Something like The Architecture of Open Source Applications tries to make it easier to understand the big picture of some successful and long lived open source applications, but you would need a match set of closed source applications to compare and contrast against.

Building a great architecture requires taking time to deeply understand the problem. Sometimes you truly need to “build one to throw away,” knowing all the cost that represents. The pressure of commercial success and unbridled growth puts short term thinking into the forefront, and often prevents throwing it away.

Open source is a thousand developers trying to fix problems they run into alone and putting it out there. When you hear about the solution you go take a look at it. If its API doesn’t work for you the project doesn’t gain a new follower and without followers there are no contributors and the project eventually withers and dies. If the API does work for others and it solves problems the project grows. You can look at it as a distributed genetic algorithm doing API first development. You rarely hear about the ideas that don’t catch on; you hear about the things  where people felt they gained value.

One of my favorite definitions of software architecture is making the decisions that are hard to change. If an open source project decides they are just going to change their underlying central abstractions for the next year, that’s a strictly technical decision. They don’t need to build a business justification, or fight to get it on the schedule. They don’t even necessarily need a consensus that it’s a good idea, they could just fork the project if they felt so strongly that was the proper solution.

At work we’ve got a central architectural abstraction that is, in my opinion, not right. I could go build a PR on my time to change it, but that change would then ripple out to a dozen other teams as they start using the new abstraction. I could help them adopt the new abstraction, but I would need a strong consensus from those teams to get the PR in. Even though I think that consensus exists in engineering, the product schedule doesn’t leave time for this sort of change. It’s hard to quantify the drag this abstraction is causing which makes that difficult. If you could use a library that was built by those who are quality obsessed or a library built by those who are trying to match schedule, which would you choose?

Book Chat: Elastic Leadership

I recently ran into a situation at work that I wasn’t sure how to resolve the specifics of the situation aren’t important to this post. I ended up rereading several books looking for some sort of kernel of knowledge that would give me some additional guidance on what to do. I started with Peopleware, moved on to Managing Humans, and finally ended up on Elastic Leadership. Here I found something to help with my problem.

The “something” was a description of how people are influenced that felt like it applied to my problem and helped break down my feeling in a way that I could describe to others. The influence description consisted of two axis, the type of influence (personal, social, and environmental), and ability vs motivation. It ended up with the six zones of influence. For example, personal-ability is influence through skills you have, while environmental-motivation is structural incentives like giving public recognition for different kinds of behaviors. Looking at the problem from the perspective of each zone helped me to articulate my problem and arrive at a course of action.

There are other useful constructs in the book as well. There is an alternative to the “Storming-Forming-Norming-Performing” model of group development. This alternative model has three stages: surviving, learning, and self-organizing. This construct is used to describe how the behavior of a manager should be different in different stages of the team’s development. When you are in the surviving phase the manager’s goal is to get the team to the learning phase. Once in the learning phase the goal is maximize learning and enable the team to gain the confidence to self-organize. I identified with this model, since it emphasizes that the role of a manager for a team in trouble is vastly different than a team in a good place.

Overall it’s an interesting read but a lot of it is what I would describe as management advice rather than leadership advice, in the sense that you need to be in a place of structural power to use a lot of it. Even then, understanding of the management perspective can help you understand the situations going on around you, like it did for me.

Engineering Initiatives

At work we have engineering initiatives that are technical projects that cross multiple teams. They include things like upgrading to a newer version of a framework across the company or building out the infrastructure for on-demand test environments. These initiatives all seem to share a common pattern. Engineering leadership decrees it should be done and then each of the individual teams add the relevant tasks to their backlog. Product then proceeds to prioritize everything above these tasks for each team. Then Engineering leadership starts asking questions of why nothing got done and begins putting pressure on individual engineering managers to get stuff done.

At this point different teams seem to react in one of two ways. They either find reasons they can’t finish the work and hence don’t want to start the work, or they try to negotiate with product on the priority of the work. None of this work was accounted for on the product roadmaps so negotiation is tough because it means bumping work that was promised to other parties. I know when my team this year was putting together its roadmap [1] I made sure that there were two good sized blocks of time for ‘Things people don’t know they need yet.’ This is the team’s way to try to add visibility for work coming in from other sources. I know one of the other teams essentially schedules 9 months of work for the year-long roadmap as a strategy to be sure they have capacity for whatever comes up from other sources. I like the solution my team uses since we can inject the two blocks into the schedule specifically so the work is better spread out across the year.

A lot of organizations need to tackle systematic architectural improvements and the engineering initiative practice is a way to structure that work. I would like to see the engineering initiatives being better molded into the high level product roadmap. Individual product owners understand the importance of the initiatives, but there isn’t anyone on that side driving it and it really comes down to each individual engineering manager or team negotiating with the product owner about the priority of these efforts and that can result in them not getting the priority they really need if the initiative becomes security critical.

[1] Given the annual nature of the business we’re in an annual planning cycle makes more sense than more “by the book” scrum style backlog management. We still do individual sprint commits but being at the roadmap for when your team’s annual cycle comes due is important.

Book Chat: The Master Algorithm

The Master Algorithm is a description of the pros and cons of different machine learning techniques and the author’s quest to unify them into a single algorithm that can tackle any kind of problem. It has sections on five major kinds of learning algorithms: nearest neighbor, naive bayes, decision trees, support vector machines (SVM), and neural networks. It then covers Alchemy, the author’s attempt to unify multiple disparate styles of learning algorithms into a single overarching implementation. Overall the book succeeds at a popular science level of description of the current status of machine learning techniques, but it didn’t satisfy my needs as someone closer to the software.

The author declared an intention to keep the amount of math to a minimum. It ended up that the author tried to describe mathematical concepts in prose, and that didn’t work as well for me as just using some formulas probably would have. I wanted either enough math that I felt that I understood what was going on fully or no math at all so I didn’t feel like I had a mediocre partial understanding.

The descriptions of the individual learning algorithms made sense with a surface read. Once I stopped to try and really understand the differences it was less clear, I think the lack of hard formulas impeded my understanding. There was a section that described a unification of several different algorithms, but turned out to be a metaphor for early attempts to unify learning algorithms where the math never worked. The inclusion of this effort and the way it was described ended up being confusing to me.

The discussion of the real, but incomplete, unification scheme in Alchemy was interesting. The implication that you would still need an advanced degree in machine learning to use it says to me there is more to do. If machine learning is truly going to change the world the means of training models needs to be opened up to at least the average software engineer, if not business users.

I feel like the author’s particular perspective put him too close to the problem to really write a more popular science style book. I think he could have written the book I was looking for with more math and a more practical software aspect. From a technical prerequisite perspective this could be an excellent text for people who are trying to come at machine learning from a non-technical background.