Book Chat: Debugging Teams

Debugging Teams is the second edition of Team Geek, rewritten to apply to a more general audience but still fundamentally the same material. It has a variety of fun and interesting anecdotes from the authors’ time running teams at Apple and Google as well as organizing a major open source project. There is some pragmatic advice about how to work within an organization as well as lead a team. Most of the book is directed towards a team leader, not a leader of a department, as would be expected by the title.

I feel as though the rewrite doesn’t make it apply to a general audience sufficiently enough. I felt that the rewrite does get away from the idea of running a software team, but doesn’t fully bring it to the any team perspective. The rewrite does get to the level of a team that works like a software team where you can iteratively deliver value. You couldn’t use much of the advice  to run a lot of other professional work where a report/result of knowledge work is delivered then you move onto another project. Some of the specific advice would still apply in these scenarios but it doesn’t work in the general management sense.

Overall it’s a fun read, but didn’t hit the target they were aiming for. It is valuable advice for someone who is running a software-like team. It still focuses on the basic practices of what to do as a leader, but doesn’t go enough into the how for me. I know I don’t personally run a team because I don’t enjoy some of the required aspects of project management, but I do really appreciate the skills and abilities required to do it well.

Advertisements

Who is the Customer for Security?

When looking at an application from a security perspective I can assess the controls involved and try to understand the threats it faces. But then comes the question, is this application secure? You can look at an application and declare it insecure easily enough. Finding a flaw in the design of the controls or the implementation makes an application insecure. Essentially all applications appear to have vulnerabilities according to this 2017 report. When you have the keys to the kingdom to protect you can go to significant lengths to protect them, like AWS does with its provable security measures.

For those of us with more limited means, we need to find a way to determine where is the line for what we should do. There isn’t a clear line for what’s good enough, a risk based approach makes too many guess, and an absolute approach doesn’t scale with organizational size. You build a system that looks secure to the best of your knowledge and monitor it to make sure it stays that way. You counter new known vulnerabilities as they come up. You can try to be proactive with scanning; you can try to put yourself in a good place to be reactive to issues that come up. You can build defenses in depth and apply any sort of new emerging technologies to the problem, but if the only restriction to how much work you put into security is how much money you have, how do you decide when you have enough?

Some searching turned up this formula

ROI=(Average loss expected / cost of countermeasures)

Conceptually it makes sense, but what is the average loss expected? The article defined it as (Number of Incidents per Year) X (Potential Loss per Incident). That’s still a guess at best, if you experienced some number of incidents in the past you can project forward. On the loss side you have reputational damage from the incident, the cost of response, any regulatory fines. So at this point you have a guess times a guess and you estimate how any action you take will impact those two guesses to determine if it is worth doing or not. That doesn’t seem like a good way to measure if you are making an impact.

I’ve been trying to think through other ways to analyze this sort of position. I keep coming around to something similar to downtime analysis. How confident are you in understanding your security and how long does it take you to react to something that needs to be remediated once you’ve identified it? This formulation gives you a different way to scope your security activity. It brings a number of different things as security related that wouldn’t make sense under the earlier formulation. Things like the scalability of your build and deployment pipeline are a security concern, because if you need to build and redeploy hundreds of services after finding out about a vulnerability in a commonly used library it would be good not to bottleneck the system.

I think this metric makes sense from a more technical perspective of the security of the system compared to the incidents and losses perspective which makes more sense from a business risk perspective. Having this additional perspective in a

Book Chat: Manage It!

Manage It! is an overview of modern project management techniques. Most of it was accessible to me as a person who has never formally run a project, but has been involved in many. The less accessible material was concentrated towards the end which I dutifully worked through thinking there might be more immediately relevant portions after. It heartily embraced three different practices: strong meeting facilitation, rolling wave planning, and avoiding schedule games.

The meeting facilitation advice started out fairly straightforward – have an agenda, stick to the topic at hand, and hold one-on-ones with the team. It then goes on to discuss some more radical advice, like don’t go to meetings that aren’t about solving problems, question why you had the meeting if it ends and nobody has any action items, and avoid serial status meetings. If your project has a problem, getting the relevant people into a room and coming out with a solution is a great way to break the impasse. Other sorts of meetings can impact the progress of a project, but to me that doesn’t make them immediately a bad idea. From the perspective of the project manager I can see that other sorts of long-term work or out of band activity can impact the potential of the project, but it seems necessary for the functioning of a healthy engineering organization. I agree that the lack of action items coming out of a meeting seems like a warning sign it wasn’t a good usage of time. If a decision was made generally one or more people would leave the room and do something because of it. Serial status meetings are more complex; if you are holding a meeting where people tell you about progress being made on initiatives where the others in the room aren’t involved or impacted, it may be a good use of your time but it’s a bad use of everyone else’s time. If you are being invited to meetings to provide status, the book advises to send status via email and skip the meeting, the idea being that if the organization doesn’t accept that behavior then it isn’t the place you should be. Daily standups are not impacted from this practice because they’re about the impediments, not just the status. Overall it seems like a good package of advice as to how to interact with meetings.

Rolling wave planning is an implementation of the idea that your plan will fail, but that the exercise of planning is valuable regardless. Your short term plan should be pretty solid but the further out, the more vague the plan gets. So, you don’t worry as much about the long term, and as you acquire more information you update the plan. This works both for changes from outside the project and things you learn from executing the project itself. The one experience I have had with explicit rolling wave planning did not go well, but I feel that was because we were trying to keep the near-term solid plan too far into the future and engaging in some devotion to the schedule.

The schedule games section was the part that felt the most real to me, it listed out 16 ways a project plan can go awry and ways to cope with each of them. I felt like I had seen the vast majority of the pitfalls out in the wild. This previous visibility involved me more in the rest of the material since I felt the authenticity of this compared to a lot of books which don’t give practical guidance on  how to get from poor practices to good ones. The section on “schedule chicken” felt particularly familiar to me after having been involved in a high stakes version many years ago. We even made progress in getting out of the situation using one of the techniques described.

I would recommend this book to someone who has already led a team or project for a little bit and is interested in doing more of it. If you’ve been doing that sort of work as your primary role it probably still has some bits of interest. If you haven’t run a team or project yet you may get something from it but I feel that for a lot of the information you need to have seen the problems in action some before you can appreciate the importance of  avoiding it.

OWASP Dependency Check

The Open Web Application Security Project (OWASP) provides a number of tools and resources to help create more secure web applications. Their dependency check tool is designed to integrate into the build step of your application and tell you if any of your downstream dependencies have known security issues. The idea is a simple but powerful one – when your application is being built it pulls in a number of libraries and those libraries pull in even more transitive dependencies. You need to look at all of them and cross-reference the National Vulnerability Database to see if any of them have known issues. While you may know to keep an eye out for announcements of security findings from libraries you use directly, but you may not easily know the transitive dependencies your system expects. It’s like a more traditional vulnerability scanner, but rather than pointing it at existing servers to find those that are vulnerable, you put it into the build pipeline to find those you are about to make vulnerable.

The tool has integrations for a number of different popular build systems for the JVM and .NET. It also has experimental support for Ruby, Python, C++, and Node. This predates the github vulnerability scanner but works in a similar way. Since I’m working with Scala day to day using SBT as my build system it isn’t supported by github. I’ve been using an SBT plugin to start integrating it into build systems in various services and libraries. The report generated provides both the severity of the vulnerability and the likelihood that it matched the vulnerability to the binary. So far the likelihood has always been correct in determining whether the binary matches the one indicated to be vulnerable, but I’m still not confident in that since most of my repos were using similar dependency sets and thus generating similar reports.

Taking an afternoon to setup the scanner, run it against a variety of repos you work with, and look through the findings seems highly worthwhile. If you’ve been keeping up with your library upgrades you likely won’t find anything earth shattering. But if you have a dependency that isn’t keeping up to date then you might be in for an interesting surprise.

Book Chat: Securing DevOps

Securing DevOps is an introduction to modern software security practices. It both suffers and succeeds from being technology- and tool- agnostic. By not picking any particular technology stack it will remain relevant for a long time, however it is not a complete solution for anyone since it gives you classes of tools to find but not a complete package for software security. If you need to start a software security program from zero this lays out a framework to get started with.

While I’ve only been doing software security full time for a few months now, I feel like the identification of the practices to engage isn’t the hard part, it’s the specifics of the implementation where I feel I want additional guidance. I know I should be doing static analysis of the code as part of my CI pipeline, but I don’t know how to handle false positives in the pipeline or what is worth failing a build because of. I don’t know what sort of custom rules I should be implementing in the scanner for my technology stack.

The book did go further into detail on the subject of setting up a logging pipeline. It describes how to set up rules to look for logins from abnormal geographic locations and how to look for abnormal query strings. The described logging platform is nothing abnormal for a midsized web application, however, I don’t know if you could have a small organization and have this level of infrastructure setup. Hooking up the ELK stack, while open source, is not easy, and the kibana portion requires a fair bit of customization and time to get everything together and working.

It feels as though we are missing a higher level of abstraction for dealing with these concerns. Perhaps, the idea that most software applications should have to go through this level of effort to get ‘standard’ security setup for a web application is reasonable. Even on the commercial tools side there seems to be a lack of complete solutions. Security information and event management (SIEM) tools try to provide this, but they each still require significant setup to get your logs in and teach the program how to interpret them. It feels like some of this could be accomplished by building more value in a web application firewall (WAF). WAFs were not fully endorsed by the book due to the author having had a bad experience with a bad configuration problem. Personally, I think a WAF seems necessary to protect against distributed denial of service style attacks.

Overall the book is an introductory to intermediate text, not the advanced practices I was looking for. If you’re bootstrapping an application security program this seems like a reasonable place to get started. If you’re trying to find new tactics for your established program, then you’ll probably be disappointed.