Hackathon

At work we recently held a hackathon where everyone who was interested in participating had 24 hours to build whatever they thought would be interesting and useful. We had 21 teams across 4 offices with ~60 participants total. I had never done a hackathon before and this seemed interesting so I registered for it without a particular idea in mind. As the start rolled around I was planning to put together a tool to simplify how we were generating configurations for containers in marathon. However, at the kickoff pizza dinner I heard another developer saying he had a plan to solve our issues concerning the lack of available conference rooms. Every afternoon there would be an hour or two where there were no meeting rooms left available, and while we’ve got more space coming it will be a while until it is available. Having been a victim of this problem before I asked if they could use another hand on their team and was graciously invited onboard.

The key insight they had was to use the logs from the wireless network to figure out who was in what office each day. Once we had an idea of who was in what office that day we could cross reference that with their calendar appointments and see what rooms were booked and didn’t need to be. There was some concern about false positives, i.e.,  we didn’t want to have the system saying that you weren’t in the office by 10 so release your room at 11 while you were stuck in traffic. So we built a hipchat integration to check with you about it.

The three of us started Thursday night at about 6 with a general divide and conquer along the three major components: data mining/parsing, calendar matching and decision, and the hipchat integration. I mostly worked on the hipchat portion. Since the bot had to reach out to specific people on it’s own volition as opposed to responding to people or messaging a fixed channel, our needs were different than what most of the prebuilt hipchat integrations are doing. I ended up doing an XMPP integration using Smack. The biggest challenge in getting this working in the context of a web service was that I needed to keep the connection to hipchat open longer than the API implied it needed to be. I found this out when my initial attempt to send a message and then close the connection failed because the message didn’t finish going through but we had closed the connection on our end. After spending several hours working through that I called it a night at about 1:30 a.m. and headed home to catch some sleep.

Getting back the next morning at about 7:30, in my office there was one lone developer who had been there working on his project all night. He had been working on porting a feature from the web app to the android app, because when he used the app he wanted that feature. I spent the first part of the morning working on getting the response from hipchat hooked up and found another interesting problem. I wasn’t able to respond to myself as the bot for whatever reason. So if the bot was using my credentials to send messages it wouldn’t see my response to it. I suspect it was because hipchat was being clever and not sending the message as a message but some sort of history, but I never was able to confirm. At 8:30 the dev who had been working on the matching stuff for our project got in and started processing live data for that day; our app immediately started spitting out rooms we thought didn’t need to be booked. I went and did a little scouting at about 10:30 to confirm the situation and matching seemed right.

We ran into a credentials snag on getting an account with the rights to unreserve other people’s meetings. So we didn’t have a full demo but the example meetings we had identified painted a pretty picture of how well it could work and the number of rooms it could free up.

When demo time rolled around we all got together to show off what we built. There was a bunch of interesting stuff put together. There was a set of living visualizations of service dependencies built by parsing urls from the system configuration data. There was a port of one of our mobile apps to Apple Watch. There were two different teams that built Alexa integrations for different portions of our products. Several teams built features for various mobile apps. One team set up a version of Netflix’s Chaos Monkey in the load test environment, including a hacked Amazon Dash button that would kill a server in that environment at the push of a button. Another team built a deploy football in the vein of the nuclear football complete with keys and switches and a little screen to display progress. Two tech writers twisted arms and got someone to build a hipchat integration to look up acronyms from a glossary they had put together on the wiki.

Overall I had a blast but ended up pretty exhausted from the ordeal. Some prizes will be given out on Monday. I’m not sure of the exact criteria for them but I wasn’t competing at all – I was enjoying the latitude to do what I thought was best. There will be one more prize given out at the company-wide all hands wherein everyone gets to vote on as the most impactful project after we get a chance to see how everything  turns out in real usage.

Both Project Management and Coaching?

This is somewhat outside of the normal material I write about, but this idea came to mind and I wanted to take an opportunity to explore it further. The context of this thought was that a mentoring program is being launched at work. Why is a manager both responsible for guiding the day-to-day execution of a project and for development of staff beneath them?
The goals of the project and the goals of personnel development are often at odds. Any specific experience a person needs or wants to get is likely something they don’t have much (if any) experience with yet, which means they won’t be as efficient or as good at that particular skill. While management taking care of your people gains you flexibility and goodwill in the long term, it has short term costs.
The product owner is separated from the scrum master, separating the “what” from the “how” in order to prevent a conflict of interest. Similarly, splitting the staff development aspect from the technical management of the project seems like it could prevent a similar conflict of interest about an individual. Maybe a stronger explicit mentorship program would handle mitigate this conflict of interest, but unless you give out a mentor who is both capable of and interested in working on staff development it wouldn’t help much. I have seen an explicit mentorship program where serving as a mentor was informally required for promotion to higher level, which resulted in people becoming mentors explicitly to check the box for their own personal benefit.
By setting up part of management to be incentivized to do staff development and achieve technical excellence, rather than completing projects or shipping features, you can create an environment that allows those closest to the system under development to do their best without the pressure to achieve explicit results. This reminds me of the idea of slack in queueing theory, where putting less work into the system means more work will come out. Once you build up staff appropriately and get everyone cross-trained, the overall outcome becomes better. Think of it as an optimization problem where you may have achieved a local maximum; getting to a higher peak would be better but the cost of valley between the peaks needs to be paid.
You could theoretically have the manager of the developers on a team be involved with the work of a different team, but it would be hard to see what those developers needed in order to help develop them in the longer term. Spending all day with a group of other developers working on the technical problems you face doesn’t really give you the insights necessary to see what a completely separate group of developers is struggling with. If you look at a problem with a new perspective, you can easily see different solutions, you could also miss important details about the problem itself and backtrack to already tread territory.
Maybe this is just my perspective based on the places I’ve worked, where the scrum master has been more of a process leader and impediment resolver than technical coach or project management. In my experience, development managers have spent a large portion of time working with the product owner to help factor stories in a more completable fashion and to derive the technical requirements from the business requirements. It always felt like the development manager spent more of their time working on those urgent but not really important things, and ignoring the important but delayable things because they were hard.
To answer the original question “Why is a manager both responsible for guiding the day-to-day execution of a project and for development of staff beneath them?” it seems to be because splitting up the responsibility differently won’t give management the day-to-day visibility into how to effectively provide useful developmental guidance. The ability to coach people on how to perform their role better requires that you spend enough time seeing them perform the role, so you can’t really be engaged in other technical activities on a day-to-day basis. Even if they had the insight to do so, it is exceedingly hard to measure staff development, which would make it hard to create goals and metrics around the activity. If you’ve had a different experience with how these roles have been broken down post a comment.

Future[Unit] Harmful

I’m not the first person to have seen this behavior but I saw it bite another engineer recently so it’s clearly not well enough known. The Unit type in Scala is a special type, there is effectively an implicit conversion from every type to Unit. If you are interested in the specifics check out this post, I’m not going to get into the specifics of how it works or the edge cases. This means that you can write the something like

val doubleWrapped: Future[Unit] = Future.successful(Future.successful(true))

and that compiles. It breaks the type safety that we have come to expect. More intriguingly if you were to do

val innerFailure: Future[Unit] = Future.successful(Future.failed(new RuntimeException))
innerFailure.value

what would you expect the result to be?

If you said

Some(Success(()))

you would be right, but that would probably not be what you were looking for. If you had a map where you needed a flatMap you would end up with compiling code that silently swallows all error conditions. If you have an appropriate set of unit tests you will notice this problem, but you can have that code that looks simple enough that it doesn’t need any unit tests.

The compiler flag -Ywarn-value-discard should give you some protection, but you need to turn it on explicitly and it may produce a fair bit of news in an existing large codebase. So keep an eye out for this issue, and be forewarned.

Generators for Infrastructure

I listened to a recent episode of .NET Rocks about CI/CD pipelines and the guest had built a yeoman generator to build out the pipeline. It is specific to the Microsoft offerings of Visual Studio Team Services and Team Foundation Services, but it is an interesting idea. There isn’t a reason you couldn’t build something similar on top of Travis, Circle, or Semaphore. I had touched on some of the other generators at work previously, but those were bootstrapping service and application code, not infrastructure. But that’s the advantage of infrastructure as code – you get all of the toolchain available to work with it.

At work we have code to build out the CI/CD pipelines but it is more of a desired state thing than a generator. It’s all built on top of a local Jenkins install, and regularly recreates all of the job definitions to prevent any changes made by hand from living very long. At previous jobs the automation for building different services and libraries were all a little different or built in a brittle template/multipart style that didn’t allow for any changes in a safe way. Our tool has been great for keeping all of the microservice jobs the same, templated in a way that makes upgrading them in groups easy enough, and even fairly testable.

The testing aspect is part of what’s most interesting, we boot up a Jenkins instance inside a container and generate the jobs into it, with some different configuration for notifications and where to put artifacts generated to blackhole all of it. The deployment jobs work using the same version running to the dev cluster so it should all be idempotent. The biggest issue I’ve seen so far is that it runs a ‘representative sample’ of jobs as part of testing; that hasn’t  always been as representative as we would want it to be especially as we add new jobs. We’ve got roughly 300 repos but only maybe 150-200 use managed builds like this. The good news is they break much less often than the unmanaged ones, but the bad news is when they break it’s usually a large swath of them that go down all at once.

AWS CodeStar was recently announced as a hosted version of this sort of tool. It creates repos, scaffolds a new application, can deal with permissions, and carries through to the provisioning of hosting as well. It has options for multiple languages and hosting options. So it seems like it is simplifying a lot of the initial setup of a service.

I’m not sure where the tipping point on the value of setting up your own infrastructure would be, but it seems like at least 100 repos depending on the complexity of each build. This sort of automation will become more important as microservices deployment continues to get bigger. If you haven’t done any sort of automation like this it’s worth looking into depending on the scope of the project.