A somewhat obscure question got asked in a chat channel at work that I knew the answer to, which helped out some other engineer. The question wasn’t anything that abnormal – it was about a weird error message coming from an internal library. Searching through the library’s code wasn’t immediately helpful since the unique part of the error message didn’t appear in the code. The reason I knew the answer wasn’t because it was easy, but because I had spent an hour investigating it the day before.
Sometimes when you see someone have an apparently impressive insight, that doesn’t necessarily mean they are better than you, they may just have had an experience which makes the answer obvious to them. This applies to all sorts of other technical activities. During the Hackathon I did a similar thing. One of the other devs on the team was integrating the portion of the code I was working on and having trouble. It was immediately obvious to me why, because I had put in the time earlier to figure it out the hard way. Your mind is a powerful pattern matching system. It immediately recognizes this:
If you think back to when you first started learning calculus, the terminology and symbols of it were complicated and foreign, but after a while you gained a certain familiarity with them and after a while they became second nature.
You may go to work and make some business web app in one particular technology stack, but there are all sorts of concepts that go with it that aren’t the business or the tech stack. You’re synthesizing things like design patterns, test driven development, RESTful web services, algorithms, or just the HTTP stack and everything that goes with that. These are all the transferable skills that can help you “cast a spell” and jump past a problem.
When I sat down to learn Scala, it wasn’t that big a task since most of the language features had equivalents I was familiar with in other languages. That let me skip forward to the nuances of those implementations and the few language features I was less familiar with. Getting experience with those ideas in the abstract let me appear as a wizard going forward since I jumped ahead on the learning curve and look the wizard. Some of the common feelings of impostor syndrome are the worry to be found out like another wizard.
At work we recently held a hackathon where everyone who was interested in participating had 24 hours to build whatever they thought would be interesting and useful. We had 21 teams across 4 offices with ~60 participants total. I had never done a hackathon before and this seemed interesting so I registered for it without a particular idea in mind. As the start rolled around I was planning to put together a tool to simplify how we were generating configurations for containers in marathon. However, at the kickoff pizza dinner I heard another developer saying he had a plan to solve our issues concerning the lack of available conference rooms. Every afternoon there would be an hour or two where there were no meeting rooms left available, and while we’ve got more space coming it will be a while until it is available. Having been a victim of this problem before I asked if they could use another hand on their team and was graciously invited onboard.
The key insight they had was to use the logs from the wireless network to figure out who was in what office each day. Once we had an idea of who was in what office that day we could cross reference that with their calendar appointments and see what rooms were booked and didn’t need to be. There was some concern about false positives, i.e., we didn’t want to have the system saying that you weren’t in the office by 10 so release your room at 11 while you were stuck in traffic. So we built a hipchat integration to check with you about it.
The three of us started Thursday night at about 6 with a general divide and conquer along the three major components: data mining/parsing, calendar matching and decision, and the hipchat integration. I mostly worked on the hipchat portion. Since the bot had to reach out to specific people on it’s own volition as opposed to responding to people or messaging a fixed channel, our needs were different than what most of the prebuilt hipchat integrations are doing. I ended up doing an XMPP integration using Smack. The biggest challenge in getting this working in the context of a web service was that I needed to keep the connection to hipchat open longer than the API implied it needed to be. I found this out when my initial attempt to send a message and then close the connection failed because the message didn’t finish going through but we had closed the connection on our end. After spending several hours working through that I called it a night at about 1:30 a.m. and headed home to catch some sleep.
Getting back the next morning at about 7:30, in my office there was one lone developer who had been there working on his project all night. He had been working on porting a feature from the web app to the android app, because when he used the app he wanted that feature. I spent the first part of the morning working on getting the response from hipchat hooked up and found another interesting problem. I wasn’t able to respond to myself as the bot for whatever reason. So if the bot was using my credentials to send messages it wouldn’t see my response to it. I suspect it was because hipchat was being clever and not sending the message as a message but some sort of history, but I never was able to confirm. At 8:30 the dev who had been working on the matching stuff for our project got in and started processing live data for that day; our app immediately started spitting out rooms we thought didn’t need to be booked. I went and did a little scouting at about 10:30 to confirm the situation and matching seemed right.
We ran into a credentials snag on getting an account with the rights to unreserve other people’s meetings. So we didn’t have a full demo but the example meetings we had identified painted a pretty picture of how well it could work and the number of rooms it could free up.
When demo time rolled around we all got together to show off what we built. There was a bunch of interesting stuff put together. There was a set of living visualizations of service dependencies built by parsing urls from the system configuration data. There was a port of one of our mobile apps to Apple Watch. There were two different teams that built Alexa integrations for different portions of our products. Several teams built features for various mobile apps. One team set up a version of Netflix’s Chaos Monkey in the load test environment, including a hacked Amazon Dash button that would kill a server in that environment at the push of a button. Another team built a deploy football in the vein of the nuclear football complete with keys and switches and a little screen to display progress. Two tech writers twisted arms and got someone to build a hipchat integration to look up acronyms from a glossary they had put together on the wiki.
Overall I had a blast but ended up pretty exhausted from the ordeal. Some prizes will be given out on Monday. I’m not sure of the exact criteria for them but I wasn’t competing at all – I was enjoying the latitude to do what I thought was best. There will be one more prize given out at the company-wide all hands wherein everyone gets to vote on as the most impactful project after we get a chance to see how everything turns out in real usage.
“Write the code you want and then make it compile” was a thought expressed on library design while I was at the NE Scala Symposium. It is a different way to describe the TDD maxim of letting the usage in tests guide the design. It is very much influenced by the extremely flexible syntax rules and DSL creation abilities in Scala. One of the talks, Can a DSL be Human? by Katrin Shechtman, took a song’s lyrics and produced a DSL that would compile them.
Since you can make any set of arbitrary semantics compile, there is no reason you can’t have the code you want for your application. There is an underlying library layer that may not be the prettiest code, or may be significantly verbose but you can always make it work. Segregating the complexity to one portion of the code base means that most of the business logic is set up in a clean fashion and that the related errors can be handled in a structured and centralized fashion.
Taking the time to do all of this for a little utility probably isn’t worth it, but the more often a library is used the more valuable this becomes. If you’ve got a library that will be used by hundreds, really refining the interface to make it match how you think would be really user friendly.
Building software that works is the easy part, building an intuitive interface and all of the comprehensive documentation so others can understand what a library can do for you is the hard part. I’m going to take this to heart with some changes coming up with a library at work.
This still doesn’t even cover the aspect of deciding what you want. There are different ways you can express the same idea. The difference between a function, a symbolic operator, or create a DSL can all express the same functionality. You can express the domain in multiple ways, case classes, enums, or a sealed trait. You can declare a trait, a free function, or an implicit class. Deciding on the right way to express all of this is the dividing line between a working library and a good library.
I’m continuing to dissect and share my experiences at the NE Scala Symposium. Several presenters referenced the “three monad laws,” which I had never heard of before. A monad must adhere to three laws:
- Left Identity
- Right Identity
The expression of left identity and right identity were new to me. I understood the concept of identity but wasn’t familiar with the distinction between right and left. Apparently the identity I was familiar with assumed a commutative relationship which means that the left and right identity are the same thing. From wikipedia “Then an element e of S is called a left identity if e ∗ a = a for all a in S, and a right identity if a ∗ e = a for all a in S. If e is both a left identity and a right identity.” Trying to think of an example, the right identity of division would be 1 but the left identity would be the value squared.
Associativity is the final law: this is the normal associativity that we’re all used to from basic math, which is expressed as X * (Y * Z) = (X * Y) * Z.
That’s the total of the laws, it’s nothing hard to accomplish. Like a lot of the pieces of functional programming each individual piece isn’t anything magical but the way they interact creates a special sort of synergy.
Traveling has always made me introspective. When I was going to the West coast regularly for work, my wife knew she would get a rambling email from me that I wrote while in the plane. Some of them were sappy, some were crazy, some were a disjointed mess. But they were all those thoughts that came out when I was stuck there with nothing but those thoughts to keep me company.
Sure I’d read, listen to a podcast, or watch something but eventually the mind would wander to that place it wanted to go. Like a very slow form of meditation. Once my mind emptied of thoughts of the day or about where I was going, I achieved this zenlike state where answers to questions would just unfold like an origami crane.
The worst part is that the answers were always fleeting. There for a moment, gone the next without the chance to fully understand the epiphany. A deep insight into the universe that you know was there, but never get to appreciate.
This post started with one of those epiphanies I was on the train on the way to New York for the NE Scala Symposium and there was this moment of clarity about a communication struggle I had been having at work. For a moment, I had a vision of the creative action plan I had been looking for, then a PA announcement came on and the thought was gone. I hadn’t recorded the thought in any way but I know it was there right outside of Trenton.
On the ride home I managed to rekindle some of the thought, but it wasn’t the same deep insight that I had originally had. Initially, my work team had been proposing to adjust this existing framework to enable a new usage. But by rephrasing it as “replacing” the entire framework and building a brand new system everyone was immediately on board. By reframing the initial idea from being a change to a new thing it got everyone on board. I think this reasoning is twofold, it would mean we can roll out the change in smaller increments and we can go back without as much effort. It’s the same work and the same expected end state, but the reception was significantly different.
I had a stray thought about fragments in JSON documents, after a colleague asked about if we had considered using GraphQL to allow allow querying subsections of a larger document. It seemed like you could reasonably index into a JSON document and reference a subsection of it in a simpler fashion. I went looking for information about this and found RFC 6901 on JSON pointers, describing exactly what I was theorizing. However, it doesn’t appear to have been adopted yet. The gist is that you can index into a document with #/<key>. Little mystery solved, but while researching this I stumbled into something else more interesting.
I found this about best practices in designing fragment IDs when creating a media type. But it discusses these best practices in relation to several schemas that were out in the wild. This had the JSON pointer example, it also had XML and PDF examples, which seemed reasonable. But it also had plain text and CSV, which was interesting as the format i’s a line number and a character/field which works but I had never really considered attempting. The more interesting one was the SVG, since it lets you index into an image. You can index into a named section or any arbitrary view box.
This was an interesting diversion that really reminded me about the amazing flexibility and foresight of some of the technologies underlying the web.
This idea came together from two different sources. First, I’ve been reading The Fifth Discipline about the creation of learning organizations. One of the elements of becoming a learning organization is Systems Thinking, which I had heard of before and seems like a great idea. Then I went to a local meetup about applying Systems Thinking to software development. We ran through a series of systems diagrams from Scaling Lean & Agile Development. These diagrams helped us to both understand Systems Thinking by analyzing a domain we were already familiar with in a structured way where we were presented the nodes of the system diagram and asked to draw connections between them.
There were several different groups each looking at the same sets of nodes and drawing different conclusions for some of the relationships. There were a few very adamant people who all felt that an increase of incentives for the developers would increase the velocity of the team, which was interesting to see since they were mostly scrum masters or software development managers with many years of experience doing the management of software projects and should have seen how that does work out. If the impact of an incentive plan on a team is significantly uncertain, it makes me curious about the sorts of teams these other leaders are running.
Through the entire process we had interesting conversations about how various aspects of the software process were related. Everyone agreed that the interaction of a strong mentoring process on the overall system would decrease the number of low skills developers on the team. There was a discussion of whether it would also directly impact the velocity of the team. Some people were adamant that it would lower velocity since the mentoring time was time that people weren’t working on building features. It’s a reasonable consideration, but it doesn’t seem to match with my specific experiences with mentoring. If you were actively spending enough time to lower your velocity significantly on mentoring activities you would be forcing the mentoring relationship instead of letting it happen organically.
The experience made me wonder if we could construct a system diagram that describes some of the other dysfunctions of many organizations. The diagram we built described (1) dysfunctions around hiring large numbers of low skill developers, (2) using certain kinds of financial incentives, and (3) the push for delivering features above all else. It didn’t describe why a lot of organizations end up in siloed configurations, “not invented here” behavior on technical systems, or lots of the other dysfunctions in multiple organizations. I made the below diagram with the intent of describing the situation about siloed organizations but without any real experience in this sort of analysis I’m not sure if I’m doing it well.
I made another simple diagram about the causes of “not invented here.”
It feels to me like these diagrams describe the dysfunctions that are common in software organizations. Expanding these diagrams to try and find the leverage points in the system might yield some greater insights into the problems. In spending some time thinking about both of the diagrams I’m not sure what other nodes should be there to further describe the problems.
I’m going to definitely do some more reading on Systems Thinking and try to expand on the thoughts behind these diagrams. If you’ve got more experience with System Thinking I’d love to hear some feedback on these charts.
Looking back at this year’s blogging, strictly by the numbers everything is much bigger than last year. Last year there were 63 pageviews by 42 visitors, this year there were 440 views by 359 visitors. Part of the increase was since I have a whole year rather than a couple of months’ worth but that doesn’t account for the whole increase.
Last year I outlined some goals for this year.
- Increase readership
- Keep up the weekly posts
- Write more technical posts
- Write about what I was doing at work some
I feel like I achieved each of these goals. The views were up considerably, for both the new posts and some of the old posts. I hit the weekly post goal, although I went through a good bit of the backlog of post ideas I had. I got some more technical posts with some of the Nuget conversion, Scala posts and the anonymous type serialization. The Nuget conversion was what I was doing at work and I got to talk about that some.
I’ve got a couple of goals going forward for 2017
- Continued increases in readership
- Keep up the weekly post cadence
- Write a longform piece and see what that looks like in this format
I don’t have any specific means to accomplish the readership increase. Last year I added references to the blog in a couple of profiles I had other places, but that seems to have only generated five referrals. Since a quarter of the views came from one post I’m slightly concerned that readership may go down. The weekly post cadence helps me keep in the writing habit and while there have been some posts that didn’t generate any views, that’s something I think I can live with. I’d like to try the longform piece since a number of the posts I’ve been writing are just pointing towards other people’s ideas and not a lot of new thoughts. To a certain point the problems I’m fighting and the solutions I’m deploying aren’t that different than what other people are doing so a lot of these ideas are already out there in the world.
This is a bit outside of the normal material I write about here, but I felt that it was something that others might appreciate as well. One of my wife’s classmates gave her a copy of Daring Greatly and said that it was part of his inspiration to be in their doctoral program. I’ve always read broadly and tried to find a way to integrate that into my life. I saw this book and wasn’t sure if it was the sort of thing for me, but I decided to read it to find out. I started reading it and maybe a third of the way through I thought to myself, “I’ve read this book before but with the word vulnerability replaced with with word authenticity.” But I kept reading and eventually something started resonating with me.
What resonated was the idea that you need to put yourself out there, give more of yourself to the situation, and say those things you are thinking. You have to do that even if it’s not be easy, because it is important and is part of what separates good from great. The willingness to say something that puts yourself in a vulnerable position to open yourself to those around you takes a bigger commitment than most people are willing to make. People are willing to say the easy thing but not the hard things that require them to push against the structure around them. It’s like the Emperor’s New Clothes, everyone sees something but the incentive structure is put together so they don’t recognize the problem. This book is all about how to structure your own thoughts to be able to push against the structure around you.
There are portions of the book written in an abstract sense and some others that are much more specific. The specific sections contain several manifestos to describe how leaders or parents should behave around those they are responsible for. It puts out there that you may not be perfect but that you strive to be better and hope that everyone else engages with you in trying to be better. Each of the manifestos describes how the person in that situation can open up to those subordinate to them to truly embrace the position they are in.
I’m not sure if this is that profound, or if it found me at the point in my life that had me open to what it was saying. However I felt moved by this. It made me feel that I should push outwards and express my opinions further. I had been expressing myself in some domains, however it is hard to put yourself out there in all areas all of the time. Sometimes you just want to wait and let things happen, but sometimes you need to make things happen. Not in the doing sense but in the living sense, where you can’t just wait for something to happen but need something to progress the situation.
The Mikado Method describes a way to discover how to accomplish a particular refactoring. The method itself asks that you first attempt to do what you want “naively” and identifying the problems with that approach. Then you roll the code base back to the original state in order to tackle one of those problems and iterate on the process until you can begin to resolve the problems in a bottom-up fashion, resulting in multiple small refactorings rather than one big one. This strategy means you can merge or push with the master branch more frequently since the codebase is regularly in a working state. This avoids the rabbit hole of making changes and more changes and never being sure how close you are to having compiling software with a passing test suite again.
The actual description of the technique and examples is only about 60 pages of the roughly 200 pages of the book; most of the rest is other tips and tricks for working on refactoring. There is also a rather long appendix on technical debt that I found expressed some ideas I had been thinking about recently; it describes four techniques for tackling the sources of technical debt.
The four techniques listed are absolve, resolve, solve, and dissolve. Absolve is essentially normalizing the practice and saying it is okay to do things this way. This would be something like lowering the automated test coverage necessary during a hard scheduled push. Resolve is reverting a change in the current processes and environments that had unintended negative effects. This would be something like getting rid of an internal bug bounty if it was being abused. Solving is changing the incentive schemes in order to put groups into alignment. For instance, having development teams on call in order to help align their incentives with the operations teams. Dissolving is the sort of radical solution that completely removes the friction between groups and makes the problem disappear completely. To continue with the previous example this would be a sort of devops culture where operations and developers are all on the same team and there is less distinction between the two. Each of these techniques could be applied to various means that create technical debt, or even to other sorts of problems.
The actual Mikado technique doesn’t seem book-worthy in the sense that it isn’t complex enough to warrant an entire book on it’s own. The other refactoring techniques weren’t anything particularly novel to people who are already familiar with Refactoring Legacy Code or other similar material. Overall it was a quick read and enjoyable but not the sort of thing I would be recommending to others strongly.