Seven More Languages in Seven Weeks

Seven More Languages in Seven Weeks is a continuation of the idea started in Seven Languages in Seven Weeks that by looking at other languages you can expand your understanding of concepts in software engineering. While you may never write production code in any of these languages, looking at the ideas that are available may influence the way you think about problems and provide better idioms for solving them.

This installment brings chapters on Lua, Factor, Elixir, Elm, Julia, MiniKanren, and Idris. Each of these languages is out on the forefront of some part of software engineering. Lua is a scripting language with excellent syntax for expressing data as code. Factor is a stack-based programming language with interesting function composition capabilities. Elixir is Ruby-like syntax on the Erlang VM. Elm is reactive functional programming targeting javascript as an output language. Julia is technical computing with a more user friendly atmosphere, and good parallelism primitives. MiniKanren is a logic programming language and constraint solver built on top of Clojure. Idris is a Haskell descendent bringing in the power of dependent types to provide provably correct functional code.

Overall it was an interesting survey of the variety of programming languages. Some I had done a bit with before (Lua, Elixir) some I had heard of before (Elm, Julia, and Idris) and some I hadn’t even heard of (Factor and MiniKanren). Each chapter was broken into three ‘days’ indicating a logical chunk of the book to tackle at once. Each day ends with a series of exercises to help make sure you understand what’s being presented.

Since these languages are out on the edge of the world in programming terms, they are evolving fairly quickly. This ended up biting the Elm example code particularly hard since large portions of it have been deprecated in the releases since then and they didn’t work on the current runtime. Compared to the lineup from the original book (Clojure, Haskell, Io, Prolog, Scala, Erlang, and Ruby) you’ve got a much broader variety of languages in the sequel, but nothing with the popularity of Ruby or the legacy install base of Erlang. Since this was written in 2014, none of these have had a massive breakout in terms of popularity and adoption, however they do seem to do well in terms of languages people want to work with.

Overall it’s an interesting take on where things could be going.  I don’t think most of the languages covered have significant mainstream appeal right now. Two of these languages seem to be more ready for the primetime than the others. Julia definitely has a niche where it could be successful. I feel like the environment is ripe for something like Elm to surge in popularity since frontend technology seems to be going through constant revisions.

Advertisements

Java Containers on Mesos

I recently ran into an interesting issue with an application running in a container. It would fire off a bunch of parallel web requests (~50) and sometimes would get but not process the results in a timely manner. This was despite the application performance monitoring we were using saying the CPU usage during the request stayed very low. After a ton of investigation, I found out a few very important facts that contradicted some assumptions I had made about how containers and the JVM interact.

  1. We had been running the containers in marathon with a very low CPU allocation (0.5) since they didn’t regularly do much computation. This isn’t a hard cap on resource usage of the container. Instead it is used by Mesos to decide which physical host should run the container and it influences the scheduler of the host machine. More information available on this in this blog post.
  2. The number of processors the runtime reports is the number of processors the host node has. It doesn’t have anything to do with a CPU allocation made to the container. This impacts all sorts of under the hood optimizations the runtime makes including thread pool sizes and JIT resources allocated. Check out this presentation for more information on this topic.
  3. Mesos can be configured with different isolation modes that control how the system behaves when containers begin to contest for resources. In my case this was configured to let me pull against future CPU allocation up to a certain point.

This all resulted in the service firing off all of the web requests on independent threads which burned through the CPU allocation for the current time period and the next. So then the results came back and weren’t processed. Immediately we changed the code to only fire off a maximum number of requests at a time. In the longer term we’re going to change how we are defining the number of threads but since that has a larger impact it got deferred until later when we could measure the impact more carefully.

Book Chat: The Pragmatic Programmer

For a long time this had been on my list of books to buy and read with a “note to self” saying to check if there was a copy of it somewhere on my bookshelf before buying one. It felt like a book I had read at some point years ago, but that I didn’t really remember anymore. Even the woodworking plane on the cover felt familiar. It felt like it was full of ideas about creating software that you love when you encounter them but are disappointingly sparse in practice. Despite being from the year 2000 it still contains a wealth of great advice on the craft of creating software.

Since it is about the craft of software, not any specific technologies or tools or styles, it aged much better than other books. That timeless quality makes the book like a great piece of hardwood furniture, it may wear a little but it develops that patina that says these are the ideas that really matter. There is an entire chapter devoted to mastering the basic tools of the trade: your editors and debuggers, as well as the suite of command line tools available to help deal with basic automation tasks. While we’ve developed a number of specialized tools to do a lot of these tasks it is valuable to remember than you don’t need to break out a really big tool to accomplish a small but valuable task.

It’s all about the fundamentals, and mastering these sorts of skills will transfer across domains and technical stacks. It was popular enough that is spawned an entire series of books – The Pragmatic Bookshelf – and while I have only written about one of them I have read a few more and they’ve all been informative.

About two-thirds of the way through the book I realized that I had indeed read it before – I had borrowed a copy of it from a coworker at my second job. He had recommended it to me as a source he had learned a lot from. I remember having enjoyed it a lot but not really appreciating the timeless quality. Probably since that would have been around 2007, it wouldn’t have seemed as old, especially since things seemed to be moving less quickly then. Maybe I just feel that way since I didn’t know enough of the old stuff to see it changing.

If you haven’t read it, go do it.

tumblr_inline_o2aushqfpx1slrvm0_1280

Being a Wizard

A somewhat obscure question got asked in a chat channel at work that I knew the answer to, which helped out some other engineer. The question wasn’t anything that abnormal – it was about a weird error message coming from an internal library. Searching through the library’s code wasn’t immediately helpful since the unique part of the error message didn’t appear in the code. The reason I knew the answer wasn’t because it was easy, but because I had spent an hour investigating it the day before.

Sometimes when you see someone have an apparently impressive insight, that doesn’t necessarily mean they are better than you, they may just have had an experience which makes the answer obvious to them. This applies to all sorts of other technical activities. During the Hackathon I did a similar thing. One of the other devs on the team was integrating the portion of the code I was working on and having trouble. It was immediately obvious to me why, because I had put in the time earlier to figure it out the hard way. Your mind is a powerful pattern matching system. It immediately recognizes this:

 

happycatOr thisftc

 

If you think back to when you first started learning calculus, the terminology and symbols of it were complicated and foreign, but after a while you gained a certain familiarity with them and after a while they became second nature.

You may go to work and make some business web app in one particular technology stack, but there are all sorts of concepts that go with it that aren’t the business or the tech stack. You’re synthesizing things like design patterns, test driven development, RESTful web services, algorithms, or just the HTTP stack and everything that goes with that. These are all the transferable skills that can help you “cast a spell” and jump past a problem.

When I sat down to learn Scala, it wasn’t that big a task since most of the language features had equivalents I was familiar with in other languages. That let me skip forward to the nuances of those implementations and the few language features I was less familiar with. Getting experience with those ideas in the abstract let me appear as a wizard going forward since I jumped ahead on the learning curve and look the wizard. Some of the common feelings of impostor syndrome are the worry to be found out like another wizard.

wizard_behind_the_curtain

Hackathon

At work we recently held a hackathon where everyone who was interested in participating had 24 hours to build whatever they thought would be interesting and useful. We had 21 teams across 4 offices with ~60 participants total. I had never done a hackathon before and this seemed interesting so I registered for it without a particular idea in mind. As the start rolled around I was planning to put together a tool to simplify how we were generating configurations for containers in marathon. However, at the kickoff pizza dinner I heard another developer saying he had a plan to solve our issues concerning the lack of available conference rooms. Every afternoon there would be an hour or two where there were no meeting rooms left available, and while we’ve got more space coming it will be a while until it is available. Having been a victim of this problem before I asked if they could use another hand on their team and was graciously invited onboard.

The key insight they had was to use the logs from the wireless network to figure out who was in what office each day. Once we had an idea of who was in what office that day we could cross reference that with their calendar appointments and see what rooms were booked and didn’t need to be. There was some concern about false positives, i.e.,  we didn’t want to have the system saying that you weren’t in the office by 10 so release your room at 11 while you were stuck in traffic. So we built a hipchat integration to check with you about it.

The three of us started Thursday night at about 6 with a general divide and conquer along the three major components: data mining/parsing, calendar matching and decision, and the hipchat integration. I mostly worked on the hipchat portion. Since the bot had to reach out to specific people on it’s own volition as opposed to responding to people or messaging a fixed channel, our needs were different than what most of the prebuilt hipchat integrations are doing. I ended up doing an XMPP integration using Smack. The biggest challenge in getting this working in the context of a web service was that I needed to keep the connection to hipchat open longer than the API implied it needed to be. I found this out when my initial attempt to send a message and then close the connection failed because the message didn’t finish going through but we had closed the connection on our end. After spending several hours working through that I called it a night at about 1:30 a.m. and headed home to catch some sleep.

Getting back the next morning at about 7:30, in my office there was one lone developer who had been there working on his project all night. He had been working on porting a feature from the web app to the android app, because when he used the app he wanted that feature. I spent the first part of the morning working on getting the response from hipchat hooked up and found another interesting problem. I wasn’t able to respond to myself as the bot for whatever reason. So if the bot was using my credentials to send messages it wouldn’t see my response to it. I suspect it was because hipchat was being clever and not sending the message as a message but some sort of history, but I never was able to confirm. At 8:30 the dev who had been working on the matching stuff for our project got in and started processing live data for that day; our app immediately started spitting out rooms we thought didn’t need to be booked. I went and did a little scouting at about 10:30 to confirm the situation and matching seemed right.

We ran into a credentials snag on getting an account with the rights to unreserve other people’s meetings. So we didn’t have a full demo but the example meetings we had identified painted a pretty picture of how well it could work and the number of rooms it could free up.

When demo time rolled around we all got together to show off what we built. There was a bunch of interesting stuff put together. There was a set of living visualizations of service dependencies built by parsing urls from the system configuration data. There was a port of one of our mobile apps to Apple Watch. There were two different teams that built Alexa integrations for different portions of our products. Several teams built features for various mobile apps. One team set up a version of Netflix’s Chaos Monkey in the load test environment, including a hacked Amazon Dash button that would kill a server in that environment at the push of a button. Another team built a deploy football in the vein of the nuclear football complete with keys and switches and a little screen to display progress. Two tech writers twisted arms and got someone to build a hipchat integration to look up acronyms from a glossary they had put together on the wiki.

Overall I had a blast but ended up pretty exhausted from the ordeal. Some prizes will be given out on Monday. I’m not sure of the exact criteria for them but I wasn’t competing at all – I was enjoying the latitude to do what I thought was best. There will be one more prize given out at the company-wide all hands wherein everyone gets to vote on as the most impactful project after we get a chance to see how everything  turns out in real usage.

Write the Code You Want

“Write the code you want and then make it compile” was a thought expressed on library design while I was at the NE Scala Symposium. It is a different way to describe the TDD maxim of letting the usage in tests guide the design. It is very much influenced by the extremely flexible syntax rules and DSL creation abilities in Scala. One of the talks, Can a DSL be Human? by Katrin Shechtman, took a song’s lyrics and produced a DSL that would compile them.

Since you can make any set of arbitrary semantics compile, there is no reason you can’t have the code you want for your application. There is an underlying library layer that may not be the prettiest code, or may be significantly verbose but you can always make it work. Segregating the complexity to one portion of the code base means that most of the business logic is set up in a clean fashion and that the related errors can be handled in a structured and centralized fashion.

Taking the time to do all of this for a little utility probably isn’t worth it, but the more often a library is used the more valuable this becomes. If you’ve got a library that will be used by hundreds, really refining the interface to make it match how you think would be really user friendly.

Building software that works is the easy part, building an intuitive interface and all of the comprehensive documentation so others can understand what a library can do for you is the hard part. I’m going to take this to heart with some changes coming up with a library at work.

This still doesn’t even cover the aspect of deciding what you want. There are different ways you can express the same idea. The difference between a function, a symbolic operator, or create a DSL can all express the same functionality. You can express the domain in multiple ways, case classes, enums, or a sealed trait. You can declare a trait, a free function, or an implicit class. Deciding on the right way to express all of this is the dividing line between a working library and a good library.

Three Monad Laws

I’m continuing to dissect and share my experiences at the NE Scala Symposium. Several presenters referenced the “three monad laws,” which I had never heard of before. A monad must adhere to three laws:

  1. Left Identity
  2. Right Identity
  3. Associativity

The expression of left identity and right identity were new to me. I understood the concept of identity but wasn’t familiar with the distinction between right and left. Apparently the identity I was familiar with assumed a commutative relationship which means that the left and right identity are the same thing. From wikipediaThen an element e of S is called a left identity if ea = a for all a in S, and a right identity if ae = a for all a in S. If e is both a left identity and a right identity.” Trying to think of an example, the right identity of division would be 1 but the left identity would be the value squared.

Associativity is the final law: this is the normal associativity that we’re all used to from basic math,  which is expressed as X * (Y * Z) = (X * Y) * Z.

That’s the total of the laws, it’s nothing hard to accomplish. Like a lot of the pieces of functional programming each individual piece isn’t anything magical but the way they interact creates a special sort of synergy.

Traveling Stories

Traveling has always made me introspective. When I was going to the West coast regularly for work, my wife knew she would get a rambling email from me that I wrote while in the plane. Some of them were sappy, some were crazy, some were a disjointed mess. But they were all those thoughts that came out when I was stuck there with nothing but those thoughts to keep me company.

Sure I’d read, listen to a podcast, or watch something but eventually the mind would wander to that place it wanted to go. Like a very slow form of meditation. Once my mind emptied of thoughts of the day or about where I was going, I achieved this zenlike state where answers to questions would just unfold like an origami crane.

The worst part is that the answers were always fleeting. There for a moment, gone the next without the chance to fully understand the epiphany. A deep insight into the universe that you know was there, but never get to appreciate.

This post started with one of those epiphanies I was on the train on the way to New York for the NE Scala Symposium and there was this moment of clarity about a communication struggle I had been having at work. For a moment, I had a vision of the creative action plan I had been looking for, then a PA announcement came on and the thought was gone. I hadn’t recorded the thought in any way but I know it was there right outside of Trenton.

On the ride home I managed to rekindle some of the thought, but it wasn’t the same deep insight that I had originally had. Initially, my work team had been proposing to adjust this existing framework to enable a new usage. But by rephrasing it as “replacing” the entire framework and building a brand new system everyone was immediately on board. By reframing the initial idea from being a change to a new thing it got everyone on board. I think this reasoning is twofold, it would mean we can roll out the change in smaller increments and we can go back without as much effort. It’s the same work and the same expected end state, but the reception was significantly different.

Document Fragments

I had a stray thought about fragments in JSON documents, after a colleague asked about if we had considered using GraphQL to allow allow querying subsections of a larger document. It seemed like you could reasonably index into a JSON document and reference a subsection of it in a simpler fashion. I went looking for information about this and found RFC 6901 on JSON pointers, describing exactly what I was theorizing. However, it doesn’t appear to have been adopted yet. The gist is that you can index into a document with #/<key>. Little mystery solved, but while researching this I stumbled into something else more interesting.

I found this about best practices in designing fragment IDs when creating a media type. But it discusses these best practices in relation to several schemas that were out in the wild. This had the JSON pointer example, it also had XML and PDF examples, which seemed reasonable. But it also had plain text and CSV, which was interesting as the format i’s a line number and a character/field which works but I had never really considered attempting. The more interesting one was the SVG, since it lets you index into an image. You can index into a named section or any arbitrary view box.

This was an interesting diversion that really reminded me about the amazing flexibility and foresight of some of the technologies underlying the web.

The Systems of Software Engineering

This idea came together from two different sources. First, I’ve been reading The Fifth Discipline about the creation of learning organizations. One of the elements of becoming a learning organization is Systems Thinking, which I had heard of before and seems like a great idea. Then I went to a local meetup about applying Systems Thinking to software development. We ran through a series of systems diagrams from Scaling Lean & Agile Development. These diagrams helped us to both understand Systems Thinking by analyzing a domain we were already familiar with in a structured way where we were presented the nodes of the system diagram and asked to draw connections between them.

There were several different groups each looking at the same sets of nodes and drawing different conclusions for some of the relationships. There were a few very adamant people who all felt that an increase of incentives for the developers would increase the velocity of the team, which was interesting to see since they were mostly scrum masters or software development managers with many years of experience doing the management of software projects and should have seen how that does work out. If the impact of an incentive plan on a team is significantly uncertain, it makes me curious about the sorts of teams these other leaders are running.

Through the entire process we had interesting conversations about how various aspects of the software process were related. Everyone agreed that the interaction of a strong mentoring process on the overall system would decrease the number of low skills developers on the team. There was a discussion of whether it would also directly impact the velocity of the team. Some people were adamant that it would lower velocity since the mentoring time was time that people weren’t working on building features. It’s a reasonable consideration, but it doesn’t seem to match with my specific experiences with mentoring. If you were actively spending enough time to lower your velocity significantly on mentoring activities you would be forcing the mentoring relationship instead of letting it happen organically.

The experience made me wonder if we could construct a system diagram that describes some of the other dysfunctions of many organizations. The diagram we built described (1) dysfunctions around hiring large numbers of low skill developers, (2) using certain kinds of financial incentives, and (3) the push for delivering features above all else. It didn’t describe why a lot of organizations end up in siloed configurations, “not invented here” behavior on technical systems, or lots of the other dysfunctions in multiple organizations. I made the below diagram with the intent of describing  the situation about siloed organizations but without any real experience in this sort of analysis I’m not sure if I’m doing it well.

I made another simple diagram about the causes of “not invented here.”

It feels to me like these diagrams describe the dysfunctions that are common in software organizations. Expanding these diagrams to try and find the leverage points in the system might yield some greater insights into the problems. In spending some time thinking about both of the diagrams I’m not sure what other nodes should be there to further describe the problems.

I’m going to definitely do some more reading on Systems Thinking and try to expand on the thoughts behind these diagrams. If you’ve got more experience with System Thinking I’d love to hear some feedback on these charts.