Looking Back Again

I had reflected back on my last job before. However, I had a thought today that got me in a reflective mood again. I miss being involved in any and all problems that came up. I was in a position that I would get tapped whenever anything went wrong and it wasn’t clear whose problem it was. I got to take it apart and find the root cause and either fix it or pass it off the team responsible for that area. The rush of having a regular stream of critical problems to deal with was intense. It had stratospheric highs, and it had some amazing frustrations that went with it; in all it was intensely involving.

I miss the puzzle of it all. I used to go to the office, read my email, pick the most immediately  pressing problem, and go look into it. The problem of it was I spent all day putting out fires and no time doing fire prevention. The knowledge to help prevent the fires was only really learnable through dealing with them. Most of the developer team wasn’t exposed to the problems in a way to learn how to avoid them. The problems were varied enough that you couldn’t just put together a brown bag session to teach them.

It’s exhilarating to be someone who gets to work on the problems but it also becomes massively stressful. The situation is bad for the organization as well. Problems get solved but the organization doesn’t really learn, so it will never get better. There was one particular incident that sticks out in my mind where another developer sent me this error message asking if I knew what to do with it. I responded immediately with an explanation of what was needed to fix the problem. The developer later asked how I knew what to do so quickly, the answer was I had spent 6 hours figuring it out the day before when someone else ran into the same problem.

I don’t miss the stress aspect of this when you’ve got problems six deep and someone asks if you’re able to help them and the answer is no. It’s difficult when the person whose problem is way down your queue wants to know what is going on with it, and you need to tell them it isn’t your priority right now. I didn’t fully understand how much stress I was under there until I had given it all up and moved on.

I managed to avoid some of the negative aspects of being a hero in software development. I kept to normal hours. I didn’t create situations that need to be fixed by my personal efforts. I didn’t try to be involved with everything. In fact, I was actively avoiding proactively getting involved in some places that looked like they were going to get into problems, because I felt that while their problem was going to prevent them from making progress, it didn’t endanger the existing system and therefore might be a learning experience.

By understanding and being able to articulate the various issues there that I did and did not enjoy, I hope to be able to optimize my involvement at my new job. Learning from your past experiences is the only real way to do better.

Dabbling in AngularJS

I got the opportunity to do some work on the front end recently. It’s my first exposure to AngularJS and it’s been a great experience. I’ve never been a big fan of JavaScript in general but this seems to be a great way to work with it. Angular is a highly opinionated framework; it specifies how to lay out the application, the patterns you should use to decompose the visual elements, the layering for calls into the backend, etc. It gives you a lot of tools for all of this: two-way data binding, dependency injection, tools for working concurrently, and testable abstractions for making remote calls.

The limitations are quite freeing since they guide you to a style that produces lots of loosely coupled small modules that are testable. The testing plus the compilation step gives a lot of confidence that the system will work when deployed. I haven’t been making any significant changes on the front end, but I feel like if I needed to set up a new project I would go with a similar stack to what we’ve got going here. I feel like I’d be happier with TypeScript since that more closely matches how I think about code. In my understanding it provides for better development environment integration than what I’ve been able to get with JavaScript so far.

The documentation has been some of the best I’ve seen so far for a framework. Each piece is described independently and there are examples of how they layer together with other. There are plenty of examples that work there in the browser, since it is all JavaScript . Even better, it all links to Plunker, which lets you edit the examples right there in your browser so you can play with it and see what happens. There have been a couple of places where the documentation has failed me. The documentation regarding promises and $q falls into the classic trap of making sense if you already understand what it is trying to tell you but is almost worthless when you don’t know what it means.

I’m still having issues with the modern toolchain for front end development. Fortunately my teammates who have are more experienced with this have gotten angular, gulp, phantomjs node, npm, bower, jasmine, and who knows what else setup. This means I can just hang new pieces onto the existing infrastructure without needing to know how to bootstrap the entire stack, which is good for getting started with it. This toolchain is complex and I’m not sure how much I need to know about each of them.

This is definitely better than my previous front end experiences, although front end work still isn’t something that I want to do lots of. Finding a tool set that I like better will make it easier to do when it does come across my plate in the future.

Book Chat:Scala for the Impatient

Scala for the Impatient is a reaction to the heft of Programming Scala. It recognized that Programming in Scala was a great resource but weighed in at 800+ pages and that it may make a better reference than introductory text. Scala for the Impatient attempts to capture the bulk of the value at a much slimmer 340 pages. Having not read Programming in Scala I can’t say if it captures the bulk of the value, but it does provide an amazing introduction to Scala.

The each chapter is labeled with a Scala level to give you an idea of how complex the topic is. This helped me since I could see that the chapter on implicits was labeled L3, the most complex level, and that most of the time you don’t need a feature like this. That helped signal to me that it was okay that I didn’t fully understand how to apply the topic right after reading, while I’m still a relative newcomer to Scala. I haven’t wanted to write any implicits so far, since I haven’t been writing anything too complex. But, it still helped when I saw some implicits in library code I’ve gone into; I have been able to follow along, although I don’t fully understand why the code needed to be put together that way.

I had issues with some of the exercises. There were some that asked you to think about why something was written in a particular manner, but there wasn’t an answer key, so it was impossible to know if you were right. I went searching on the internet and found people discussing the exercises but I would have appreciated a more authoritative and definitive answer. There were also places where it asked you to implement something, where it clearly meant for you to use some particular feature that was discussed in the chapter, but it wasn’t evident how you would implement it using those features.

I really enjoyed the ability to download of all of the example code, and code snippets from the examples. It made it easy to experiment with the example code and see what happens, without having to copy it all over. The errata was helpful, but there was a lot of it and it interrupts the flow of reading the book, but it doesn’t reduce the value of the book. Overall, out of the things I did to start learning Scala it was the most comprehensive. It has been my biggest time investment in deliberate learning for Scala so far and I’m glad I did it. I’ve got a copy of Programming Scala which I intend to get to eventually, but its significant heft may make it wait for a while before I dig into that to give a comparison.

Scala Intricacies

I ran into a couple of little syntax bits today that all added up to a bunch of wasted time. First was a call by name argument:

def passByName( t: => Long) = {}

This results in t essentially becoming Unit => Long that gets implicitly called each time t is used. This was not immediately obvious, and like other similar bits of syntax, it was difficult to search for. The name of the feature, pass by name, doesn’t help since it doesn’t immediately explain what’s going on.

The second piece of syntax was the ability to have multiple parameter lists to enable a special kind of currying:

def currying(a:Long)(b:Int):String = {(a + b).toString}

This is equivalent to:

def currying(a:Long):Int=>String = {(x:Int) => (x + a).toString}

This syntax is more compact and clearer. It also gets better as you add more items. It also interoperates well with implicit parameters since you can attach the implicits to the last parameter list and have them fill in but you get a clean function signature for everything else.

The third piece of syntax was an inheritance issue that happens when a trait extends a class. Normally a trait is mixed in using the `with` keyword and classes are extended using the `extends` keyword. At first I didn’t realize you could extend a class with a trait, having seen the self trait syntax:

trait Bar { self: Foo =>}

But I did so unknowingly and when I attempted to mixin that trait in a way that was illegal, I got a great error message which helped resolve what to do next, but didn’t explain why what I did was wrong.

Having two different syntaxes that allow for very similar but slightly different behavior seemed confusing. I found this post describing the slight differences between the two different syntaxes and what they mean. It mostly seems to be a style thing, the self trait syntax lets you extend whatever class you want but it needs to eventually have the self class. This may have some consequences regarding linearization although it didn’t come up in the referenced post or what I was doing. The trait extending class syntax requires that you extend from it directly. It also does give some additional options for overriding methods since they can call the super implementations.

The last piece was how context bounds work on ClassTag types. When you define a method like:

def myMethod[T:ClassTag]

it implicitly creates an additional argument changing the signature to:

def myMethod[T](implicit tag: ClassTag[T])

This argument propagates through to any other usages of T and enables the usage of tag. Figuring this out from the codebase itself didn’t work out well. Searching for associated phrases was unhelpful since it isn’t really inheritance and most of the documentation is listed under TypeTag and not ClassTag since the two work similarly. The other issue was that the majority of the documentation I saw during my initial readings referenced manifests, which were deprecated in favor of the ClassTag and TypeTag. I managed to get some assistance from some of my coworkers to figure this one out, but it wasn’t an easily searchable or understandable concept since it took inheritance syntax to create implicit arguments.

All four of these syntax bits are useful but have some very specific nuances. Like a lot of Scala language features, they are all useful and well implemented individually. As a group though, there is a lot of unique and quirky  syntax to learn. I’ll keep chronicling all of the various bits and pieces that I find that warrant more thought.  

Scala Futures and Stack Traces

I’ve been digging into a real Scala codebase for a while now and I’ve been enjoying a lot of it. The trait system in practice works way better than it looks like it would, mostly since all the pre-research I’d done involved people talking about the difficult bits because the normal use cases work out easily (and are therefore uninteresting). The integration with tools and libraries worked out great in most ways; however, the error message you get when a Java class and a Scala class have conflicting names wasn’t ideal even if it worked. The import rename syntax (ex. import scala.collection.mutable.{Map => MMap}) fixed the problem easily enough.

The big gripe I’ve had so far is with debugging future-based programs. The big issue is that when an exception gets thrown in a future, the stack trace doesn’t give any idea of what scheduled the future. The problem has been recognized as a Scala bug, but it doesn’t seem to be getting dealt with since it was opened in 2014 and hasn’t been updated since 2015. This is also partly a tooling issue since the step in functionality doesn’t work the way you’d want it to. Stepping into a method that returns a future doesn’t actually step in. It just schedules that action to occur, it doesn’t actually happen yet and will happen on another thread when it does.

A possible solution would be if the step-in action in the debugger put a breakpoint inside the method being scheduled with some sort of conditional. Not sure what the downside would be to this particular solution. I don’t see a reasonable way to add this as a plugin or a script in some other way. Forking IntelliJ itself doesn’t seem like a good idea either since that’s the community edition, and that doesn’t support all of the plugins we are using plus all of the ongoing support issues with that.

The one gripe against the stack doesn’t out weigh all of the positive aspects. The concise syntax is great. While I haven’t had any opportunity to really dig into the performance aspects of working with futures yet, I expect they will hold up as expected. The type system is expressive and safe. Higher order functions aren’t as much of a differentiating factor over Java with lambdas in Java 8 but they aren’t integrated into the libraries the way they were in C# or are in Scala. Overall, I’m excited with my choice and will keep digging in.

Flow and Learning

Since I recently changed jobs, I’ve been learning a ton of new things. Moving from Windows/C#/Visual Studio/ASP.Net/SQL Server to OS X/Scala/IntelliJ/Play/MongoDB is a lot of change, normally you’d have some sort of strong point to leverage off of, but for me it is all new. I’ve always had an introspective personality, and the introspective part of me is looking at this and thinking that I got in over my head. The logical portion of my brain says the learning curve is expected and I’m working my way up it, but it is difficult to reconcile the two perspectives.


I had found the above diagram describing how challenge and skill interact and can cause boredom or frustration. The context of this diagram was how games try to stay in the flow channel, and the player response on either side of the channel. The more immediately relevant version was the one where it described the experience of starting a new job.


This diagram matches with my experience here, especially since I took a bigger hit to my applied skill level due to the radical change in the technical stack. This puts me further into the frustration section than other similar job change experiences, and causes the sort of anxiety that I had been grappling with.

I know I’m making sure to keep an eye on my stress levels and everyone at work understands the challenges that are being encountered since they went through most of the same ones at some point. I changed some of my habits around what I’ve been reading, less dense material more fiction. I changed some other habits, since I cut my commute down significantly I’ve been trying to make sure to use that time wisely and get some additional exercise to deal with the stress in a positive fashion.

By putting together the rationalization of what is happening I hope to assuage my own insecurities. The mental load of the insecurities can take attention away from learning and doing your best, making the insecurities a self-fulfilling prophecy. I hope this account of the feelings I’ve been encountering helps others to recognize that they aren’t anything abnormal to feel, but that you can’t let the negative feelings control your mind.

Reactive Manifesto

I ran across the Reactive Manifesto while I was doing some reading on Scala. I touched on this idea back in the Virtual Conference post with the presentation What does it mean to be Reactive? by Erik Meijer. His 45 minute presentation tried to explain the nuances of this concept. The manifesto is all about taking the basic concept of what reactive programming is and making the big ideas accessible.

The four aspects of reactive applications – responsive, resilient, elastic, and message-driven – all come together to reinforce one another. Responsive is straightforward: the application should return results quickly. Resilient is that the system is designed to resist faults and remain responsive. Elastic means the system can react to the changing input rate and add or remove resources to keep up with demand. Message-driven is that the system operates in an asynchronous manner. These four aspects work together to build a cohesive style of application.

The message-driven nature of a reactive application allows the system to utilize the concept of back pressure. Back pressure allows the system to control the flow of requests to ensure that the system remains responsive while the elastic aspect kicks in and applies additional resources. The message-driven nature also means that the resiliency has additional options since the services are decoupled further.

The reactive name may be new but the concepts are old. HTTP could be described as reactive. It is responsive since you can guarantee the timeout settings. The HTTP server is designed to be resilient and able to fail fast, with descriptive error codes that indicate how to react. A 404 is different from a 503 and the caller can react differently to the two of them. The system is elastic since you have ways available to scale horizontally. HTTP is clearly message-driven with the request and response paradigm.

This ties into the Scala ecosystem via Scala.react and other libraries to enable reactive programming on the platform. Scala is a good fit for reactive programming because the functional paradigm makes it easier to work in a reactive manner. Martin Odersky, the designer of Scala, also was an early proponent of reactive programming so that created a fertile breeding ground for working on reactive concepts.

Using this paradigm at a smaller scale seems like a great way to create modular systems. This pattern also compliments the microservices architecture. I’m looking forward to working with these concepts as I get more experience with Scala.

Thinking Back and Looking Forward

I’m writing this after my last day at my previous job. It has me reflecting a lot on the three years I was there. I learned some interesting things and met a lot of great people. There were aspects of the job I really loved, and there were the aspects of the job that I would love to never think about again. I was reflecting back on my accomplishments there and I was surprised how much I had done in some aspects and how little I had done in others.

One of the memories that stuck out to me was the response to some production performance problems we had in July of 2014. I was new to the organization, which made it my first chance to see how it responded to a crisis. I learned a lot about the organization then. It was great to see how disparate teams came together and worked the problem. It also gave me an opportunity to better understand the organizational structure, good and bad. While investigating, I ended up digging into something that wasn’t the cause of  the problem, but demonstrated numerous other little problems that had been festering. There, I managed to make a big impact with different higher-ups once I showed them all of the hidden problems I dug out. It was especially interesting for me to handle from a workplace navigation perspective, since all of the problems were outside of things my team was nominally responsible for.

An area where I tried to make an impact and spent a lot of effort was the build pipeline. The ‘continuous integration’ that was set up by the configuration management team had numerous problems. The basic workflow was that they would – on request or on a timer – take C# code, compile it, and commit the DLLs back to SVN. When I first saw this, I noted that using source control to store the binary artifacts that were created from the repo was odd. The configuration management team didn’t have any interest in changing this, since the entire deployment process was based around deploying DLLs from SVN; it’s understandable they were resistant since changing this piece would require changing everything. This limited a lot of my options on how to change the build pipeline without getting involved in deployment as well. I spent a long time trying to convince the configuration management team to run tests before committing back to SVN, but the server they were building on failed a number of the tests when they tried that. It turned out that the team that owned the code under test didn’t even know those tests existed. This led into an effort to find all of the tests in the whole system and get them passing again. We made progress on these goals but there ended up being a functional area nobody owned, and the rest of my team didn’t feel like we had the capacity to take ownership of it. This left the effort in a sort of stalemate: progress had been made and we could run a lot of tests, but there couldn’t be a clean build unless we started making exceptions. We held in limbo for months, where management continued to debate  who owned that functional area. This eventually led to the effort dying off. That was a massive blow to my morale, since to me it implied a lack of focus on quality; that wasn’t at all their intent, but that doesn’t reduce the frustration after I had gotten so far and made so much progress. I recognize that I may have taken it too personally or read motivation into it on management’s part that wasn’t there, which is an important lesson I’m taking away. I also learned a great deal about how to navigate the corporate environment and how to use soft power to influence change, but most of that was too late to make this change successful; now I know better how to approach this type of major change management from the start.

I don’t want to give the impression that everything was bad – a lot of aspects of the work were great and interesting. I had a great experience working with a system that is much larger than anything I had worked with before. The whole system was hundreds of times larger in terms of computational resources than anything I had worked on before. The codebase itself was approaching ten million lines of code across 10+ different programming languages. Digging into all of this was an amazing adventure to understand how it all went together. Very few people were looking at the system as a whole and I got to come in, learn and understand all of this and help those who were trying to build other pieces of it.

I also had the opportunity while there to mentor some junior engineers and it was quite rewarding. I helped them get up to speed on the code base, which is an easy enough activity. But I also spent a great deal of time working with them on code reviews and helping them understand how to build better software. Helping them to understand the difference between something that works and something that is truly good was really rewarding as they started to see the difference.

My new job is a much different situation than my last job. At the last job I was there to help them clean up the technical mess they knew they had; their issues are common to large organization that have grown through acquisition, and I do appreciate that they wanted to improve. The technical mess wasn’t the whole problem; as you can see from these two anecdotes above there were multiple groups who weren’t in sync and a lack of understanding of each other. The new place is much smaller so it’s unlikely that I’ll have to navigate as many different competing groups, or be assigned such a narrow portfolio. My new company is also much newer so there hasn’t been the time to build up the layers of technical debt. From a technical perspective, at the new job I hope to master a new language and stack and build out some RESTful web services. I’d also like to help build a high performing team and mentor some junior developers, since I found that  really rewarding The person who mentor me to see that difference between “works” and “good” happens to be working at the new place, so that will be a great relationship to renew now that I’ve had an opportunity to play that role. Sometimes the journey is the destination afterall.

More Scala

I’ve been doing a ton of reading on Scala and jotting down some observations on what I’ve learned. Some of it is simple but not immediately obvious; some of it is those places where the reading left a desire for more information. Hopefully this will save some time for others trying to pick this up. If not, at least it will be a reference for me.

Some notes from going through Scala School

  • The relationship between partial application and currying is still not totally clear to me; it seems like you curry to create a series of functions then partially apply them in sequence to achieve the full result.
  • The differences between methods and functions are unclear; the tutorial assures me that’s okay but I don’t like it. It seems that functions are strictly things that inherit from function.
  • Type aliases seem like a great way to build new types that imply a collection of existing types.
    • type SocketFactory = SocketAddress => Socket
  • I need a better understanding of objects in scala. Not the normal OO object, the Scala object seems to be closer to the singleton pattern or a static.
  • Guard matching syntax seems interesting, e.g.,
    • case i if i == 1 => “one”
    • I wonder what syntax you can use in that expression. Hopefully function calls are legal.
  • Tuple Accessors are 1 base not 0 based.
  • Destructuring bindings to decompose tuples is awesome, they let you break down tuples quickly and neatly.
    • val (char, digit) = tuple
  • I’m a big fan of GetOrElse on Option, very concise and clear.
  • Partial functions are another idea I haven’t seen before.
    • It is a function that knows it can only produce results for a subset of inputs.
    • They’re the underlying implementation in case statements which allows case statements to be used like functions!
    • Not sure what else I could do with them.
  • I think I’ve figured out F-bounded polymorphism, essentially you define that the generic type must implement the type you are defining. This is so that you can constrain the generic type to pass it into the base class.
  • Structural types look like duck typing but uses reflection so we should avoid using them.😦
  • Manifests are compiler generated code to counter type erasure. I wonder how much code bloat this causes.
  • Found the DecorateAsJava trait, nice usage of implicit conversions to make interop with java easier.
  • Spies seem like a nice feature of the testing framework. Seems like you could build inconsistent behavior in an object like a list that says it has some number of objects but doesn’t have an entry at that point.
  • The section on traits did not make the mixin functionality clear. You can mixin a trait as part of the constructor of an object not just as part of the class declaration! That’s not quite C# extension methods but close enough for some usages.
    • Found implicit classes elsewhere that can be used almost exactly like extension methods.

Notes from Scala Exercises

  • The operator, ie fat arrow =>, shows as one character in some of the examples which is confusing.
    • Similar problems with the skinny arrow ->
  • Backticks in pattern matching allow matching against other variables.
    • def patternEquals(i: Int, j: Int) = j match {
       case `i` ⇒ true
       case _ ⇒ false
  • There seems to be usages of implicits that work like extension methods, but you need more syntax.
    • class KoanIntWrapper(val original: Int) {
       def isOdd = original % 2 != 0

      implicit def thisMethodNameIsIrrelevant(value: Int) = new KoanIntWrapper(value)
  • The section on for expressions did not really help my understanding.
    • val xValues = 1 to 4
      val yValues = 1 to 2
      val coordinates = for {
       x ← xValues
       y ← yValues
      } yield (x, y)
    • In this example it isn’t clear why the two lists are evaluated in any particular order.
  • Scala Fiddle was a great help for running some of the examples and being able to try some variations.
  • I’m not sure how I feel about the conversion between methods and operators where certain kinds of methods can be used as prefix/infix/unary operators. Having seen a C++ codebase with off operator implementations I realize the pain it can cause; maybe since you can name them however you want they’ll read better.

I ended up going through the Play Framework Tutorial as well since it’s part of the stack I’ll be using.

  • Looks a lot like ASP.Net MVC.
  • Session data signed to prevent tampering.
  • @With annotation and allowing composition of actions.
  • Twirl temples look a lot like razor templates except that you add dynamic elements in Scala not C#.
  • Play seems like it just composites together best practices from various other JVM libraries bringing together JPA, Twirl, Jackson, Akka, Guice, and logback.
  • The WithApplication class seems like a way to make building testable controllers easier.
  • Similarly with WithServer class will help test some fully remote calls.

Overall I’m feeling pretty psyched. At this point I feel pretty comfortable reading the sample code. The exercises were more about reading it and understanding it rather than writing your own code.

Since the language is so big I’m not sure which parts are where I should be focusing. I feel like I need to start building something with it at this point to understand how to decompose a class into traits. I’m not sure the Project Euler problems will supply the appropriate domain to try and build these more complex inheritance hierarchies. I’m also not certain if I’ll be building good Scala code or just mashing together different styles in a way that wouldn’t scale. Maybe Scalastyle can help with that, but the rules don’t seem to be the kind of thing I’m looking for.

I’m going to forge ahead with this list of Scala problems since I hope that being able to contrast my solutions with the provided ones will give me some level of feedback on what I’m doing. Then probably move into the Project Euler problems afterwards.

Starting Scala

I’m starting a new job soon which is going to be entirely Scala programming. I’m trying to dig in a little bit and get started with a leg up. I’m going to outline my basic plan here and post updates as time goes on. I’ve done a tiny bit of Scala before when learning some about Spark. I’ve done some Java work in the past as well so the ecosystem is a little bit familiar. So I’m not coming in from nothing.

The place I’m going to be working uses Macs so getting off of Windows seems like the right way to get started. I had previously used Eclipse when working on JVM based languages. The last time I had used Eclipse though it seemed like more had changed than was the same as I remembered. So I’m going to take a different tack this time and try IntelliJ. So the overall development environment is going to be an Ubuntu VM and use IntlliJ on it.

I plan to start with Scala School to make sure I’ve got all the basics down. Once I feel I’ve got that put together I’ll try Scala Exercises for some more guided exercises. Then I’m going to do some of the Project Euler problems I had done in F# since I understand the problems and can focus on learning the idiomatic Scala way of doing things.

I figure this should get me up to speed on Scala pretty quickly. We’ll see how well it goes.