Scala Futures and Stack Traces

I’ve been digging into a real Scala codebase for a while now and I’ve been enjoying a lot of it. The trait system in practice works way better than it looks like it would, mostly since all the pre-research I’d done involved people talking about the difficult bits because the normal use cases work out easily (and are therefore uninteresting). The integration with tools and libraries worked out great in most ways; however, the error message you get when a Java class and a Scala class have conflicting names wasn’t ideal even if it worked. The import rename syntax (ex. import scala.collection.mutable.{Map => MMap}) fixed the problem easily enough.

The big gripe I’ve had so far is with debugging future-based programs. The big issue is that when an exception gets thrown in a future, the stack trace doesn’t give any idea of what scheduled the future. The problem has been recognized as a Scala bug, but it doesn’t seem to be getting dealt with since it was opened in 2014 and hasn’t been updated since 2015. This is also partly a tooling issue since the step in functionality doesn’t work the way you’d want it to. Stepping into a method that returns a future doesn’t actually step in. It just schedules that action to occur, it doesn’t actually happen yet and will happen on another thread when it does.

A possible solution would be if the step-in action in the debugger put a breakpoint inside the method being scheduled with some sort of conditional. Not sure what the downside would be to this particular solution. I don’t see a reasonable way to add this as a plugin or a script in some other way. Forking IntelliJ itself doesn’t seem like a good idea either since that’s the community edition, and that doesn’t support all of the plugins we are using plus all of the ongoing support issues with that.

The one gripe against the stack doesn’t out weigh all of the positive aspects. The concise syntax is great. While I haven’t had any opportunity to really dig into the performance aspects of working with futures yet, I expect they will hold up as expected. The type system is expressive and safe. Higher order functions aren’t as much of a differentiating factor over Java with lambdas in Java 8 but they aren’t integrated into the libraries the way they were in C# or are in Scala. Overall, I’m excited with my choice and will keep digging in.


Flow and Learning

Since I recently changed jobs, I’ve been learning a ton of new things. Moving from Windows/C#/Visual Studio/ASP.Net/SQL Server to OS X/Scala/IntelliJ/Play/MongoDB is a lot of change, normally you’d have some sort of strong point to leverage off of, but for me it is all new. I’ve always had an introspective personality, and the introspective part of me is looking at this and thinking that I got in over my head. The logical portion of my brain says the learning curve is expected and I’m working my way up it, but it is difficult to reconcile the two perspectives.


I had found the above diagram describing how challenge and skill interact and can cause boredom or frustration. The context of this diagram was how games try to stay in the flow channel, and the player response on either side of the channel. The more immediately relevant version was the one where it described the experience of starting a new job.


This diagram matches with my experience here, especially since I took a bigger hit to my applied skill level due to the radical change in the technical stack. This puts me further into the frustration section than other similar job change experiences, and causes the sort of anxiety that I had been grappling with.

I know I’m making sure to keep an eye on my stress levels and everyone at work understands the challenges that are being encountered since they went through most of the same ones at some point. I changed some of my habits around what I’ve been reading, less dense material more fiction. I changed some other habits, since I cut my commute down significantly I’ve been trying to make sure to use that time wisely and get some additional exercise to deal with the stress in a positive fashion.

By putting together the rationalization of what is happening I hope to assuage my own insecurities. The mental load of the insecurities can take attention away from learning and doing your best, making the insecurities a self-fulfilling prophecy. I hope this account of the feelings I’ve been encountering helps others to recognize that they aren’t anything abnormal to feel, but that you can’t let the negative feelings control your mind.

Reactive Manifesto

I ran across the Reactive Manifesto while I was doing some reading on Scala. I touched on this idea back in the Virtual Conference post with the presentation What does it mean to be Reactive? by Erik Meijer. His 45 minute presentation tried to explain the nuances of this concept. The manifesto is all about taking the basic concept of what reactive programming is and making the big ideas accessible.

The four aspects of reactive applications – responsive, resilient, elastic, and message-driven – all come together to reinforce one another. Responsive is straightforward: the application should return results quickly. Resilient is that the system is designed to resist faults and remain responsive. Elastic means the system can react to the changing input rate and add or remove resources to keep up with demand. Message-driven is that the system operates in an asynchronous manner. These four aspects work together to build a cohesive style of application.

The message-driven nature of a reactive application allows the system to utilize the concept of back pressure. Back pressure allows the system to control the flow of requests to ensure that the system remains responsive while the elastic aspect kicks in and applies additional resources. The message-driven nature also means that the resiliency has additional options since the services are decoupled further.

The reactive name may be new but the concepts are old. HTTP could be described as reactive. It is responsive since you can guarantee the timeout settings. The HTTP server is designed to be resilient and able to fail fast, with descriptive error codes that indicate how to react. A 404 is different from a 503 and the caller can react differently to the two of them. The system is elastic since you have ways available to scale horizontally. HTTP is clearly message-driven with the request and response paradigm.

This ties into the Scala ecosystem via Scala.react and other libraries to enable reactive programming on the platform. Scala is a good fit for reactive programming because the functional paradigm makes it easier to work in a reactive manner. Martin Odersky, the designer of Scala, also was an early proponent of reactive programming so that created a fertile breeding ground for working on reactive concepts.

Using this paradigm at a smaller scale seems like a great way to create modular systems. This pattern also compliments the microservices architecture. I’m looking forward to working with these concepts as I get more experience with Scala.

Thinking Back and Looking Forward

I’m writing this after my last day at my previous job. It has me reflecting a lot on the three years I was there. I learned some interesting things and met a lot of great people. There were aspects of the job I really loved, and there were the aspects of the job that I would love to never think about again. I was reflecting back on my accomplishments there and I was surprised how much I had done in some aspects and how little I had done in others.

One of the memories that stuck out to me was the response to some production performance problems we had in July of 2014. I was new to the organization, which made it my first chance to see how it responded to a crisis. I learned a lot about the organization then. It was great to see how disparate teams came together and worked the problem. It also gave me an opportunity to better understand the organizational structure, good and bad. While investigating, I ended up digging into something that wasn’t the cause of  the problem, but demonstrated numerous other little problems that had been festering. There, I managed to make a big impact with different higher-ups once I showed them all of the hidden problems I dug out. It was especially interesting for me to handle from a workplace navigation perspective, since all of the problems were outside of things my team was nominally responsible for.

An area where I tried to make an impact and spent a lot of effort was the build pipeline. The ‘continuous integration’ that was set up by the configuration management team had numerous problems. The basic workflow was that they would – on request or on a timer – take C# code, compile it, and commit the DLLs back to SVN. When I first saw this, I noted that using source control to store the binary artifacts that were created from the repo was odd. The configuration management team didn’t have any interest in changing this, since the entire deployment process was based around deploying DLLs from SVN; it’s understandable they were resistant since changing this piece would require changing everything. This limited a lot of my options on how to change the build pipeline without getting involved in deployment as well. I spent a long time trying to convince the configuration management team to run tests before committing back to SVN, but the server they were building on failed a number of the tests when they tried that. It turned out that the team that owned the code under test didn’t even know those tests existed. This led into an effort to find all of the tests in the whole system and get them passing again. We made progress on these goals but there ended up being a functional area nobody owned, and the rest of my team didn’t feel like we had the capacity to take ownership of it. This left the effort in a sort of stalemate: progress had been made and we could run a lot of tests, but there couldn’t be a clean build unless we started making exceptions. We held in limbo for months, where management continued to debate  who owned that functional area. This eventually led to the effort dying off. That was a massive blow to my morale, since to me it implied a lack of focus on quality; that wasn’t at all their intent, but that doesn’t reduce the frustration after I had gotten so far and made so much progress. I recognize that I may have taken it too personally or read motivation into it on management’s part that wasn’t there, which is an important lesson I’m taking away. I also learned a great deal about how to navigate the corporate environment and how to use soft power to influence change, but most of that was too late to make this change successful; now I know better how to approach this type of major change management from the start.

I don’t want to give the impression that everything was bad – a lot of aspects of the work were great and interesting. I had a great experience working with a system that is much larger than anything I had worked with before. The whole system was hundreds of times larger in terms of computational resources than anything I had worked on before. The codebase itself was approaching ten million lines of code across 10+ different programming languages. Digging into all of this was an amazing adventure to understand how it all went together. Very few people were looking at the system as a whole and I got to come in, learn and understand all of this and help those who were trying to build other pieces of it.

I also had the opportunity while there to mentor some junior engineers and it was quite rewarding. I helped them get up to speed on the code base, which is an easy enough activity. But I also spent a great deal of time working with them on code reviews and helping them understand how to build better software. Helping them to understand the difference between something that works and something that is truly good was really rewarding as they started to see the difference.

My new job is a much different situation than my last job. At the last job I was there to help them clean up the technical mess they knew they had; their issues are common to large organization that have grown through acquisition, and I do appreciate that they wanted to improve. The technical mess wasn’t the whole problem; as you can see from these two anecdotes above there were multiple groups who weren’t in sync and a lack of understanding of each other. The new place is much smaller so it’s unlikely that I’ll have to navigate as many different competing groups, or be assigned such a narrow portfolio. My new company is also much newer so there hasn’t been the time to build up the layers of technical debt. From a technical perspective, at the new job I hope to master a new language and stack and build out some RESTful web services. I’d also like to help build a high performing team and mentor some junior developers, since I found that  really rewarding The person who mentor me to see that difference between “works” and “good” happens to be working at the new place, so that will be a great relationship to renew now that I’ve had an opportunity to play that role. Sometimes the journey is the destination afterall.