Book Chat: Scala With Cats

Scala with Cats is a free ebook put together to help introduce the Cats library and it’s programming style to developers. It is targeted to Scala developers with about a year of experience with the language, but if you were using the language in a very Java-like way you may not be prepared for this book. That caveat aside, it brings an accessible introduction to the library and it’s style of programming.

I can’t go back and have read this before having read Functional Programming in Scala but it seems like either order would work fine. They both talk about the same basic concepts around purely functional programming. They come at it from two different perspectives; Scala with Cats is about how the category theory-inspired structures in the library can be used to solve problems, whereas Functional Programming in Scala is leading you towards those same category theory-inspired structures but getting you to find the patterns yourself.

I really appreciated the last set of exercises in Scala with Cats where it had you implement this concept. It starts out out as a fully concrete class then converting it into more and more generic structures. First, by adding some type classes to become generic to the specific types. Then, by abstracting over the intermediate data structure and converted the structure to its own type class. Finally, by abstracting over the data structure even further by replacing it with another type class.

I think this style of programming has some definitive pros. The idea behind the vocabulary is good, even if the terms chosen obscure some of the intent. The extensive usage of type classes adds an additional layer of polymorphism that lets a library author abstract over portions of the implementation to make it future-proof. The Scala implementation of type classes makes this feel awkward at points since the imports around implicit instances are less obvious around what is happening. I feel like I need to spend some time with a real application written in this style to try to see what the negatives are to working with it. I can see the issues with learning to work in this style, but I’m uncertain about what the negatives are once you’ve gotten used to this style.

Advertisements

Type Aliases in Scala

I had an interesting conversation recently with one of the junior engineers on my team about when to use type aliases. Normally when I get asked for advice I’ve thought about the topic or at least have a rule of thumb I use for myself. Here all I could manage to express was that I don’t use type aliases but not for any particular reason. I felt I should do better than that and promised to get some better advice and see what we can do with that.

Having thought it through a little, here’s the guidance I gave. You can use type aliases to do type refinement to constrain an existing type. So, you could constrain that integer to only positive integers. Instead of assuming that some arbitrary integer is positive, or checking it in multiple places you can push that check to the edge of your logic. This gives you better compile time checks that your logic is correct and that error conditions have been handled.

They can also be used to attach a name to a complex type. So instead of having an

Either[List[Error], Validated[BusinessObject]]

being repeated through the codebase you can name it something more constructive to the case. This also allows hiding some of the complexities of a given type. So if, for example, you had a function that returns multiply nested functions itself like

String => (Int, Boolean) => Foo[T] => Boolean

it can wrap all that up into a meaningful name.

None of this is really a good rule for a beginner but it feels like it wraps up the two major use cases that I was able to find. I ended up going back to the engineer that prompted the question, with “use type aliases when it makes things clearer and is used consistently.” Neither of us were really happy with that idea. There are clearly more use cases that make sense but we weren’t able to articulate them. We’re both going to try it in some code and come back around to the discussion later and see where that gets us.

Scala Varargs and Partial Functions

I ran into a piece of code recently that looked like

foo(bar,
   {case item:AType => …}
   {case item:AnotherType => …}
{case item:YetAnotherType => …}
// 10 more cases removed for simplicity
)

I was immediately intrigued because that was a very odd construction and I was confused why someone would write a function accepting this many different partial functions and what they were up to. I went to look at the signature and found the below.

def foo(bar: ADTRoot, conditions: PartialFunction[ADTRoot, Map[String, Any]]*):  Map[String, Any]

It was using the partial functions to pick items from the algebraic data type (ADT) and merge them into the map. More interestingly it used the the ability of the partial function to identify if it can operate on the type that bar happened to be. Overall it was interesting combination of language features to create a unique solution.

Part of is that the ADT was missing some abstractions that should have been there to make this sort of work easier, but even then we would have had three cases not a dozen. I’m not sure if this pattern is a generalizable solution or even desirable if it is, but it got me thinking about creative ways to combine language features provided by Scala.

Book Chat: Functional Programming in Scala

I had been meaning to get a copy of this for a while, then I saw one of the authors, Rúnar Bjarnason, at NEScala 2017 giving a talk on adjunctions. Before seeing this talk I had been trying to wrap my head around a lot of the Category Theory underpinning functional programming, and I thought I had been making progress. Seeing the talk made me recognize two facts. First, there was a long way for me togo. Second, there were a lot of other people who also only sort of got it and were all there working at understanding the material. At the associated unconference he gave a second talk which was much more accessible than the linked one. Sadly there is no recording, but I started to really feel like I got it. Talking with some of the other attendees at the conference they all talked about Functional Programming in Scala in an awe inspiring tone about how it helped them really get functional programming, and the associated category theory.

The book is accessible to someone with minimal background in this, so I came in a somewhat overqualified for the first part but settled in nicely for the remaining three parts. It’s not a textbook, but it does come with a variety of exercises and an associated repo with stubs for the questions and answers to the exercises. There is also a companion pdf with chapter notes and hints about how to approach some of the exercises that can help you get moving in the right direction if stuck.

Doing all of the exercises while reading the book is time consuming. Sometimes I would go read about a half a page and do the associated exercises and spend more than an hour at it. The entire exercise was mentally stimulating regardless of the time I committed to the exercise, but it was draining. Some of the exercises were even converted to have a web-based format that is more like unit testing at Scala Exercises.

I made sure I finished the book before going back to NEScala this year. Rúnar was there again, and gave more or less the same category theory talk as the year before, but this time around I got most of what was going on in the first half of the talk. In fact, I was so pleased with myself, that I missed a key point in the middle when I realized how much of the talk I was successfully following. I ended up talking with one of the organizers who indicated he encouraged Runar to give this same talk every year since it is so helpful to get everyone an understanding of the theoretical underpinnings of why all this works.

This book finally got me to understand the underlying ideas of how this works as I built the infrastructure for principled functional programming. It leaned into the complexity and worked through it whereas other books (like Functional Programming in Java) tried to avoid the complexity and focus on the what not the why. This was the single best thing I did to learn this information.

NEScala 2018

I attended NEScala 2018 recently for the second time and wanted to discuss the experiences I had there. It’s three loosely affiliated conferences across three different days. The first day was an unconference, the second was NEScala proper, and the third day was a Typelevel summit. I saw a bunch of great talks and met other practitioners who all brought a different perspective to the same sorts of problems I work with every day, as well as some people who have radically different problems.

There were a pair of talks from presenters at Twitter on how they deal with their monorepo using Scalafix and Pants. These were interesting solutions to the problems of the monorepo. During the transition to the microservices at my current job the code base has shattered into hundreds of repositories, which comes with problems, and you sometimes look and wonder if doing this another way would solve those problems. This was a clear reminder that there are problems on the other side that are just as difficult – just different.

The sections on Http4s(talk) and sttp were especially interesting to see the way they tackled HTTP servers and clients as purely functional structures. HTTP was in my mind difficult to describe purely functionally because it is all about referentially untransparent actions. Sttp was especially interesting because we had built a similar abstraction at work in the last year and seeing how others made different tradeoffs was interesting.

The big takeaway for me was that functional programming purity is a pragmatic thing. Functional programming is a tool to tackle complexity in software, but it’s not the only tool available to do that. There are ways to use local effects to wrap small bits of imperative code, but outside of the function where the imperative code lives, none of the callers can tell. You have a thin imperative wrapper on the outside and possibly little imperative nuggets on the inside that resolve performance issues and occasionally improve algorithmic readability, but the whole program retains the composability and readability of an immutable program.

Functional Programming Katas

Based upon my success with the F# koans I went looking for some more covering the functional side of Scala programming. I’ve found a couple so far, which I’ve completed with varying levels of success.

First was the Learn FP repo is a github repo to checkout and has some code to fill in to make some tests pass. The exercise asks you to provide the implementations of various type classes for different types. There were some links to other articles about the topics but otherwise it was just code. The first part of this was fairly straightforward; I had some trouble with State and Writer but otherwise persevered until I hit the wall at Free. I ended up breaking down and looking up the completed solutions provided in a different branch to find that IntelliJ indicates that the correct solution doesn’t compile (Turns out the IntelliJ Scala plugin has an entire scala compiler in it, and it’s rough around the edges). That frustrated me for a while but I eventually managed to power through, and thankfully the rest of the exercises didn’t suffer from the same problem.

Next was the Cats tutorial. I had done some of the other exercises here when first learning Scala and that had been pretty helpful. This has a neat interactive website to run the code you fill in, but it makes it harder to experiment more with the code. This seemed like a reasonable place to start to cover a lot of the major type classes in Cats. It has you look at sample code and fill in what it would evaluate to. It was good but I had two issues with it. First, there are multiple blanks to fill in some of the sections and it evaluates all of them as a group and doesn’t provide any feedback helping you know which one you got wrong. Second, it’s a lot of looking at other code and describing what it does, no writing of code in this style yourself. Overall it helped me feel more comfortable with some of the terminology, but didn’t produce that “ah ha” moment I was looking for regarding the bigger picture.

Then I went to the Functional Structures Refactoring Kata, which is an application to be refactored into a more functional style with samples in multiple languages.The authors provide a ‘solution’ repo with refactored code to compare to. The issue I had with this exercise is that other than going to look at the solution there isn’t a real way to tell when you’re done.  Even then, some of the ways they factored their solution are opinion based. While seeing that opinion is interesting they don’t really explain the why of their decisions.

The last tutorial I tried was the Functional Programming in Scala exercises. It’s from the same people as the Cats tutorial above and is based on the exercises in the book Functional Programming in Scala. I managed to get about halfway through it without having read the book. While there is some prose in between exercises, it doesn’t adequately explain all of the concepts. While I’m reading the book I will come back to this and do the rest of the exercises.

Overall I would strongly recommend the Learn FP repo, and recommend the Cats tutorial. I would pass on Functional Structures Refactoring Kata. I’ll hold judgment on Functional Programming in Scala until I can try it with  the book. While these were largely good starts, I still haven’t had that conceptual breakthrough I’m looking for on how to use all of these pieces in practice.

Type Classes

I have been trying to understand type classes for a while and having a hard time figuring out exactly what they mean or how to effectively use them. The idea that they are a means to extend the behavior of a class in a consistent matter makes sense in an abstract way. But I’m not sure how exactly they accomplish that goal, or what a type class is or isn’t. A coworker had offhandedly referred to a type class as “a trait with a type parameter,” but it feels like there has to be more to it than that. This post is a sort of journal of my efforts to figure out exactly what a type class is.

The wikipedia page wasn’t that helpful to me since it defines type classes in terms of other unknown terms. It had one useful tidbit of information from my perspective: in Scala, type classes are implemented with implicits. Following some more links I ended up at the Cats documentation, which has a bunch of example type classes and code using them that I found useful. This got me thinking about whether I had seen anything that had a signature that looked like it might be using a type class. I remembered the sum method on List, which had stuck in my mind because it was unclear as to how it knew how to sum the items and what types it would be legal to sum.

This definitely looks like a type class given our definition so far. We have an implicit argument that is a trait with a type parameter. It is being used to extend numeric types to give them orderings and perform arithmetic operations, but it also signals that the type is numeric. The type class is also being used to constrain what types the sum method is available for, since if the implicit is not available it won’t compile. This constraint also plays nicely with the context bound syntax.

So we’ve got an example and some rules, but that’s not really a definition. I went looking for some more examples in the Cats codebase since that is full of type classes. Each of the individual type classes in Cats definitely follows the pattern of a trait with a type parameter. I think the missing piece for my understanding, lay in what the methods on the type class are. The methods all seem to take at least one argument of the type parameter so that appears to be a reasonable constraint on the functions.

Type classes are different than an implicit class since you can constrain type signatures with it, but they both let you add new functionality to an existing type. The implicit lookup progress imposes some constraints on the implicit class, such as only taking one non-implicit argument, which the type class methods can bypass. You could write an implicit class to expose the type class in a more fluent way like.

 implicit class NumericWrapper[T:Numeric](x: T) {
  def plus(b: T): T = {
    val n = implicitly[Numeric[T]]
    n.plus(x, b)
  }
}

Type classes seem like they would be more useful in a language without multiple inheritance. Since in Scala I can have a type implementing multiple traits that already have implementation associated with them, just mixing in more code seems like an easier way around the extension problems. I found this proposal for adding type classes to C#, which seems very cool and in line with the sorts of powerful abstractions they’ve been trying to add to the language. Seeing a different syntax for using type classes without implicits being involved helped me understand what they really are.

Going back to my initial goal of figuring out what a type class is and how it works, I think I’ve figured out both. It is a generic type that adds functionality to that type but defers the implementation of that functionality. Then you specify something that expects the type class and brings the implementation of the functionality and the data together. In Scala it accomplishes this using type bounds to specify the type class and implicit parameters to pass the type class into the implementation. I’m still not sure when I would want to write my own type class as opposed using other polymorphic concepts, but I’m now confident about using existing type classes and even breaking out some of the Cats-based ones as opposed to just some of the inbuilt ones.

Book Chat: Programming in Scala

I had previously mentioned Programming in Scala when discussing Scala for the Impatient, saying that Scala for the Impatient was written as a reaction to the ~800 page bulk of Programming in Scala for people who wanted just enough to get started with. After having read Programming in Scala it feels like criticism of its length is fair. The first half of the book was massive overkill for people who had experience in any C derived language or any other sort of object oriented language. There were sections that were marked off as optional reading  if you were familiar with Java because the behavior being described was similar. In comparison, the second half of the book was a wonderful experience even for an experienced programmer, since there were in depth explanations of all of the advanced language features.

Some of the things I learned were simple. For instance, that regular expressions can be used as extractors, which is a straightforward idea. Or, that predef is implicitly imported everywhere. Or that the arrow operator is actually defined as an implicit conversion in predef and not an explicit part of the language.

Other sections were more complex. The rules for how for expressions get mapped into other syntax were similar to what I had figured out, but the rule about how conditionals and assignments within the expression are evaluated added a lot of clarity to what I had learned by doing. I learned the ways you can use type bounds to improve variance indicators. The authors also discussed the transform method on futures that will be available in Scala 2.12, which has me excited to get to that upgrade.

There were some other things covered that even after a considerable study I’m not sure I understand. I understand the syntax for refinement types but I don’t think I understand the value even after the provided in-depth example using currencies. There was also an in-depth discussion of how the designers arrived at CanBuildFrom in the collections package. CanBuildFrom enables extraction of common operations from many collections but returns a collection of that same type and not some supertype. It makes sense in an abstract sense, but I don’t think I could implement a similar pattern without copying it directly out of the book.

Despite the book’s heft, there were a couple of topic I would have liked to know more about. I was hoping for a discussion of the reflection capabilities provided by manifests, type tags, and class tags, but since they are just library pieces and not integral to the language they weren’t covered. There were some oblique references to how bytecode gets generated from various Scala structures, but I was hoping for more insight into how to make interfaces that are less susceptible to breaking changes under the hood even when the Scala side looks fine.

Overall it’s a good read and not as long to read as you would think a book this size would be. It’s easily divided up into small sections so you can easily sit down and read a page or two and make progress over time.

Continuation Passing Style

I have been doing some work with a library that creates guards around various web endpoints. They have different kinds of authentication and authorization rules, but are all written in a continuation passing style. The idea of the continuation passing style is that you give some construct a function to ‘continue’ execution with once it does something. If you’ve ever written an event handler, that was a continuation. The usages all look somewhat like

secureActionAsync(parseAs[ModelType]) { (userInfo, model) => user code goes here }

There was some discussion around whether we wanted to do it like that or with a more traditional control flow like

secureActionAsync(parseAs[ModelType]) match {
    case Allowed(userInfo, model) => user code goes here then common post action code
    case Unallowed => common error handling code
}

The obvious issue with the second code sample is the need to call the error handling code and post action code by hand. This creates an opportunity to fail to do so or to do so incorrectly. The extra routine calls also distract from the specific user code that is the point of the method.  

There were some additional concerns about the testability and debuggability of the code in the continuation passing style. The debugging side does have some complexity but it isn’t any more difficult to work through than a normal map call which is already common in the codebase. The testability aspect is somewhat more complex though. The method can be overwritten with a version that always calls the action, but it still needs to do the common post action code. The common post action code may or may not need to be mocked. If it doesn’t need to be mocked this solution works great. If the post action code needs to be mocked then putting together a method that will set up the mocks can simplify that issue.

This usage of a continuation helps collect the cross cutting concern and keeps it all on one place. You could wrap up this concern other ways, notably with something like like AspectJ. The issue with doing it with something like AspectJ is that it is much less accessible than the continuation passing style. AspectJ for this problem is like using a bazooka on a fly, it can solve it but the level of complexity introduced isn’t worth it.

Future[Unit] Harmful

I’m not the first person to have seen this behavior but I saw it bite another engineer recently so it’s clearly not well enough known. The Unit type in Scala is a special type, there is effectively an implicit conversion from every type to Unit. If you are interested in the specifics check out this post, I’m not going to get into the specifics of how it works or the edge cases. This means that you can write the something like

val doubleWrapped: Future[Unit] = Future.successful(Future.successful(true))

and that compiles. It breaks the type safety that we have come to expect. More intriguingly if you were to do

val innerFailure: Future[Unit] = Future.successful(Future.failed(new RuntimeException))
innerFailure.value

what would you expect the result to be?

If you said

Some(Success(()))

you would be right, but that would probably not be what you were looking for. If you had a map where you needed a flatMap you would end up with compiling code that silently swallows all error conditions. If you have an appropriate set of unit tests you will notice this problem, but you can have that code that looks simple enough that it doesn’t need any unit tests.

The compiler flag -Ywarn-value-discard should give you some protection, but you need to turn it on explicitly and it may produce a fair bit of news in an existing large codebase. So keep an eye out for this issue, and be forewarned.