Vault

Recently I’ve been working on rolling out a Vault implementation at work and to migrate all of our existing secrets over. It is a tool designed to secure secret data and control access to it. It also offers a variety of ways to handle dynamic secrets for things like database credentials. The dynamic database credentials are are an interesting security feature; any particular set of database credentials can be shut off at any point if compromised and are effectively rotated each time a new instance starts up. It can also act as a certificate authority. This is all built on top of a configurable set of backends and HA clustering setups.

One of the most interesting things is the unsealing process. The system starts sealed, where all of the secrets are inaccessible. The unseal process requires a majority of key fragments to be provided to unseal the vault. This is an implementation of Shamir’s Secret Sharing which i sa cool concept. In the enterprise version, it also provides an auto-unsealing mechanism built on top of AWS Key Management Service.

The REST API is pretty good and most major languages have a third party client available already. The third party clients have different levels of compatibility with all of the features of the system; since it is a plugin based system they don’t necessarily support everything. Sadly, the UI also doesn’t support all of the features, which makes doing some basic testing about how the system works more painful.

Vault seems like a very good tool chest for dealing with secrets, but I would like a more opinionated system about how to do this. I can build my own system on top of it but would like to have integrated support for creating a key of some type and storing it securely. Similarly, its scheme to provide transit encryption requires a lot of work on my side if I wanted to use it. Despite these areas for improvement I’m still excited to get it integrated into our systems.

Advertisements

Book Chat: The Architecture of Open Source Applications Volume 2

The Architecture of Open Source Applications Volume 2 has writeups describing the internal structure and evolution of nearly two dozen different open source projects, ranging from tools to web servers to web services. This is different from volume one, which didn’t have any web service-like software, which is what I build day to day. It is interesting to see the differences between what I’m doing and how something like MediaWiki powers Wikipedia.

Since each section has a different author the book doesn’t have a consistent feel to it or even a consistent organization to the sections on each application. It does however give space to allow some sections to spend a lot of time discussing the past of the project to explain how it evolved to the current situation. If looked at from the perspective of a finished product some choices don’t make sense, but the space to explore the history shows that each individual choice was a reasonable response to the challenges being engaged with at the time. The history of MediaWiki is very important to the current architecture whereas something like SQLAlchemy(a Python ORM) has evolved more around how it adds new modules to enable different databases and their specific idiosyncrasies.

I found the lessons learned that are provided with some of the projects to be the best part of the book. They described the experience of working with codebases over the truly long term. Most codebases I work on are a couple of years old while most of these were over 10 years old as of the writing of the book, and are more than 15 years old now. Seeing an application evolve over longer time periods can truly help validate architectural decisions.

Overall I found it an interesting read, but it treads a fine line between giving you enough context on the application to understand the architecture, and giving you so much context that the majority of the section is on the “what” of the application. I felt that a lot of the chapters dealt too much with the “what” of the application. Some of the systems are also very niche things where it’s not clear how the architecture choices would be applicable to designing other things in the future, because nobody would really start a new application in the style. If you have an interest in any of the applications listed check out the site and see the section there, and buy a copy to support their endeavours if you find it interesting.

Long Term Open Source Success

I was having a discussion with some of my coworkers at our engineering book club, talking about the first few chapters of Clean Architecture. We were discussing whether anyone had worked at a place that took a really long-term view of the architecture of a system. Most people didn’t think they had, I thought I might have once but it was hard to say if the company’s leadership had that view or that they got lucky with some key engineers just making it happen. Unfortunately the company ran into some business issues, with the very large competitors in the industry picking off their largest customers. From there I posited a different question to the group: is the architectural difference between open source software and commercial software one of the reasons for its long term success?

There ended up being a palpable pause and the idea at once made sense to everyone, but nobody was sure of how to try to confirm or deny the thought. We all agreed open source, at least initially, is built by software devotees for personal reasons. Whether it is for personal usage, to learn something, or to prove out an idea, people take time out of their day and build a piece of software and put it out there for the world to see. It’s finished when it’s finished and if you aren’t enjoying working on it then it’s no big loss to just set it aside and do something else. There was some discussion around the open source code with paid support model and whether that had more to do with the development of large chunks of open source software. There was a discussion about a resurgence in popularity for postgres because it was feature packed and solid, and whether its continuation in flexibility was because of an underlying architectural quality difference, given that it does not have corporate backing like other common relational databases.

From an ideological point of view, the idea that software succeeds in the long term because of better architecture is greatly pleasing. I would love to see data bearing that idea out, but I don’t know how you would get access to an appropriately large selection of equivalent projects. Something like The Architecture of Open Source Applications tries to make it easier to understand the big picture of some successful and long lived open source applications, but you would need a match set of closed source applications to compare and contrast against.

Building a great architecture requires taking time to deeply understand the problem. Sometimes you truly need to “build one to throw away,” knowing all the cost that represents. The pressure of commercial success and unbridled growth puts short term thinking into the forefront, and often prevents throwing it away.

Open source is a thousand developers trying to fix problems they run into alone and putting it out there. When you hear about the solution you go take a look at it. If its API doesn’t work for you the project doesn’t gain a new follower and without followers there are no contributors and the project eventually withers and dies. If the API does work for others and it solves problems the project grows. You can look at it as a distributed genetic algorithm doing API first development. You rarely hear about the ideas that don’t catch on; you hear about the things  where people felt they gained value.

One of my favorite definitions of software architecture is making the decisions that are hard to change. If an open source project decides they are just going to change their underlying central abstractions for the next year, that’s a strictly technical decision. They don’t need to build a business justification, or fight to get it on the schedule. They don’t even necessarily need a consensus that it’s a good idea, they could just fork the project if they felt so strongly that was the proper solution.

At work we’ve got a central architectural abstraction that is, in my opinion, not right. I could go build a PR on my time to change it, but that change would then ripple out to a dozen other teams as they start using the new abstraction. I could help them adopt the new abstraction, but I would need a strong consensus from those teams to get the PR in. Even though I think that consensus exists in engineering, the product schedule doesn’t leave time for this sort of change. It’s hard to quantify the drag this abstraction is causing which makes that difficult. If you could use a library that was built by those who are quality obsessed or a library built by those who are trying to match schedule, which would you choose?

Book Chat: 97 Things Every Software Architect Should Know

97 Things Every Software Architect Should Know is a selection of pithy articles with titles like “There Is No One-Size-Fits-All Solution” and “Communication is King; Clarity, and Leadership, Its Humble Servant.” To me they were all common sense ideas in the big picture. None of them were wrong, but none of them are anything anyone who has done the role well or is really ready to do the role shouldn’t be aware of already.

If you ended up thrust into a software architecture position without having a strong idea of how to do it check this out, it will give you some of the big picture stuff to check out. Otherwise just skip it.

Serverless Architecture

The serverless architecture is an architectural pattern where you create a function to be run when an event occurs. This event can be a message being placed in a queue, or an email being received, or a call to a web endpoint. The idea is that you don’t need to think about how the code is being hosted which allows scale out to happen without you being involved. This is great for high availability, cost control, and reducing operational burdens.

The serverless architecture seems to result in setting up a number of integration databases. Imagine the route /foos/:id – there is a get and a put available. Hosting these in a serverless fashion means that you’ve got two independent pieces of code interacting with a single database. In my mind this isn’t significantly different from having two services interacting with the same database.

I went looking around for anyone else discussing this seemingly obvious problem, and found this aside from Martin Fowler comparing them to stored procedures. The stored procedure comparison seems apt since most endpoints seem to me to be wrappers around database calls, your basic CRUD stuff. These endpoints, just like most stored procedures, are a fairly simple query. Stored procedures started out as a good way to isolate your SQL to a particular layer, and enable you to change the underlying design of the database. If you’ve never worked in a large codebase with stored procedures they can evolve in lots of negative ways. You can end up with stored procedures that have business logic in the database, stored procedures that evolved and have 12 arguments (some of which were just ignored and only there for backwards compatibility), or six variations of stored procedure but it being unclear what the differences are. These are all downsides to stored procedures I have seen in real code bases.

I can imagine the serverless architecture equivalent of these problems, in that you would have an endpoint querying some database and be unaware of that shared database eventually the shared database becomes an impediment to development progress. Serverless architecture may still be interesting for a truly stateless endpoint. But, even then you could end up with the put and get endpoints not being in sync as to what the resource looks like. There isn’t a good way yet to package serverless applications. The idea of orchestrating an entire applications worth of endpoints seems a daunting task. Using something like GraphQL that radically limits the number of endpoints being exposed would simplify the deployment orchestration. While GraphQL has had a significant adoption it isn’t the solution for every problem.

Given these issues I don’t see the appeal of the serverless architecture for general application development and using it to back rest endpoints. For pulling from a queue, processing emails, or processing other event sources and interacting with web services it seems a good solution since there would be a single input source and it just produces additional events or other web service calls.

The serverless architecture for standard web apps seems like a choice that could result in a mountain of technical debt going forward. This is especially likely because the serverless architecture is most attractive to the smallest organizations due to the potential hosting savings. Those organizations are the least capable of dealing with the complexity. These are the same organizations that are prone to monolithic architectures or to integration databases since they have a short-term view of events and limited resources to deal with the issues.

I don’t know what the eventual fallout from applications built with this architecture will be but, I do suspect it will employ a lot of software engineers for many years to dig applications out of the pain inflicted.

Book Chat: The Pragmatic Programmer

For a long time this had been on my list of books to buy and read with a “note to self” saying to check if there was a copy of it somewhere on my bookshelf before buying one. It felt like a book I had read at some point years ago, but that I didn’t really remember anymore. Even the woodworking plane on the cover felt familiar. It felt like it was full of ideas about creating software that you love when you encounter them but are disappointingly sparse in practice. Despite being from the year 2000 it still contains a wealth of great advice on the craft of creating software.

Since it is about the craft of software, not any specific technologies or tools or styles, it aged much better than other books. That timeless quality makes the book like a great piece of hardwood furniture, it may wear a little but it develops that patina that says these are the ideas that really matter. There is an entire chapter devoted to mastering the basic tools of the trade: your editors and debuggers, as well as the suite of command line tools available to help deal with basic automation tasks. While we’ve developed a number of specialized tools to do a lot of these tasks it is valuable to remember than you don’t need to break out a really big tool to accomplish a small but valuable task.

It’s all about the fundamentals, and mastering these sorts of skills will transfer across domains and technical stacks. It was popular enough that is spawned an entire series of books – The Pragmatic Bookshelf – and while I have only written about one of them I have read a few more and they’ve all been informative.

About two-thirds of the way through the book I realized that I had indeed read it before – I had borrowed a copy of it from a coworker at my second job. He had recommended it to me as a source he had learned a lot from. I remember having enjoyed it a lot but not really appreciating the timeless quality. Probably since that would have been around 2007, it wouldn’t have seemed as old, especially since things seemed to be moving less quickly then. Maybe I just feel that way since I didn’t know enough of the old stuff to see it changing.

If you haven’t read it, go do it.

tumblr_inline_o2aushqfpx1slrvm0_1280

Session Fun

I’ve been working on getting a new session infrastructure set up for the web application I’m working on. We ended up going with a stateful session stored in mongoDB along with some endpoints to query the session with. This design has a couple of nice aspects – all of the logic about if a session is active or not can live inside of one specific service and a session can be terminated if needed.

Building a session infrastructure is a fairly common activity, but we are building a session that is used by multiple services and we can’t roll all of them out simultaneously. So we’ve been building a setup that can process both the new session and the old session as a way to have a backwards compatible intermediate step. This is creating some interesting flow issues. There are two specific issues I wanted to discuss: (1) how to maintain the activity of the new session during backwards compatibility and (2) processing of identity federation.

Maintaining the new session from applications that have not been updated is nuanced. Since, by rule, you haven’t made changes to the applications that haven’t been updated, you can’t add any calls. In our case there was a call made from all of the application frontends to a specific service, so we are using that to piggyback keeping the session alive. We looked into a couple of other options but didn’t find anything easy. We considered rolling out a heartbeat to the various application frontends but that would require an extra round of updates, testing, and deploys for code that was likely to all be ripped out when we were done.

The federation flow is extra complex because a federation in the new scheme is not that different from some of the session passing semantics under the old session scheme. This ends up mixing together the case where there is just a federation occurring and the case where the new session has timed out and the old session is still valid. This creates an awkward compromise; you’d like to be able to say that if the new session has timed out the entire session is expired, but if you can’t tell the difference between the two cases that’s not possible. This means that the new session expiration can’t be any longer than the old session expiration. But it also solved the problem with maintaining the new session while in applications that haven’t been updated yet.

The one problem solved the other problem which was a nice little win.

Book Chat: AWS in Action

Amazon Web Services in Action is a great introduction to the basics of AWS. It mostly discusses IaaS services, but touches on some of the PaaS services too. It also covers a good portion of what’s in the Architecting on AWS course if you were considering that. I was hoping for coverage of AWS Lambda, API Gateway and EC2 Container Service but they weren’t included. There were references to when to use AWS Cloudfront but it was never introduced like the other services were.

It’s a very quick read, even though it weighs in about 400 pages. There are lots of detailed examples including specific information about if the example was covered under the free tier of services and how to be sure to roll back everything. Lots of screenshots of consoles and the CLI interface and plenty of code samples of using the various SDKs, mostly in node.

If you are already familiar with AWS this probably isn’t the right book for you. It doesn’t cover the architecture aspect in depth, just simple examples of how to combine the various services. For my taste it doesn’t sufficiently cover how to decide when to use a PaaS solution vs rolling your own at the IaaS level. There is one little chart in the portion covering Elastic Beanstalk about the benefits of it vs other less managed options.

Overall it wasn’t the book I was looking for. But it is the sort of thing that can be helpful to lots of people who want to try and understand how to apply their existing architectural knowledge to the AWS platform.

Book Chat: Zero Bugs and Program Faster

Zero Bugs and Program Faster by Kate Thompson is a book that’s hard to describe. None of it is a really novel way of looking at creating software but it’s all of those things that you would expect to describe when you think about how to do programming well. It’s a breezy and fun read that is divided into enough small sections that you can read it in however much time you have available.

The book is structured in two parts. The first part is a series of short vignettes about programming. Some of which are more direct, like the chapter on ACID; some are more abstract, like the chapter entitled “The Many Sides of the Elephant.” I appreciated the dual chapters of “Do It Now” and “Do It Later” that are about how you can’t always do it now but you shouldn’t always defer it either. None of it was a mind shattering revelation but it was all solid advice about programming.

The second part is extracts from various programs to demonstrate a lot of different ideas. The code samples in the second part were generally significantly older, mostly in assembly or C. The low level nature of the examples made it more difficult for me to appreciate. Seeing Altair assembly from the 70s that’s notable for being clever and concise won’t help me build a better web service today.

If you are the sort of person who is reading lots of programming books, you will appreciate the book, however you may not get much from it. If you aren’t the kind of person who reads lots of programming books some of the more oblique points may be obscured. I don’t have anything bad to say about it but don’t know who I would recommend the book to.

Tagging in Macwire and Type Currying

We’ve been using the Macwire dependency injection framework and I recently got to use the tagging functionality for the first time. This was a great feature that enabled me to instantiate multiple instances of the same type and have the framework differentiate between the instances when choosing which to inject.

First, you define a trait to be the marker; it doesn’t need anything in it. Then, at the injection site it becomes

BaseType @@ Marker

When setting up the initial site, instead of being

wire[BaseType]

it becomes

wire[BaseType].taggedWith[Marker]

That’s all there is to the base case. You can chain multiple tags in a couple of different ways.

wire[BaseType].taggedWith[Marker].andTaggedWith[OtherMarker]

or

wire[BaseType].taggedWith[Marker with OtherMarker]

 

I combined all of this with another neat construction of the type system.

trait Foo[Marker](dependency:Bar @@ Marker)

Once I set up the Foo type like this it lets me take a marked dependency in a generic way.

I would like to see some sort of syntax like

preference[Marker].wire[BaseType]

So I don’t need to make changes to the type being instantiated in order to control the preference of what instance gets supplied. There is also the module syntax, where you can create a module for each set and then nest those modules to get the injections as desired; but I don’t want to have to organize my code in a specific way in order to have it work correctly.

All of this had me thinking about currying/partial application and the application of generic types. You can take a type with multiple generic parameters and apply some of them and leave the rest for later. It’s not something that happens often but it came to mind with the way I was using generic types there and it was an interesting insight into the symmetry between types and data.