Back in March 2021, Anvil released SpectaQL to the open source community with the goal of becoming the de-facto, go-to library to use for auto-generating static HTML documentation for any GraphQL API. So far we're very pleased with the progress we've made:
- It gets downloaded thousands of times per week on NPM
- It has over 500 stars on Github
- It has a healthy amount of user support and Github issue activity
- There have been several articles written about it, like this one and this one
Despite the success of SpectaQL over the last year or so, it was still in a "pre 1.0" stage as we learned more and more about what worked well, heard from users about what they wanted, and of course fixed a whole bunch of bugs. After all that time, feedback, and thought we decided that it was time to devote some resources to make a major internal overhaul, improve some existing features and APIs, and create or expose some new features and APIs that prepare SpectaQL for the future. We are happy to announce the culmination of all that work with the release of SpectaQL 1.0!
In the rest of this article, I'm going to outline some of the major changes, enhancements, and new features, as well as the decision process around some of the things we've done. I hope you find it informative and interesting.
Why did we decide to do it? And why now?
In order to increase their reliability and usefulness, most open source projects, and especially libraries and packages, follow the Semantic Versioning (aka Semver) specification. Perhaps the most important takeaway from Semver is that if you make any "breaking changes" to your library, you should release those changes as a new Major version—e.g. go from 1.x.x => 2.0.0. While the possibility of breaking changes without a major version change was something I warned users about while SpectaQL was on version 0.x.x, I was still hesitant to cause any breaking changes due to the amount of usage the package was getting.
As a developer who uses open source packages myself, I know that upgrading package dependencies by a "patch" or "minor" amount is usually no big deal, but that upgrading a package by a "major" amount is something that requires careful consideration and oftentimes some (major) code changes and real work. For this reason, I wanted to be careful about introducing breaking changes too often, so I waited for more understanding.
After what felt like enough experience, feedback, discussion, thought, and research there ended up being many things that I wanted to change about SpectaQL that would be considered "breaking changes." That time and perspective also allowed me to be more confident that the APIs and ergonomics of the changes I wanted to make would prove to be smart and complete enough to avoid having to make any breaking changes for some time. And so I began developing SpectaQL 1.0. Here is a somewhat exhaustive list of the "breaking changes" intended to help users migration from 0.x.x to 1.0.0. I'll focus the rest of this article on some of the major and more interesting parts of this work.
Internal DNA based on JSON Schema and Swagger/OpenAPI
SpectaQL was originally forked from a project called DociQL to add some enhancements there. And DociQL was originally forked from a project called Spectacle to make it work with GraphQL. And (finally) Spectacle was designed to work with Swagger/OpenAPI. Swagger/OpenAPI is a specification designed to help standardize the description of REST APIs with many routes, and uses JSON Schema as the way to define Data Types/Models. This means that SpectaQL had to wrestle with several layers of history and transformations; making SpectaQL's enhancements fit into the DociQL world, which in turn had to make things fit into the Spectacle/Swagger/OpenAPI world.
This presented many challenges, but there were a few in particular that made things feel like they were really trying to fit a round peg into a square hole:
- Unlike REST APIs, GraphQL does not have many different endpoints or paths to represent. It usually has just 1 which accepts any valid Queries or Mutations.
- GraphQL already has a Schema definition and strict Typing built into the language, so conversion to JSON Schema felt unnecessary.
- GraphQL already had a number of tools to work with and interact with schemas in useful and interesting ways.
These are just a few of the big ones, but there were other reasons as well, and eventually it became clear that all signs pointed to re-writing SpectaQL's internals to use pure GraphQL tools—no more Swagger/OpenAPI, and no more JSON Schema. The responses and comments on this issue I opened up confirmed my feelings.
This re-writing of the internals took up the bulk of the time to get SpectaQL 1.0 ready, and in the end, provided little-to-no immediate User-facing changes. However, it was well worth the trouble as it dramatically simplified the codebase, making it much easier to make other changes and enhancements. Things that used to take a great deal of experience in and understanding of the codebase can now be easily understood and figured out by a much more casual observer of the code. This has already been proven by receiving pull requests from users wanting to fix bugs or add new features with just a few intuitive lines of code.
Simpler schema manipulation, and the creation of Microfiber
One major feature of SpectaQL is its ability to filter (or "hide") things in your GraphQL schema that you don't want documented. Most schemas have at least a few Types, Queries, or Mutations that are for administrators only, or that are sensitive for various reasons. We put a lot of effort into making this possible for the original launch of SpectaQL, but as time went on it became increasingly challenging to implement complex logic. For example, if you tell SpectaQL to hide your custom type Foo, you probably also want to hide all references to Foo: Queries or Mutations that return Foo, fields that return Foo, arguments of type Foo, etc. This was very tricky to do in the old system.
After starting from scratch with a purely GraphQL mindset and landscape, these sorts of advanced manipulations became much easier to reason about and implement. Since there was no existing tool for it, we ended up creating a standalone package just for querying and manipulating the GraphQL schema in the ways that we needed. We then decided that others might find this tool useful, so we recently open sourced it as Microfiber. You can read my blog post about its release here.
Increase customizabilty and reduce forking via "themes"
By far the biggest user-facing changes that were made for SpectaQL 1.0 were around allowing for customization. In the pre-1.0 world, there was a limited, difficult, disjointed set of ways to customize the CSS, JavaScript, and output. Even worse, there was no real way to customize the data arrangement / grouping or to really control the HTML structure. All of these shortcomings would either leave some users frustrated, or perhaps looking elsewhere for another solution for them. Even worse than a user deciding not to use SpectaQL would be having a user need to fork the project in order to make the changes they need. Forking, we reasoned, was a sign that we'd not given enough flexibility and power to the users. With all of this in mind, we decided that the goal of our customization system in SpectaQL would be to reduce the need to fork the project in order to get any sort of reasonable customization that a user might want.
We accomplished this through the creation of a "theme" system that gives users the following:
- Complete control of the HTML templates (SpectaQL uses Handlebars) to render any DOM structure you like.
- Complete control over the SCSS/CSS supplied along with your HTML.
- Complete control over the JavaScript files that are loaded with your HTML.
- Complete control over the GraphQL data that gets passed to the HTML template engines, allowing you to group, order, and arrange the data however you like.
Some other cool things to note about themes:
- SpectaQL comes with 3 built-in themes to choose from.
- User-created theme files and structure will be overlaid on top of the default theme files and structure so that the theme only needs to contain files that are supplemental or that should be overwritten in the default theme. This makes it easy for the user to provide minor tweaks to the default theme without a lot of code and without missing out on any future updates and bug fixes made to the original/underlying theme.
If you're interested in learning more about SpectaQL's theming system, please check out the docs.
Summary
While SpectaQL owes a great deal to its original DNA, over time some of that legacy code became a hindrance. Converting GraphQL to Swagger/OpenAPI was tough to reason about and follow. Converting GraphQL to JSON Schema also provided challenges. A "feel good" way to support customizing all aspects of the output was also just not there. Over time, it became clear that some old patterns were not necessary, and that things would be a lot better off after a major refactoring. Time, feedback, and research were also necessary before being able to design and implement a robust customization system via "themes."
The big takeaway? While you don't want to be hasty and end up rolling out major breaking changes too frequently, you should also not be afraid to act once you feel you've gathered enough understanding and a critical mass of things that need to change for the project to succeed. For SpectaQL, this was a major investment of time and effort but already it has borne much fruit. We feel confident that we made the right decisions and have set up SpectaQL for long-term success. Happy coding!