1. API versioning

This document details the versioning scheme used for wire-server’s HTTP APIs. This applies equally to:

  • the public-facing API (defined in wire-api)

  • the federation API (defined in wire-api-federation).

1.1. Supported and development versions

An API version is a natural number, represented as vN, where N is the version. For example, version 5 is denoted v5.

A backend advertises a set of supported API versions, divided into a set of stable API versions and a set of development API versions. These sets can be discovered via the GET /api-version endpoint, which returns a JSON object of the form:

{ "supported": [0, 1, 2, 3, 4],
  "development": [4],

where the ... stands for other fields, which are irrelevant to the purposes of API versioning. Development versions (usually one), are listed explicitly. Stable versions are all supported versions that are not development.

Stable versions map to well-defined and fixed API contracts, which will not change over time. That is to say, if two different instances of the backend both list version 3 in supported, a client can assume that they will accept the same exact requests when version 3 is used, and handle them in exactly the same way.

On the other hand, development versions give no such guarantees. When making requests using a development version on some backend, the client needs to be aware of the corresponding API contract used by that specific backend.

Of course, development versions are useful while building a new API, but are not suitable for production. Backends deployed to production environments should disable development versions (and not advertise them in /api-version).

Similarly, clients that are meant for production use can decide to ignore development versions on their backend. This is not strictly necessary, but it can be used as a safeguard against mistakes in deployment.

The /api-version endpoint returns information about the public-facing (client) API. The corresponding information for the federation API is available at /federation/api-version.

1.2. Making requests to a particular version

An API at version N can be accessed by prepending /vN/ to endpoint paths. So, for example, to access the endpoint /conversations for version 3, the correct path is /v3/conversations.

To support clients that have not yet implemented versioning, backends that support version 0 will also accept unversioned requests, which will automatically be rewritten to /v0/ requests.

Some endpoints are unversioned. The backend will also accept versioned requests to them, but they all behave identically regardless of the version prefix. The unversioned endpoints are:

  • the /api-version endpoint itself; this is so that a client can dynamically determine which version to use based on the information returned by the backend;

  • the /access endpoint; this is so that access cookie paths can be set to the same value regardless of the version, which avoids invalidating logins across version upgrades.

1.3. Server implementation

On the server side, we normally have a single development version and multiple stable versions. When we make a change to the development version, no further actions are needed.

However, the development version is disabled in production, so whenever we need those changes to actually be deployed, we have to perform a version bump. That means a few things:

  • freezing the current contract of the development version (see below);

  • turning the development version into a stable version;

  • creating a new development version, which is an exact copy of the previous development version at the time of the bump.

To make this process easier, and to avoid unreasonable code duplication in the definition of the endpoints, API versions are maintained as a single routing table, appropriately tagged with version information, as described below.

For each endpoint, we record a range of versions in which this endpoint exists. Note that an endpoint is assumed to behave identically regardless of the version it is being invoked at.

This might seem like a very severe limitation, but in practice most endpoints evolve by adding optional fields, which means that the implementation can simply assume that it being called with the most recent version, and that implicitly will provide support for all older versions as well.

Endpoint changing significantly between two versions are not directly supported by this system. It is still possible to implement them, but they will appear as different endpoints (and have different names) on the same path, and with non-overlapping version ranges.

1.3.1. Version bump checklist

When making the client API version bump, i.e., when finalising a version, there are several steps to make apart from deciding what endpoint changes are part of the version. In these example we assume that version V6 should be finalized and V7 should be created:

  • Run wire-server and download the swagger.json of the current development version, e.g. with the following command: curl localhost:8080/v6/api/swagger.json | jq > swagger-v6.json and copy the file to services/brig/docs/swagger-v6.json.

  • In wire-api extend the Version type with a new version by appending the new version to the end.

  • In the same Version module update the developmentVersions value to list only the new version.

  • In services/brig/src/Brig/API/Public.hs

    • update versionedSwaggerDocsAPI so that the finalized version points to the pregenerated swagger

    • and internalEndpointsSwaggerDocsAPI so that the finalized version V6, the new version V7, as well as the unversioned path point to the swagger of the internal API, and the previous latest stable version V5 points to an empty swagger.

  • Set the version for gDefaultAPIVersion in integration/test/Testlib/Env.hs to 7.

  • Consider updating the backendApiVersion value in Stern, which is unit-tested by checking if it is listed as supported in the response to GET /api-version.

1.3.2. Examples of endpoint evolution

In the following, we present some examples of API changes and how they might be realised in practice. In all examples, we assume a supported API version 1, and a development version 2, and an endpoint which is identical in both versions that now needs to be updated. Adding an optional field or parameter

This is the most common scenario, and the system is optimised to handle this efficiently. We simply go ahead and implement the new version of the endpoint, while leaving the version ranges unaffected. This works because the new implementation is also able to function as an implementation of the older version. Adding a new endpoint

We add the new endpoint to the routing table, and set its version range to only include the development version. The supported version is unaffected. Removing an endpoint

We keep the endpoint in the routing table, but reduce its version range to exclude the development version. The implementation is unaffected. Making an incompatible change (e.g. removing a field)

This is the most complicated scenario, but hopefully the rarest. It can only be done indirectly, as a combination of the two scenarios above: we add a new endpoint on the same path, set its version range to only the development version, and then reduce the version range of the existing endpoint.

The new and old endpoint can share code in most cases, but this needs to be arranged manually. One can use type families to handle differences in version. It is advised to use global versions to index the handlers and related types in these cases, rather than making up an endpoint-specific numbering.

For example, the GET /foo endpoint at version 1 changes the format of its response. Before the change, the routing entry looks something like:

Named "get-foo" ("foo" :> Get '[JSON] Foo)

No version range is specified, resulting in the endpoint being available on all API versions.

Afterwards, we will have two entries:

Named "get-foo@v1" (Until V2 :> "foo" :> Get '[JSON] (Foo V1))
Named "get-foo" (From V2 :> "foo" :> Get '[JSON] (Foo V2))

where Foo has now been turned into a type (or data) family with appropriate JSON instances to reflect the change in data format. The handler implementation can be polymorphic in the version and therefore shared, but it does have to be specified twice when instantiating the server.

1.3.3. Swagger documentation

Since backwards-compatible changes in endpoints are lumped into single definitions, it is not possible to automatically generate accurate Swagger documentation for older versions from the global API routing table.

Therefore, we keep pre-generated Swagger descriptions for all supported versions in the source, and serve them directly.

When a development version N is frozen into a supported one, its automatically generated Swagger description has to be manually saved to services/brig/docs/vN.json, and kept around for as long as version N remains supported.

The Swagger description for the development version is generated by pre-processing the global routing table and removing all entries that do not contain the development version in their range.

1.4. Client implementation

Like servers, clients define a set of versions that they are able to make requests with. However, since they ultimately make the decision of which version to use, there is no distinction in clients between supported and development versions.

Of course, using a version which is marked as development on the backend means utilising an unspecified API contract, so it should not be done in production.

1.4.1. Version negotiation

Before making a request to the server, the client needs to have negotiated which version to use. The recommended flow is as follows:

  • query GET /api-version and retrieve the set of supported and development versions;

  • decide whether using a development version is unacceptable, and if so, take it out of consideration;

  • use the latest (i.e. numerically largest) version that the client supports;

  • if no backend-supported version is supported by the client, return a versioning error (either ask the user or the backend administrator to upgrade, depending on which versions are higher).

1.4.2. Examples of endpoint evolution

Just like for the server above, we present some examples of changes in endpoints, and to go about implementing the corresponding client changes. In the following, we again assume that the changed endpoint was initially equal in both version 1 and 2, with 2 being a development version. Adding an optional field or parameter

Client only need to implement version 2, since the change is backwards- and forwards-compatible. Of course, clients might need to be written in such a way as to handle the scenario in which the server ignores the extra field or parameter. Adding a new endpoint

Clients can use the new endpoint, but need to handle the case where the endpoint is not available, either by catching a 404 error, or by pre-emptively using the old endpoint when the negotiated version is known not to contain it. Removing an endpoint

Clients would just stop using the removed endpoint. Of course, this usually means refactoring whatever code was using it so that some other combination of endpoints is used instead. Incompatible changes to an endpoint

Again, this is a combination of the previous two scenarios. After negotiation, clients need to determine which version of the endpoint can be used, and act accordingly.

1.4.3. Federation client in wire-server

In wire-server itself, changes in the federation API have to be reflected not only in the implementation of the corresponding handlers, but also in client invocations.

To that end, the module Wire.API.Federation.Client provides a FederatorClient monad which is integrated with the client functionality of Servant. To create an action in the FederatorClient monad, we use fedClient in Wire.API.Federation.API, e.g.:

fedClient @'Brig @"get-user-clients"

returns a function of type A -> FederatorClient 'Brig B, where A and B are respectively the input and output of the get-user-clients endpoint. Running such an action will automatically perform version negotiation and then send the corresponding request.

When invoking an endpoint as a federation client, we need to make sure that all supported versions are covered. The FederatorClient monad has an Alternative instance which can be useful for this purpose: an action will fail (before even performing any request) if it refers to an endpoint whose version range does not contain the version that was negotiatted.

For example, suppose that get-user-clients disappears in version 2, and clients are now supposed to use an endpoint called get-clients-ng, with slightly different input and output types. Then the client invocation will look something like:

fedClient @'Brig @"get-user-clients" input <|>
(adaptOutput <$> fedClient @'Brig @"get-clients-ng" (adaptInput input)

where adaptInput and adaptOutput are (pure) functions that convert the input of the old endpoint into the input of the new, and the output of the new endpoint into the output of the old, respectively.

Many variations on this theme are possible. For example, one could choose to write adapting functions in terms of the new input/output types, or even use a mixed approach. The adapting functions need not be pure in general, and they might even perform further RPC calls.

1.5. Versioning changes in events

Making incompatible changes to events is also sometimes necessary, or at least desirable. Unfortunately, there is no direct way to make API versioning preserve compatibility with older clients when the format of events changes. This is because the format of events is decided when they are generated, which is of course before they are fetched by the clients. By the time the backend is made aware of the version supported by a client, it is too late to change any logic of event generation or serialisation.

However, there are ways to alter the event API in incompatible ways without breaking older clients. Namely, we can tie a change X in the format of an event to a specific api version N. This means that in order for a client to support version N or later, it has to be able to consume events in any format, before or after X.

If clients respect this constraint, then the backend only needs to keep the old format around for as long as version N-1 is supported, and can apply change X as soon as version N-1 is dropped.

Conversely, clients need to be advised on when it is ok for them to drop their legacy event parsing code. Unfortunately, determining this point in time is complicated by the fact that legacy events may be retained by a backend for some time after it has been upgraded to a version that emits events in the new format. Therefore, this has to be worked out on a case by case basis.

More precisely: When a new version Q of a backend is released, if we can ensure that no version lower than N is running anywhere in production, and hasn’t been for a time at least as long as the maximum event retention time, then we can drop the requirement for clients to be able to read events in the legacy format, as long as they support only versions larger or equal to Q.

1.5.1. Example timeline

  • While version 3 is in development: a new format for an event is introduced in the code base, but not yet used for output events, the new format is documented for clients.

  • Version 3 is finalised: events are still produced using the old format; clients that implement v3 are able to parse both event formats.

  • Versions 4 to 6 are finalised. No changes to events.

  • Support for version 2 is dropped while version 7 is in development. The old format can be removed from the code base, and the backend can start producing events in the new format. No changes in clients are required.

  • Version 7 to 9 are finalised. No further changes to events.

  • Version 2 or lower is not used in production anymore. Version 10 is currently in development. The old event format is removed from the documentation. Clients that support only version 10 or above are not required to understand the old format anymore.