API Design 303: Pagination

Minimize response times for requests and improve the app-user experience

Pagination is a ubiquitous method for handling large datasets and responses in the browser-based Web but developers of API-based apps also turn to pagination in order to minimize response times for requests and generally improve the end-user experience. In this lesson, we will explore some common methods for enabling pagination in APIs.

While the benefits of using pagination in client apps are clear, actually implementing pagination can be challenging. This is because there are a range of methods API designers can use to make it possible for client app developers to use pagination – and choosing a method often involves making a tradeoff between usability and flexibility.

There are two main design decisions involved in implementing pagination for APIs: 1. how to partition the response data that the API generates; 2. how the client application will select and navigate to the data page it needs. For page selection, including a query parameter in the URI is the recommended method with the majority of use cases.

For partitioning response data, on the other hand, there are various methods you might use, each of which has its own advantages and disadvantages, depending on the use case. (However, whichever method you choose, being consistent across the API will make for a better overall experience, from the developer and end-user perspectives alike.)

Partitioning methods include:

• HTTP range header
• Fixed data pages
• Flexible data pages
• Offset and count
• Default values

Below, we explore these partitioning methods as well as providing some advice on key navigation considerations.

HTTP Range Header

The HTTP protocol has a built-in feature for splitting up data, known as the range header. Using this method, the client application requests a specific byte range from the server and the server returns that data. While it is tempting to use an HTTP range header, the range header was not designed to operate on the level that is necessary for API design.

The HTTP range header is based on byte range, making it very useful for binary data. However, it is not as useful for text-based data. Therefore, in practice, paginating data in this way is likely to be significantly less meaningful to any developer building a Web or mobile application against the API.

Fixed Data Pages

Of all the methods an API designer can use to facilitate pagination in client applications, using fixed data pages is probably the easiest method to grasp conceptually. This simply means splitting results into pages of a predetermined size and partitioning scheme then having the client application request a specific page.

With fixed data pages, the developer writes an app that requests a specific page of data by page number and the server partitions the data and responds. This method is easy to navigate – you just pick a page. But for this gain in usability, we lose flexibility: the page size is the same for everyone and there is little ability for developers to customize their results.

Determining the right size of data to select per page can be difficult because the answer depends on a number of factors, including the platform that the app is deployed on, the user experience you want the app to project and even the type of network that the app is using to make its calls.

Flexible Data Pages

Another option, which provides client application developers with additional flexibility, is to give them control of the data page sizes, rather than having these sizes dictated by the server. In this scenario, the client application not only requests a specific page but it also indicates what the page size should be.

While using flexible data pages preserves ease of navigation and increases flexibility by splitting up the data according to the client application’s needs, it makes the implementation more difficult for the developer by increasing the interface complexity (with more parameters) as well as the partitioning complexity.

Offset & Count

To provide the greatest level of flexibility for client applications, you can eliminate the concept of pages altogether and use an offset scheme. Rather than splitting data up into discrete pages, you consider data as a collection of items and allow the client application to request a specific starting index as well as the number of items the server should return.

This is obviously the most complex approach from the developer perspective because devs can no longer think abstractly in terms of pages. Not only does the developer have to think about which specific item should be requested but the client application must also pay attention to a “cursor†position and keep track of where the next offset should be.

Default Values

While each of these methods has advantages and disadvantages, another approach manages to retain the flexibility of flexible data pages or offsets while retaining the usability of fixed data pages. This is achieved by starting with one of the flexible methods and putting default values in place when there is an absence of page instructions and specific values are not indicated.

By doing this, you can take something like an offset mechanism and prescribe default values that will satisfy 90 percent of developers (who will want pagination in a standard way) but still provide the 10 percent who have special requirements with additional controls that will allow them to be very precise.

Navigation Considerations Just like human users, applications using your API will need a way to navigate through these pages. To make navigation easier, you need to provide metadata with each page response. For instance, this may include the number of pages or items available, the last time the dataset was updated or the current location within the dataset.

In addition to this basic information, providing hyperlinks to other pages within the dataset will make it easier for applications to navigate through your API. In particular, it will mean they will not have to construct their own URLs or calculate their own offsets. When making these links, remember to make the interface simple for devs by using common naming standards.