How To Create Pagination In DynamoDB

How to easily paginate across multiple results in a DynamoDB query.

pagination with dynamodb

Ifyou’ve used DynamoDB enough times, you’ll notice something interesting.

The entire database system is designed to make your queries efficient.

DynamoDB is known to “not let you design a bad query”.

It does this with several different methods I discussed in previous articles like this one.

The last concept I discuss in that article is that in order to make your queries faster, DynamoDB will limit the results a query can return to 1MB.

This means that if you need to get more data from a given query, whose result exceeds 1MB, you need to paginate the data.

In this article, I’ll guide you through doing exactly that.

Pagination Properties In DynamoDB

https://www.cloudtechsimplified.com/pagination-in-dynamodb/

DynamoDB provides you with two properties to help you paginate results in your queries:

• LastEvaluatedKey: DynamoDB returns this key with a value if there are more results available. Once you reach the end, this key isn’t in the response anymore.

• ExclusiveStartKey: You can set this attribute in your query to the LastEvaluatedKey from the previous query response to get the next set (or page) of results.

Implementing Pagination

To paginate across results in an incremental fashion — such as having “page numbers” in your app and displaying a new page of results on each page — you can simply use the LastEvaluatedKey from your previous query and insert it into your new query.

Let’s imagine we have 100 items in our “users” table.

We want to get the first 10 on “page 1" and the next 10 on “page 2”, and so on:

We start by querying the first 10 user items:

TableName: "users",
KeyConditionExpression: "#city = 'Montreal' "
Limit: 10

This query fetches all items where the city is equal to “Montreal” and limits the items to 10.

The result of this query will look like this:

{
  "Items": [
    {
      "userId": { "S": "u#101" },
      "name": { "S": "John Doe" },
      "city": { "S": "Montreal" }
    },
    {
      "partitionKey": { "S": "u#102" },
      "name": { "S": "John Smith" },
      "city": { "S": "Montreal" }
    },
    ...//the other items
  ],
  "Count": 10,
  "ScannedCount": 10,
  "LastEvaluatedKey": {
    "userId": { "S": "u#110" },
  }
}

The response returns the Items and a key LastEvaluatedKey which contains the last item that was returned.

Now in the next query, you can specify this LastEvaluatedKey. This will tell DynamoDB to start the query scan after this key, and for another 10 items (or the Limit you define in your query).

The next query would look like this:

TableName: "users",
KeyConditionExpression: "#city = 'Montreal' "
Limit: 10,
ExclusiveStartKey: {
  "userId": { "S": "u#110" },
}

And the next query result will have the following 10 items:

{
  "Items": [
    {
      "userId": { "S": "u#111" },
      "name": { "S": "Jack Kane" },
      "city": { "S": "New York" }
    },
    {
      "partitionKey": { "S": "u#112" },
      "name": { "S": "jessica Banner" },
      "city": { "S": "Los Angeles" }
    },
    ...//the other 8 items
  ],
  "Count": 10,
  "ScannedCount": 10,
  "LastEvaluatedKey": {
    "userId": { "S": "u#120" },
  }
}

This repeats until the query result contains no more items.

When this happens the results will return an empty array of Items and no LastEvaluatedKey:

{
  "Items": [],
  "Count": 0,
  "ScannedCount": 0
}

Getting multiple pages at once (Infinite Scroll)

You can also implement an “infinite scrolling” UI with DynamoDB queries and pagination.

https://addyosmani.com/blog/infinite-scroll-without-layout-shifts/

For this you’ll need to use a loop to fetch multiple pages of results in one query. This will keep the previous data when scrolling down a list (as opposed to displaying a new set of data with a “pages” UI).

For this, you can use the following code that will loop through multiple queries. As long as the lastEvaluatedLey exists the loop will keep fetching more items from DynamoDB.

function paginateQuery(params) {
  let items = [];
  let lastEvaluatedKey = null;

  do {
    const queryCommand = new QueryCommand({
      ...params,
      ExclusiveStartKey: lastEvaluatedKey,
    });

    const data = await client.send(queryCommand);
    
    items = items.concat(data.Items || []);
    
    lastEvaluatedKey = data.LastEvaluatedKey;

  } while (lastEvaluatedKey); 

  return items; 
}

The code above will loop through queries until the lastEvaluatedKey is undefined and will return all of these items in the items array.

This enables you to keep the previous items alongside the new items, like for a infinite scrolling feature.

Conclusion

Pagination in DynamoDB is essential to understand to handle large datasets efficiently.

To ensure queries remain efficient and fast we can fetch data incrementally using the LastEvaluatedKey and ExclusiveStartKey.

The use cases range from “load more” functionalities to pagination UIs and infinite scrolling features.

---

👋 My name is Uriel Bitton and I’m committed to helping you master Serverless, Cloud Computing, and AWS.

🚀 If you want to learn how to build serverless, scalable, and resilient applications, you can also follow me on Linkedin for valuable daily posts.

For help building your DynamoDB database, I provide 15 minute consultations here.

Thanks for reading and see you in the next one!