> ## Documentation Index
> Fetch the complete documentation index at: https://docs.command.cleargrid.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pagination & Incremental Loads

> Cursor encoding, initial loads, incremental loads, and the variable page-size behavior

The Exports API uses **cursor pagination** rather than offset/page numbering. Cursors are stable under concurrent writes: as long as you walk forward by passing back the cursor you were given, you never duplicate or skip a row even if data is being written while you read. The same endpoint handles both the initial bootstrap (read everything once) and incremental delta loads (read only what changed since last time).

## How the cursor works

* Each response returns a `body.pagination.nextCursor`. It is an opaque, base64url-encoded token. **Do not parse, decode, or construct cursors client-side** — pass them back verbatim as the `cursor` query parameter on the next request.
* Each cursor encodes the boundary row's `(cursor_field_value, _id)`. The cursor field varies by dataset — see the [table on the Overview page](/api-reference/exports/overview#phase-1-datasets-at-a-glance). For most datasets it is `updated_at`; for `D21` and `D23` it is `communicated_at`.
* Cursors are stable: passing the same cursor twice returns the same page.
* When `nextCursor` is `null`, you have reached the end of the data for your current filter.

## Load patterns

<AccordionGroup>
  <Accordion title="Initial load pattern">
    Omit `since`. Walk forward by passing back `nextCursor` until it returns `null`.

    ```bash theme={null}
    cursor=""
    while true; do
      resp=$(curl -s "https://stage-v3-api.cleargrid.ai/admin/v3/lenders/${LENDER}/exports/S2/D07?limit=500&cursor=${cursor}" \
        -H "Authorization: Bearer ${TOKEN}")
      echo "$resp" | jq '.body.rows[]'
      cursor=$(echo "$resp" | jq -r '.body.pagination.nextCursor // empty')
      [ -z "$cursor" ] && break
    done
    ```
  </Accordion>

  <Accordion title="Incremental load pattern">
    Store `max(updated_at)` (or whichever cursor field that dataset uses) from the last full load. On the next run, pass that timestamp as `since`. Walk forward as above.

    ```bash theme={null}
    since="2026-05-22T10:00:00Z"
    cursor=""
    while true; do
      resp=$(curl -s "https://stage-v3-api.cleargrid.ai/admin/v3/lenders/${LENDER}/exports/S2/D07?limit=500&since=${since}&cursor=${cursor}" \
        -H "Authorization: Bearer ${TOKEN}")
      echo "$resp" | jq '.body.rows[]'
      cursor=$(echo "$resp" | jq -r '.body.pagination.nextCursor // empty')
      [ -z "$cursor" ] && break
    done
    ```

    `since` is treated as an **inclusive lower bound** (`>=`) and `until` as an **exclusive upper bound** (`<`) on the cursor field. If your client is unsure how a range was interpreted, treat the `body.meta.since` and `body.meta.until` values echoed back in the response as authoritative.
  </Accordion>

  <Accordion title="Variable page size on some datasets">
    Datasets in **S4 Communications** (and some in S3) are filtered to your tenant through a parent collection using an internal `$lookup` join. For those, a single response can contain **fewer rows than your `limit`** even when more data exists beyond the page. The cursor still advances correctly — keep walking until `nextCursor` is `null`.

    **Do not infer "end of data" from `rows.length < limit`.** Only `nextCursor: null` reliably means the end.

    Affected datasets (TIER 2 — variable page size): `D03`, `D17`, `D18`, `D19`, `D21`, `D22`, `D23`. You can tell which datasets behave this way from `body.meta.tenantPathKind` — a value of `"via"` indicates the join-based filter, while `"direct"` datasets return a constant page size until the last page.

    **Window the largest `via` datasets.** A few of these — notably [`D23`](/api-reference/exports/datasets/d23-manual-communication-logs) (tens of millions of rows) and `D03` — are large enough that an **unbounded** request (no `since`/`until`) can exceed the API gateway timeout and return a [`504`](/api-reference/exports/errors). For these, always pass a time window and walk forward in slices (e.g. month-by-month), paging the cursor to `null` within each slice before advancing to the next. A smaller `limit` also helps each page return faster.
  </Accordion>

  <Accordion title="Sort order and tiebreaking">
    Rows are sorted by the dataset's cursor field **ascending** by default. Pass `sort=desc` to reverse. Ties on the cursor field are broken by `_id` ascending (or descending, matching `sort`). The cursor encodes both the cursor-field value and `_id`, so pagination stays stable even when many rows share the same timestamp.
  </Accordion>

  <Accordion title="Resuming after an interrupted load">
    Persist the most recent `nextCursor` after each page has been fully processed. On restart, resume by passing it back as `cursor`. There is **no per-cursor TTL** — a cursor stored for a week and resumed later will still work.
  </Accordion>

  <Accordion title="Filter-wide totals (optional)">
    Pass `?includeTotal=true` on the fetch endpoint to get `pagination.total` — the total number of rows that match your filter, independent of `cursor` and `limit`. Useful for progress tracking and capacity planning.

    Three things to know:

    * **`via` (TIER 2) datasets require a time window.** For any dataset filtered through a parent collection — `D03`, `D17`, `D18`, `D19`, `D21`, `D22`, `D23` — you must pass `since` and/or `until`, otherwise `total` comes back as `null` with `totalNote: "window_required"`. This guards against an unbounded join.
    * **Counts are capped at a server-side limit.** For very large result sets (e.g. millions of rows in a wide time window), `total` is returned as a lower bound with `totalExact: false` and `totalNote: "capped"`. Narrow the time window for an exact count.
    * **Counts have a wall-clock budget.** If the count query exceeds the internal timeout, `total` is `null` with `totalNote: "timeout"`. The rest of the response (`rows`, `nextCursor`) is unaffected — only the count failed. Retry later or narrow the window.

    Don't pass `includeTotal=true` on every page request — it's an extra query. Recommended pattern: pass it once at the start of the load to size the work, then omit it on subsequent pages.
  </Accordion>
</AccordionGroup>

<Warning>
  Do not decode or attempt to construct the cursor. Its internal format is an implementation detail and may change without notice. Always treat it as an opaque string and pass it back exactly as received.
</Warning>

***

<Snippet file="snippets/support-info.mdx" />