Skip to main content
The Exports API uses cursor pagination rather than offset/page numbering. Cursors are stable under concurrent writes: as long as you walk forward by passing back the cursor you were given, you never duplicate or skip a row even if data is being written while you read. The same endpoint handles both the initial bootstrap (read everything once) and incremental delta loads (read only what changed since last time).

​
How the cursor works

  • Each response returns a body.pagination.nextCursor. It is an opaque, base64url-encoded token. Do not parse, decode, or construct cursors client-side — pass them back verbatim as the cursor query parameter on the next request.
  • Each cursor encodes the boundary row’s (cursor_field_value, _id). The cursor field varies by dataset — see the table on the Overview page. For most datasets it is updated_at; for D21 and D23 it is communicated_at.
  • Cursors are stable: passing the same cursor twice returns the same page.
  • When nextCursor is null, you have reached the end of the data for your current filter.

​
Load patterns

Omit since. Walk forward by passing back nextCursor until it returns null.
cursor=""
while true; do
  resp=$(curl -s "https://stage-v3-api.cleargrid.ai/admin/v3/lenders/${LENDER}/exports/S2/D07?limit=500&cursor=${cursor}" \
    -H "Authorization: Bearer ${TOKEN}")
  echo "$resp" | jq '.body.rows[]'
  cursor=$(echo "$resp" | jq -r '.body.pagination.nextCursor // empty')
  [ -z "$cursor" ] && break
done
Store max(updated_at) (or whichever cursor field that dataset uses) from the last full load. On the next run, pass that timestamp as since. Walk forward as above.
since="2026-05-22T10:00:00Z"
cursor=""
while true; do
  resp=$(curl -s "https://stage-v3-api.cleargrid.ai/admin/v3/lenders/${LENDER}/exports/S2/D07?limit=500&since=${since}&cursor=${cursor}" \
    -H "Authorization: Bearer ${TOKEN}")
  echo "$resp" | jq '.body.rows[]'
  cursor=$(echo "$resp" | jq -r '.body.pagination.nextCursor // empty')
  [ -z "$cursor" ] && break
done
since is treated as an inclusive lower bound (>=) and until as an exclusive upper bound (<) on the cursor field. If your client is unsure how a range was interpreted, treat the body.meta.since and body.meta.until values echoed back in the response as authoritative.
Datasets in S4 Communications (and some in S3) are filtered to your tenant through a parent collection using an internal $lookup join. For those, a single response can contain fewer rows than your limit even when more data exists beyond the page. The cursor still advances correctly — keep walking until nextCursor is null.Do not infer “end of data” from rows.length < limit. Only nextCursor: null reliably means the end.Affected datasets (TIER 2 — variable page size): D03, D17, D18, D19, D21, D22, D23. You can tell which datasets behave this way from body.meta.tenantPathKind — a value of "via" indicates the join-based filter, while "direct" datasets return a constant page size until the last page.Window the largest via datasets. A few of these — notably D23 (tens of millions of rows) and D03 — are large enough that an unbounded request (no since/until) can exceed the API gateway timeout and return a 504. For these, always pass a time window and walk forward in slices (e.g. month-by-month), paging the cursor to null within each slice before advancing to the next. A smaller limit also helps each page return faster.
Rows are sorted by the dataset’s cursor field ascending by default. Pass sort=desc to reverse. Ties on the cursor field are broken by _id ascending (or descending, matching sort). The cursor encodes both the cursor-field value and _id, so pagination stays stable even when many rows share the same timestamp.
Persist the most recent nextCursor after each page has been fully processed. On restart, resume by passing it back as cursor. There is no per-cursor TTL — a cursor stored for a week and resumed later will still work.
Pass ?includeTotal=true on the fetch endpoint to get pagination.total — the total number of rows that match your filter, independent of cursor and limit. Useful for progress tracking and capacity planning.Three things to know:
  • via (TIER 2) datasets require a time window. For any dataset filtered through a parent collection — D03, D17, D18, D19, D21, D22, D23 — you must pass since and/or until, otherwise total comes back as null with totalNote: "window_required". This guards against an unbounded join.
  • Counts are capped at a server-side limit. For very large result sets (e.g. millions of rows in a wide time window), total is returned as a lower bound with totalExact: false and totalNote: "capped". Narrow the time window for an exact count.
  • Counts have a wall-clock budget. If the count query exceeds the internal timeout, total is null with totalNote: "timeout". The rest of the response (rows, nextCursor) is unaffected — only the count failed. Retry later or narrow the window.
Don’t pass includeTotal=true on every page request — it’s an extra query. Recommended pattern: pass it once at the start of the load to size the work, then omit it on subsequent pages.
Do not decode or attempt to construct the cursor. Its internal format is an implementation detail and may change without notice. Always treat it as an opaque string and pass it back exactly as received.