Class KeysetPager

java.lang.Object

com.kingsrook.qqq.backend.core.actions.tables.KeysetPager

public class KeysetPager
extends Object

Read a table a page at a time using keyset ("seek") paging, rather than skip/limit: each page is

WHERE (filter) AND pk > lastKey ORDER BY pk ASC LIMIT pageSize

so the cost of a page stays flat no matter how deep into the table it is, and a run can be resumed from the last key it handled (startingAfter(Serializable), getLastKey()). Works against any backend that supports criteria and ordering on the table's single primary key field. Pass a transaction (withTransaction(com.kingsrook.qqq.backend.core.actions.QBackendTransaction)) and every page is read inside it - e.g., inside a consistent-snapshot transaction. Requirements and edge cases:

• The primary key must be unique (it is, for a real primary key) and non-null. Rows with a null key can't be reached by a "key > lastKey" criteria, so they are not returned.
• If the primary key's type is not orderable (e.g., a BLOB or boolean - rare), forEachPage(com.kingsrook.qqq.backend.core.utils.lambdas.UnsafeConsumer<java.util.List<com.kingsrook.qqq.backend.core.model.data.QRecord>, com.kingsrook.qqq.backend.core.exceptions.QException>) falls back to a single streamed query, chunked into pages of at most the page size. That can't be resumed; and the iterator styles require an orderable key.
• The filter given may not have its own skip or limit, as those can't be combined with paging. Its order-bys are replaced by the primary key order.
• Every page is fetched with the same QueryInput settings - use withQueryInputCustomizer(java.util.function.Consumer<com.kingsrook.qqq.backend.core.model.actions.tables.query.QueryInput>) (or withRawRead(com.kingsrook.qqq.backend.core.model.actions.tables.query.RawReadReason)) to set things such as fieldNamesToInclude (which must include the primary key), join, or the heavy-field flag.

Example:

 KeysetPager.forTable("person")
    .withFilter(new QQueryFilter(new QFilterCriteria("isEmployed", EQUALS, true)))
    .withPageSize(5000)
    .withTransaction(snapshotTransaction)
    .forEachPage(page -> write(page));
 

Field Summary

Fields

Modifier and Type	Field	Description
static final int	DEFAULT_PAGE_SIZE

Method Summary

Modifier and Type	Method	Description
void	forEachPage(UnsafeConsumer<List<QRecord>,QException> consumer)	Run the consumer on each page, in primary key order, until the records are exhausted.
static KeysetPager	forTable(String tableName)	Start building a pager for a table.
Serializable	getLastKey()	The primary key of the last record of the last page that was fully handled (that is, that forEachPage(com.kingsrook.qqq.backend.core.utils.lambdas.UnsafeConsumer<java.util.List<com.kingsrook.qqq.backend.core.model.data.QRecord>, com.kingsrook.qqq.backend.core.exceptions.QException>)'s consumer returned from, or that the iterator moved past) - or the startingAfter(java.io.Serializable) key, if no page has been handled yet.
long	getPagesRead()	The number of pages (queries that returned records) read so far.
long	getRecordsRead()	The number of records read so far.
boolean	isPrimaryKeyOrderable()	Is the table's primary key of a type that can be keyset-paged?
Iterator<List<QRecord>>	pageIterator()	Iterate over pages (lazily - each is queried as the iterator is advanced).
Iterator<QRecord>	recordIterator()	Iterate over individual records, fetching a page at a time.
KeysetPager	startingAfter(Serializable lastKey)	Resume: only return records whose primary key is greater than this one (e.g., the getLastKey() of an earlier, interrupted run).
KeysetPager	withFilter(QQueryFilter filter)	Only page through records matching this filter.
KeysetPager	withPageSize(int pageSize)	The maximum number of records per page.
KeysetPager	withQueryInputCustomizer(Consumer<QueryInput> queryInputCustomizer)	Adjust the QueryInput used for each page, e.g., fieldNamesToInclude, or shouldFetchHeavyFields.
KeysetPager	withRawRead(RawReadReason rawReadReason)	Read every page as a raw read - see QueryInput.withRawRead(RawReadReason).
KeysetPager	withTransaction(QBackendTransaction transaction)	Run every page's query inside this transaction (e.g., a snapshot).

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Details

DEFAULT_PAGE_SIZE

public static final int DEFAULT_PAGE_SIZE

See Also:

• Constant Field Values

Method Details

forTable

public static KeysetPager forTable(String tableName)

Start building a pager for a table.

withFilter

public KeysetPager withFilter(QQueryFilter filter)

Only page through records matching this filter.

withPageSize

public KeysetPager withPageSize(int pageSize)

The maximum number of records per page.

withTransaction

public KeysetPager withTransaction(QBackendTransaction transaction)

Run every page's query inside this transaction (e.g., a snapshot). The pager neither commits nor closes it.

startingAfter

public KeysetPager startingAfter(Serializable lastKey)

Resume: only return records whose primary key is greater than this one (e.g., the getLastKey() of an earlier, interrupted run).

withQueryInputCustomizer

public KeysetPager withQueryInputCustomizer(Consumer<QueryInput> queryInputCustomizer)

Adjust the QueryInput used for each page, e.g., fieldNamesToInclude, or shouldFetchHeavyFields. Don't set the filter or transaction here.

withRawRead

public KeysetPager withRawRead(RawReadReason rawReadReason)

Read every page as a raw read - see QueryInput.withRawRead(RawReadReason). The permission check and audit happen for each page's query.

getLastKey

public Serializable getLastKey()

The primary key of the last record of the last page that was fully handled (that is, that forEachPage(com.kingsrook.qqq.backend.core.utils.lambdas.UnsafeConsumer<java.util.List<com.kingsrook.qqq.backend.core.model.data.QRecord>, com.kingsrook.qqq.backend.core.exceptions.QException>)'s consumer returned from, or that the iterator moved past) - or the startingAfter(java.io.Serializable) key, if no page has been handled yet. Pass it to startingAfter(java.io.Serializable) to resume after a failure.

getPagesRead

public long getPagesRead()

The number of pages (queries that returned records) read so far.

getRecordsRead

public long getRecordsRead()

The number of records read so far.

isPrimaryKeyOrderable

public boolean isPrimaryKeyOrderable()
                              throws QException

Is the table's primary key of a type that can be keyset-paged?

Throws:

QException

forEachPage

public void forEachPage(UnsafeConsumer<List<QRecord>,QException> consumer)
                 throws QException

Run the consumer on each page, in primary key order, until the records are exhausted. Pages are at most the page size; all but the last are full.

Throws:

QException

pageIterator

public Iterator<List<QRecord>> pageIterator()
                                     throws QException

Iterate over pages (lazily - each is queried as the iterator is advanced). The last key is updated when the iterator moves on from a page, to the next page or to the end. A QException while querying is thrown as a QRuntimeException.

Throws:

QException

recordIterator

public Iterator<QRecord> recordIterator()
                                 throws QException

Iterate over individual records, fetching a page at a time. See pageIterator().

Throws:

QException