Class ChunkedProcessPlan

java.lang.Object
com.kingsrook.qqq.backend.core.jobs.plans.ChunkedProcessPlan
All Implemented Interfaces:
JobPlan
public class ChunkedProcessPlan extends Object implements JobPlan
The CHUNKED plan for streamed ETL processes (spec section 2.4, plan 1): bulk insert, edit and delete, record actions and data processes. Planning (as the launching user): - TABLE_FILTER / RECORD_IDS: primary-key ranges over the frozen filter, one page of chunkSize keys each (KeysetPager). Rows added later are not picked up, as today. - STORAGE_FILE: the upload is streamed ONCE into chunk objects in the jobs storage table - JSON Lines of rows with their ORIGINAL row numbers (plus the header row) - so each chunk reads only its own object and a restart never rescans the file. No temp file. - ROW_SET (retry failed rows): the failed rows only, from the parent job's chunk objects (files) or by key (tables). Running a chunk runs the process's OWN steps (headless, frontend steps skipped) restricted to the chunk: queryFilterJson AND the key range, or the chunk object as the bulk-load file. Failed rows are reported by the transform and load steps through JobChunkScope. With SAME_TRANSACTION the chunk's writes, its row errors and its DONE mark commit in one transaction.