Class AbstractLoadStep

java.lang.Object
com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractLoadStep
Direct Known Subclasses:
LoadViaDeleteStep, LoadViaInsertOrUpdateStep, LoadViaInsertStep, LoadViaUpdateStep, NoopLoadStep, PauseQuartzJobsProcess, ResumeQuartzJobsProcess, RunRecordScriptLoadStep
public abstract class AbstractLoadStep extends Object
Base class for the Load (aka, store) logic of Streamed ETL processes. Records are to be read out of the input object's Records field, and after storing, should be written to the output object's Records, noting that when running as a streamed-ETL process, those input invalid input: '&' output objects will be instances of the StreamedBackendStep{Input,Output} classes, that will be associated with a page of records flowing through a pipe. Also - use the transaction member variable!!!

  • Field Details

  • Constructor Details

    • AbstractLoadStep

      public AbstractLoadStep()

  • Method Details

    • runOnePage

      public abstract void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
      Do the load logic for one page of records
      Throws:
      QException

    • preRun

      public void preRun(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
      Allow subclasses to do an action before any pages of records are processed.
      Throws:
      QException

    • postRun

      public void postRun(BackendStepPostRunInput runBackendStepInput, BackendStepPostRunOutput runBackendStepOutput) throws QException
      Allow subclasses to do an action after the run is complete - after the last page of records is passed in.
      Throws:
      QException

    • openTransaction

      public Optional<QBackendTransaction> openTransaction(RunBackendStepInput runBackendStepInput) throws QException
      Throws:
      QException

    • setTransaction

      public void setTransaction(Optional<QBackendTransaction> transaction)
      Setter for transaction

    • getTransaction

      public Optional<QBackendTransaction> getTransaction()
      Getter for transaction

    • getOverrideRecordPipeCapacity

      public Integer getOverrideRecordPipeCapacity(RunBackendStepInput runBackendStepInput)
      Allow this load step to specify the capacity of the pipe being used by the process. The specific use-case for which this is being added is, in the case of a process with many records being extracted, if the load job is too slow, then the pipe can get filled, and the extractor (who puts records into the pipe) can time out waiting for capacity in the pipe to open up, while a slow loader is consuming the records. In other words, for a slow loader, setting a lower pipe capacity can help prevent time-out errors ("Giving up adding record to pipe...")

    • getTransformStep

      public AbstractTransformStep getTransformStep()
      Getter for transformStep

    • setTransformStep

      public void setTransformStep(AbstractTransformStep transformStep)
      Setter for transformStep

    • withTransformStep

      public AbstractLoadStep withTransformStep(AbstractTransformStep transformStep)
      Fluent setter for transformStep