Class AbstractTransformStep

java.lang.Object
com.kingsrook.qqq.backend.core.processes.implementations.etl.streamedwithfrontend.AbstractTransformStep
All Implemented Interfaces:
ProcessSummaryProviderInterface
Direct Known Subclasses:
AbstractMergeDuplicatesTransformStep, AbstractTableSyncTransformStep, BulkDeleteTransformStep, BulkEditTransformStep, BulkInsertTransformStep, GarbageCollectorTransformStep, NoopTransformStep
public abstract class AbstractTransformStep extends Object implements ProcessSummaryProviderInterface
Base class for the Transform logic of Streamed ETL processes. Records are to be read out of the input object's Records field, and after storing, should be written to the output object's Records, noting that when running as a streamed-ETL process, those input invalid input: '&' output objects will be instances of the StreamedBackendStep{Input,Output} classes, that will be associated with a page of records flowing through a pipe.

  • Constructor Details

    • AbstractTransformStep

      public AbstractTransformStep()

  • Method Details

    • runOnePage

      public abstract void runOnePage(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
      Do the transform logic for one page of records
      Throws:
      QException

    • preRun

      public void preRun(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
      Allow subclasses to do an action before any pages of records are processed.
      Throws:
      QException

    • postRun

      public void postRun(BackendStepPostRunInput runBackendStepInput, BackendStepPostRunOutput runBackendStepOutput) throws QException
      Allow subclasses to do an action after the run is complete - after the last page of records is passed in.
      Throws:
      QException

    • setTransaction

      public void setTransaction(Optional<QBackendTransaction> transaction)
      Setter for transaction

    • getTransaction

      public Optional<QBackendTransaction> getTransaction()
      Getter for transaction

    • getOverrideRecordPipeCapacity

      public Integer getOverrideRecordPipeCapacity(RunBackendStepInput runBackendStepInput)
      Allow this transform step to specify the capacity of the pipe being used by the process. The specific use-case for which this is being added is, in the case of a process with many records being extracted, if the transform job is too slow, then the pipe can get filled, and the extractor (who puts records into the pipe) can time out waiting for capacity in the pipe to open up, while a slow loader is consuming the records. In other words, for a slow transformer, setting a lower pipe capacity can help prevent time-out errors ("Giving up adding record to pipe...")