Introduction

QQQ is a metadata-driven application framework for Java engineers building business software. An application describes its tables, fields, relationships, processes, authentication and permissions in a QInstance. Backend modules perform data operations; middleware exposes the application through HTTP, CLI or other supported entry points; a separately versioned dashboard supplies the standard administration UI.

QQQ 4.0 status and prerequisites

QQQ 4.0.0 requires Java 21 and Maven 3.8 or later. The repository README records current availability. Existing applications should follow the 4.0 migration guide, which covers package renames and removed APIs.

Describe the application

Register backend connection metadata, then tables and fields that use those backends. Add relationships, possible-value sources, processes, reports and widgets as needed. Metadata can be produced by Java classes and annotations; the repository also contains metadata loaders for supported file-based definitions. Application code supplies business rules and integrations that the built-in actions do not provide.

QQQ includes relational database modules, MongoDB, filesystem storage and API backend support. Backend capabilities differ: check the selected module before relying on joins, transactions, aggregation or other optional behavior. Configure middleware and the dashboard to expose the metadata as a usable application; defining a table alone does not launch a server.

Configure access explicitly

Choose and configure an authentication provider, such as OAuth2 or Auth0, including its required credentials and endpoints. Register it for the intended authentication scope, configure permission rules and any record-level security keys, and test permitted and denied access. Selecting a provider does not automatically define application authorization. Mock and fully anonymous authentication are useful for deliberately limited examples; the sample’s explicit mock option is intended for local sample data.

Processes and extension points

QQQ provides CRUD actions, reporting, scheduled jobs, record automations and process steps. Applications implement backend steps, customizers and widget renderers where their business behavior is needed. Streamed ETL processes separate extraction, transformation/validation and writes; preview and validation must not perform the final data changes. Production QBits, including the separate ESB QBit, have their own delivery and compatibility requirements and are outside this core release train.

Start with the sample

Follow the sample application’s README to build and run its H2-backed tables, greeting process, dashboard and CLI. The framework build needs Docker for its integration tests. Use the metadata and action reference sections below when replacing sample behavior with your own application.

Meta Data Production

The first thing that an application built using QQQ needs to do is to define its meta data. This basically means the construction of a QInstance object, which is populated with the meta data objects defining the backend(s), tables, processes, possible-value sources, joins, authentication provider, etc, that make your application.

There are various styles that can be used for how you define your meta data, and for the most part they can be mixed and matched. They will be presented here based on the historical evolution of how they were added to QQQ, where we generally believe that better techniques have been added over time. So, you may wish to skip the earlier techniques, and jump straight to the end of this section. However, it can always be instructive to learn about the past, so, read at your own pace.

Examples below are application fragments: imports and explicitly omitted application classes must be supplied. AuthScope is com.kingsrook.qqq.backend.core.model.metadata.authentication.AuthScope. Fully anonymous authentication illustrates metadata registration; configure an appropriate authentication provider and permissions before deploying an application.

Omni-Provider

At the most basic level, the way to populate a QInstance is the simple and direct approach of creating one big class, possibly with just one big method, and just doing all the work directly inline there.

This may (clearly) violate several good engineering principles. However, it does have the benefit of being simple - if all of your meta-data is defined in one place, it can be pretty simple to find where that place is. So - especially in a small project, this technique may be worth continuing to consider.

Re: "doing all the work" as mentioned above - what work are we talking about? At a minimum, we need to construct the following meta-data objects, and pass them into our QInstance:

  • QAuthenticationMetaData - how (or if!) users will be authenticated to the application.

  • QBackendMetaData - a backend data store.

  • QTableMetaData - a table (within the backend).

  • QAppMetaData - an organizational unit to present the other elements in a UI.

Here’s what a single-method omni-provider could look like:

About the simplest possible single-file meta-data provider
public QInstance defineQInstance() throws QException
{
    QInstance qInstance = new QInstance();

    qInstance.registerAuthenticationProvider(AuthScope.instanceDefault(), new QAuthenticationMetaData()
       .withName("anonymous")
       .withType(QAuthenticationType.FULLY_ANONYMOUS));

    qInstance.addBackend(new QBackendMetaData()
       .withBackendType(MemoryBackendModule.class)
       .withName("memoryBackend"));

    qInstance.addTable(new QTableMetaData()
        .withName("myTable")
        .withPrimaryKeyField("id")
        .withBackendName("memoryBackend")
        .withField(new QFieldMetaData("id", QFieldType.INTEGER)));

    qInstance.addApp(new QAppMetaData()
       .withName("myApp")
       .withSectionOfChildren(new QAppSection().withName("mySection"),
          qInstance.getTable("myTable")));

   return (qInstance);
}

Multi-method Omni-Provider

The next evolution of meta-data production comes by just applying some basic better-engineering principles, and splitting up from a single method that constructs all the things, to at least using unique methods to construct each thing, then calling those methods to add their results to the QInstance.

Multi-method omni- meta-data provider
public QInstance defineQInstance() throws QException
{
    QInstance qInstance = new QInstance();
    qInstance.registerAuthenticationProvider(AuthScope.instanceDefault(), defineAuthenticationMetaData());
    qInstance.addBackend(defineBackendMetaData());
    qInstance.addTable(defineMyTableMetaData());
    qInstance.addApp(defineMyAppMetaData(qInstance));
    return qInstance;
}

public QAuthenticationMetaData defineAuthenticationMetaData()
{
    return new QAuthenticationMetaData()
       .withName("anonymous")
       .withType(QAuthenticationType.FULLY_ANONYMOUS);
}

public QBackendMetaData defineBackendMetaData()
{
    return new QBackendMetaData()
       .withBackendType(MemoryBackendModule.class)
       .withName("memoryBackend");
}

// implementations of defineMyTableMetaData() and defineMyAppMetaData(qInstance)
// left as an exercise for the reader

Multi-class Providers

Then the next logical evolution would be to put each of these single meta-data producing objects into its own class, along with calls to those classes. This gets us away from the "5000 line" single-class, and lets us stop using the word "omni":

Multi-class meta-data providers
public QInstance defineQInstance() throws QException
{
    QInstance qInstance = new QInstance();
    qInstance.registerAuthenticationProvider(AuthScope.instanceDefault(), new AuthMetaDataProvider().defineAuthenticationMetaData());
    qInstance.addBackend(new BackendMetaDataProvider().defineBackendMetaData());
    qInstance.addTable(new MyTableMetaDataProvider().defineTableMetaData());
    qInstance.addApp(new MyAppMetaDataProvider().defineAppMetaData(qInstance));
    return qInstance;
}

public class AuthMetaDataProvider
{
    public QAuthenticationMetaData defineAuthenticationMetaData()
    {
        return new QAuthenticationMetaData()
           .withName("anonymous")
           .withType(QAuthenticationType.FULLY_ANONYMOUS);
    }
}

public class BackendMetaDataProvider
{
    public QBackendMetaData defineBackendMetaData()
    {
        return new QBackendMetaData()
           .withBackendType(MemoryBackendModule.class)
           .withName("memoryBackend");
    }
}

// implementations of MyTableMetaDataProvider and MyAppMetaDataProvider
// left as an exercise for the reader

MetaDataProducerInterface

As the size of your application grows, if you’re doing per-object meta-data providers, you may find it burdensome, when adding a new object to your instance, to have to write code for it in two places - that is - a new class to produce that meta-data object, AND a single line of code to add that object to your QInstance. As such, a mechanism exists to let you avoid that line-of-code for adding the object to the QInstance.

This mechanism involves adding the MetaDataProducerInterface to all of your classes that produce a meta-data object. This interface is generic, with a type parameter that will typically be the type of meta-data object you are producing, such as QTableMetaData, QProcessMetaData, or QWidgetMetaData, (technically, any class which implements TopLevelMetaDataInterface). Implementers of the interface are then required to override just one method: T produce(QInstance qInstance) throws QException;

Once you have your MetaDataProducerInterface classes defined, then there’s a one-time call needed to add all of the objects produced by these classes to your QInstance - as shown here:

Using MetaDataProducerInterface
public QInstance defineQInstance() throws QException
{
    QInstance qInstance = new QInstance();
    MetaDataProducerHelper.processAllMetaDataProducersInPackage(qInstance,
      "com.mydomain.myapplication");
    return qInstance;
}

public class AuthMetaDataProducer implements MetaDataProducerInterface<QAuthenticationMetaData>
{
    @Override
    public QAuthenticationMetaData produce(QInstance qInstance)
    {
        return new QAuthenticationMetaData()
           .withName("anonymous")
           .withType(QAuthenticationType.FULLY_ANONYMOUS);
    }
}

public class BackendMetaDataProducer implements MetaDataProducerInterface<QBackendMetaData>
{
    @Override
    public QBackendMetaData produce(QInstance qInstance)
    {
        return new QBackendMetaData()
           .withBackendType(MemoryBackendModule.class)
           .withName("memoryBackend");
    }
}

// implementations of MyTableMetaDataProvider and MyAppMetaDataProvider
// left as an exercise for the reader

Failing fast on producer errors

By default, MetaDataProducerHelper logs a warning and skips any class it finds but can’t use as a producer, and any producer whose produce method throws. The application then starts without that meta-data. Classes it can’t use include:

  • a MetaDataProducerInterface without a public no-arg constructor, or whose constructor throws;

  • a @QMetaDataProducingEntity class that is not a QRecordEntity, or whose TABLE_NAME is not a String;

  • a @QMetaDataProducingPossibleValueEnum enum that is not a PossibleValueEnum;

  • a @ChildTable that asks for a @ChildRecordListWidget without its @ChildJoin, or whose child entity has no field with a possible value source naming the parent table.

To make those errors stop startup instead, turn on fail-fast before processing producers, either on the QInstance, or with the system property qqq.metaData.failOnProducerError=true. With either one on, the helper throws a QException naming the failing class, with the original error as its cause.

Failing fast on meta-data producer errors
QInstance qInstance = new QInstance()
   .withFailOnMetaDataProducerError(true);
MetaDataProducerHelper.processAllMetaDataProducersInPackage(qInstance,
   "com.mydomain.myapplication");

An AbstractMetaDataProducerBasedQQQApplication (such as MetaDataProducerBasedQQQApplication) creates its own QInstance, so set the flag on the application with withFailOnMetaDataProducerError(true), or on the QApplicationLauncherConfig that launches it (see QQQ Middleware: Javalin web server).

MetaDataProducerMultiOutput

It is worth mentioning, that sometimes it might feel like a bridge that’s a bit too far, to make every single one of your meta-data objects require its own class. Some may argue that it’s best to do it that way - single responsibility principle, etc. But, if you’re producing, say, 5 widgets that are all related, and it’s only a handful of lines of code for each one, maybe you’d rather produce them all in the same class. Or maybe when you define a table, you’d like to define its joins and widgets at the same time.

This approach can be accomplished by making the type argument for your MetaDataProducerInterface be MetaDataProducerMultiOutput - a simple class that just wraps a list of other MetaDataProducerOutput objects.

Returning a MetaDataProducerMultiOutput
public class MyMultiProducer implements MetaDataProducerInterface<MetaDataProducerMultiOutput>
{
    @Override
    public MetaDataProducerMultiOutput produce(QInstance qInstance)
    {
       MetaDataProducerMultiOutput output = new MetaDataProducerMultiOutput();

       output.add(new QBackendMetaData()
          .withName("memoryBackend")
          .withBackendType(MemoryBackendModule.class));
       output.add(new QTableMetaData()
          .withName("myTable")
          .withBackendName("memoryBackend")
          .withPrimaryKeyField("id")
          .withField(new QFieldMetaData("id", QFieldType.INTEGER)));

       return (output);
    }
}

Aside: TableMetaData with RecordEntities

At this point, let’s take a brief aside to dig deeper into the creation of a QTableMetaData object. Tables, being probably the most important meta-data type in QQQ, have a lot of information that can be specified in their meta-data object.

At the same time, if you’re writing any custom code in your QQQ application (e.g., any processes or table customizers), where you’re working with records from tables, you may prefer being able to work with entity beans (e.g., java classes with typed getter & setter methods), rather than the default object type that QQQ’s ORM actions return, the QRecord, which carries all of its values in a Map (where you don’t get compile-time checks of field names or data types). QQQ has a mechanism for dealing with this - in the form of the QRecordEntity class.

So - if you want to build your application using entity beans (which is recommended, for the compile-time safety that they provide in custom code), you will be writing a QRecordEntity class for each of your tables, which will look like:

QRecordEntity example
public class MyTable extends QRecordEntity
{
   public static final String TABLE_NAME = "myTable";

   @QField(isEditable = false, isPrimaryKey = true)
   private Integer id;

   @QField()
   private String name;

   // no-arg constructor and constructor that takes a QRecord
   // getters & setters (and optional fluent setters)
}

The point of introducing this topic here and now is, that a QRecordEntity can be used to shortcut to defining some of the attributes in a QTableMetaData object. Specifically, in a MetaDataProducer<QTableMetaData> you may say:

QTableMetaDataProducer using a QRecordEntity
public QTableMetaData produce(QInstance qInstance) throws QException
{
  return new QTableMetaData()
     .withName(MyTable.TABLE_NAME)
     .withFieldsFromEntity(MyTable.class)
     .withBackendName("memoryBackend");
}

That withFieldsFromEntity call is one of the biggest benefits of this technique. It allows you to avoid defining all of the fields in you table in two places (the entity and the table meta-data).

MetaData Producing Annotations for Entities

If you are using QRecordEntity classes that correspond to your tables, then you can take advantage of some additional annotations on those classes, to produce more related meta-data objects associated with those tables. The point of this is to eliminate boilerplate, and simplify / speed up the process of getting a new table built and deployed in your application.

Furthermore, the case can be made that it is beneficial to keep the meta-data definition for a table as close as possible to the entity that corresponds to the table. This enables modifications to the table (e.g., adding a new field/column) to only require edits in one java source file, rather than necessarily requiring edits in two files.

@QMetaDataProducingEntity

This is an annotation meant to be placed on a QRecordEntity subclass, which you would like to be processed by an invocation of MetaDataProducerHelper, to automatically produce some meta-data objects.

This annotation supports:

  • Creating table meta-data for the corresponding record entity table. Enabled by setting produceTableMetaData=true.

    • One may customize the table meta data that is produced automatically by supplying a class that extends MetaDataCustomizerInterface in the annotation attribute tableMetaDataCustomizer.

    • In addition to (or as an alternative to) the per-table MetaDataCustomizerInterface that can be specified in @QMetaDataProducingEntity.tableMetaDataCustomizer, when an application calls MetaDataProducerHelper.processAllMetaDataProducersInPackage, an additional MetaDataCustomizerInterface can be given, to apply a common set of adjustments to all tales being generated by the call.

  • Making a possible-value-source out of the table. Enabled by setting producePossibleValueSource=true.

  • Processing child tables to create joins and childRecordList widgets

@ChildTable

This is an annotation used as a value that goes inside a @QMetaDataProducingEntity annotation, to define child-tables, e.g., for producing joins and childRecordList widgets related to the table defined in the entity class.

@ChildJoin

This is an annotation used as a value inside a @ChildTable inside a @QMetaDataProducingEntity annotation, to control the generation of a QJoinMetaData, as a ONE_TO_MANY type join from the table represented by the annotated entity, to the table referenced in the @ChildTable annotation.

@ChildRecordListWidget

This is an annotation used as a value that goes inside a @QMetaDataProducingEntity annotation, to control the generation of a QWidgetMetaData - for a ChildRecordList widget.

QRecordEntity with meta-data producing annotations and a table MetaDataCustomizer
@QMetaDataProducingEntity(
   produceTableMetaData = true,
   tableMetaDataCustomizer = MyTable.TableMetaDataCustomizer.class,
   producePossibleValueSource = true,
   childTables = {
      @ChildTable(
         childTableEntityClass = MyChildTable.class,
         childJoin = @ChildJoin(enabled = true),
         childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Children"))
   }
)
public class MyTable extends QRecordEntity
{
   public static final String TABLE_NAME = "myTable";

   public static class TableMetaDataCustomizer implements MetaDataCustomizerInterface<QTableMetaData>
   {
      @Override
      public QTableMetaData customizeMetaData(QInstance qInstance, QTableMetaData table) throws QException
      {
         String childJoinName = QJoinMetaData.makeInferredJoinName(TABLE_NAME, MyChildTable.TABLE_NAME);

         table
            .withUniqueKey(new UniqueKey("name"))
            .withIcon(new QIcon().withName("table_bar"))
            .withRecordLabelFormat("%s")
            .withRecordLabelFields("name")

            .withSection(new QFieldSection("identity", new QIcon().withName("badge"), Tier.T1,
               List.of("id", "name")))
            // add sections for the remaining application fields
            .withSection(new QFieldSection("children", new QIcon().withName("account_tree"), Tier.T2)
               .withWidgetName(childJoinName))

            .withExposedJoin(new ExposedJoin()
               .withLabel("Children")
               .withJoinPath(List.of(childJoinName))
               .withJoinTable(MyChildTable.TABLE_NAME));

         return (table);
      }
   }

   @QField(isEditable = false, isPrimaryKey = true)
   private Integer id;

   // remaining fields, constructors, getters & setters left as an exercise for the reader and/or the IDE
}

The class given in the example above, if processed by the MetaDataProducerHelper, would add the following meta-data objects to your QInstance:

  • A QTableMetaData named myTable, with all fields annotated as @QField from the QRecordEntity class, and with additional attributes as set in the TableMetaDataCustomizer inner class.

  • A QPossibleValueSource named myTable, of type TABLE, with myTable as its backing table.

  • A QJoinMetaData named myTableJoinMyChildTable, as a ONE_TO_MANY type, between those two tables.

  • A QWidgetMetaData named myTableJoinMyChildTable, as a CHILD_RECORD_LIST type, that will show a list of records from myChildTable as a widget, when viewing a record from myTable.

Other MetaData Producing Annotations

Similar to these annotations for a RecordEntity, a similar one exists for a PossibleValueEnum class, to automatically write the meta-data to use that enum as a possible value source in your application:

@QMetaDataProducingPossibleValueEnum

This is an annotation to go on a PossibleValueEnum class, which you would like to be processed by MetaDataProducerHelper, to automatically produce a PossibleValueSource meta-data based on the enum.

PossibleValueEnum with meta-data producing annotation
@QMetaDataProducingPossibleValueEnum(producePossibleValueSource = true)
public enum MyOptionsEnum implements PossibleValueEnum<Integer>
{
   ENABLED, DISABLED;

   @Override
   public Integer getPossibleValueId()
   {
      return ordinal() + 1;
   }

   @Override
   public String getPossibleValueLabel()
   {
      return name();
   }
}

The enum given in the example above, if processed by the MetaDataProducerHelper, would add the following meta-data object to your QInstance:

  • A QPossibleValueSource named MyOptionsEnum, of type ENUM, with MyOptionsEnum as its backing enum.

Meta Data Types

QInstance

An application in QQQ is defined as a set of Meta Data objects. These objects are all stored together in an object called a QInstance.

A QInstance can be constructed in Java, populated by metadata producers, or loaded from configuration files. In this Java fragment, the application supplies the three metadata-building methods:

Adding meta-data for two tables and one process to a QInstance
QInstance qInstance = new QInstance();
qInstance.addTable(definePersonTable());
qInstance.addTable(defineHomeTable());
qInstance.addProcess(defineSendPersonHomeProcess());

ConfigFilesBasedQQQApplication supports metadata files such as YAML and JSON. MetaDataProducerBasedQQQApplication loads Java metadata producers. See [MetaDataProduction] for the Java patterns.

Directory loading is recursive and ordered by filename. Every file must contain valid top-level YAML/JSON metadata: missing directories, unsupported files, unknown properties and mapping failures throw QException rather than leaving a successful partial load. For a packaged resource, MetaDataLoaderHelper.readMetaDataFile(instance, inputStream, fileName) reads one object with the same validation; the caller closes the stream and adds the returned object to the instance. The sample’s Person table uses this path, so its YAML definition works outside a source checkout.

The middleware and/or frontends being used in an application will drive how the QInstance is connected to the running server/process. For HTTP applications, pass a configured AbstractQQQApplication to QApplicationJavalinServer. Here, application supplies the instance; the caller handles QException. See the Javalin server guide for a complete entry point.

Starting a QQQ Javalin application server
QApplicationJavalinServer server = new QApplicationJavalinServer(application);
server.start();

QInstance Setup:

These are the methods that one is most likely to use when setting up (defining) a QInstance object:

  • add(TopLevelMetaDataInterface metaData) - Generic method that takes most of the meta-data subtypes that can be added to an instance, such as QBackendMetaData, QTableMetaData, QProcessMetaData, etc. There are also type-specific methods (e.g., addTable, addProcess, etc), which one can call instead - this would just be a matter of personal preference.

  • withRecordSearchProvider(QCodeReference codeReference) - Optional RecordSearchProviderInterface implementation that record search (the v1 POST /search endpoint) uses for the tables it claims, such as a search-index module. See searchFields in Tables for details.

  • withHomeAppName(String homeAppName) - Optional name of the app whose dashboard a frontend shows as the instance’s home page (Next renders it at /app/). Validation requires it to name an app in the instance. The metadata response includes homeAppName only for users who may access that app.

QInstance Usage:

Generally you will set up a QInstance in your application’s startup flow, and then place it in the server (e.g., javalin). But, if, during application-code runtime, you need access to any of the meta-data in the instance, you access it via the QContext object’s static getQInstance() method. This can be useful, for example, to get a list of the defined tables in the application, or fields in a table, or details about a field, etc.

It is generally considered risky and/or not a good idea at all to modify the QInstance after it has been validated and a server is running. Future versions of QQQ may in fact restrict modifications to the instance after validation.

Backends

A key component of QQQ is its ability to connect to various backend data stores, while providing the same interfaces to those backends - both User Interfaces, and Programming Interfaces.

For example, out-of-the-box, QQQ can connect to:

  • RDBMS (Relational Database Management Systems, such as MySQL)

  • File Systems (Amazon S3 or local disk)

  • JSON Web APIs (using a custom mapping class per-API backend).

  • MongoDB document storage, via MongoDBBackendMetaData

  • PostgreSQL and SQLite through their dedicated RDBMS backend modules

  • In-Memory data stores

All QQQ Tables in a QQQ instance must belong to a backend. As such, any instance using tables (which would be almost all instances) must define 1 or more backends.

QBackendMetaData

Backends are defined in a QQQ Instance in a QBackendMetaData object. These objects will have some common attributes, but many different attributes based on the type of backend being used.

QBackendMetaData Properties:

  • name - String, Required - Unique name for the backend within the QQQ Instance.

  • backendType - String, Required - Identifier for the backend type being defined.

    • This attribute is typically set in the constructor of a QBackendMetaData subclass, and as such does not need to be set when defining a backend meta data object.

  • enabledCapabilities and disabledCapabilities - Sets, containing Capability enum values. Basic rules that apply to all tables in the backend, describing what actions (such as Delete, or Count) are supported in the backend.

    • By default, QQQ assumes that a backend supports most capabilities, with one exception being QUERY_STATS.

    • When the backend disables a capability, a table-level enabled entry overrides that exclusion. Otherwise, a table-level disabled entry disables it. Avoid contradictory enabled and disabled entries. A flag cannot add an operation that the selected backend implementation does not implement.

    • Capability flags control the operations exposed in frontend metadata. The count and export HTTP routes refuse tables without TABLE_COUNT or TABLE_EXPORT; other direct table HTTP operations do not enforce these flags, so use permissionRules to deny access. Broader runtime capability enforcement is deferred in issue #546.

  • usesVariants - Boolean, default false - Control whether or not the backend supports the concept of "Variants".

    • Supporting variants means that tables within the backend can exist in alternative "variants" of the backend. For example, this might mean a sharded or multi-tenant database backend (perhaps a different server or database name per-client). Or this might mean using more than one set of credentials for connecting to an API backend - each of those credential sets would be a "variant".

    • Configure backendVariantsConfig: its variant type key, options table/filter or lookup function, and mapping from backend settings to fields on a variant record. See BackendVariantsConfig and validate the configuration with QInstanceValidator.

In a QQQ application, one will typically not create an instance of QBackendMetaData directly, but instead will create an instance of one of its subclasses, specific to the type of backend being used. The currently available list of such classes are:

RDBMSBackendMetaData

The meta data required for working with tables in an RDBMS (relational database) backend are defined in an instance of the RDBMSBackendMetaData class.

RDBMSBackendMetaData Properties:

  • vendor - String, Required - Database vendor. Currently supported values are: aurora, mysql, h2.

  • jdbcUrl - String, Optional - Full JDBC URL for connecting to the database.

    • If this property is provided, then following properties (which are the components of a JDBC URL) are ignored. In other words, you can either provide the jdbcUrl, or the individual components that make it up.

  • hostName - String, Conditionally Required - Hostname or ip address of the RDBMS server.

  • port - Integer, Conditionally Required - Port used to connect to the RDBMS server.

  • databaseName - String, Conditionally Required - Name of the database being connected to, within the RDBMS server.

  • username - String, Conditionally Required - Username for authenticating in the database server.

  • password - String, Conditionally Required - Password for authenticating in the database server.

Examples

/*******************************************************************************
** Full example of constructing an RDBMSBackendMetaData
*******************************************************************************/
public class ExampleDatabaseBackendMetaDataProducer extends MetaDataProducer<QBackendMetaData>
{
   public static final String NAME = "rdbmsBackend";

   /*******************************************************************************
    ** Produce the QBackendMetaData
    *******************************************************************************/
   @Override
   public QBackendMetaData produce(QInstance qInstance)
   {
      ///////////////////////////////////////////////////////////////////////
      // read settings from either a .env file or the system's environment //
      ///////////////////////////////////////////////////////////////////////
      QMetaDataVariableInterpreter interpreter = new QMetaDataVariableInterpreter();
      String vendor       = interpreter.interpret("${env.RDBMS_VENDOR}");
      String hostname     = interpreter.interpret("${env.RDBMS_HOSTNAME}");
      String port         = interpreter.interpret("${env.RDBMS_PORT}");
      String databaseName = interpreter.interpret("${env.RDBMS_DATABASE_NAME}");
      String username     = interpreter.interpret("${env.RDBMS_USERNAME}");
      String password     = interpreter.interpret("${env.RDBMS_PASSWORD}");

      return (new RDBMSBackendMetaData()
         .withName(NAME)
         .withVendor(vendor)
         .withHostName(hostname)
         .withPort(ValueUtils.getValueAsInteger(port))
         .withDatabaseName(databaseName)
         .withUsername(username)
         .withPassword(password)
         .withCapability(Capability.QUERY_STATS));
   }
}
Secret references

QMetaDataVariableInterpreter resolves these references (several may be chained with ??; the first that resolves wins):

  • ${env.NAME} - an environment variable (or a .env file entry).

  • ${prop.NAME} - a Java system property.

  • ${file./absolute/path} - the trimmed contents of a secret file mounted by the orchestrator, such as a Kubernetes secret volume (${file./var/run/secrets/storage/access-key}) or a Docker secret (${file./run/secrets/db-password}). The file is only read, never written, and is re-read on every interpretation, so a rotated secret is picked up when the value is next interpreted. The path must be absolute and inside an allowed root after symbolic links are resolved. The roots default to /var/run/secrets and /run/secrets; set the qqq.fileReference.roots system property or the QQQ_FILE_REFERENCE_ROOTS environment variable (comma-separated) to change them. Files over 1 MiB are refused, and file contents are never logged. An interpreter that has value maps (one that resolves user-supplied values such as ${input.X} in query filters or report formulas) does not read files unless setFileReferencesEnabled(true) is called on it. Prefer environment variables or a secret manager where they fit; file references exist for orchestrator-mounted secrets, not for application data.

  • ${literal.TEXT} - the literal TEXT.

S3BackendMetaData

The meta data required for working with tables in an Amazon S3 backend are defined in an instance of the S3BackendMetaData class.

S3BackendMetaData Properties:

  • bucketName - String, Required - Bucket name to connect to inside AWS S3.

  • accessKey - String - Access key for connecting to S3.

  • secretKey - String - Secret key for connecting to S3.

  • sessionToken - String - Session token of temporary (STS) credentials, used with accessKey and secretKey. A session token without both keys is an error. The token is fixed once set: when it expires, give the backend the new credentials (a new client is built and cached for them).

  • credentialsProviderCode - QCodeReference - A class implementing the AWS SDK’s AwsCredentialsProvider (with a no-arg constructor), for credentials that refresh themselves, such as an STS assume-role provider. When set, it takes precedence over the keys and the session token. With neither keys nor a provider, the AWS SDK’s default credentials chain applies (environment, profile, web identity, container and instance roles), which refreshes its own temporary credentials.

  • region - String, Required - AWS region containing the Bucket in S3.

  • basePath - String, Required - Base path to the files within the S3 bucket.

accessKey, secretKey, sessionToken, region, endpoint and caCertificatesPem accept the secret references described above, for example .withSessionToken("${file./var/run/secrets/storage/session-token}").

FilesystemBackendMetaData

The meta data required for working with tables in a (local) filesystem backend are defined in an instance of the FilesystemBackendMetaData class.

FilesystemBackendMetaData Properties:

  • basePath - String, Required - Base path to the backend’s files.

APIBackendMetaData

The meta data required for working with tables in a web API are defined in an instance of the APIBackendMetaData class.

QQQ provides a minimal, reasonable default implementation for working with web APIs, making assumptions such as using POST to insert records, and GET with a primary key in the URL to get a single record. However, in our experience, almost all APIs are implemented differently enough, that a layer of custom code is required. For example, query/search endpoints are almost always unique in how they take their search parameters, and how they wrap their output.

To deal with this, a QQQ API Backend can define a custom class in the actionUtil property of the backend meta-data, as a subclass of BaseAPIActionUtil, which ideally can override methods at the level where unique functionality is needed. For example, an application need not define the full method for executing a Query against an API backend (which would need to make the HTTP request (actually multiple, to deal with pagination)). Instead, one can just override the buildQueryStringForGet method, where the unique details of making the request are defined, and maybe the jsonObjectToRecord method, where records are mapped from the API’s response to a QRecord.

See BaseAPIActionUtil for the complete override contract and module tests for request/response examples.

APIBackendMetaData Properties:

  • baseUrl - String, Required - Base URL for connecting to the API.

  • contentType - String, Required - value of Content-type header included in all requests to the API.

  • actionUtil - QCodeReference, Required - Reference to a class that extends BaseAPIActionUtil, where custom code for working with this API backend is defined.

  • customValues - Map of String → Serializable - Application-defined additional name=value pairs available to the backend action utility.

  • authorizationType - Enum, Required - Specifies how authentication is provided for the API. The value here, dictates which other authentication-related properties are required. Possible values are:

    • API_KEY_HEADER - Uses the apiKey property in an HTTP header named API-Key. In the future, when needed, QQQ will add a property, tentatively named apiKeyHeaderName, to allow customization of this header name.

    • API_TOKEN - Uses the apiKey property in an Authorization header, as: "Token " + apiKey

    • BASIC_AUTH_API_KEY - Uses the apiKey property, Base64-encoded, in an Authorization header, as "Basic " + base64(apiKey)

    • BASIC_AUTH_USERNAME_PASSWORD - Combines the username and password properties, Base64-encoded, in an Authorization header, as "Basic " + base64(username + ":" + password)

    • OAUTH2 - Implements an OAuth2 client credentials token grant flow, using the properties clientId and clientSecret. By default, the id & secret are sent as query-string parameters to the API’s oauth/token endpoint. Alternatively, if the meta-data has a customValue of setCredentialsInHeader=true, then the id & secret are posted in an Authorization header (base-64 encoded, and concatenated with ":").

    • API_KEY_QUERY_PARAM - Uses the apiKey property as a query-string parameter, with its name taken from the apiKeyQueryParamName property.

    • CUSTOM - Has a no-op implementation at the QQQ layer. Assumes that an override of protected void handleCustomAuthorization(HttpRequestBase request) be implemented in the backend’s actionUtil class.

  • apiKey - String, Conditional - See authorizationType above for details.

  • clientId - String, Conditional - See authorizationType above for details.

  • clientSecret - String, Conditional - See authorizationType above for details.

  • username - String, Conditional - See authorizationType above for details.

  • password - String, Conditional - See authorizationType above for details.

  • apiKeyQueryParamName - String, Conditional - See authorizationType above for details.

Memory backend

The memory backend stores QRecords in the process. Its Query, Count and Aggregate paths use active personalized READ rules, including supported INNER and LEFT joins. Joined READ checks run before LEFT null extension. These operations do not imply SQL support for every operator, join type or value conversion.

Query honors selected join aliases, fieldNamesToInclude and shouldFetchHeavyFields; see QueryInput. Supported joined virtual fields use alias-local function inputs, and Boolean subfilters retain the same execution context. Virtual selection, filtering, sorting and Aggregate operands retain their function-specific behavior. Unselected joined fields are not returned. Count can use a private key for distinct counting without restoring it to public metadata. Aggregate can read heavy operands internally; its computed results do not receive Query result masking.

Memory Aggregate retains Java equality for groups/distinct values and double-based DECIMAL AVG. Decimal scale and array identity can distinguish values, and precision differs from native SQL. See AggregateAction for source-window, empty-result, type and provider boundaries.

The legacy MemoryRecordStore setting buildJoinCrossProductFromJoinContext=false is no longer an executable query mode: reads fail with a checked QException. The getter, setter, fluent setter and BUILD_JOIN_CROSS_PRODUCT_FROM_JOIN_CONTEXT_DEFAULT field remain available. Keep the default true and remove any configuration that selects false; true uses the join plan that includes security rules.

Apps

QQQ User Interfaces (e.g., Material Dashboard) generally organize their contents via Apps. Apps are a lightweight construct in QQQ - basically just containers for other objects.

Specifically, Apps can contain:

QAppMetaData

Apps are defined in a QQQ Instance in QAppMetaData objects. Apps must consist of either 1 or more QQQ Widgets, or 1 or more Sections, which are expected to contain 1 or more QQQ Tables, QQQ Processes, or QQQ Reports.

QAppMetaData Properties:

  • name - String, Required - Unique name for the app within the QQQ Instance.

  • label - String - User-facing label for the app, presented in User Interfaces. Inferred from name if not set.

  • permissionRules - QPermissionRules - Permissions to apply to the app. See Permission Rules for details.

  • children - List of QAppChildMetaData - Objects contained within the app. These can be QQQ Tables, QQQ Processes, QQQ Reports or other QQQ Apps.

    • See the example below for some common patterns for how these child-meta data objects are added to an App.

  • parentAppName - String - For an app which is a child of another app, the parent app’s name is referenced in this field.

    • Note that this is generally automatically set when the child is added to its parent, in the addChild method.

  • icon - QIcon - An icon to display in a UI for the app. See Icons.

  • widgets - List of String - A list of names of QQQ Widgets to include in the app.

  • sections - List of QAppSection - A list of QAppSection objects, to create organizational subdivisions within the app.

    • As shown in the example below, the method withSectionOfChildren can be used to fluently add a new QAppSection, along with its child objects, to both an app and a section all at once.

QAppSection

A QAppSection is an organizational subsection of a QQQ App.

  • name - String, Required - Unique name for the section within its app.

  • label - String - User-facing label for the section, presented in User Interfaces. Inferred from name if not set.

  • icon - QIcon - An icon to display in a UI for the section. See Icons.

  • tables - List of String - A list of names of QQQ Tables to include in the section.

  • processes - List of String - A list of names of QQQ Processes to include in the section.

  • reports - List of String - A list of names of QQQ Reports to include in the section.

Examples

/*******************************************************************************
** Full example of constructing a QAppMetaData object.
*******************************************************************************/
public class ExampleAppMetaDataProducer extends MetaDataProducer<QAppMetaData>
{

   /*******************************************************************************
    ** Produce the QAppMetaData
    *******************************************************************************/
   @Override
   public QAppMetaData produce(QInstance qInstance) throws QException
   {
      return (new QAppMetaData()
         .withName("sample")
         .withLabel("My Sample App")
         .withIcon(new QIcon().withName("thumb_up"))
         .withWidgets(List.of(
            UserWelcomeWidget.NAME,
            SystemHealthLineChartWidget.NAME))
         .withSectionOfChildren(new QAppSection().withName("peoplePlacesAndThings"),
            qInstance.getTable(People.TABLE_NAME),
            qInstance.getTable(Places.TABLE_NAME),
            qInstance.getTable(Things.TABLE_NAME),
            qInstance.getProcess(AssociatePeopleWithPlacesProcess.NAME))
         .withSectionOfChildren(new QAppSection().withName("math").withLabel("Mathematics"),
            qInstance.getProcess(ComputePiProcess.NAME),
            qInstance.getReport(PrimeNumbersReport.NAME),
            qInstance.getReport(PolygonReport.NAME)));
   }


   /*******************************************************************************
    ** Since this meta-data producer expects to find other meta-data objects in the
    ** QInstance, give it a sortOrder higher than the default (which we'll expect
    ** the other objects used).
    *******************************************************************************/
   @Override
   public int getSortOrder()
   {
      return (Integer.MAX_VALUE);
   }

}

Tables

One of the most common types of object in a QQQ Instance is the Table. In the most common use-case, a QQQ Table may be the in-app representation of a Database table. That is, it is a collection of records (or rows) of data, each of which has a set of fields (or columns).

QQQ also allows other types of data sources (QQQ Backends) to be used as tables, such as File systems, API’s, etc. All of these backend types present the same interfaces (both user-interfaces, and application programming interfaces), regardless of their backend type.

QTableMetaData

Tables are defined in a QQQ Instance in QTableMetaData objects. All tables must reference a QQQ Backend, a list of fields that define the shape of records in the table, and additional data to describe how to work with the table within its backend.

QTableMetaData Properties:

  • name - String, Required - Unique name for the table within the QQQ Instance.

  • label - String - User-facing label for the table, presented in User Interfaces. Inferred from name if not set.

  • backendName - String, Required - Name of a QQQ Backend in which this table’s data is managed.

  • fields - Map of String → QQQ Field, Required - The columns of data that make up all records in this table.

  • primaryKeyField - String, Conditional - Name of a QQQ Field that serves as the primary key (unique identifier) for records in this table.

    • Whether a primary key field is required or not depends on the backend type that the table belongs to.

  • uniqueKeys - List of UniqueKey - Definition of additional unique keys or constraints (from an RDBMS point of view) from the table. e.g., sets of columns which must have unique values for each record in the table. The properties of the UniqueKey object are:

    • fieldNames - List of String, Required - List of field names from this table.

    • label - String - Optional label to be shown to users with error messages (e.g., for violation of this unique key).

  • backendDetails - QTableBackendDetails or subclass - Additional data to configure the table within its QQQ Backend.

    • For example, for an RDBMS-type backend, the name of the table within the database.

    • vs. a FileSystem backend, this may be the sub-path where files for the table are stored.

    • Use the backend module’s table-details class; see the module README and metadata source for its storage-specific fields.

  • automationDetails - QTableAutomationDetails - Configuration of automated jobs that run against records in the table, e.g., upon insert or update.

  • customizers - Map of String → QCodeReference - References to custom code that are injected into standard table actions, that allow applications to customize certain parts of how the table works.

    • Allowed values for keys in this map come from the role property of the TableCustomizers enum.

    • Based on the key in this map, the QCodeReference used as the value must be of the appropriate java type, as specified in the expectedType property of the TableCustomizers enum value corresponding to the key.

    • Example:

// MyPreInsCustomizer is the application class extending AbstractPreInsertCustomizer.
table.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(MyPreInsCustomizer.class));

  • isHidden - Boolean, default false - Option to hide the table from all User Interfaces.

  • parentAppName - String - Name of a QQQ App that this table exists within.

    • This field generally does not need to be set on the table when it is defined, but rather, is set when the table gets placed within an app.

  • icon - QIcon - Icon associated with this table in certain user interfaces. See Icons.

  • recordLabelFormat - String - Java Format String, used with recordLabelFields to produce a label shown for records from the table.

  • recordLabelFields - List of String, Conditional - Used with recordLabelFormat to provide values for any format specifiers in the format string. These strings must be field names within the table.

    • Example of using recordLabelFormat and recordLabelFields:

// given these fields in the table:
new QFieldMetaData("name", QFieldType.STRING)
new QFieldMetaData("birthDate", QFieldType.DATE)

// We can produce a record label such as "Darin Kelkhoff (1980-05-31)" via:
.withRecordLabelFormat("%s (%s)")
.withRecordLabelFields(List.of("name", "birthDate"))

  • searchFields - List of String - Fields that record search matches a search term against (RecordSearchAction, and the v1 POST /search endpoint that the dashboard’s global search uses). String fields (STRING, TEXT, HTML) match when they contain the term, ignoring case; integer fields (INTEGER, LONG) match when they equal a numeric term. A table without search fields is not searched. Search fields must be visible, non-password fields of the table, and the table’s backend must not use variants. Record search only searches tables the user may read, applies record security locks, and returns a limited number of records per table. The table meta data lists searchFields only to users who may read the table, so a frontend searches only when some table declares them.

.withSearchFields("firstName", "lastName", "email")

+ A record search provider can search some tables in place of the default "contains" query - for example, a module backed by a search index such as OpenSearch. Register one on the instance with QInstance.withRecordSearchProvider(new QCodeReference(YourProvider.class)), where YourProvider implements RecordSearchProviderInterface:

+

  • boolean claimsTable(QTableMetaData table) - whether the provider searches this table. It is asked only about tables that are being searched (tables with search fields that the user may read and that support query), so a provider-searched table still needs searchFields (frontends use them to find searchable tables, and they are the fallback).

  • RecordSearchOutput search(RecordSearchInput input) - search the claimed tables. The input has the trimmed search term, exactly the claimed tables the user may read as tableNames, the effective limitPerTable, and the caller’s input source. Return a RecordSearchResult with at least tableName and recordId (primary key) for each hit, most relevant first within each table.

+ Unclaimed tables keep the default search, and both sets of results are returned together, in table order, so the POST /search response does not change. Provider hits are read again by primary key through QueryAction, so record security locks apply, records deleted since they were indexed drop out, and record labels are computed as usual. For each table, up to RecordSearchAction.MAX_PROVIDER_IDS_PER_TABLE (100) ids are considered in the provider’s order, and the first limitPerTable the user may read are returned; a provider may return more than limitPerTable ids per table to make up for hits that drop out. If the provider throws (or cannot be loaded), a warning is logged and its tables are searched with the default search, so an unavailable search index does not fail search. A new provider instance is loaded for each search; keep connections and other shared state outside it. Without a provider, record search is unchanged.

  • sections - List of QFieldSection - Mechanism to organize fields within user interfaces, into logical sections. If any sections are present in the table meta data, then all fields in the table must be listed in exactly 1 section. If no sections are defined, then instance enrichment will define default sections.

  • associatedScripts - List of AssociatedScript - Definition of user-defined scripts that can be associated with records within the table.

  • enabledCapabilities and disabledCapabilities - Set of Capability enum values - Overrides from the backend level, for capabilities that this table does or does not possess.

  • associations - List of Association - tables whose records can be managed along with records from this table. See below for details.

  • recordSecurityLocks - List of RecordSecurityLock - locks that apply to records in the table - e.g., to control what users can or cannot access records in the table. See RecordSecurityLock below for details.

  • permissionRules - QPermissionRules object - define the permission/access rules for the table. See Permission Rules for details.

  • auditRules - QAuditRules object - define the audit rules for the table. See QAuditRules below for details.

  • cacheOf - CacheOf object - specify that this table serves as a "cache of" another table. See CacheOf object below for details.

  • exposedJoins - List of ExposedJoin objects - optional list of joined tables that are to be exposed in User Interfaces. See ExposedJoin object below for details.

The supplementalMetaData map attaches module-specific table settings. For application APIs, see ApiTableMetaData.

Unique-key validation

QQQ checks declared unique keys before INSERT and key-changing UPDATE. Sparse updates retain omitted key components for validation; an explicit null remains null, and any null component is distinct. Unchanged self keys are allowed, but another stored owner is a conflict, including owners hidden by record security. Validation does not grant READ access or return stored key values. Within a request, accepted typed key tuples remain reserved for later records. If repeated updates target the same primary key and any fragment changes a key, every occurrence is rejected; combine those fragments into one record. Non-key-only updates retain their existing behavior.

Native backend lookup must provide complete stored-key visibility. Custom backend query implementations provide lookupUniqueKey(UniqueKeyLookup.Input); API providers implement the corresponding method on their configured BaseAPIActionUtil. Unsupported lookups fail before writing. These checks compare typed values before backend conversion; matching storage constraints are still needed for final stored-value uniqueness under concurrent writes, numeric rounding and database collations. INSERT customizers after the unique-key phase remain trusted and must preserve the constraint. The existing trusted INSERT skipUniqueKeyCheck option retains its behavior.

QFieldSection

When users view records from a QQQ Table in a UI, fields are organized on the screen based on the QFieldSection objects in the table’s meta-data.

QFieldSection Properties:

  • name - String, Required - unique identifier for the section within its table.

  • label - String - User-facing label for the section, presented in User Interfaces. Inferred from name if not set.

  • tier - enum - importance of the fields in section for the table. Different tiers may be presented differently in UI’s. Only a single T1 section is allowed per-table. Possible values are: T1, T2, and T3.

  • icon - QIcon - Icon associated with this section in certain user interfaces. See Icons.

  • isHidden - Boolean, default false - Option to hide the table from all User Interfaces.

  • gridColumns - Integer - Option to specify how many columns in a grid layout the section should use. For the Material-Dashboard frontend, this is a grid of 12.

  • fieldNames - List of String, Conditional - List of names of QQQ Fields from this table to be included in this section.

  • widgetName - String, Conditional - Name of a QQQ Widget to be displayed in this section.

    • Note that exactly one of fieldNames or widgetName must be used.

QTableAutomationDetails

Records in QQQ can have application-defined custom actions automatically asynchronously executed against them after they are inserted or updated. The configuration to enable this functionality is assigned to a table in a QTableAutomationDetails object.

QTableAutomationDetails Properties:

  • statusTracking - AutomationStatusTracking object, Required - define how QQQ should keep track, per record, its status (e.g., pending-insert-automations, running-update-automations, etc). Properties of AutomationStatusTracking object are:

    • type - enum, Required - what type of tracking is used for the table. Possible values are:

      • FIELD_IN_TABLE - specifies that the table has a field which stores an AutomationStatus id.

      • Additional types may be defined in the future, such as ONE_TO_ONE_TABLE or SHARED_TABLE.

    • fieldName - String, Conditional - for type=FIELD_IN_TABLE, this property specifies the name of the QQQ Field in the table that stores the AutomationStatus id.

  • providerName - String, Required - name of an Automation Provider within the QQQ Instance, which is responsible for running the automations on this table.

  • overrideBatchSize - Integer - optional control over how many records from the table are processed in a single batch/page. For tables with "slow" actions (e.g., one that may need to make an API call per-record), using a smaller batch size (say, 50) may be required to avoid timeout errors.

  • actions - List of TableAutomationAction - list of the actions to perform on new and updated records in the table. Properties are:

    • name - String, Required - unique identifier for the action within its table.

    • triggerEvent - enum, Required - indicate which event type (POST_INSERT, POST_UPDATE, or PRE_DELETE (which is not yet implemented)) the action applies to.

    • priority - Integer, default 500 - mechanism to control the order in which actions on a table are executed, if there are more than one. Actions with a smaller value for priority are executed first. Ties are broken in an undefined manner.

    • filter - QQueryFilter - optional filter that gets applied to records when they match the triggerEvent, to control which records have the action ran against them.

    • includeRecordAssociations - Boolean, default false - for tables that have associations, control whether or not a record’s associated records are loaded when records are fetched and passed into the action’s custom code.

    • values - Map of String → Serializable - optional application-defined map of name=value pairs that can be passed into the action’s custom code.

    • processName - String, Conditional - name of a QQQ Processes in the QQQ Instance which is executed as the custom-code of the action.

    • codeReference - QCodeReference, Conditional - reference to a class that implements RecordAutomationHandlerInterface, to be executed as the custom-code of the action.

      • Note, exactly one of processName or codeReference must be provided.

Association

An Association is a way to define a relationship between tables, that facilitates, for example, a parent record having a list of its child records included in it when it is returned from a Query. INSERT and UPDATE can process associated records supplied with the parent; replacement UPDATE also deletes omitted stored members. Cascading DELETE follows the canonical declared association graph and complete stored membership, independently of supplied child records or UI visibility. See DeleteAction for selection, results and transaction boundaries.

Association Properties:

  • name - String, Required - unique name for the association within this table. Used as the key in the associatedRecords map within QRecord objects for this table.

  • associatedTableName - String, Required - name of a QQQ Table, which is the associated table.

  • joinName - String, Required - name of a QQQ Join in the instance, which defines how the tables are joined.

Full frontend table metadata exposes each association as name, associatedTableName and its resolved join, including the declared joinOns field pairs. Names are independent of target table names: pets may target pet, and multiple groups may target the same table. Use the exact association name to index associatedRecords; query exposedJoins are a separate contract. Child metadata and permission flags remain independent. Managed widget sections bind through widgetName and the widget’s frontend defaults (manageAssociationName for ChildRecordList or associationName for RowBuilder).

A successfully queried association is returned as an array, including [] when that group has no records. Base-only, unrequested and personalized-away groups remain absent; a failed query does not manufacture an empty group. Preserve this distinction in write clients: omitting a group leaves it unchanged, while an explicit empty replacement removes that group’s stored children subject to the existing write permissions.

Expanded reads use the declared parent and child endpoints in either join direction. Each relationship tuple is converted to the child’s declared field types before matching; a tuple containing null has no read membership. Required relationship fields missing from query results cause an error instead of a misleading empty group. Nested association selections use literal names, including names containing regular-expression characters. Nested INSERT and UPDATE use the same oriented field pairs. Replacement UPDATE finds omitted children using the stored old relationship, then assigns retained/new children the normalized prospective values. Omitted parent components remain stored, including hidden/heavy fields; private fetched values are not added to the public patch or child response. A null old tuple has no members. A null prospective tuple permits an empty replacement, but rejects child writes before parent mutation. Cascading DELETE uses the same declared tuples and complete native membership, deleting descendants before ancestors; UPDATE removes omitted children before changing their parent. An incomplete or rejected child deletion stops that parent mutation. Cyclic cascades are rejected. Caller transactions provide rollback where supported; these operations do not create an automatic transaction. Custom backend requirements and remaining recursive HTTP copy limits are described in the migration guide.

RecordSecurityLock

A RecordSecurityLock is the mechanism through which users can be allowed or denied access to read and/or write records, based on values in the record, and values in the user’s session. Record security locks must correspond to a Security Key Type.

For example:

  • An instance may have a security key type called clientId.

  • Users may have 1 or more clientId values in their Session, or, they may have an "All Clients" key in their session (e.g., for internal/admin users).

  • For some tables, it may be required to limit visibility to records based on a user’s clientId key. To do this. a RecordSecurityLock would be applied to the table, specifying the clientId field corresponds to the clientId security key.

  • With these settings in place, QQQ will prevent users from viewing records from this table that do not have a matching key, and will similarly prevent users from writing records with an invalid key value.

    • For example, in an RDBMS backend, all SELECT statements generated against such a table will have an implicit filter, such as AND client_id = ? based on the user’s security key values.

RecordSecurityLock Properties:

  • securityKeyType - String, Required - name of a Security Key Type in the Instance.

  • fieldName - String, Required - name of a QQQ Field in this table (or a joined table, if joinNameChain is set), where the value for the lock is stored.

  • joinNameChain - List of String - if the lock value is not stored in this table, but rather comes from a joined table, then this property defines the path of joins from this table to the table with the lock field.

  • nullValueBehavior - enum, default: DENY - control how records with a null value in the lock field should behave. Possible values are:

    • DENY - deny all users access to a record with a null value in the lock field (unless the user has an all-access key - see Security Key Type)

    • ALLOW - allow all users access to a record with a null value in the lock field.

    • ALLOW_WRITE_ONLY - allow all users to write records with null in the lock field, but deny reads on records with null in the lock field (also excepted by all-access keys).

  • lockScope - enum, default: READ_AND_WRITE - control what types of operations the lock applies to. Possible values are:

    • READ_AND_WRITE - control both reading and writing records based on the user having an appropriate security key.

    • WRITE - restrict writes to users satisfying this lock; this lock does not restrict reads.

    • READ - restrict reads to users satisfying this lock; this lock does not itself restrict writes. Record-lock scope does not grant table permissions or bypass other applicable locks. Nested MultiRecordSecurityLock AND/OR groups retain their Boolean structure after filtering for the operation’s applicable scopes.

QAuditRules

The audit rules on a table define the level of detail that is automatically stored in the audit table (if any) for DML actions (Insert, Update, Delete).

QAuditRules Properties:

  • auditLevel - enum, Required - level of details that are audited. Possible values are:

    • NONE - no automatic audits are stored for the table.

    • RECORD - only record-level audits are stored for the table (e.g., a message such as "record was edited", but without field-level details)

    • FIELD - full field-level audits are stored (e.g., including all old & new values as audit details).

CacheOf

One QQQ Table can be defined as a "cache of" another QQQ Table by assigning a CacheOf object to the table which will function as the cache. Currently, only UNIQUE_KEY_TO_UNIQUE_KEY loads records from the source. The two primary-key-based modes below are declared but unfinished, and cacheSourceMisses has no effect. These existing limitations are deferred under issue #545. Configure record-security rules on both source and cache tables when access is restricted; caching does not supply those application policies.

CacheOf Properties:

  • sourceTable - String, Required - name of the other QQQ Table that is the source of data in this cache.

  • expirationSeconds - Integer - optional number of seconds that a cached record is allowed to exist before it is considered expired, and must be re-fetched from the source table.

  • cachedDateFieldName - String, Conditional - used with expirationSeconds to define the field in this table that is used for storing the timestamp for when the record was cached.

  • useCases - List of CacheUseCase - what caching use-cases are to be implemented.

Properties of CacheUseCase are:

  • type - Enum, Required - the type of use-case. Possible values are:

    • PRIMARY_KEY_TO_PRIMARY_KEY - the primary key in the cache table equals the primary key in the source table.

    • UNIQUE_KEY_TO_PRIMARY_KEY - a unique key in the cache table equals the primary key in the source table.

    • UNIQUE_KEY_TO_UNIQUE_KEY - a unique key in the cache table equals a unique key in the source table.

  • cacheSourceMisses - Boolean, default false - whether or not, if a "miss" happens in the source, if that fact gets cached.

  • cacheUniqueKey - UniqueKey, conditional - define the fields in the cache table that define the unique key being used as the cache key.

  • sourceUniqueKey - UniqueKey, conditional - define the fields in the source table that define the unique key being used as the cache key.

  • doCopySourcePrimaryKeyToCache - Boolean, default false - specify whether or not the value of the primary key in the source table should be copied into records built in the cache table.

  • excludeRecordsMatching - List of QQueryFilter - optional filter to be applied to records before they are cached. If a record matches the filter, then it will not be cached.

ExposedJoin

Query screens in QQQ applications can potentially allow users to both display fields from joined tables, and filter by fields from joined tables, for any QQQ Join explicitly defined as an Exposed Join.

The reasoning why not all joins are implicitly exposed is that in many applications, the full join-graph can sometimes be overwhelming, surprisingly broad, and not necessarily practically useful. This could be subject to change in the future, e.g., given a UI that allowed users to more explicitly add additional join tables…​

ExposedJoin Properties:

  • label - String, Required - how the joined table should be presented in the UI.

  • joinTable - String, Required - name of the QQQ Table that is joined to this table, and is being exposed as a join in the UI.

  • joinPath - List of String, Required - names of 1 or more QQQ Joins that describe how to get from this table to the join table.

AssociatedScript

A QQQ Table can have end-user defined Script records associated with individual records in the table by use of the associatedScripts property of the table’s meta-data.

The "types" of these scripts (e.g., how they are used in an application) are wholly application-designed & managed. QQQ provides the mechanism for UI’s to present and manage such scripts (e.g., the Developer Mode screen in the Material Dashboard), as well as an interface to load & execute such scripts RunAssociatedScriptAction).

Warning

Script Test executes against live application data. Explicit write calls such as qqq.update(…​) persist changes using the caller’s permissions. Writes completed before a later script error can remain committed. Test does not save its draft revision or execution logs, and local changes to an input record are not saved unless the script explicitly writes them. Test does not isolate Java calls or external service effects. Use disposable records and a test environment for code that can change data.

AssociatedScript Properties:

  • fieldName - String, Required - name of a QQQ Field in the table which stores the id of the associated script record.

  • scriptTypeId - Serializable (typically Integer), Required - primary key value from the "scriptType" table in the instance, to designate the type of the Script.

  • scriptTester - QCodeReference - reference to a class which implements TestScriptActionInterface, that can be used by UI’s for running an associated script to test it.

Supplemental Meta Data

QQQ Frontend Material Dashboard

When running a QQQ application with the QQQ Frontend Material Dashboard module (QFMD), there are various pieces of supplemental meta-data which can be assigned to a Table, to modify some behaviors for the table in this UI.

Default Quick Filter Field Names

QFMD’s table query has a "Basic" mode, which will always display a subset of the table’s fields as quick-access filters. By default, the "Tier 1" fields on a table (e.g., fields in a Section that is marked as T1) will be used for this purpose.

However, you can customize which fields are shown as the default quick-filter fields, by providing a list of field names in a MaterialDashboardTableMetaData object, placed in the table’s supplementalMetaData.

table.withSupplementalMetaData(new MaterialDashboardTableMetaData()
    .withDefaultQuickFilterFieldNames(List.of("id", "warehouseId", "statusId", "orderDate")));
Go To Field Names

QFMD has a feature where a table’s query screen can include a "Go To" button, which a user can hit to open a modal popup, into which the user can enter a record’s identifier, to be brought directly to the record matching that identifier.

To use this feature, the table must have a List of GotoFieldNames set in its MaterialDashboardTableMetaData object in the table’s supplementalMetaData.

Each entry in this list is actually a list of fields, e.g., to account for a multi-value unique-key.

table.withSupplementalMetaData(new MaterialDashboardTableMetaData()
    .withGotoFieldNames(List.of(
        List.of("id"),
        List.of("partnerName", "partnerOrderId"))));

Processes

Besides QQQ Tables, the other most common type of object in a QQQ Instance is the Process. Processes are "custom" actions (e.g., defined by the application developers, rather than QQQ) that users and/or the system can execute against records. Processes generally are made up of two types of sub-objects:

  • Screens - i.e., User Interfaces (e.g., for gathering input and/or showing output).

  • BackendSteps - Java classes (of type BackendStep) that execute the logic of the process.

QProcessMetaData

Processes are defined in a QQQ Instance in a QProcessMetaData object. In addition to directly building a QProcessMetaData object setting its properties directly, there are a few common process patterns that provide Builder objects for ease-of-use. See StreamedETLWithFrontendProcess below for a common example

QProcessMetaData Properties:

  • name - String, Required - Unique name for the process within the QQQ Instance.

  • label - String - User-facing label for the process, presented in User Interfaces. Inferred from name if not set.

  • icon - QIcon - Icon associated with this process in certain user interfaces. See Icons.

  • tableName - String - Name of a QQQ Table that the process is associated with in User Interfaces (e.g., Action menu).

  • isHidden - Boolean, default false - Option to hide the process from all User Interfaces.

  • basepullConfiguration - BasepullConfiguration - config for the common "Basepull" pattern, of identifying records with a timestamp greater than the last time the process was ran. See below for details.

  • permissionRules - QPermissionRules object - define the permission/access rules for the process. See Permission Rules for details.

  • steps and stepList - Map of String → QStepMetaData and List of QStepMetaData - Defines the screens and backend code that makes up the process.

    • stepList is the list of steps in the order that they will be executed (that is to say - this is the default order of execution - but it can be customized - see How to customize a Linear process flow for details).

    • steps is a map, including all steps from stepList, but which may also include steps which can used by the process if its backend steps make the decision to do so, at run-time (e.g., using How to customize a Linear process flow).

    • A process’s steps are normally defined in one of two was:

      • 1) by a single call to .withStepList(List<QStepMetaData>), which internally adds each step into the steps map.

      • 2) by multiple calls to .withStep(QStepMetaData), which adds a step to both the stepList and steps map.

    • If a process also needs optional steps (for a How to customize a Linear process flow), they should be added by a call to .withOptionalStep(QStepMetaData), which only places them in the steps map.

  • stepFlow - enum, default LINEAR - specifies the the flow-control logic between steps. Possible values are:

    • LINEAR - steps are executed in-order, through the stepList. A backend step can customize the nextStepName or re-order the stepList, if needed. In a frontend step, a user may be given the option to go back to a previous step as well.

    • STATE_MACHINE - steps are executed as a Finite State Machine, starting with the first step in stepList, but then proceeding based on the nextStepName specified by the previous step. Thus allowing much more flexible flows.

  • schedule - QScheduleMetaData - set up the process to run automatically on the specified schedule. See below for details.

  • minInputRecords - Integer - not used…​

  • maxInputRecords - Integer - not used…​

The supplementalMetaData map attaches module-specific process settings. For application APIs, see ApiProcessMetaData.

QStepMetaData

This is the base class for the two types of steps in a process - screens and backend code. There are some shared attributes of both of them, defined here.

QStepMetaData Properties:

  • name - String, Required - Unique name for the step within the process.

  • label - String - User-facing label for the step, presented in User Interfaces. Inferred from name if not set.

  • stepType - String - Deprecated.

QFrontendStepMetaData

For processes with a user-interface, they must define one or more "screens" in the form of QFrontendStepMetaData objects.

QFrontendStepMetaData Properties:

  • components - List of QFrontendComponentMetaData - a list of components to be rendered on the screen.

  • formFields - List of String - list of field names used by the screen as form-inputs.

  • viewFields - List of String - list of field names used by the screen as visible outputs.

  • recordListFields - List of String - list of field names used by the screen in a record listing.

  • format - Optional String - directive for a frontend to use specialized formatting for the display of the process.

    • Consult frontend documentation for supported values and their meanings.

  • backStepName - Optional String - For processes using LINEAR flow, if this value is given, then the frontend should offer a control that the user can take (e.g., a button) to move back to an earlier step in the process.

QFrontendComponentMetaData

A screen in a process may consist of multiple "components" - e.g., help text, and a form, and a list of records. Each of these components are defined in a QFrontendComponentMetaData.

QFrontendComponentMetaData Properties:

  • type - enum, Required - The type of component to display. Each component type works with different keys in the values map. Possible values for type are:

    • EDIT_FORM - Displays a list of fields for editing, similar to a record edit screen. Requires that formFields be populated in the step.

    • VIEW_FORM - Displays a list of fields for viewing (not editing), similar to a record view screen. Requires that viewFields be populated in the step.

    • HELP_TEXT - Block of help text to be display on the screen. Requires an entry in the component’s values map named "text".

    • HTML - Block of custom HTML, generated by the process backend. Expects a process value named html.

    • DOWNLOAD_FORM - Presentation of a link to download a file generated by the process. Expects downloadFileName and either serverFilePath or the pair storageTableName / storageReference. In 4.0, trusted backend code must register the generated file with ProcessFileDownload before returning these values; see Downloading generated files.

    • GOOGLE_DRIVE_SELECT_FOLDER - Special form that presents a UI from Google Drive, where the user can select a folder (e.g., as a target for uploading files in a subsequent backend step).

    • BULK_EDIT_FORM - For use by the standard QQQ Bulk Edit process.

    • BULK_LOAD_FILE_MAPPING_FORM, BULK_LOAD_VALUE_MAPPING_FORM, or BULK_LOAD_PROFILE_FORM - For use by the standard QQQ Bulk Load process.

    • VALIDATION_REVIEW_SCREEN - For use by the QQQ Streamed ETL With Frontend process family of processes. Displays a component prompting the user to run full validation or to skip it, or, if full validation has been ran, then showing the results of that validation.

    • PROCESS_SUMMARY_RESULTS - For use by the QQQ Streamed ETL With Frontend process family of processes. Displays the summary results of running the process.

    • WIDGET - Render a QQQ Widget. Requires that widgetName be given as a value for the component.

    • RECORD_LIST - Deprecated. Showed a grid with a list of records as populated by the process.

  • values - Map of String → Serializable - Key=value pairs, with different expectations based on the component’s type. See above for details.

QBackendStepMetaData

Process Backend Steps are where custom (at this time, Java, but in the future, potentially, from any supported language) code is executed, to provide the logic of the process. QQQ comes with several common backend steps, such as for extracting lists of records, storing lists of records, etc.

QBackendStepMetaData Properties:

  • code - QCodeReference, Required - Reference to the code to be executed for the step. The referenced code must implement the BackendStep interface.

  • inputMetaData - QFunctionInputMetaData - Definition of the data that the backend step expects and/or requires. Sub-properties are:

    • fieldList - List of QQQ Fields - Optional list of fields used by the process step. In general, a process does not have to specify the fields that its steps use. It can be used, however, for example, to cause a defaultValue to be applied to a field if it isn’t specified in the process’s input. It can also be used to cause the process to throw an error, if a field is marked as isRequired, but a value is not present. For a field configured by the server, set isEditable = false. When RunProcessInput.inputSource is QInputSource.USER, caller-supplied values for those fields are rejected before saved process state is restored or merged. Metadata defaults and previously stored backend values remain available.

    • recordListMetaData - RecordListMetaData object - Identifies the table whose records a backend step needs. If the step has no records, the framework queries that table using input record IDs or the process callback’s filter. This automatic query inherits the process input source; USER calls require READ permission on the table and joined tables.

The legacy Javalin process endpoints and v1 process init/step endpoints mark incoming calls as QInputSource.USER. Application API process calls retain the SYSTEM default and their configured API parameter mapping and customizers; changing this provenance is deferred in #451. Direct Java calls default to QInputSource.SYSTEM; code forwarding external input must explicitly select USER. This setting governs process input validation and automatic record loading. Queries or mutations created inside a custom backend step choose their own input source and permissions independently. On USER process inputs, supplied values are treated as caller input when validating non-editable fields. Application API result ownership is bound independently of process input source; restoring SYSTEM provenance does not make asynchronous results public.

Application API process custom permission checkers now run once after API input mapping, metadata defaults and PreRunApiProcessCustomizer, before synchronous or asynchronous process execution. This lets the built-in ReportProcessPermissionChecker authorize the effective reportName. Standard process permission checks still reject denied calls before the pre-run customizer. Customizers run before input-dependent authorization; they must authorize any sensitive side effects themselves.

Process state ownership

HTTP process state and asynchronous jobs belong to the originating application instance, authentication session, backend variants and process. Status, record retrieval, resume, cancellation and process-dependent possible-value lookups require that scope and current process permission. Clients must retain the server-issued session cookies or the same authentication reference across requests; a process or job UUID alone does not grant access. Rejected requests are checked before saved inputs are merged or mutated.

Trusted Java callers may use RunProcessAction.getState(…​) and AsyncJobManager.getJobStatus(…​) internally. Code serving external requests must use getStateForUser(…​) and getJobStatusForUser(…​) instead. For custom asynchronous process starters, call AsyncJobManager.withProcessUUID(…​) before startJob(…​) to associate the job with its process. Ownership is server-only and is not serialized into client responses. These scopes apply to live state in the current JVM; they do not add durable or shared process storage.

Cancellation and process lifecycle

CancelProcessAction runs the optional cancel step, subject to the same ownership and permission checks as process resume. Without a cancel step it logs a no-op. Existing missing/unknown-state handling remains; malformed UUID error classification is a separate Low issue (#481).

The proposed one-shot/terminal lifecycle, active-callback signalling, completed/repeated cancellation rejection and concurrent/no-resume rules are deferred in #478. The action does not establish those guarantees. Existing AsyncJobManager.cancelJob and AsyncJobCallback.wasCancelRequested() support cooperative work; custom code must observe the signal and release its owned resources. A cancel hook’s exception remains visible to its caller.

New parent-to-child cancellation, publication boundaries and ETL boundary checks are deferred in #479 and #480. No new callback helper API or forced interruption is introduced. See ETL cancellation and transactions for existing transaction limits.

Saved-view and saved-report ownership

Owned saved views and saved reports can be edited or deleted only by their stored owner. Submitting, clearing or omitting userId does not bypass this check, and an existing owned record cannot be transferred by editing that field. The check also applies to trusted inner actions using SYSTEM input; SYSTEM source alone is not an ownership bypass.

Mutations use a private native backend read, with the caller’s transaction, to obtain ownership and report content. Hidden fields and returned-record presentation do not provide authorization values. The snapshot is not copied into action responses. Backends must supply the requested stored fields; missing or incomplete snapshots fail closed. No lock against concurrent external ownership changes is introduced.

Partial report updates validate the stored content merged with the submitted fields. They preserve omitted content and normalize an explicitly submitted query filter without writing old fields back. Deleting a report requires ownership but does not revalidate its JSON content. Existing null-owner behavior remains: such records retain their legacy maintenance semantics, and the report USER_ID default may assign the caller on update. Applications should assign an owner when creating private assets.

storeSavedView and deleteSavedView check the caller’s insert/edit/delete table permissions and return process errors when the underlying action fails. A missing ID, denied mutation or invalid new view must not be treated as a successful save/delete.

The built-in saved-asset mutation policy remains owner-only for owned records. The generic READ_WRITE sharing label does not by itself grant another user permission to mutate these saved assets. Recipient visibility and sharing configuration are separate from this ownership check.

Sharing existing records

After applying the configured asset-owner check, editSharedRecord and deleteSharedRecord require shareId to identify a share whose configured asset foreign key matches recordId. A missing or mismatched pair returns an error before mutation, including when the same user owns both assets. Clients must retain the selected asset and its share ID together.

Both asset-owner and share-relationship checks require backend Count support; unsupported backends fail before mutation. Ownership is checked against the stored owner field and current user, rather than the returned display value. A hidden owner field is supported; removing it from personalized metadata can still deny the operation. The relationship check uses a filtered backend count, so hidden fields and returned-record presentation customizers do not supply the relationship used for this check. It does not make the check and later mutation atomic against concurrent reassignment. Applications still need appropriate record-security rules; saved views require explicit ShareableTableMetaData configuration for generic sharing. This check grants no new read or edit permissions.

Configured saved-view sharing

After SavedViewsMetaDataProvider defines the tables, configure ShareableTableMetaData on savedView with shared table sharedSavedView, asset field savedViewId, scope field scope, owner field userId, and a user audience using userId. The sample sharing fixture demonstrates this existing configuration API. Register SharingMetaDataProvider for the sharing processes. For private views, supply a user-level record lock on userId, using the caller’s user ID as the security-key value, with the standard READ_AND_WRITE scope and DENY null policy. That configuration lets owners read and manage their views and shares; recipients can read granted views but cannot manage shares or edit/delete the owned view. Removing a share removes recipient access, even if a quick-view preference remains. Quick-view labels, counts and ordering are scoped to each user’s preference record.

The default user audience accepts application-supplied user IDs. Configure an audience source table to validate recipient existence, and a possible-value source for an interactive user picker. Provider registration does not create a directory or authenticate users. The stock dashboard offers read-only sharing; the generic READ_WRITE enum remains available to applications but does not change built-in saved-asset ownership. Custom user-lock scopes and null policies retain their configured semantics: a WRITE-only lock does not restrict reads, and a READ-only lock does not provide write protection.

Joined record-security checks read native stored owner and join values privately with the caller’s transaction. Hidden/heavy fields and returned-record display transformations cannot authorize a grant or prevent a valid owner grant. Declared null memory values retain the configured null policy; custom query backends must supply requested owner and join fields, and incomplete projections fail closed. No atomic guarantee against an external ownership change is added. Application and tenant security still require explicit configuration.

Saved-view dependent cleanup

Automatic dependent-share and quick-view preference cleanup is deferred in #504. Saved-view deletion retains its established behavior; applications must manage any remaining dependent rows according to their own retention policy. No provider privacy-lock exemption or canonical audit-key fallback is introduced. Stored-owner authorization remains required.

Configured saved-report sharing

SavedReportsMetaDataProvider defines sharing metadata but does not install record-security locks. Configure them through its existing table enricher or before instance validation, and register SharingMetaDataProvider for the sharing processes. Bind the security key to the authenticated user’s actual ID. This example adds the standard owner/recipient rules; keep any additional application or tenant locks:

String userKey = "savedReportUser";
String shareJoin = SavedReportsMetaDataProvider.SHARED_SAVED_REPORT_JOIN_SAVED_REPORT;
String ownerJoin = "savedReportJoinSharedSavedReport";
instance.addSecurityKeyType(new QSecurityKeyType().withName(userKey));
instance.addJoin(instance.getJoin(shareJoin).flip().withName(ownerJoin));
RecordSecurityLock user = new RecordSecurityLock()
   .withSecurityKeyType(userKey).withFieldName("userId");

instance.getTable(SavedReport.TABLE_NAME).withRecordSecurityLock(
   new MultiRecordSecurityLock().withOperator(MultiRecordSecurityLock.BooleanOperator.OR)
      .withLock(user.clone())
      .withLock(user.clone().withLockScope(RecordSecurityLock.LockScope.READ)
         .withFieldName("sharedSavedReport.userId").withJoinNameChain(List.of(shareJoin))));
instance.getTable(SharedSavedReport.TABLE_NAME).withRecordSecurityLock(
   new MultiRecordSecurityLock().withOperator(MultiRecordSecurityLock.BooleanOperator.OR)
      .withLock(user.clone().withFieldName("savedReport.userId").withJoinNameChain(List.of(ownerJoin)))
      .withLock(user.clone().withLockScope(RecordSecurityLock.LockScope.READ)));

The default lock scope is READ_AND_WRITE and its null policy is DENY. Owners can manage reports and all their grants; recipients can read shared reports and their own grant rows. Recipients cannot manage grants or edit/delete an owned report, including when a generic READ_WRITE label is stored. Multiple grants do not duplicate the report list. This configuration does not create an authenticated session, recipient directory or dashboard picker. Configure an audience source table when IDs must be validated against a directory.

Rendering uses the records selected for the process. Configure table record-security locks to limit that selection; the sharing recipe above applies those locks to owner and recipient reads. The proposed additional private-owner/grant-scope check and revalidation after a paused step are deferred in #451, with its candidate correctness issue #446. Ownership rules for editing and deleting remain separate from permission to run a report.

Saved-report provider registration retains its existing associations; automatic dependent-share cleanup is deferred in #504. This does not define schedule/job, rendered-history or archive retention. The native sample uses H2 for reports/grants, memory for history and owned filesystem storage for generated bytes.

First-party sharing fixture and dashboard limitations

The sample’s explicit qqq.sample.sharing=true option requires qqq.sample.mockAuthentication=true. It supplies synthetic Alice, Bob and Casey sessions, native H2 asset/grant/directory rows, and the generic sharing processes. This local fixture is not production authentication and does not establish schedule/history/archive retention or complete output isolation.

Configure audiencePossibleValueSourceName on each shareable table and set the user audience’s sourceTableName to the recipient directory. Directory primary keys are the authenticated IDs. The sample TABLE possible-value source uses overrideIdField="audienceId" to produce picker values such as user:sample:bob; the intended contract preserves the full ID after the first colon. The source table validates the ID and supplies its label. The dashboard currently grants READ_ONLY.

The dashboard path has deferred Medium/Low limitations: recipient ID truncation, missing-column preview failure, and mock identity/https://github.com/QRun-IO/qqq/issues/332[display behavior]. Local fixes and dependent browser checks were preserved for a future release and removed from the 4.0 release. Earlier passing browser results apply only to that preserved candidate. Native backend sharing/ownership fixtures remain separate evidence; the sample inventory retains outstanding acceptance gaps.

Scheduled reports

Scheduled reports establish an automated session for the stored user before rendering, including when invoked manually. Application-configured table/process permissions govern schedule management and selection. The added blanket owner-only CRUD/manual-run and nested-process permission policies are deferred in #452. Nested rendering retains its trusted process default and forwards stored report inputs, including configured output values. It does not add a separate raw-table READ prerequisite merely because it runs on a schedule. Caller-session restoration remains deferred in #450.

The default automated session is a fresh ordinary QSession with a new UUID and the requested user ID. Only the selected backend-variant map is copied. Scheduler permissions, security keys, tokens, session identifiers, custom values and frontend values are not inherited. Mock, Auth0 and OAuth2 providers call customizeAutomatedSessionForUser(…​) to establish owner-specific context. Integrations must load the owner’s current grants and supply authentication that their provider can validate. Permissions alone do not create a valid Auth0/OAuth2 session; unsupported automation fails rather than borrowing the scheduler’s credentials.

Requested email delivery uses configured sender and provider metadata. A later delivery failure can leave completed archive bytes while history records FAILED, retaining the established status behavior. Its proposed COMPLETE interpretation remains deferred in #461. See scheduled process failures for the existing log-based scheduler boundary. Record-backed activation, rescheduling, deactivation and deletion use explicitly configured linked jobs; see saved-report schedule lifecycle for configuration and failure-reporting limits.

QStateMachineStep

Processes that use flow = STATE_MACHINE should use process steps of type QStateMachineStep.

A common pattern seen in state-machine processes, is that they will present a frontend-step to a user, then always run a given backend-step in response to that screen which the user submitted. Inside that backend-step, custom application logic will determine the next state to go to, which is typically another frontend-step (which would then submit data to its corresponding backend-step, and continue the FSM).

To help facilitate this pattern, factory methods exist on QStateMachineStep, for constructing the commonly-expected types of state-machine steps:

  • frontendThenBackend(name, frontendStep, backendStep) - for the frontend-then-backend pattern described above.

  • backendOnly(name, backendStep) - for a state that only has a backend step. This might be useful as a “reset” step, to run before restarting a state-loop.

  • frontendOnly(name, frontendStep) - for a state that only has a frontend step, which would always be followed by another state, which must be specified as the defaultNextStepName on the QStateMachineStep.

BasepullConfiguration

A "Basepull" process is a common pattern where an application needs to perform some action on all new (or updated) records from a particular data source. To implement this pattern, one needs to store the timestamp of when the action runs, then query the source for records where a date-time field is after that timestamp.

QQQ helps facilitate this pattern by automatically retrieving and updating that timestamp field, and by building a default query filter based on that timestamp.

This is done by adding a BasepullConfiguration object to a process’s meta-data.

BasepullConfiguration Properties:

  • tableName - String, Required - Name of a QQQ Table in the QQQ Instance where the basepull timestamps are stored.

  • keyField - String, Required - Name of a QQQ Field in the basepull table that stores a unique identifier for the process.

  • keyValue - String - Optional value to be stored in the keyField of the basepull table as the unique identifier for the process. If not set, then the process’s name is used.

  • lastRunTimeFieldName - String, Required - Name of a QQQ Field in the basepull table that stores the last-run time for the process.

  • hoursBackForInitialTimestamp - Integer - Optional number of hours to go back in time (from now) for the first time that the process is executed (i.e., if there is no timestamp stored in the basepull table).

  • timestampField - String, Required - Name of a QQQ Field in the table being queried against the last-run timestamp.

QScheduleMetaData

QQQ can automatically run processes using an internal scheduler, if they are configured with a QScheduleMetaData object.

QScheduleMetaData Properties

  • repeatSeconds - Integer, Conditional - How often the process should be executed, in seconds.

  • repeatMillis - Integer, Conditional - How often the process should be executed, in milliseconds. Mutually exclusive with repeatSeconds.

  • initialDelaySeconds - Integer, Conditional - How long between when the scheduler starts and the process should first run, in seconds.

  • initialDelayMillis - Integer, Conditional - How long between when the scheduler starts and the process should first run, in milliseconds. Mutually exclusive with initialDelaySeconds.

  • variantRunStrategory - enum, Conditional - For processes than run against QQQ Tables that use a QQQ Backend with Variants enabled, this property controls if the various instances of the process should run in PARALLEL or in SERIAL.

  • variantBackend - enum, Conditional - For processes than run against QQQ Tables that use a QQQ Backend with Variants enabled, this property specifies the name of the QQQ Backend.

StreamedETLWithFrontendProcess

A common pattern for QQQ processes to exhibit is called the "Streamed ETL With Frontend" process pattern. This pattern is to do an "Extract, Transform, Load" job on a potentially large set of records.

The records are Streamed through the process’s steps, meaning, QQQ runs multiple threads - a producer, which is selecting records, and a consumer, which is processing records. As such, in general, records can be processed a page at a time. Custom steps must also bound their caches and retained data to avoid exhausting server memory.

These processes also have a standard user-interface for displaying a summary of what the process will do (and has done), with a small number of records as a preview. The goal of the summary is to give the user the big-picture of what the process will do (e.g., X records will be inserted; Y records will be updated), along with a small view of some details on the records that will be stored (e.g., on record A field B will be set to C).

This type of process uses 3 backend steps, and 2 frontend steps, as follows:

  • preview (backend) - does just a little work (limited # of rows), to give the user a preview of what the final result will be - e.g., some data to seed the review screen.

  • review (frontend) - a review screen, which after the preview step does not have a full process summary, but can generally tell the user how many records are input to the process, and can show a preview of a small number of the records.

  • validate (backend) - optionally (per input on review screen), does like the preview step, but does it for all records from the extract step.

  • review (frontend) - a second view of the review screen, if the validate step was executed. Now that the full validation was performed, a full process summary can be shown, along with a some preview records.

  • execute (backend) - processes all the rows - does all the work - stores data in the backend.

  • result (frontend) - a result screen, showing a "past-tense" version of the process summary.

These backend steps are defined within QQQ, meaning they themselves do not execute any application-defined custom code. Instead, these steps use the following secondary backend steps:

  • Extract - Fetch the rows to be processed. Used by preview (but only for a limited number of rows), validate (without limit), and execute (without limit).

  • Transform - Do whatever transformation is needed to the rows. Done on preview, validate, and execute. Always works with a page of records at a time. Since it is called on the preview & validate steps, it should NOT ever store any data (unless it does a specific check to confirm that it is being used on an execute step).

  • Load - Store the records into the backend, as appropriate. Only called by the execute step. Always works with a page of records at a time.

The meta-data for a StreamedETLWithFrontendProcess uses several input fields on its steps. As such, it can be somewhat clumsy and error-prone to fully define a StreamedETLWithFrontendProcess. To improve this programmer-interface, an inner Builder class exists within StreamedETLWithFrontendProcess (generated by a call to StreamedETLWithFrontendProcess.processMetaDataBuilder()).

StreamedETLWithFrontendProcess.Builder methods:

  • withName(String name) - Set the name for the process.

  • withLabel(String label) - Set the label for the process.

  • withIcon(QIcon icon) - Set an Icon to be display with the process in the UI.

  • withExtractStepClass(Class<? extends AbstractExtractStep>) - Define the Extract step for the process. If no special extraction logic is needed, ExtractViaQuery.class is often a reasonable default. In other cases, ExtractViaQuery can be a reasonable class to extend for a custom extract step.

  • withTransformStepClass(Class<? extends AbstractTransformStep>) - Define the Transform step for the process. If no transformation logic is needed, NoopTransformStep.class can be used (though this is not very common).

  • withLoadStepClass(Class<? extends AbstractLoadStep>) - Define the Load step for the process. Several standard implementations exist, such as: LoadViaInsertStep.class, LoadViaUpdateStep.class, and LoadViaDeleteStep.class.

  • withTableName(String tableName) - Specify the name of the QQQ Table that the process should be associated with in the UI.

  • withSourceTable(String sourceTable) - Specify the name of the QQQ Table to be used as the source of records for the process.

  • withDestinationTable(String destinationTable) - Specify the name of the QQQ Table to be used as the destination for records from the process.

  • withSupportsFullValidation(Boolean supportsFullValidation) - By default, all StreamedETLWithFrontendProcesses do allow the user to choose to run the full validation step. However, in case cases it may not make sense to do so - so this method can be used to turn off that option.

  • withDoFullValidation(Boolean doFullValidation) - By default, all StreamedETLWithFrontendProcesses will prompt the user if they want to run the full validation step or not. However, in case cases you may want to enforce that the validation step always be executed. Calling this method will remove the option from the user, and always run a full validation.

  • withTransactionLevelAutoCommit(), withTransactionLevelPage(), and withTransactionLevelProcess() - Change the transaction-level used by the process. By default, these processes are ran with a single transaction for all pages of their execute step. But for some cases, doing page-level transactions can reduce long-transactions and locking within the system.

  • withPreviewMessage(String previewMessage) - Sets the message shown on the validation review screen(s) above the preview records.

  • withReviewStepRecordFields(List<QFieldMetaData> fieldList) -

  • withFields(List<QFieldMetaData> fieldList) - Adds additional input fields to the preview step of the process.

  • withBasepullConfiguration(BasepullConfiguration basepullConfiguration) - Add a BasepullConfiguration to the process.

  • withSchedule(QScheduleMetaData schedule) - Add a QScheduleMetaData to the process.

How to customize a Linear process flow

As referenced in the definition of the QProcessMetaData Properties, by default, (with flow = LINEAR) a process will execute each of its steps in-order, as defined in the stepList property. However, a Backend Step can customize this flow as follows:

  • RunBackendStepOutput.setOverrideLastStepName(String stepName)

    • QQQ’s RunProcessAction keeps track of which step it "last" ran, e.g., to tell it which one to run next. However, if a step sets the OverrideLastStepName property in its output object, then the step named in that property becomes the effective "last" step, thus determining which step comes next.

  • RunBackendStepOutput.updateStepList(List<String> stepNameList)

    • Calling this method changes the process’s runtime definition of steps to be executed. Thus allowing a completely custom flow. It should be noted, that the "last" step name (as tracked by QQQ within RunProcessAction) does need to be found in the new stepNameList - otherwise, the framework will not know where you were, for figuring out where to go next.

The following fragments share one enclosing application class. Supply PROCESS_NAME, replace each omitted QCodeReference target with the application’s BackendStep class, and put the process-building statements inside the metadata producer method.

Class fields and metadata-method statements for a customized linear flow:
// for a case like this, it would be recommended to define all step names in constants:
public final static String STEP_START = "start";
public final static String STEP_A     = "a";
public final static String STEP_B     = "b";
public final static String STEP_C     = "c";
public final static String STEP_1     = "1";
public final static String STEP_2     = "2";
public final static String STEP_3     = "3";
public final static String STEP_END   = "end";

// also, to define the possible flows (lists of steps) in constants as well:
public final static List<String> LETTERS_STEP_LIST = List.of(
  STEP_START, STEP_A, STEP_B, STEP_C, STEP_END);

public final static List<String> NUMBERS_STEP_LIST = List.of(
  STEP_START, STEP_1, STEP_2, STEP_3, STEP_END);

// when we define the process's meta-data, we only give a "skeleton" stepList -
// we must at least have our starting step, and we may want at least one frontend step
// for the UI to show some placeholder(s):
QProcessMetaData process = new QProcessMetaData()
   .withName(PROCESS_NAME)
   .withStepList(List.of(
      new QBackendStepMetaData().withName(STEP_START)
         .withCode(new QCodeReference(/*...*/)),
      new QFrontendStepMetaData()
         .withName(STEP_END)
   ));

// the additional steps get added via `withOptionalStep`, which only puts them in
// the process's stepMap, not its stepList!
process.withOptionalStep(new QFrontendStepMetaData().withName(STEP_A));
process.withOptionalStep(new QBackendStepMetaData().withName(STEP_B)
   .withCode(new QCodeReference(/*...*/)));
process.withOptionalStep(new QFrontendStepMetaData().withName(STEP_C));

process.withOptionalStep(new QBackendStepMetaData().withName(STEP_1)
   .withCode(new QCodeReference(/*...*/)));
process.withOptionalStep(new QFrontendStepMetaData().withName(STEP_2));
process.withOptionalStep(new QBackendStepMetaData().withName(STEP_3)
   .withCode(new QCodeReference(/*...*/)));
Example of a process backend step adjusting the process’s runtime flow:
/***************************************************************************
** look at the value named "which". if it's "letters", then make the process
** go through the stepList consisting of letters; else, update the step list
** to be the "numbers" steps.
**
** Also - if the "skipSomeSteps" value is give as true, then set the
** overrideLastStepName to skip again (in the letters case, skip past A, B
** and C; in the numbers case, skip past 1 and 2).
***************************************************************************/
public static class StartStep implements BackendStep
{
  @Override
  public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
  {
     Boolean skipSomeSteps = runBackendStepInput.getValueBoolean("skipSomeSteps");

     if("letters".equals(runBackendStepInput.getValueString("which")))
     {
        runBackendStepOutput.updateStepList(LETTERS_STEP_LIST);
        if(BooleanUtils.isTrue(skipSomeSteps))
        {
           runBackendStepOutput.setOverrideLastStepName(STEP_C);
        }
     }
     else
     {
        runBackendStepOutput.updateStepList(NUMBERS_STEP_LIST);
        if(BooleanUtils.isTrue(skipSomeSteps))
        {
           runBackendStepOutput.setOverrideLastStepName(STEP_2);
        }
     }
  }
}
How to allow a process to go back

The simplest option to allow a process to present a "Back" button to users, thus allowing them to move backward through a process (e.g., from a review screen back to an earlier input screen), is to set the property backStepName on a QFrontendStepMetaData.

If the step that is executed after the user hits "Back" is a backend step, then within that step, runBackendStepInput.getIsStepBack() will return true (but ONLY within that first step after the user hits "Back"). It may be necessary within individual processes to be aware that the user has chosen to go back, to reset certain values in the process’s state.

Alternatively, if a frontend step’s "Back" behavior needs to be dynamic (e.g., sometimes not available, or sometimes targeting different steps in the process), then in a backend step that runs before the frontend step, a call to runBackendStepOutput.getProcessState().setBackStepName() can be made, to customize the value which would otherwise come from the QFrontendStepMetaData.

Widgets

Widgets are the most customizable UI components in QQQ. They can be used either on App Home Screens (e.g., as Dashboard screens), or they can be included into Record View screens.

QQQ defines several types of widgets, such as charts (pie, bar, line), numeric displays, application-populated tables, or even fully custom HTML.

QWidgetMetaData

A Widget is defined in a QQQ Instance in a QWidgetMetaData object.

QWidgetMetaData Properties:

  • name - String, Required - Unique name for the widget within the QQQ Instance.

  • type - String, Required - Specifies the UI & data type for the widget. Use a WidgetType enum value’s getType() method.

  • label - String - User-facing header or title for a widget.

  • tooltip - String - Text contents to be placed in a tooltip associated with the widget’s label in the UI.

  • gridColumns - Integer - for a desktop-sized screen, in a 12-based grid, how many columns the widget should consume.

  • codeReference - QCodeReference, Required - Reference to the custom code, a subclass of AbstractWidgetRenderer, which is responsible for loading data to render the widget.

  • footerHTML - String - HTML String, which, if present, will be displayed in the footer of the widget (not supported by all widget types).

  • isCard - boolean, default false - Requests card styling in frontends that support it.

  • showReloadButton - boolean, default true - Requests a control to reload the widget data.

  • showExportButton - boolean, default false - Requests an export control for widget types/frontends that support export.

  • dropdowns - List<WidgetDropdownData> - Selection controls whose values are passed to the renderer.

  • storeDropdownSelections - boolean, default false - Requests that the frontend retain dropdown selections. Material currently forwards this setting through parent widgets; ordinary HTML widgets do not save selections across refresh. This limitation is deferred under #554.

  • collapsible - Declares widget collapse behavior, but the native frontend widget metadata response currently omits it. Material therefore cannot enable the declared collapse controls through that response; #555 is deferred.

  • icons - Map<String, QIcon> - Named icons used by the selected widget type.

  • defaultValues - Map<String, Serializable> - Default named values available to widget rendering.

ChartData is a reusable backend payload for concrete chart widgets. The CHART enum value does not have a standalone Material renderer; choose a concrete metadata type such as barChart, lineChart or pieChart.

LocationData serializes image, title, description, location and footer values; the LOCATION enum has no standalone Material renderer. The US map currently displays fixed demo markers rather than supplied marker values; #560 tracks this deferred limitation.

A customComponent widget names its JavaScript bundle with defaultValues.componentSourceUrl and its global export with defaultValues.componentName. The bundle registers window.ComponentName.ComponentName; the component receives an envelope containing props.widgetMetaData, props.widgetData and props.record, plus the frontend bridge/context. An unavailable bundle displays a loading error. Only load application-owned, trusted component code.

A process widget rendered by ProcessWidgetRenderer sets defaultValues.processName. Query parameters become default process values; running the process still requires its own permissions. The sample’s widget browser fixture demonstrates both execution and denial.

Child-record widgets use ChildRecordListRenderer.widgetMetaDataBuilder(join). User requests require READ on the parent, child and queried join tables; paginated totals use the same personalized visibility as the child query. The configured join remains authoritative over a request’s joinName. Empty parents and create-screen metadata can render without child rows.

CronUIWidgetRenderer.buildWidgetMetaData configures a record’s cron-expression field and optional time zone. Rendering an existing record requires source-table READ permission and uses the caller’s personalized visibility; describing a supplied cronExpression without an id does not read a record. Missing or invisible records return not-found. On record edit screens, place the editor in a widget section without fieldNames; fields still need declarations in another section (which may be hidden). A section containing fieldNames renders ordinary form inputs. This also applies to dynamicForm widget sections.

There are also some subclasses of QWidgetMetaData, for some specific widget types:

ParentWidgetMetaData Properties:

  • title - String - A metadata property retained by this subclass. The standard ParentWidgetRenderer does not copy it to its output; use the widget label or explicitly set the label in a custom renderer.

  • childWidgetNameList - List<String> - Names of child widgets. Empty lists render no children; Material omits children absent from the current user’s permitted widget metadata.

  • childProcessNameList - List<String> - Retained metadata property; the standard ParentWidgetRenderer does not render this list. Use process widgets in childWidgetNameList to show processes.

  • layoutType - enum of GRID or TABS, default GRID - Both layouts render permitted children. Restoring a saved tab selection after refresh is currently limited by #559 (deferred).

QNoCodeWidgetMetaData Properties:

  • values - List<AbstractWidgetValueSource> - Definitions of the data values to obtain while rendering.

  • outputs - List<AbstractWidgetOutput> - Definitions of the output widgets built from those values.

No-code value sources support query fields, counts, aggregates, named Java ad-hoc functions and calculations. HTTP query/count/aggregate sources preserve the caller’s access context and enforce READ permissions on source tables and joined filter tables. A failed value source leaves that value unavailable while other outputs can still render.

WidgetHtmlLine emits authored HTML through Velocity; escape untrusted text in the value provider before inserting it into the template. Ad-hoc values use a named Java class implementing java.util.function.Function with a public no-argument constructor.

See Rendering Widgets for complete pie and composite renderers. The NoCodeWidgetRenderer and no-code metadata classes define the supported declarative value/output combinations.

Fields

QQQ Fields define

QFieldMetaData

QFieldMetaData Properties:

  • name - String, Required - Unique name for the field within its container (table, process, etc).

  • label - String - User-facing label for the field, presented in User Interfaces.

  • type - enum of QFieldType, Required - Data type for values in the field.

  • backendName - String - Name of the field within its backend.

    • For example, in an RDBMS-backed table, a field’s name may be written in camel case, but its backendName written with underscores.

  • isRequired - boolean, default false - Indicator that a value is required in this field.

  • isEditable - boolean, default true - Indicator that users may edit values in this field.

  • displayFormat - String, default %s - Java Format Specifier string, used to format values in the field for display in user interfaces. Used to set values in the displayValues map within a QRecord.

    • Recommended values for displayFormat come from the DisplayFormat interface, such as DisplayFormat.CURRENCY, DisplayFormat.COMMAS, or DisplayFormat.DECIMAL2_COMMAS.

  • defaultValue - Value to use for the field if no other value is given. Type is based on the field’s type.

  • possibleValueSourceName - String - Reference to a QQQ Possible Value Source to be used for this field. Values in this field should correspond to ids from the referenced Possible Value Source.

  • maxLength - Integer - Maximum string length in Java UTF-16 units. A supplementary Unicode character occupies two units. Only applicable for fields with type=STRING. Needs to be used with a FieldBehavior of type ValueTooLongBehavior.

Field Behaviors

Additional behaviors can be attached to fields through the use of the behaviors attribute, which is a Set of 0 or more instances of implementations of the FieldBehavior interface. Note that in some cases, these instances may be enum constants, but other times may be regular Objects.

QQQ provides a set of common field behaviors. Applications can also define their own field behaviors by implementing the FieldBehavior interface, and attaching instances of their custom behavior classes to fields.

ValueTooLongBehavior

Used on String fields. Requires the field to have a maxLength set. Depending on the chosen instance of this enum, before a record is Inserted or Updated, if the value in the field is longer than the maxLength, then one of the following actions can occur:

  • TRUNCATE - Truncate to at most maxLength UTF-16 units, keeping surrogate pairs intact.

  • TRUNCATE_ELLIPSIS - Reserve three units for …​ and truncate the value without splitting a surrogate pair. For a nonnegative maximum shorter than three, return only the periods that fit. The result can be shorter than maxLength to preserve a complete pair.

  • ERROR - An error will be reported, and the record will not be inserted or updated.

  • PASS_THROUGH - Nothing will happen. This is the same as not having a ValueTooLongBehavior on the field.

Examples of using ValueTooLongBehavior
   new QFieldMetaData("sku", QFieldType.STRING)
      .withMaxLength(40)
      .withBehavior(ValueTooLongBehavior.ERROR);

   new QFieldMetaData("reason", QFieldType.STRING)
      .withMaxLength(250)
      .withBehavior(ValueTooLongBehavior.TRUNCATE_ELLIPSIS);
ValueRangeBehavior

Used on Numeric fields. Specifies min and/or max allowed values for the field. For each of min and max, the following attributes can be set:

  • minValue / maxValue - the number that is the limit.

  • minAllowEqualTo / maxAllowEqualTo - boolean (default true). Controls if < (>) or ≤ (≥).

  • minBehavior / maxBehavior - enum of ERROR (default) or CLIP.

    • If ERROR, then a value not within the range causes an error, and the value does not get stored.

    • else if CLIP, then a value not within the range gets "clipped" to either be the min/max (if allowEqualTo), or to the min/max plus/minus the clipAmount

  • minClipAmount / maxClipAmount - Default 1. Used when behavior is CLIP (only applies when not allowEqualTo).

Examples of using ValueRangeBehavior
    new QFieldMetaData("noOfShoes", QFieldType.INTEGER)
      .withBehavior(new ValueRangeBehavior().withMinValue(0));

    new QFieldMetaData("price", QFieldType.DECIMAL)
      .withBehavior(new ValueRangeBehavior()
         // set the min value to be >= 0, and an error if an input is < 0.
         .withMinValue(BigDecimal.ZERO)
         .withMinAllowEqualTo(true)
         .withMinBehavior(ValueRangeBehavior.Behavior.ERROR)
         // set the max value to be < 100 - but effectively, clip larger values to 99.99
         // here we use the .withMax() method that takes 4 params vs. calling 4 .withMax*() methods.
         .withMax(new BigDecimal("100.00"), false, ValueRangeBehavior.Behavior.CLIP, new BigDecimal("0.01"))
    );
DynamicDefaultValueBehavior

Used to set a dynamic default value to a field when it is being inserted or updated. For example, instead of having a hard-coded defaultValue specified in the field meta-data, and instead of having to add, for example, a pre-insert custom action.

  • CREATE_DATE - On inserts, replaces the input value with the current Instant for DATE_TIME fields or the server’s current local calendar date for DATE fields. Updates preserve the supplied or stored creation value.

  • MODIFY_DATE - On inserts and updates, replaces the input value using the same DATE_TIME/DATE rules. UpdateInput.withOmitModifyDateUpdate(true) suppresses automatic modification-date assignment for that update.

  • USER_ID - On inserts and updates, sets missing or blank input fields to the current user’s nonblank ID. Blank follows StringUtils.hasContent (null, empty or trim-empty strings). An omitted field in a sparse update receives this default; an explicit nonblank value is preserved. Reads preserve stored values and never substitute the reader’s ID.

  • NONE - Disables this behavior on the field, including automatic inference for fields named createDate and modifyDate.

Note that the QInstanceEnricher will, by default, add the CREATE_DATE and MODIFY_DATE DynamicDefaultValueBehavior options to any fields named "createDate" or "modifyDate". Setting configAddDynamicDefaultValuesToFieldsNamedCreateDateAndModifyDate=false affects only that QInstanceEnricher instance. Normal validation can run another enricher with its defaults, so use an explicit DynamicDefaultValueBehavior.NONE on a field when the opt-out must survive validation.

Examples of using DynamicDefaultValueBehavior
   new QFieldMetaData("createDate", QFieldType.DATE_TIME)
      .withBehavior(DynamicDefaultValueBehavior.CREATE_DATE);

   new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)
      .withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE);

   new QFieldMetaData("createdByUserId", QFieldType.STRING)
      .withBehavior(DynamicDefaultValueBehavior.USER_ID);
DateTimeDisplayValueBehavior

By default, in QQQ, fields of type DATE_TIME are stored in UTC, and their values in a QRecord is a java Instant instance, which is always UTC. However, frontends will prefer to display date-time values in the user’s local Time Zone whenever possible.

Using DateTimeDisplayValueBehavior allows a DATE_TIME field to be displayed in a different Time Zone. An example use-case for this would be displaying airplane flight times, where you would want a flight from California to New York to display Pacific Time for its departure time, and Eastern Time for its arrival.

An instance of DateTimeDisplayValueBehavior can be configured to either use a hard-coded time ZoneId (for example, to always show users UTC, or a business’s home-office time zone). Or, it can be set up to get the time zone to use from another field in the table.

Examples of using DateTimeDisplayValueBehavior
new QTableMetaData().withName("flights").withFields(List.of(
   new QFieldMetaData("departureTimeZoneId", QFieldType.STRING),
   new QFieldMetaData("arrivalTimeZoneId", QFieldType.STRING),

   new QFieldMetaData("departureTime", QFieldType.DATE_TIME)
      .withBehavior(new DateTimeDisplayValueBehavior()
         .withZoneIdFromFieldName("departureTimeZoneId")),

   new QFieldMetaData("arrivalTime", QFieldType.DATE_TIME)
      .withBehavior(new DateTimeDisplayValueBehavior()
         .withZoneIdFromFieldName("arrivalTimeZoneId")),

   new QFieldMetaData("ticketSaleStartDateTime", QFieldType.DATE_TIME)
      .withBehavior(new DateTimeDisplayValueBehavior()
         .withDefaultZoneId("UTC"))));
CaseChangeBehavior

A field can be made to always go through a toUpperCase or toLowerCase transformation, both before it is stored in a backend, and after it is read from a backend, by adding a CaseChangeBehavior to it:

The behavior also normalizes string filter operands, including unique-key and primary-key lookups. It uses Java’s default-locale case conversion. Read normalization changes the returned value without rewriting storage; caller-owned filters are preserved. Active normalization requires a string-like field (STRING, TEXT, HTML or PASSWORD). NONE preserves the value.

Examples of using CaseChangeBehavior
new QTableMetaData().withName("item").withFields(List.of(

   new QFieldMetaData("sku", QFieldType.STRING)
      .withBehavior(CaseChangeBehavior.TO_UPPER_CASE),

   new QFieldMetaData("username", QFieldType.STRING)
      .withBehavior(CaseChangeBehavior.TO_LOWER_CASE)));
WhiteSpaceBehavior

Whitespace policies apply on writes, reads and string filter operands, with the same string-like field requirement. TRIM uses Java String.trim, TRIM_LEFT and TRIM_RIGHT use stripLeading and stripTrailing, and REMOVE_ALL_WHITESPACE removes characters for which Character.isWhitespace is true. These Java operations have different Unicode whitespace rules. NONE preserves the value. Whitespace and case behaviors can be combined on a field; read normalization does not modify stored values.

Field Adornments

FieldAdornment objects (QFieldMetaData.withFieldAdornment(…​)) tell frontends how to present a field’s value: a type (an AdornmentType) plus a map of values whose keys depend on the type. They are passed through, as-is, in v1 table meta-data (fields.<name>.adornments).

FILE_PREVIEW

A FILE_PREVIEW adornment renders a field as a thumbnail (for images) or a file-type icon (for other files), wherever the frontend shows the value: query grids, record views, child-record lists and widgets. Unless previewable is false, clicking (or pressing Enter on) the thumbnail opens a preview: the full-size image with zoom, a PDF, a video or audio player, or a file icon with open and download links. The preview shows the file name, type, size and image dimensions when known, and the arrow keys step through the other previewable values on the same screen.

The adorned field’s own value is the default source for both the thumbnail and the full-size file. It can be a URL (an absolute https: URL, or a same-origin path), or, for a BLOB field that also has a FILE_DOWNLOAD adornment, the download URL the backend makes for it. Every value is optional. The …​Field values name other fields of the same table, and the frontend reads them from the same record, so they must be fields the record is queried with.

Value key (AdornmentType.FilePreviewValues) Meaning

thumbnailUrlField (THUMBNAIL_URL_FIELD)

Field holding a small rendition’s URL, used for thumbnails. Default: the adorned field.

fullSizeUrlField (FULL_SIZE_URL_FIELD)

Field holding the full-size (or original) URL, used in the preview and for open/download. Default: the adorned field.

fileNameField (FILE_NAME_FIELD)

Field holding the file name. Default: the display value, else the last segment of the URL.

mimeTypeField (MIME_TYPE_FIELD)

Field holding the MIME type, e.g. image/png, application/pdf, video/mp4.

defaultMimeType (DEFAULT_MIME_TYPE)

MIME type to assume without a mimeTypeField value, e.g. image/*. Without either, the type is guessed from the file extension.

altTextField (ALT_TEXT_FIELD)

Field holding the image’s alternative text. Default: the file name.

captionField (CAPTION_FIELD)

Field holding a caption, shown in the preview.

widthField, heightField (WIDTH_FIELD, HEIGHT_FIELD)

Fields holding the image’s pixel dimensions, shown in the preview and used to reserve the thumbnail’s aspect ratio.

sizeBytesField (SIZE_BYTES_FIELD)

Field holding the file size in bytes.

thumbnailMaxSize (THUMBNAIL_MAX_SIZE)

Largest edge of a thumbnail in lists, in CSS pixels, from 16 to 512. Default: 40.

previewable (PREVIEWABLE)

false shows the thumbnail without opening a preview. Default: true.

QInstanceValidator checks that each …​Field value names a field of the table, that thumbnailMaxSize is in range, that previewable is a boolean and defaultMimeType is type/subtype, and that a BLOB field also has a FILE_DOWNLOAD adornment or a thumbnailUrlField / fullSizeUrlField.

Frontends load these URLs into <img>, <video>, <audio> and <iframe> elements, so they must be allowed by the dashboard’s Content-Security-Policy. The default policy allows images from 'self', data:, blob: and any https: origin, but media and frames only from 'self'. A cross-origin PDF is therefore offered as open/download links instead of an inline preview, and cross-origin video or audio falls back to those links if the policy blocks it.

Thumbnails from a rendition field, and a full-size preview from the original
new QFieldMetaData("originalUrl", QFieldType.STRING)
   .withFieldAdornment(AdornmentType.FilePreviewValues.newFieldAdornment()
      .withValue(AdornmentType.FilePreviewValues.THUMBNAIL_URL_FIELD, "thumbnailUrl")
      .withValue(AdornmentType.FilePreviewValues.FILE_NAME_FIELD, "fileName")
      .withValue(AdornmentType.FilePreviewValues.MIME_TYPE_FIELD, "mimeType")
      .withValue(AdornmentType.FilePreviewValues.WIDTH_FIELD, "width")
      .withValue(AdornmentType.FilePreviewValues.HEIGHT_FIELD, "height")
      .withValue(AdornmentType.FilePreviewValues.SIZE_BYTES_FIELD, "fileSize")
      .withValue(AdornmentType.FilePreviewValues.CAPTION_FIELD, "caption"));

Possible Value Sources

A QPossibleValueSource supplies allowed values and display labels for a field. Register the source in the QInstance and refer to its name from the field metadata. Sources can read another table, use a fixed enum value list, or run custom lookup code; select the type that matches the data and its access requirements.

QPossibleValueSource

  • name is the unique source name; label is its human-readable label.

  • type selects TABLE, ENUM or CUSTOM.

  • idType describes the identifier field type.

  • valueFormat and valueFields control how the display label is built. valueFormatIfNotFound and valueFieldsIfNotFound provide missing-value formatting.

  • TABLE sources use tableName, with optional overrideIdField, searchFields and orderByFields.

  • ENUM sources use enumValues; PossibleValueEnum implementations provide getPossibleValueId() and getPossibleValueLabel().

  • CUSTOM sources use customCodeReference.

A source using another registered table
QPossibleValueSource source = QPossibleValueSource.newForTable("customer");

Use QPossibleValueSource.newForEnum(name, enumValues) for enums. See the source factories and properties for the complete contract. Query/Get callers must request possible-value translation when they need labels in record display values.

For TABLE sources, overrideIdField works for search results and direct label translation. Selected-value searches using idList currently still filter by the table primary key; an alternate string identifier over an integer primary key can fail conversion. This existing limitation is deferred under issue #544. Use a primary-key-based possible-value source for selected-value lookup until it is resolved.

Joins

A QJoinMetaData is a meta-data object that tells QQQ, essentially “it is possible for these 2 tables to join, here’s how to do it”.

Joins can be used then, in an application, in a number of possible ways:

  • In a QQQ Table, we can specify joins to be “exposed”, e.g., made available to users on a query screen

  • Also in a Table, as part of an “Association”, which sets up one table as a “parent” of another, such that you can store (and fetch) the child-records at the same time as the parent

    • A common use-case here may be an order & lineItem table - such that QQQ can generate an API uses to allow you to post an order and its lines in a single request, and they get stored all together

  • In defining the security field (record lock) on a table, sometimes, it isn’t a field directly on the table, but instead comes from a joined table (possibly even more than just 1 table away).

    • For example, maybe a lineItem table, doesn’t have a clientId, but needs secured by that field

      • so its recordLock can specify a “joinNameChain” that describes how to get from lineItem to order.clientId

  • The QueryAction can take (through its QueryInput object) zero or more QueryJoin objects, which must make a reference (implicitly or explicitly) to a QJoinMetaData. See the section on QueryJoins for more details.

QJoinMetaData

Joins are defined in a QQQ Instance in a QJoinMetaData object.

In this object, we have the concept of a "leftTable" and a "rightTable". There isn’t generally anything special about which table is on the "left" and which is on the "right". But the remaining pieces of the QJoinMetaData do all need to line-up with these sides.

For example:

  • The Type (one-to-one, one-to-many, many-to-one) - where the leftTable comes first, and rightTable comes second (e.g., a one-to-many means 1-row in leftTable has many-rows in rightTable associated with it)

  • In a JoinOn object, the 1st field name given is from the leftTable; the second fieldName from the rightTable.

QJoinMetaData Properties:

  • name - String, Required - Unique name for the join within the QQQ Instance.

    • One convention is to name joins based on (leftTable + "Join" + rightTable).

    • If you do not wish to define join names yourself, the method withInferredName() can be called (which defers to public static String makeInferredJoinName(String leftTable, String rightTable)), to create a name for the join following the (leftTable + "Join" + rightTable) convention.

  • leftTable - String, Required - Name of a QQQ Table in the QInstance.

  • rightTable - String, Required - Name of a QQQ Table in the QInstance.

  • type - enum, Required - cardinality between the two tables in the join.

    • e.g., ONE_TO_ONE, ONE_TO_MANY (indicating 1 record in the left table may join to many records in the right table), or MANY_TO_ONE (vice-versa).

    • Note that there is no MANY_TO_MANY option, as a many-to-many is built as multiple QJoinMetaData’s going through the intermediary (intersection) table.

  • joinOns - List<JoinOn>, Required - fields used to join the tables. Note: In the 2-arg JoinOn constructor, the leftTable’s field comes first. Alternatively, the no-arg constructor can be used along with .withLeftField().withRightField()

  • orderBys - List<QFilterOrderBy> - Optional list of order-by objects, used in some framework-generated queries using the join. The field names are assumed to come from the rightTable.

A join declaration describes the relationship; it does not by itself fetch joined fields or guarantee that related writes are atomic. Configure the query joins or associations that use it and verify transaction support in the selected backend. See QueryJoin for query examples.

Security Key Types

In QQQ, record-level security is provided by using a lock & key metaphor.

The use-case being handled here is:

  • A user has full permission on a table (query, insert, update, and delete).

  • However, they should only be allowed to read a sub-set of the rows in the table.

    • e.g., maybe it’s a multi-tenant system, or the table has user-specific records.

The lock & key metaphor is realized by the user being associated with one or more "Keys" (as values in their session), and records in tables being associated with one or more "Locks" (as values in fields). In this default READ_AND_WRITE example, access requires satisfying the record’s applicable security locks. A lock can instead apply only to reads or writes; see the lockScope properties under QQQ Table. Table permissions remain independent.

For a practical example, picture a multi-tenant Order Management System,where all orders are assigned to a "client". Users (customers) should only be able to see orders associated with the client which that user works for.

In this scenario, the order table would have a "lock" on its clientId field. Customer-users would have a clientId key in their session. When the QQQ backend did a search for records (e.g., an SQL query) it would implicitly (without any code being written by the application developer) filter the table to only allow the user to see records with their clientId.

To implement this scenario, the application would define the following pieces of meta-data:

  • At the QQQ-Instance level, a SecurityKeyType, to define a domain of possible locks & keys within an application.

    • An application can define multiple Security Key Types. For example, maybe clientId and userId as key types.

  • At the per-table level, a RecordSecurityLock, which references a security key type, and how that key type should be applied to the table.

    • For example, what field stores the clientId value in the order table.

  • Finally, when a user’s session is constructed via a QQQ Authentication provider, security key values are set, based on data from the authentication backend.

Additional Scenarios

All Access Key

A Security Key Type’s allAccessKeyName satisfies locks of that type when the session contains the named key with a value of true. This satisfies that lock even as an alternative within an OR group; other required lock branches and table permissions still apply.

Null Value Behaviors

In a record security lock, different behaviors can be defined for handling rows with a null key value.

For example:

  • Sometimes orders may be loaded into the OMS system described above, where the application doesn’t yet know what client the order belongs to. In this case, the application may need to ensure that such records, with a null value in clientId are hidden from customer-users, to avoid potentially leaking a different client’s data.

    • This can be accomplished with a record security lock on the order table, with a nullValueBehavior of DENY.

    • Furthermore, if internal/admin users should be given access to such records, then the security key type can be configured with a nullValueBehaviorKeyName (e.g., "clientIdNullValueBehavior"), which can be set per-user to allow access to records, overriding the table lock’s specified nullValueBehavior.

      • This could also be done by giving internal/admin users an allAccessKey, but sometimes that is not what is required.

  • Maybe a warehouse locations table is assigned a clientId once inventory for a client is placed in the location, at which point in time, only the client’s users should be allowed to see the record. But, if no client has been assigned to the location, and clientId is null, then you may want to allow any user to see such records.

    • This can be accomplished with a record security lock on the Warehouse Locations table, with a nullValueBehavior of ALLOW.

QSecurityKeyType

A Security Key Type is defined in a QQQ Instance in a QSecurityKeyType object.

QSecurityKeyType Properties:

  • name - String, Required - Unique name for this security key type within the QQQ Instance.

  • allAccessKeyName - String - Optional name of the all-access security key associated with this key type.

  • nullValueBehaviorKeyName - String - Optional name of the null-value-behavior overriding security key associated with this key type.

    • Note, name, allAccessKeyName, and nullValueBehaviorKeyName are all checked against each other for uniqueness. A QInstanceValidationException will be thrown if any name collisions occur.

  • possibleValueSourceName - String - Optional reference to a possible value source from which value for the key can come.

Reports

QQQ can generate reports based on QQQ Tables defined within a QQQ Instance. Users can run reports, providing input values. Alternatively, application code can run reports as needed, supplying input values.

Report data access

ReportInput defaults to SYSTEM. Developer-defined named reports and built-in saved-report generation retain their trusted source behavior. The direct HTTP route checks the named report permission; processes can use ReportProcessPermissionChecker for the selected report. Report permission can expose approved output without granting raw-table READ.

The report input source is not uniformly propagated into nested count/query inputs. Applications can set query policy through DataSourceQueryInputCustomizer; the returned input is respected. Automatic source propagation, additional source permissions and forced post-customizer privacy are deferred in #448. Custom record sources must authorize their own data; static suppliers remain trusted application code.

Table exports

ExportAction exports table data as XLSX, JSON, CSV or TSV. LIST_OF_MAPS is a test utility with static, process-wide storage; it is not a concurrent production destination. Table exports add a label column for fields with a possible-value source.

ReportFormat.fromString retains case-insensitive parsing of all enum values, including the test-only LIST_OF_MAPS; null, blank and unknown names are rejected. File HTTP routes can fail with a server error for that internal format; validation placement is deferred in #490. The static map collector requires headers; header-free support is deferred in #489. XLSX, JSON, CSV and TSV support header-free output. A zero-row limit produces a readable empty export. CSV data values double embedded quotes and replace line feeds with spaces. TSV data values use backslash escapes for tabs, line feeds, carriage returns and backslashes; JSON and XLSX preserve line feeds and tabs in field values. Header/title escaping remains a known limitation tracked in #471.

The standard streamed POI XLSX implementation preserves numeric cell values but does not implement dynamic per-view numeric DisplayFormat styling. Explicit TABLE report-column format overrides are also incomplete. That presentation support, including DEFAULT clearing and style reuse, is deferred in #472. Existing date/time and template styling behavior remains.

The HTTP table-export route checks base-table READ. USER exports also check READ on selected and filter/order-referenced joined tables before preflight counting and again before query execution; an earlier preflight does not cache this authorization. This closes #493. Direct Java callers remain responsible for base-table authorization. Broader core authorization and private-field predicate restrictions are deferred in #494. Existing query output privacy and explicit REVEAL remain; hidden header omission is deferred in #495. Explicit SYSTEM calls retain trusted behavior.

Exports retain the original limit policy: ExportInput.limit replaces the query-filter limit, including null. An existing caller filter is shared and its limit is modified; its offset remains. Filter isolation, query-only limits, generic action-reuse revalidation and effective-window counting are deferred in #488. Use a fresh export action for independent requests.

Spreadsheet preflight retains the backend count interface and original maximum-data-count check. It can reject an export by counting rows outside USER personalization; that correction is deferred in #496. It does not adjust for the effective skip/limit window or subtract title/header rows. Window corrections remain #488; title/header capacity and physical writer guards remain #465. Preflight and generation are not a transactional snapshot.

Direct HTTP report downloads

Direct /reports/{reportName} downloads retain named-report permission checks and the trusted default for server-defined sources. Responses stream while the report is generated. A failure after delivery starts can leave partial output; callers should discard incomplete reports. Completed-response buffering is deferred in #497. This is separate from saved-report generation before archive storage, which protects existing archive bytes.

Report counts and progress

The existing preliminary count for each table-backed TABLE view supplies progress totals and checks the selected format’s data-row limit. ReportOutput.totalRecordCount sums those counts by data-source name. It is not a guaranteed count of emitted rows: query customizers, pagination, view transforms, custom/static sources and summary-only reports can differ. Counts do not include layout rows. Use a fresh GenerateReportAction for each report.

The proposed processed-row definition, count-state reset, transformed-row guard and query-preparation changes are deferred in #464. Query customizers retain their existing execution after transform setup, inside the asynchronous producer. This release does not require applications to move transform-provided query setup.

Summary source and totals

Known summary limitations remain in this release. When a source also has TABLE views, summary collection runs during each table pass and receives that table’s transformed records. Multiple table views can duplicate summary aggregates, and a table-only transform can change summary inputs. Source-isolation corrections are deferred in #475.

Base and variance total maps are shared across views, so reports with unrelated summary sources can combine their totals (#476). Total inputs are collected only when a relevant view requests a displayed total row; hiding that row can leave ${total.*} formulas without their required values (#477). These Medium fixes and their new tests are preserved for a future release; there is no new snapshot or transaction guarantee.

Summary ordering retains its existing flat sort, including when includeSummarySubTotals is true. With multiple grouping fields, subtotals can be separated from their related rows or precede them. Hierarchical ordering and guaranteed children-before-subtotal placement are deferred in #486. The grand total stays last; this ordering limitation does not by itself establish incorrect aggregate arithmetic.

JSON column names

JSON output normalizes column labels by removing spaces and converting the leading acronym or capital to lower case, for example First Count becomes firstCount. A missing label uses the field name unchanged. Normalized labels are cached by field name, so a later view using the same field with a different label can reuse the earlier key. A blank label can produce an empty key. Per-view label caching and a blank-label fallback are deferred in #487. Use consistent, nonblank labels and distinct resulting keys.

Exporter lifecycle and spreadsheet bounds

ExportStreamerInterface retains its existing finish() contract for final writing and resource closure; no AutoCloseable or new cleanup hook is required in 4.0. Destination closure depends on the implementation. The standard POI exporter closes its wrapped destination during successful finalization and may close it on record failure; actions also close the destination on success. Direct-stream callers must discard failed output and follow their storage backend’s abort/cleanup requirements. Exported bytes cannot be rolled back by the exporter. Saved reports finish local generation before opening archive storage as described below.

The existing table-export loop can finish successfully after a producer/query failure (#473). Report producer cleanup after consumer failure is deferred in #474; the original shared loop still propagates producer errors, flushes final buffered records and supports its existing record-limit cancellation. Parent-to-child cancellation and new finalization boundaries are deferred in #479. Applications must handle cooperative cancellation explicitly and discard partial direct output after failures. The 4.0 release introduces no new loop callback API or guaranteed child cleanup.

XLSX worksheets support at most 1,048,576 rows and 16,384 columns. Titles, headers, data and totals share that row budget. Existing preflight checks do not guarantee that every physical streamed row/cell fits, especially when records change or transforms expand output. Physical writer guards are deferred in #465. Destination-ownership rules and a common cleanup API are deferred in #466 and #467; header-cause diagnostics and allocation cleanup are deferred in #468 and #469.

Saved-report storage and failures

Saved reports generate into a temporary local file and close it before opening archive storage through StorageAction.createOutputStream. Size the runtime’s temporary disk for concurrent report outputs. A generation failure removes the temporary file without creating or replacing the archive. A completed local file is copied through the existing storage stream API; custom storage backends do not need a new method.

Opening, copying or closing the destination still follows the selected backend’s streaming behavior. A failure during that stage can leave partial output or replace an earlier object. No atomic replacement or transaction across storage, history and email is promised. Stronger publication guarantees are deferred in #459.

New cancellation checks before storage, download registration, public access and email delivery are deferred in #479. Operations already in progress are not rolled back. The existing render-and-deliver lifecycle marks caught failures as FAILED when its history update succeeds, even if archive bytes were already written. History validation/error-handling improvements and a separate rendering-only status remain deferred in #460 and #461.

Requested email requires trusted sender/provider configuration; absent configuration is an error. Local filesystem download URLs are encoded file URIs, and email HTML escapes both the link and visible filename. A local file URI is usable only where that filesystem is accessible; use a suitable storage/download service for remote recipients.

Native pivot sources and customization

The default XLSX implementation emits native pivot definitions and requests refresh on load. A supporting spreadsheet application calculates the pivot results; QQQ does not precompute native pivot cells. Use SUMMARY views for values calculated during report generation.

Ordinary and empty TABLE-backed native pivots retain their existing behavior, including header-only ranges for empty sources. Known source-range, cache-XML, SUMMARY-source indexing/rendering and instance-validation limitations are tracked in #484. A title can enter the source range, reserved XML characters in labels can break cache XML, SUMMARY sources can fail, and the validator can demand unrelated columns on a PIVOT view. Pivot sources reaggregate emitted rows; total or subtotal rows can therefore double count when included.

View customization retains its existing TABLE-only timing after output preparation. Generalized customization before template creation is deferred in #482; column objects remain shared by view clones (#483). Early checks for incomplete runtime metadata are deferred in #485. Existing unknown pivot-source lookup still rejects a missing source view. CSV, TSV and JSON omit native PIVOT views and render the supported views.

QReportMetaData

Reports are defined in a QQQ Instance with a QReportMetaData object. Reports are defined in terms of their sources of data (QReportDataSource), and their view(s) of that data (QReportView).

QReportMetaData Properties:

  • name - String, Required - Unique name for the report within the QQQ Instance.

  • label - String - User-facing label for the report, presented in User Interfaces. Inferred from name if not set.

  • processName - String - Name of a QQQ Process used to run the report in a User Interface.

  • inputFields - List of QQQ Field - Optional list of fields used as input to the report.

    • The values in these fields can be used via the syntax ${input.NAME}, where NAME is the name attribute of the inputField.

    • For example:

// given this inputField:
new QFieldMetaData("storeId", QFieldType.INTEGER)

// its run-time value can be accessed, e.g., in a query filter under a data source:
new QFilterCriteria("storeId", QCriteriaOperator.EQUALS, List.of("${input.storeId}"))

// or in a report view's title or field formulas:
.withTitleFields(List.of("${input.storeId}"))
new QReportField().withName("storeId").withFormula("${input.storeId}")

  • dataSources - List of QReportDataSource, Required - Definitions of the sources of data for the report. At least one is required.

  • views - List of QReportView, Required - Definitions of the report output views. At least one is required.

QReportDataSource

Data sources for QQQ Reports can either reference QQQ Tables within the QQQ Instance, or they can provide custom code in the form of a CodeReference to a Supplier, for use cases such as a static data tab in an Excel report.

QReportDataSource Properties:

  • name - String, Required - Unique name for the data source within its containing Report.

  • sourceTable - String, Conditional - Reference to a QQQ Table in the QQQ Instance, which the data source queries data from.

  • queryFilter - QQueryFilter - If a sourceTable is defined, then the filter specified here is used to filter and sort the records queried from that table when generating the report.

  • queryJoins - List of QueryJoin - Joins used by table-backed queries.

  • queryInputCustomizer - QCodeReference - Implements DataSourceQueryInputCustomizer to customize each prepared query input. The returned query input retains the policy selected by the application customizer.

  • customRecordSource - QCodeReference - Implements ReportCustomRecordSourceInterface to supply records through the report pipe.

  • staticDataSupplier - QCodeReference, Conditional - Reference to custom code which can be used to supply the data for the data source, as an alternative to querying a sourceTable.

    • Must be a JAVA code type

    • The referenced class must implement the interface: Supplier<List<List<Serializable>>>. Each inner list supplies one row, mapped to column0, column1, and so on.

QReportView

Report Views control how the source data for a report is organized and presented to the user in the output report file. If a DataSource describes the rows for a report (e.g., what table provides what records), then a View may be thought of as describing the columns in the report. A single report can have multiple views, specifically, for the use-case where an Excel file is being generated, in which case each View creates a tab or sheet within the xlsx file.

QReportView Properties:

  • name - String, Required - Unique name for the view within its containing Report.

  • label - String - Used as a sheet (tab) label in Excel formatted reports.

  • type - enum of TABLE, SUMMARY, PIVOT. Required - Defines the type of view being defined.

    • TABLE views are a simple listing of the records from the data source.

    • SUMMARY views are essentially pre-computed Pivot Tables. That is to say, the aggregation done by a Pivot Table in a spreadsheet file is done by QQQ while generating the report. In this way, a non-spreadsheet report (e.g., JSON or CSV) can have summarized data, as though it were a Pivot Table in a live spreadsheet.

    • PIVOT views produce native pivot tables with an output format that supports them, including the Excel POI implementation. Define pivotTableSourceViewName to identify the source view. Formats without native pivot support do not render a native PIVOT view; use SUMMARY for computed aggregate data. See ReportFormat capabilities.

  • dataSourceName - String, Required for TABLE and SUMMARY - Reference to a DataSource within the report, that is used to provide the rows for the view.

  • varianceDataSourceName - String - Optional reference to a second DataSource within the report, that is used in SUMMARY type views for computing variances.

    • For example, given a Data Source with a filter that selects all sales records for a given year, a Variance Data Source may have a filter that selects the previous year, for doing comparissons.

  • summaryFields - List of String, Conditional - For SUMMARY views, specify the source fields used to group rows. These fields appear before the calculated columns in the output.

    • For example, in a summary view of orders, you may "pivot" on the customerId field, to produce one row per-customer, with aggregate data for that customer.

  • pivotTableSourceViewName - String, Required for PIVOT - Name of the source view whose worksheet supplies pivot data. Ordinary TABLE sources are supported; SUMMARY-source limitations are tracked in #484.

  • pivotTableDefinition - PivotTableDefinition, Required for PIVOT - Defines rows and columns using PivotTableGroupBy, and values using PivotTableValue with a field name and aggregation function.

  • titleFormat - String - Java Format String, used with titleFields (if given), to produce a title row, e.g., first row in the view (before any rows from the data source).

  • titleFields - List of String, Conditional - Used with titleFormat, to provide values for any format specifiers in the format string. Syntax to reference a field (e.g., from a report input field) is: ${input.NAME}, where NAME is the name attribute of the inputField.

    • Example of using titleFormat and titleFields:

// given these inputFields:
new QFieldMetaData("startDate", QFieldType.DATE)
new QFieldMetaData("endDate", QFieldType.DATE)

// a view can have a title row like this:
.withTitleFormat("Weekly Sales Report - %s - %s")
.withTitleFields(List.of("${input.startDate}", "${input.endDate}"))

  • includeHeaderRow - boolean, default true - Indication that first row of the view should be the column labels.

    • If true, then header row is put in the view.

    • If false, then no header row is put in the view.

  • includeTotalRow - boolean, default false - Indication that a totals row should be added to a SUMMARY view. Each column formula is evaluated against the total aggregates; a ratio is recomputed from those aggregates rather than summed from displayed rows.

    • If true, then totals row is put in the view.

    • If false, then no totals row is put in the view.

  • includeSummarySubTotals - boolean, default false - For a SUMMARY view with several summaryFields, include subtotal rows for higher grouping levels.

  • columns - List of QReportField - Definition of the columns to appear in the view. Instance validation requires columns or a view customizer, including for native PIVOT views; that validation limitation is tracked in #484. Only TABLE views currently run the customizer. See section on QReportField for details.

  • orderByFields - List of QFilterOrderBy, optional - For a SUMMARY view, how to sort computed rows.

  • recordTransformStep - QCodeReference, subclass of AbstractTransformStep - Custom code reference for TABLE views that transforms records after they are queried from the data source and before output. When a source also has TABLE views, SUMMARY collection receives records during those table passes, including their transforms; source isolation is deferred in #475. Can be used to transform or customize values, or to look up additional values to add to the report.

  • viewCustomizer - QCodeReference, implementation of interface Function<QReportView, QReportView> - Custom code reference that can be used to customize the report view, at runtime. Runs for TABLE views after grouping and output preparation. It can customize that view’s columns and labels, but template/pivot references may already use the declared metadata. The view clone shares column objects, so mutating an existing column can change the declared metadata. Generalized timing and column isolation are deferred in #482 and #483. Implement ReportViewCustomizer when the customizer needs ReportInput.

QReportField

Report Fields define the fields (AKA columns) of data that appear in a report view. These fields can either be direct references to fields from the report’s data sources, or values computed using formula defined in the QReportField.

QReportField Properties:

  • name - String, required - Unique identifier for the field within its ReportView. In general, will be a reference to a field from the ReportView’s DataSource unless a *formula is given (for SUMMARY type views), the field is marked as isVirtual, or the field is marked as showPossibleValueLabel).

  • label - String - Optional text label to identify the field, for example, in a header row. If not given, may be derived from field, where possible.

  • type - QFieldType

  • formula - String, conditional - Required for SUMMARY type views. Defines the formula to be used for computing the value in this field. Function expressions start with =; thisRow variables refer to earlier calculated columns. DIVIDE and DIVIDE_SCALE return null for a zero denominator.

    • For example:

.withName("reportEndDate").withFormula("${input.endDate}")

.withName("count").withFormula("${pivot.count.id}")

.withName("percentOfTotal").withFormula("=DIVIDE(${pivot.count.id},${total.count.id})")

.withName("sumCost").withFormula("${pivot.sum.cost}")

.withName("sumCharge").withFormula("${pivot.sum.charge}")

.withName("profit").withFormula("=MINUS(${pivot.sum.charge},${pivot.sum.cost})")

.withName("totalCost").withFormula("=DIVIDE_SCALE(${pivot.sum.cost},${pivot.count.id},2)")

.withName("revenuePer").withFormula("=DIVIDE_SCALE(${pivot.sum.charge},${pivot.count.id},2)")

.withName("marginPer").withFormula("=MINUS(DIVIDE_SCALE(${pivot.sum.charge},${pivot.count.id},2),DIVIDE_SCALE(${pivot.sum.cost},${pivot.count.id},2))")

.withName("thisWeekMargin").withFormula("=SCALE(DIVIDE(${thisRow.profit},${pivot.sum.charge}),2)")

.withName("previousWeekProfit").withFormula("=MINUS(${variancePivot.sum.charge},${variancePivot.sum.cost})")

.withName("previousWeekMargin").withFormula("=SCALE(DIVIDE(${thisRow.previousWeekProfit},${variancePivot.sum.charge}),2)")

.withName("marginThisVsPrevious").withFormula("=SCALE(MINUS(${thisRow.thisWeekMargin},${thisRow.previousWeekMargin}),3)")

.withName("exception").withFormula("""
   =IF(LT(${thisRow.thisWeekMargin},0),Negative Margin,IF(LT(${thisRow.marginThisVsPrevious},0),Margin Decreased,""))""")

  • displayFormat String - Requested presentation format. Streamed POI numeric styling and explicit TABLE source-field overrides remain incomplete; see #472.

  • isVirtual Boolean, default false - Use the report column’s own metadata instead of resolving a table field. Supply its type and label. Static supplier columns use positional names column0, column1, and so on.

  • showPossibleValueLabel Boolean, default false - To show a translated value for a Possible Value field (e.g., a name or other value meaningful to a user, instead of a foreign key).

  • sourceFieldName String - Used for the scenario where a possibleValue field is included in a report both as the foreign key (raw, id value), and the translated "label" value. In that case, the field marked with showPossibleValueLabel = true should be given a different name, and should use sourceFieldName to indicate the field that has the id value.

    • For example:

// this field would have the "raw" warehouseId values
// e.g., integers - foreign keys to a warehouse table.  Generally useful for machines to know.
new QReportField("warehouseId")
   .withLabel("Warehouse Id"),

// this field would have the translated values from the warehouse PossibleValueSource
// for example, maybe the name field from the warehouse table.  A string, useful for humans to read.
new QReportField("warehouseName")
   .withSourceFieldName("warehouseId")
   .withShowPossibleValueLabel(true)
   .withLabel("Warehouse Name"),

Icons

A QIcon supplies an icon name or image path to metadata such as apps, tables, fields and widgets.

QIcon

Icons are defined in a QQQ Instance in a QIcon object.

For example, new QIcon().withName("table_bar") selects a named icon. Rendering and available icon names depend on the frontend.

QIcon Properties:

  • name - String - Name of an icon from the Material UI Icon set

    • Note that icon names from the link above need to be converted from CamelCase to underscore_case…​

  • path - String - Path to a file served under the application’s web root, to be used as the icon.

Either name or path must be specified. If both are given, then name is used.

Permission Rules

QPermissionRules controls access to metadata objects such as tables and processes. Authentication identifies a session; permission rules determine which operations that session may perform. Instance defaults and object-specific rules are combined by PermissionsHelper. Configure and test both allowed and denied operations rather than treating a visible menu item as an authorization check.

QPermissionRules

  • level selects NOT_PROTECTED, HAS_ACCESS_PERMISSION, READ_WRITE_PERMISSIONS, or READ_INSERT_EDIT_DELETE_PERMISSIONS.

  • denyBehavior is HIDDEN or DISABLED for presentation of denied metadata.

  • permissionBaseName customizes the base used to construct permission names.

  • customPermissionChecker references an implementation of CustomPermissionChecker for application-specific checks.

An object rule requiring separate read/insert/edit/delete permissions
QPermissionRules rules = new QPermissionRules()
   .withLevel(PermissionLevel.READ_INSERT_EDIT_DELETE_PERMISSIONS)
   .withDenyBehavior(DenyBehavior.HIDDEN);

Attach the rule to the relevant metadata object and supply the matching permissions in the session. QPermissionRules.defaultInstance() uses NOT_PROTECTED and HIDDEN; it is not a secure application-wide policy by itself. Record-level security keys are a separate part of access control.

Joined HTTP reads

The legacy and V1 Query/Count endpoints require READ for the base table and every table requested through joins, qualified filter fields or ordering. Aliases and intermediate tables in a join path do not bypass this check. Legacy Get applies the same rule to requested joins and associated-record expansion, including nested associations. Making a join exposed or granting WRITE does not grant READ. Denied requests return HTTP 403 without records or counts. A join added solely to enforce a record-security lock remains internal to that enforcement; a client who also requests its data must have READ permission.

These checks apply at the HTTP boundary. Trusted core actions still support internal application operations without automatically requiring the public endpoint’s table grants. This distinction does not replace authentication or record-level security.

HTTP Query/Count inputs use QInputSource.USER, and associated-record queries inherit their parent’s input source. A personalizer that removes fields for USER reads therefore applies to associated children and grandchildren as well. Association authorization follows the personalized association metadata; removed associations are not expanded.

If personalization fails while validating a joined filter field, the filter is rejected. An error must not restore the table’s unrestricted field definitions.

See QPermissionRules and permission evaluation and permission-name construction.

Permission Enforcement in Core Actions

Before QQQ 4.1.0-RC.4, table, process, widget and report permissions were checked only by the middlewares (Javalin, the application API, picocli, Slack), before they called a core action. Any other entry point - an MCP server, an AI assistant, a custom route, a script - that called QueryAction, InsertAction, RunProcessAction and so on directly skipped those checks (#1061). FieldSecurityLock was validated at startup but never enforced (#1062).

The core actions can now enforce these themselves.

Which actions

Action What is checked when enforcement applies

QueryAction, GetAction, CountAction, AggregateAction

Table READ; READ on every joined table (explicit joins, joins implied by table.field criteria, exposed-join paths); no criteria, sort, group-by, aggregate, unique-key or fieldNamesToInclude reference to a field denied by a FieldSecurityLock. Denied fields are removed from the output (values and display values, including joined and associated records).

InsertAction / UpdateAction / DeleteAction

Table INSERT / EDIT / DELETE. Insert and update reject records that carry a value (even null) for a denied field; delete rejects a filter on one, and its filter needs READ on any joined table. Associated child records need the child table’s permissions too (insert: INSERT; update: INSERT/EDIT for supplied children, DELETE for omitted ones; delete: DELETE on cascaded children), and their field locks apply. Denied fields are removed from the output records.

ReplaceAction

INSERT and EDIT (plus DELETE when performDeletes is true), and the insert/update field rules.

ExportAction

As for query; locked fields are left out of the columns.

RunProcessAction

The process permission (custom permission checkers receive the process values).

RenderWidgetAction / GenerateReportAction

The widget / report permission, for widgets and reports defined in the instance (the same meta data object). Caller-supplied meta data is refused under ENFORCE; with the instance setting on it runs without the trusted scope, so each of its queries is checked.

TableMetaDataAction

Fields denied by a FieldSecurityLock are omitted from the table meta data, so frontends hide them.

Record security locks are unchanged: they were already enforced inside the backend modules and DML actions, for every call.

When enforcement applies

Each action input has a PermissionEnforcement:

  • DEFAULT (the default) - enforced only for an outermost call (one not running inside another core action), and only when the instance setting is on.

  • ENFORCE - always enforced, even for a nested call and even when the instance setting is off. Use it from any code that acts for a user outside the middlewares: MCP servers, AI assistants, custom routes.

  • SKIP - never enforced. It can only be set with input.withPermissionBypass("reason"); the reason is required and logged.

// an assistant or MCP tool acting for the current user
QueryInput queryInput = new QueryInput("order").withFilter(filter);
queryInput.setPermissionEnforcement(PermissionEnforcement.ENFORCE);
new QueryAction().execute(queryInput);

// a deliberate, auditable bypass for one internal lookup
new QueryInput("configuration").withPermissionBypass("read app configuration at startup");

// a block of trusted application work (e.g., a public route serving published content)
CorePermissionEnforcer.runTrusted("public site API: published posts", () -> new QueryAction().execute(input));

Calls nested inside a core action are framework-internal and trusted: process steps, table customizers, audits, record-change listeners, possible-value translation, widget renderers, report data sources, and async work they start (the trusted scope is carried by QContext.capture()). This keeps QQQ’s existing model - a user who may run a process can have that process read or write tables the user can’t access directly.

Sessions of type QSystemUserSession pass every check (as before). The framework’s own scheduler, polling automations and SQS queue poller also mark their process runs with withPermissionBypass. The ESB keeps its explicit process-permission check for its run-as user.

Possible-value lookups (SearchPossibleValueSourceAction, QPossibleValueTranslator) are trusted, because a possible-value label is visible to anyone who can see the field that references it (unchanged behavior). Under ENFORCE, a caller-supplied filter on a possible-value search still may not use locked fields of the possible-value source’s table. A record label built from a locked field is cleared.

The instance setting

qInstance.setEnforcePermissionsInCoreActions(true);

If the instance doesn’t set it, the system property qqq.enforcePermissionsInCoreActions or the environment variable QQQ_ENFORCE_PERMISSIONS_IN_CORE_ACTIONS (true/false) is used; otherwise it is off. In QQQ 4.1.x it is off by default: only inputs that request ENFORCE are checked, so existing applications see no change. The default is planned to become on in QQQ 4.2; set it explicitly to choose.

When the setting is on, the middlewares' own checks still run first and give the same answers, so allowed requests behave as before and denied requests get the same 403. What changes for applications:

  • Code that calls core actions directly - custom routes, scripts, scheduled jobs that run with a non-system session - is now subject to the session’s permissions. Run such code as the system user, or mark it with withPermissionBypass / CorePermissionEnforcer.runTrusted where the bypass is intended.

  • Fields with a FieldSecurityLock are hidden from users the lock denies (they were visible before). Check existing locks before turning the setting on: their default behavior is DENY.

  • A filter or sort on a locked field returns 403 instead of results.

Field security locks

A FieldSecurityLock on a field names a security key type, a default behavior (DENY by default) and override values. For the current session, the lock’s default behavior applies unless the session has one of the override values for that key type, which flips it. A session with the key type’s all-access key, and the system user session, can always access the field. A lock on a table’s primary key is ignored.

new QFieldMetaData("salary", QFieldType.DECIMAL)
   .withFieldSecurityLock(new FieldSecurityLock()
      .withSecurityKeyType("clearance")
      .withDefaultBehavior(FieldSecurityLock.Behavior.DENY)
      .withOverrideValues(List.of("hr")));

Sensitive fields

QFieldMetaData.withIsSensitive(true) marks a field whose values should not leave the application in bulk or go to third parties (PII, secrets). It does not restrict reads in the UI or API (use a FieldSecurityLock for that).

  • ExportAction leaves sensitive fields out unless ExportInput.withIncludeSensitiveFields(true).

  • Frontend field meta data includes isSensitive, so frontends can mask values.

  • AI assistant and MCP integrations must not send sensitive values to a model.

Dry runs

InsertInput, UpdateInput and DeleteInput have withDryRun(true). A dry run applies permissions (when enforced), runs the pre-action customizers with isPreview = true, and runs all validations (required fields, unique keys, record security locks, record existence), but stores nothing: no backend write, association changes, audits, record-change listeners or post-action customizers.

  • Insert returns the would-be records (no generated keys) with their errors and warnings.

  • Update returns each would-be record: the old record with the input’s values applied (when the backend can fetch old records), with errors and warnings.

  • Delete returns DeleteOutput.dryRunPrimaryKeys (a filter is always resolved to keys) plus recordsWithErrors / recordsWithWarnings; deletedRecordCount stays 0. A dry run does not validate the cascading delete of associated child records.

Pre-action customizers that have side effects should check isPreview.

Classified Value Tokens

PII and PHI values never appear in core audit details. Effective classification comes from AiDataClassifications, or an instance’s AuditValueProtectorInterface. Unclassified audit details keep their existing behavior.

Formats and binding

The UTF-8 MAC input is table + "\u001F" + field + "\u001F" + String.valueOf(pk) + "\u001F" + canonical(value). Context tokens join the supplied binding parts and canonical value with the same separator; the first part is a nonempty domain, for example email. Context binding parts cannot contain the separator. No additional prefix is added to the agreed record binding.

  • hmac:v1:<keyId>:<mac>: HMAC-SHA256 with the instance key.

  • shred:v1:<keyRef>:<mac>: HMAC-SHA256 with the record-field key.

  • changed:v1: no key or an application policy that stores no derivative.

  • wiped:v1:<ruleId>: the new value for fields changed by an update carrying WipeAuditFlag.

  • shredded:v1:<ruleId>: a separate key-destruction detail written by AuditShreddedEvent.shreddedDetail.

MACs use base64url without padding. Identifiers use ASCII letters, digits, dot, underscore or hyphen; complete tokens stay below 200 characters. Null values stay absent. Canonical strings use ValueUtils, with BigDecimal trailing zeros removed and plain notation; Instant uses ISO-8601 UTC, dates ISO and booleans lowercase. Binding values are case-sensitive and are not trimmed.

Keys and verification

Register a ClassifiedValueKeySource using ClassifiedValueKeySourceMetaData.ofOrWithNew(instance).withKeySourceInstance(source) (or withKeySource(QCodeReference)). Its nested KeyAndId holds key bytes and ID. Record tokens prefer keyForRecordField(table, field, pk, true), then instanceKey(), then changed:v1. Context tokens use only the instance key. The default reads base64 QQQ_AUDIT_HMAC_KEY and optional QQQ_AUDIT_HMAC_KEY_ID (default env1) through the metadata variable interpreter. Same-named Java properties support tests. Secret values are never logged. Missing keys produce one warning on first token use.

verifyRecordField(table, field, pk, candidate, token) and verifyContext(parts, candidate, token) compare complete keyed tokens in constant time. Marker tokens and null candidates never verify. Shred verification requests create=false: a destroyed key is never recreated and there is no instance-key fallback for a shred token. The current SPI exposes one instance key; verification of historical rotated instance keys requires selecting the corresponding key source externally. Keep verification actions access-controlled in the application.

Register audit policy through AuditValueProtectionMetaData.ofOrWithNew(instance).withProtectorInstance(protector). The protector returns tokens; core constructs the value-free Set/Changed/Removed message. Wipe updates omit old values entirely. Pass the key-destruction transaction to the five-argument AuditShreddedEvent.shreddedDetail overload so key destruction and its audit commit together.

Logs and events

QLogger.registerRedactor / unregisterRedactor maintain a thread-safe registry. Redactors run on string log values and completed messages, including formatted parameters and exception text. QRecord log rendering masks classified values independently of user tier and omits record labels, display values and backend metadata. Records with unknown table metadata are fully masked. Ordinary API JSON serialization is unchanged; use QRecord log pairs for record logging.

ESB table publications omit classified values by default, including old values and classified primary-key subjects. UPDATE payloads retain names in changedFields. EsbTablePublication.withIncludeClassified(true) explicitly opts a destination in. If the same destination has multiple matching registrations, all must opt in. Outbound webhook consumers inherit the default safe payload.

Shared test vectors

Test key (bytes 00 through 1f), base64: AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8=. This is a public fixture, never a production key.

Input Expected token

Record: patient / systolic / 42 / BigDecimal("120.00")

hmac:v1:env1:YJ7YulFxAOci7Cm0pX0fOAZwmcEVTzBXaAQrmFD2fk8

Same record and value, record-field key ID dek42

shred:v1:dek42:YJ7YulFxAOci7Cm0pX0fOAZwmcEVTzBXaAQrmFD2fk8

Context: email / welcome / recipient / message-7 / [email protected]

hmac:v1:env1:p1BggBbb8ytFgS5YeM6eBiriGgC1j3x0YGjics5qVeM

Record: patient / systolic / 42 / Instant("2026-10-13T12:34:56Z")

hmac:v1:env1:y5go1-Gynl0OjSDFoDaUiXNTWm6ci_3IHOn7HBxuc4k

Record Picker for Large Possible Value Sources

A possible-value field whose source is a large table (100,000+ media, products, customers) needs more than an inline dropdown. A table-type QPossibleValueSource can declare a QPossibleValuePicker. Every field that uses the source then offers a searchable, filterable, paged picker dialog in the frontend, with thumbnails where the table has images. The inline typeahead stays for quick picks.

Layers

Layer Responsibility

QQQ core

QPossibleValuePicker metadata on the source; picker requests in SearchPossibleValueSourceAction (search, allow-listed filters and sorts, keyset or offset paging, limit + 1 instead of a count, display values per option); validation in QInstanceValidator.

Javalin middleware (v1)

The picker object of the possible-values request body (table, process and standalone routes); values, displayValues and hasMore in the response; picker and pickerTableName on field metadata.

Frontend (QQQ Next UI)

Thumbnails in the inline typeahead and on the selection, Browse/Change, Preview and View; the lazily loaded picker dialog.

Application

Declares the picker on the source, chooses its fields, and adds the indexes its filters and sorts need.

Metadata

qInstance.getPossibleValueSource("media").withPicker(new QPossibleValuePicker()
   .withThumbnailField("thumbnailUrl")              // image URL per option and for the selection
   .withPreviewField("galleryUrl")                  // larger image for the preview pane
   .withSubtitleFields(List.of("mediaTypeId", "mimeType"))
   .withDetailFields(List.of("width", "height", "fileSize", "altText"))
   .withFilterFields(List.of("mediaTypeId", "isActive", "createDate", "width"))
   .withSearchFields(List.of("name", "fileName"))   // default: the source's searchFields
   .withSort("Newest", "id", false)                 // first sort is the default
   .withSort("Name", "name", true)
   .withPageSize(48)
   .withUploadProcessName("UploadCmsImageFile")     // optional: add a record from the picker...
   .withUploadResultIdField("mediaId"));            // ...and select the id this output value holds

Every named field must exist on the source table (the validator checks it), pageSize is 1 to 200, and an upload process must exist and name its result value.

API contract

Picker requests use the existing v1 possible-values routes (POST /table/{table}/possibleValues/{field}, the process and standalone routes), with a picker object in the body:

{
  "searchTerm": "harbor",
  "values": { "siteId": 1 },
  "picker": {
    "filter": {
      "criteria": [ { "fieldName": "mediaTypeId", "operator": "EQUALS", "values": [2] } ],
      "orderBys": [ { "fieldName": "id", "isAscending": false } ]
    },
    "afterId": 1234,
    "limit": 48
  }
}

  • filter.criteria may only name the picker’s filterFields and compare to values, never to another field. Other fields get 400, so a picker cannot probe columns it does not declare.

  • filter.orderBys may only name the picker’s sort fields or the primary key. Without one, the first declared sort applies.

  • afterId: keyset paging. When the ordering is the primary key alone, the next page continues after this id (id < afterId descending, > ascending), which is an index range scan. Other orderings use skip.

  • limit: options per page (default pageSize, capped at 200). The action reads limit + 1 rows to set hasMore and never counts.

  • ids with picker looks up the selection with its display values (the thumbnail of the current value).

  • The field’s own possibleValueSourceFilter (with values interpolation) still applies, AND`ed with the picker filter and the search (`CONTAINS over the search fields).

The response adds, per option, the picker’s display fields, raw and formatted where they differ, and hasMore:

{
  "options": [
    { "id": 1235, "label": "Harbor at dusk",
      "values": { "thumbnailUrl": "https://cdn.example/m/1235/t.jpg", "galleryUrl": "https://cdn.example/m/1235/g.jpg", "mediaTypeId": 2, "width": 1600 },
      "displayValues": { "mediaTypeId": "Photo", "width": "1,600" } }
  ],
  "hasMore": true
}

Requests without picker are unchanged (prefix search, no values, no hasMore).

Security

Picker searches run like other possible-value searches: the caller needs READ, INSERT or EDIT on the table that has the field (or permission to run the process), and the search runs in the core’s authorized scope for the source table. Record security locks on the source table still apply, so a user sees only the records they may. Under ENFORCE, the picker filter may not use fields denied by a FieldSecurityLock. Display values of denied fields are left out of values.

Scale

  • No counts: hasMore comes from reading one extra row.

  • Keyset paging for the default "newest first" ordering. Infinite scroll never uses OFFSET there.

  • Offset paging only for other orderings (name, size), which users rarely scroll far.

  • Search is a case-insensitive CONTAINS (LOWER(column) LIKE '%term%'). On PostgreSQL with 100,000 media it reads the matching site’s rows once and keeps the newest page, in about 30 ms. Applications can add a trigram index (pg_trgm) on LOWER(column) when tables grow further.

  • Applications should index each offered ordering together with the filters that are always applied, e.g. (site_id, id DESC), (site_id, name) and (site_id, file_size DESC). On PostgreSQL with 100,000 media (kof22), every ordered page then reads an index range in under 1.5 ms of SQL, and keyset page N costs the same as page 1.

Frontend

The inline field shows the selection’s thumbnail with Change, Preview (full-size lightbox) and View (the record, in a new tab). The typeahead asks for 20 options with thumbnails. The dialog loads on demand (import()), so it adds nothing to route first loads. It provides:

  • debounced server-side search

  • filter controls generated from the source table’s field types (possible-value fields get a select of their own values; booleans Yes/No; numbers and dates a range; strings contains)

  • the declared sorts

  • grid and list views with lazily loaded thumbnails

  • infinite scroll with a Load more fallback

  • a preview pane with the detail fields

  • upload-and-select through the upload process

  • single or multiple selection

  • filters, sort and view remembered per field in local storage

  • keyboard navigation (arrows by row, Enter or double-click to confirm, Space to toggle in multiple selection)

  • a focus trap and Escape to close

  • full screen on phones

Alerts (producer SPI)

QQQ core has a small producer API for alerts ("manage by exception"): any qbit or application can report that a person should do something, resolve it when the condition clears, and beat a dead-man’s switch. Storing, routing and notifying is done by an alerts implementation (such as qbit-alerts). With none installed, every call goes to LoggingAlertSink, which logs (rate-limited), so a producer behaves the same with or without the implementation and needs no dependency beyond qqq-backend-core.

Everything lives in the package com.kingsrook.qqq.backend.core.alerts.

Raising, resolving and beating

QAlerts.raise(new AlertSignal("cdn.purge.dead")
   .withDedupKey("provider:" + providerId)        // stable: repeats refresh one open alert
   .withSeverity(AlertSeverity.ERROR)              // optional: defaults to the type's severity
   .withTitle("3 CDN purges failed after retries") // external-safe: no personal data
   .withSummary("Content corrects itself when the edge TTL expires.")
   .withDetails(detailsText)                       // internal: in-app only, never sent out
   .withRecordRefs(AlertRecordRef.of("cdn_purge_request", ids))
   .withLink(AlertLink.toTable("cdn_purge_request", filter, "Failed purges"))
   .withAttribute("provider", providerName));

QAlerts.resolve("cdn.purge.dead", "provider:" + providerId, "A later purge succeeded");

QAlerts.heartbeat("backups.job.nightly-db");

  • Idempotent on (alertTypeName, dedupKey). The dedup key defaults to the type name (one alert per type).

  • Bounded: at most 100 record refs (beyond that count keeps the true total) and 20 attributes.

  • Never throws and never blocks. Bad input, a sink that throws, or a sink that cannot load is logged and counted (QAlerts.getProducerFailureCount()); nothing reaches the caller.

  • Transaction-aware. raise(signal, transaction) and resolve(resolution, transaction) defer the call with QBackendTransaction.addAfterCommitCallback, and it is dropped on rollback, so an alert never describes a write that did not happen. Without an explicit transaction, the one in QContext (if any) is used. Heartbeats are never deferred.

  • Unknown types are kept. A type that no QAlertTypeMetaData declares is still passed to the sink (with a warning), so a typo never loses a signal.

AlertSeverity is INFO, WARNING, ERROR, CRITICAL; AlertSeverity.fromLabel maps other systems' labels.

Notifying users

An alert is a problem for someone to fix. A notification is a message for people’s bell (and, by their preference, email): a job that failed or finished, a share, a security notice, a release note.

QAlerts.notify(new NotificationSignal("Your export failed")   // required, plain text
   .withBody("**Orders export** stopped after 1,200 rows.")  // markdown
   .withSeverity(NotificationSeverity.ERROR)                  // default INFO
   .withCategory(NotificationCategory.JOB)                    // default SYSTEM
   .withSourceKey("process:" + processUUID)                   // dedup: same key and audience updates one
   .withLink(AlertLink.toUrl("/app/orders", "Orders"))
   .withAudience(NotificationAudience.user(userId)));         // required

  • NotificationSeverity is INFO, SUCCESS, WARNING, ERROR, CRITICAL. NotificationCategory is SYSTEM, JOB, SECURITY, RELEASE, MARKETING, SHARE, ALERT (users mute or choose email per category).

  • NotificationAudience is allUsers(), roles(…​), permission(name), users(…​) or user(id); the implementation resolves it to users when it sends.

  • Optional: sourceName, sourceTableName and sourceRecordId (the record it is about), publishAt, expireAt, sticky (a banner until dismissed) and up to 20 attributes.

  • The guarantees are the same as raise: it never throws or blocks, it waits for the commit (notify(signal, transaction) or the transaction in QContext) and is dropped on rollback, and a missing title or audience is logged and counted.

  • AlertSinkInterface.notify is a default method that only debug-logs, so sinks written before it still work. LoggingAlertSink logs notifications at INFO (WARN for ERROR and CRITICAL), rate-limited per category and source key (or title).

Declaring alert types

Declare each kind of problem a module can report, so an implementation can list and tune every type every installed module produces:

qInstance.add(new QAlertTypeMetaData()
   .withName("cdn.purge.dead")                       // lowercase, dotted
   .withLabel("CDN purge failed")
   .withGroupName("cdn")
   .withDefaultSeverity(AlertSeverity.ERROR)
   .withDescription("A purge failed after its last retry.")
   .withHowToFix("Open the failed purge and choose Retry purge.")
   .withSubjectTableName("cdn_purge_request")       // optional: drives visibility
   .withVisibilityPermission("cdn.view")            // optional extra gate
   .withResolveMode(ResolveMode.PRODUCER)            // or MONITOR, QUIET_PERIOD (withQuietPeriod), MANUAL
   .withDefaultMonitor(AlertDefaultMonitor.heartbeat("cdn.dispatcher", "cdn.dispatcher", 300, 60))
   .withExternalDetailCeiling(DetailLevel.STANDARD));

Types are stored in the supplemental instance meta-data AlertsInstanceMetaData (QAlertTypeMetaData.of(qInstance, name) finds one). The source qbit is set automatically when the type is produced by a qbit. Validation checks that names are unique and match ^(\.[a-z0-9-])+$, that the subject table exists, that a visibility permission is not blank, that a QUIET_PERIOD type has a positive quiet period, and that a heartbeat default monitor has a positive expectedEverySeconds.

Process failures and missed runs, by meta-data

process.withSupplementalMetaData(new AlertProcessMetaData()
   .withAlertOnFailure(AlertSeverity.ERROR)   // raise process.<name>.failed when a run fails
   .withExpectedEvery(Duration.ofHours(24))); // declare process.<name>.missed, with a heartbeat monitor

Enrichment declares the types (the process name in lowercase-hyphen form, e.g. process.nightly-export.failed) and registers AlertProcessLifecycleListener: a failed run raises the failure alert (the exception message goes in the internal details only), and a completed run resolves it and beats process.<name>.

Installing an implementation

An implementation registers its sink, which must be fast and thread-safe (it is loaded once per instance and called on the producer’s thread, after commit):

AlertsInstanceMetaData.ofOrWithNew(qInstance).withSink(new QCodeReference(MyAlertSink.class));

See the Alerts qbit design (#1090).

Record collaboration contracts

Record collaboration is opt-in. Core publishes metadata, enforces record references, and provides extension SPIs; feature storage and business behavior belong to qbits.

Declaring features

RecordFeaturesMetaData.ofOrWithNew(instance)
   .withEnabled(RecordFeature.FAVORITES, true);

RecordFeaturesMetaData.ofOrWithNew(app)
   .withEnabled(RecordFeature.COMMENTS, true);

RecordFeaturesMetaData.ofOrWithNew(table)
   .withSetting(RecordFeature.COMMENTS, "format", "markdown")
   .withPermissionMode(RecordFeature.COMMENTS, FeaturePermissionMode.EDIT_OR_GRANTED)
   .withOwnership(new RecordOwnership().withOwnerFieldName("ownerUserId"));

RecordFeaturesMetaData is the shared holder and instance supplemental metadata. RecordFeaturesAppMetaData and RecordFeaturesTableMetaData adapt the holder to the other supplemental base classes. All use the name recordFeatures.

Resolution merges the instance, matching apps in instance iteration order, and table declarations, then runtime overrides in that same scope order. Settings merge by setting name. Features default off and permission modes default to EDIT_OR_GRANTED. A false withFeatureLocked(feature, false) at any applicable level cannot be overridden. Tables require one primary key and GET or QUERY; collaboration_* tables cannot receive record features.

Register an override provider with withOverrideProvider(new QCodeReference(…​)) on the instance holder. Its runtimeProviders list accepts RuntimeMetaDataProviderInterface implementations. After enrichment, and after runtime changes commit, call RuntimeMetaDataOverlay.refresh(instance). Each refresh publishes a new monotonic generation, at least as high as the source providers' generations, and invalidates resolutions. An invalid provider contribution retains its previous accepted overrides and logs the failure. There are no runtime field contributions in this phase.

Frontend table metadata exposes enabled features, public settings and table capabilities. /qqq/v1/metaData and table metadata carry metaDataGeneration. GET /qqq/v1/table/{tableName}/{primaryKey}/features first reads the parent with the current session and returns record capabilities. Instance summaryProviders may fill counts, favorite/follow state, ownership, presence and extensions; core owns the final feature and capability answers. Without providers, optional values remain null and maps are empty.

Permissions

Enabled permission-bearing features add table catalog permissions using the table’s effective permission base: .comment, .attach, .tag, .share, .task and .assign. Every capability requires READ. EDIT_OR_GRANTED accepts either the explicit grant or table EDIT plus the parent’s WRITE locks. GRANTED_ONLY requires the explicit grant. Unprotected tables follow core’s NOT_PROTECTED convention. Features without a suffix require parent visibility.

Core’s generic TableFeaturePermissionsMetaData registry also serves features outside collaboration. For example:

TableFeaturePermissionsMetaData.ofOrWithNew(instance)
   .withPermission(new TableFeaturePermission().withSuffix("analyze").withLabel("Analyze"));
PermissionsHelper.hasTableFeaturePermission(table, "analyze");

A null appliesTo includes every non-hidden table. A null impliedBy (or a callback returning null) requires the explicit grant, alongside table READ. An implication can return READ, EDIT, INSERT or DELETE and uses the table’s normal permission-level mapping and custom checker. Re-registering a suffix replaces its prior definition. Callbacks implement serializable interfaces; avoid capturing an instance or non-serializable state. The registry’s clone() deep-copies entries and callbacks through Java serialization. QInstance itself does not expose a clone API or implement Java Serializable.

QInstanceEnricher automatically registers all six collaboration permissions when feature metadata exists at instance, app or table scope. Registration is idempotent and uses the current resolved feature state and permission mode. If declarations are installed after enrichment, enrich again before using them.

Record references

extensionTable.withSupplementalMetaData(new RecordReferenceTableMetaData()
   .withTableNameField("tableName")
   .withRecordIdField("recordId")
   .withFeature(RecordFeature.COMMENTS)
   .withMirroredLocks(true)
   .withAuthorField("createdByUserId"));

RecordReferenceSecurityConfig.ofOrWithNew(instance)
   .withMirroredSecurityKeyTypes(List.of("clientId"))
   .withCleanupProviders(List.of(new QCodeReference(ParentCleanup.class)));

Store primary keys as strings (new RecordReference(tableName, primaryKey).primaryKeyAsString()). The reference fields and optional author field must exist. Core filters queries, counts and aggregates to readable parent tables with the feature enabled. Queries with top-level AND criteria for one parent table and an EQUALS/IN parent key also check each parent’s visibility. Gets and every write check the parent. Updates check both the stored and proposed reference. Updating or deleting another author’s row additionally requires RecordReferenceSecurityHelper.ADMIN_PERMISSION (collaboration.admin).

Configured direct parent locks are copied into existing lockKey_<keyType> fields; non-alphanumeric characters in the key type become underscores. Declare ordinary record locks on these extension fields with null behavior ALLOW. Joined locks are enforced by the parent check and are not mirrored. Supplied mirror values are replaced by parent values on inserts and updates.

Enrichment of RecordReferenceSecurityConfig registers the record-change listener. A qbit installing metadata later can call RecordReferenceSecurityHelper.register(instance). The listener updates derived mirror columns through the backend with the parent’s transaction, without treating propagation as a user edit of another author’s row. Use a transaction on the same backend for atomic parent and mirror updates; required integrity-listener failures propagate and the caller must roll back. Cleanup providers run after commit (immediately after a successful nontransactional delete); rollback discards their callbacks. No session elevation is introduced.

Unpinned queries over unmirrored parent locks still need the post-filter, paging refill and approximate-count behavior described in collaboration spec 3.2 item 4. That work is explicitly deferred from this phase (a) implementation.

Owner, assignee, and My records

Declare ownership using existing STRING or INTEGER user-id fields. Validation checks both declared fields against each table, including ownership inherited from an instance or app; the table’s ownership declaration replaces the inherited declaration.

RecordFeaturesMetaData.ofOrWithNew(table)
   .withEnabled(RecordFeature.ASSIGNMENT, true)
   .withOwnership(new RecordOwnership()
      .withOwnerFieldName("ownerUserId")
      .withAssigneeFieldName("assigneeUserId"));

Either field may be omitted. When ASSIGNMENT is enabled after resolution and at least one field is declared, frontend table metadata includes builtInViews with a view named myRecords, labeled My records. Its filter is ownerUserId = CurrentUser OR assigneeUserId = CurrentUser, omitting an undeclared field. An empty ownership declaration, a disabled feature, or a feature hard limit publishes no view. This is a metadata preset; it does not insert or modify persisted user saved views. recordFeatures.ownership exposes the effective ownerFieldName and assigneeFieldName.

CurrentUser extends AbstractFilterExpression and evaluates to QContext.getQSession().getUser().getIdReference(). Like Now, it is stored as a JSON object with a type discriminator: {"type":"CurrentUser"}. The existing criteria deserializer reconstructs the expression by class name. Clients display this value as Me and retain the expression when saving a view, rather than saving a specific user id. The typed BuiltInViewV1 schema describes the ownership view and its expression values; the older string-only QueryFilter response contract is unchanged.

new QQueryFilter(new QFilterCriteria("ownerUserId", QCriteriaOperator.EQUALS, new CurrentUser()));

EQUALS, NOT_EQUALS, and IN work for STRING and INTEGER user-id fields. Core query, count, and aggregate actions resolve a copy before backend execution, including nested filters, so a shared filter remains session-relative. If the session, user, or user id is absent, a criterion containing CurrentUser becomes FALSE, including NOT_EQUALS and mixed IN lists; it never selects unowned rows or all rows by accident. Other criteria in an OR filter retain their normal meaning. Existing permissions and record locks still apply.

The filter variable interpreter also recognizes the exact string ${currentUser} and converts it to a deferred CurrentUser expression. Query/count/aggregate accept this spelling directly as well, including filters restored from saved-view JSON. There is no string interpolation inside larger values. == Virtual criteria and tag search

Core query hooks let a qbit implement tags without storing the tag list on each parent record. Register an implementation of VirtualFieldCriteriaExpanderInterface on a virtual field:

new QVirtualFieldMetaData("tags", QFieldType.STRING)
   .withCriteriaExpander(new QCodeReference(TagCriteriaExpander.class));

// An expander can return this filter for a parent table's tags criterion:
new QQueryFilter(new QFilterCriteria("id", QCriteriaOperator.IN,
   new QSubQuery("tag_link", "recordId", linkFilter)));

An expander makes the field queryable without a field function, including the frontend isQueryCriteria flag. It does not make the field selectable: use a field function or a post-query customizer for displayed values. Query, count and aggregate expand criteria before validation. Each replacement remains a nested filter under the original AND/OR operator. Expansion is one pass; a replacement containing another expander criterion is rejected. The expander decides which operators it supports and throws QUserFacingException for unsupported operators. CONTAINS_ALL is reserved for expander virtual fields; an implementation can return an AND of membership sub-queries.

QSubQuery is the sole value of an IN or NOT_IN criterion. Its selected field and filter fields are validated against the inner table. Same-backend RDBMS queries generate a native SELECT with their own aliases, parameters and record security locks. Other backends, and queries across backend names, materialize the selected values through QueryAction with permissions explicitly enforced and the current session’s security. The bound is QSubQuery.MAX_MATERIALIZED (10,000 rows); exceeding it raises “This filter matches too many records to combine across databases.” Empty IN matches nothing; empty NOT_IN matches everything. The context exposes the instance, session and transaction. Inner queries reuse the transaction only within the same backend; a different backend cannot share that transaction.

Include the virtual field in the table’s searchFields to enable default quick search. Its expandSearchTerm filter is ORed with normal text-field matches. Returning null means no match. After trimming, a term beginning with and at least one following character asks only expanders, with SearchTermKind.TAG and the prefix removed. Ordinary terms use SearchTermKind.TEXT. The expander owns exact versus partial matching. A table without an expander search field returns no matches for a tag term. Record-search providers receive terms unchanged.

Record followers SPI

Register a RecordFollowersProviderInterface on instance-level metadata only:

RecordFeaturesMetaData.ofOrWithNew(instance)
   .withFollowersProvider(new QCodeReference(MyFollowersProvider.class));

List<RecordFollower> followers = RecordFollowers.get(instance,
   new RecordReference(tableName, recordId), FollowEvent.ALERT);

The provider returns followers who want the requested CHANGE, COMMENT, or ALERT event; treat ALERT like CHANGE unless finer preferences are available. RecordFollower carries userId, notifyOnChange, notifyOnComment, and an optional reason. Core validates that the code reference loads and implements the SPI. App- and table-level provider registrations are not used. The helper returns empty if no provider is registered or FOLLOW is disabled by RecordFeaturesResolver; provider failures log a warning and return empty.

Followers are not filtered by read access. Before sending record data, callers must check each recipient using RecordAccessCheckerInterface, or pass sourceTableName and sourceRecordId to QAlerts.notify, which re-checks access.

Realtime invalidation and record versions

Core provides the transport and optimistic version preflight. Presence state and other collaboration business behavior belong to qbits. Events are best-effort hints; clients refetch records through the normal permission-checked endpoints.

Connecting

Open GET /qqq/v1/realtime/stream with the same authentication as other v1 requests. The response is text/event-stream. The first event is hello:

id: <connectionId>:1
event: hello
data: {"connectionId":"<connectionId>","heartbeatSeconds":25,"metaDataGeneration":1}

Then post POST /qqq/v1/realtime/subscriptions:

{"connectionId":"<connectionId>","add":["record:order:42","table:order","user:me","instance"],"remove":[]}

The response contains active and refused arrays. Refusals do not explain whether a record exists. Record channels require a successful GetAction with the session; table channels require READ. Only user:me can subscribe to user events; the server maps it to user:<sessionUserId> and returns the alias in active. Another session’s connection has the same 404 as an unknown connection. Removals apply before additions.

Each event carries an id <connectionId>:<sequence>. Reconnect with Last-Event-ID. A retained cursor resumes the same connection and subscriptions, first sending hello at the acknowledged cursor, then events after it in order. The default replay window is 120 seconds; buffers additionally cap at 10,000 events. A missing, expired, evicted, invalid, or other-pod cursor causes resync; refetch displayed data and use the new hello’s connection id. When the connection still exists, subscriptions are retained. Replay checks permissions again and re-filters field names. Load-balancer affinity improves replay continuity; it is not needed for live multi-pod delivery.

Comment pings arrive every 25 seconds by default. Session validity is checked at that interval; expiry sends session-ended and closes the connection. Explicit v1 logout and back-channel logout also publish a session-ended hint across pods. Permissions are rechecked every five minutes; denied channels are removed with channel-revoked. The default limits are 200 channels per connection and 20 connections per session; opening another connection closes the oldest. Slow transports have bounded queues and are disconnected; clients reconnect and may need to resync.

Events

Event

JSON data

hello

connectionId, heartbeatSeconds, metaDataGeneration

record.changed

table, pk, version, changedBy, fields (names only)

record.deleted

table, pk

table.changed

table

metadata.changed

generation

resync

Empty object; refetch current state

session-ended

Empty object; authenticate again

channel-revoked

channel

Each core event has its own Realtime*EventV1 OpenAPI component. The generic RealtimeEventV1 documents event, channel and data for qbit-defined events such as presence.changed and comment.added; the SSE wire data is the event’s data object.

Core publishes record changes after commit only for tables whose resolved features include PRESENCE or FOLLOW. Rollbacks publish nothing. Table hints are coalesced to at most one per table per second per instance in a pod. The hub applies frontend metadata personalization, field security and hidden-field rules to every subscriber’s field-name list. Record values are never included. Overlay refresh publishes metadata.changed after installing the new generation.

Configuring a transport

The default is InMemoryRealtimeBus: synchronous fan-out on virtual threads with isolated, logged listener failures. One bus is cached per QInstance. Configure with:

instance.withSupplementalMetaData(new RealtimeMetaData()
   .withReplaySeconds(120)
   .withHeartbeatSeconds(25)
   .withMaxChannelsPerConnection(200)
   .withMaxConnectionsPerSession(20)
   .withRecheckMinutes(5));

For multiple pods, use the qqq-esb implementation:

instance.withSupplementalMetaData(new RealtimeMetaData()
   .withBus(new QCodeReference(EsbRealtimeBus.class)));
EsbInstanceMetaData.of(instance).withRealtimeDestinationName("qqq.realtime");

Declare that destination as a TOPIC with a provider. With exactly one configured ESB provider, an undeclared topic name is resolved on that provider automatically. Every bus opens a non-durable, non-shared consumer through EsbConnectionManager, so every pod receives every hint. Publishing is non-persistent and best-effort; broker outages are logged and never fail record writes. Listener setup retries and consumers are recreated on provider reconnection. No backlog is replayed by the broker. Runtime-service shutdown closes the instance bus after producers drain; embedded hosts not using the launcher should call RealtimeBus.close(instance). The Javalin server closes its hub and streams when stopped or metadata is replaced.

Publishing from a qbit

RealtimeBus.get(instance).publish(new RealtimeMessage()
   .withChannel("record:order:42")
   .withEvent("comment.added")
   .withData(Map.of("table", "order", "pk", 42, "commentId", 1001)));

Qbits define their own event names and typed payload schemas. Publish identifiers and hints, never record contents or secrets. Defer write-related events with transaction.addAfterCommitCallback(…​). Do not implement presence in the core bus. Close handles returned by subscribe when their owner stops.

Optimistic record versions

QTableMetaData.withRecordVersionField("revision") chooses a version field. Enrichment defaults to the field with the MODIFY_DATE behavior (including conventional modifyDate fields). An explicit empty string disables checking. Custom version fields must be advanced by the application or backend.

Core callers can use UpdateInput.withExpectedVersions(Map.of(primaryKey, version)). The action reuses old rows when available, compares values using the version field’s type, and adds a RecordVersionConflictError to stale records so they are not written. Other records in a batch follow the normal per-record error behavior.

The v1 PATCH endpoint retains its flat JSON or multipart record body. Supply the optional query parameter expectedVersion (URL-encode ISO-8601 timestamps):

PATCH /qqq/v1/table/order/42?expectedVersion=2026-10-13T15%3A00%3A00Z
{"status":"approved"}

A mismatch returns HTTP 409 RecordConflictV1: currentVersion, changedFields, modifiedByUserId, and modifiedAt. Nullable details are available when the table provides them. modifiedByUserId uses a field of that name, falling back to modifiedBy; modifiedAt uses the MODIFY_DATE field. Changed names are best-effort differences between submitted values and the current record, not an audit history. Omitting the version preserves the existing unconditional update behavior.

This is a core preflight check, not a backend atomic compare-and-swap: another writer can race between the read and write. Applications requiring serialization must additionally use their backend’s transaction/locking guarantees.

Services

QQQ Middleware: Javalin web server

QQQ provides a standard implementation of a middleware layer - that is - code that exists between the QQQ backend and user interface. This implementation is a web server built using the Javalin framework, packaged and deployed in the qqq-middleware-javalin maven module

QQQ 4.1 RC1 uses Java 21, Javalin 7.2.3 and Jetty 12.1.13. The Javalin 7 configuration examples below target this release.

The de facto way to create a QQQ application server is to write a class which uses an instance of one of the subclasses of QApplicationJavalinServer.

For example, if your application metadata is defined in a directory of yaml files, your server class could be implemented as:

ConfigFileBasedQQQApplication usage example
public class MyServer
{
   public static void main(String[] args) throws QException
   {
      if(args.length != 1)
      {
         throw new IllegalArgumentException("Usage: MyServer <metadata-directory>");
      }
      ConfigFilesBasedQQQApplication application = new ConfigFilesBasedQQQApplication(args[0]);
      QApplicationJavalinServer server = new QApplicationJavalinServer(application);
      server.start();
   }
}

Pass an existing metadata directory. Startup exceptions propagate so an invalid configuration produces a nonzero process exit. For metadata bundled in a JAR, use resource streams as described in QInstance; source-directory paths are not available in a deployed artifact.

A similar class exists if your metadata is produced by a package of Java MetaDataProducer objects: MetaDataProducerBasedQQQApplication.

QApplicationJavalinServer

This class provides the bridge between your QQQ Application (e.g., your metadata) and the QQQ Middleware layer served by a Javalin web server. It has several properties to control behaviors:

  • Integer port - (default 8000) - port to use for serving HTTP.

  • boolean serveFrontendMaterialDashboard - (default true) whether to serve the javascript frontend provided in the maven artifact qqq-frontend-material-dashboard.

  • boolean serveLegacyUnversionedMiddlewareAPI - (default true) whether to serve a version the original implementation of the QQQ middleware, which current version of qqq-frontend-material-dashboard are compatible with.

  • List<AbstractMiddlewareVersion> middlewareVersionList - (default contains MiddlewareVersionV1) - list of newer, formally versioned implementations of the QQQ middleware interface to be served.

  • Consumer<JavalinConfig> javalinConfigCustomizer - (default null) - optional configuration-time hook for route, handler and other Javalin setup. Access it with getJavalinConfigCustomizer, setJavalinConfigCustomizer or withJavalinConfigCustomizer.

  • Consumer<Javalin> javalinConfigurationCustomizer - (default null) - optional observer of the created service before startup, for example to capture its reference. Despite its historical name, this hook does not register routes or handlers in Javalin 7; use javalinConfigCustomizer for that work.

  • List<QJavalinRouteProviderInterface> additionalRouteProviders - (default null) - list of fully custom implementations of QJavalinRouteProviderInterface, to add additional endpoints to the javalin server.

    • Note, you may first want to consider using JavalinRouteProviderMetaData instead - see below.

  • QJavalinMetaData javalinMetaData - (default null) - optional alternative place to define JavalinMetaData (vs. defining it in the QInstance). Note that if it is set in both places, the one in the QApplicationJavalinServer is used.

QApplicationLauncher

QApplicationLauncher.run(application, config) is one entry point that starts everything an application has configured, in this order: a QApplicationJavalinServer for the application; the QScheduleManager, when a process, queue or table automation has a schedule, or the instance has the scheduledJob table; and each runtime service registered with QInstance.withRuntimeService(QCodeReference) (an implementation of QRuntimeServiceInterface), in registration order. They all use the one QInstance the server builds (getQInstance() on the launcher or the server). A JVM shutdown hook, or the launcher’s stop(), stops them in reverse order. If one fails to start, the ones already started are stopped, and run throws.

QApplicationLauncher usage example
public static void main(String[] args) throws QException
{
   QApplicationLauncher.run(new MetaDataProducerBasedQQQApplication(MyMetaDataProducer.class), new QApplicationLauncherConfig()
      .withServerCustomizer(server -> server.withPort(8080))
      .withFailOnMetaDataProducerError(true));
}

QApplicationLauncherConfig also sets the session the schedule manager runs as (withSystemUserSessionSupplier, default a QSystemUserSession), and can turn off the shutdown hook (withRegisterShutdownHook(false)). withFailOnMetaDataProducerError(true) makes a producer that fails stop the launch (see [MetaDataProduction]); it needs an application that extends AbstractMetaDataProducerBasedQQQApplication.

Route registration and service observation

Javalin 7 registers routes, before/after handlers and error/exception handlers during configuration. Custom QJavalinRouteProviderInterface implementations use acceptJavalinConfig(JavalinConfig) for this setup, calling config.routes. getJavalinEndpointGroup() still returns io.javalin.apibuilder.EndpointGroup; QQQ installs that group with config.routes.apiBuilder(…​). acceptJavalinService(Javalin) observes the created service and must not be used to register handlers.

For a server-level registration callback, configure the server before calling start():

server.withJavalinConfigCustomizer(config ->
   config.routes.get("/custom/status", ctx -> ctx.result("ready")));

A service observer remains separate. For example, an existing AtomicReference<Javalin> serviceReference can still be populated with server.withJavalinConfigurationCustomizer(serviceReference::set). Preserve authentication and handler scopes when moving provider setup; the registration hook does not itself grant access to a route.

JavalinMetaData

Certain behaviors of a QQQ Javalin server are configured in a declarative manner by adding a QJavalinMetaData object to the supplementalMetaData in your QInstance (or, as mentioned above, by setting it directly on the QApplicationJavalinServer):

  • List<JavalinRouteProviderMetaData> routeProviders - (default null) optional list of custom route providers to add to the Javalin server. See below for details.

  • String uploadedFileArchiveTableName - (default null) - reference to a QQQ Table in your application instance, needed to support the Bulk Load process, as well as any other processes which need to accept an uploaded file as input.

  • boolean loggerDisabled - (default false)

  • Function<QJavalinAccessLogger.LogEntry, Boolean> logFilter - (default null)

  • boolean queryWithoutLimitAllowed - (default false)

  • Integer queryWithoutLimitDefault - (default 1000)

  • Level queryWithoutLimitLogLevel - (default INFO)

Process file uploads

The legacy process endpoints accept multipart files through uploadedFileArchiveTableName. Each multipart field becomes a list of StorageInput values for the backend step. QQQ checks process access before archiving, then validates every filename before writing the first file. Submit a nonblank filename without directory separators, colons or control characters; . and .. are rejected. Each accepted file uses its own generated archive directory. The archive is internal storage: a permitted process can consume its uploads without granting users direct table READ/WRITE access. Downloads still require the explicit grants described in Downloading generated files.

Configure upload limits with Javalin’s native multipart settings before starting the server. config.http.maxRequestSize applies to buffered request-body access and does not set the multipart upload limit. For example:

server.withJavalinConfigCustomizer(config ->
{
   config.jetty.multipartConfig.maxFileSize(10, io.javalin.config.SizeUnit.MB);
   config.jetty.multipartConfig.maxTotalRequestSize(20, io.javalin.config.SizeUnit.MB);
});

Invalid filenames and multipart parser rejections return HTTP client errors, before archive writes or process execution. This includes requests with chunked transfer encoding. Ordinary backend process failures retain their existing process error envelope. Archive retention and deletion remain application responsibilities; storage errors or later process failures do not provide transactional rollback of archived files.

JavalinRouteProviderMetaData

This type of metadata allows you to add additional http route providers to your Javalin instance, e.g., for serving static files or for running custom code from your application (in the form of QQQ Processes) to respond to HTTP requests.

  • String hostedPath - (required)

  • String fileSystemPath - (required for a static router)

  • String processName - required for a dynamic, process-based router. Must be a process name within the QQQ Instance. See below for additional details

  • List<String> methods - optional list of HTTP methods for a process-based route: GET, POST, PUT, PATCH, or DELETE. Defaults to GET when omitted or empty.

  • QCodeReference routeAuthenticator - Optional reference to a class that implements RouteAuthenticatorInterface, to provide security authentication to all requests handled by the route provider.

    • A default implementation is provided as SimpleRouteAuthenticator, which requires that a user session be present to access paths served by the route provider.

  • QCodeReference contextHandler - Optional reference to a RouteProviderContextHandlerInterface implementation for a process-based route. Extend DefaultRouteProviderContextHandler to augment its request and response mappings.

Process-based route provider processes

If you define a JavalinRouteProviderMetaData with a processName (e.g., to serve dynamic HTTP responses from your javalin server), the process that you implement will be called to respond to any HTTP requests received by the javalin server which match the hostedPath and methods that are specified in the metadata.

The QQQ javalin server will marshal request data from the javalin context into the process’s payload, conforming to the shape of the ProcessBasedRouterPayload class. Similarly, the http response will be built by taking values from the process’s output/state conforming to the fields in that class. As such, it is recommended to use a ProcessBasedRouterPayload instance, as show in this example:

The default DefaultRouteProviderContextHandler stores the HTTP method process-input value as a String such as "GET" or "POST". Read it with input.getValueString("method") or ProcessBasedRouterPayload.getMethod(); do not cast the underlying value to Javalin HandlerType. Custom context handlers define their own input mapping.

The default mapping also supplies path parameters, repeated query/form values, cookies, the raw String body, and a requestHeaders input map. Set responseHeaders explicitly for non-JSON output because the application server defaults to application/json. Responses may use responseString, responseBytes, or a responseStorageInput process-output value containing a StorageInput; these are checked in that order. For redirects, set redirectURL and optionally statusCode (default 302).

Process-based router usage example (including ProcessBasedRouterPayload)
public class MyDynamicSiteProcessStep implements BackendStep
{
   @Override
   public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
   {
      ProcessBasedRouterPayload payload = input.getProcessPayload(ProcessBasedRouterPayload.class);
      String path = payload.getPath();
      payload.setResponseHeaders(Map.of("Content-Type", "text/plain; charset=utf-8"));
      payload.setResponseString("You requested: " + path);
      output.setProcessPayload(payload);
   }
}

OAuth2 back-channel logout

Register /qqq/v1/oidc/backchannel-logout with the application’s configured OAuth2 provider. QQQ accepts form-encoded logout_token notifications signed with RS256, using discovery from baseUrl and the issuer externalBaseUrl when configured (otherwise baseUrl). Configure clientId and an existing userSessionTableName. Tokens must contain the matching issuer/client audience, iat, exp, jti, the back-channel logout event, and sub or sid; nonce is forbidden. Issued-at age is limited to five minutes with at most 60 seconds of future clock skew. Nimbus validates signatures and registered claims. Unsupported configuration and invalid, expired or replayed tokens return HTTP 400 before privileged session access. Successful notifications return HTTP 200, including when no sessions match. See the OpenID Connect specification.

QQQ removes sessions only from the configured table whose stored JWT access token has the same issuer and matches every supplied subject/session claim. Unreadable or absent stored tokens and other issuers remain untouched. Replay detection is bounded to 10,000 events within the current application process, retained for six minutes; capacity fails closed rather than evicting recent events. It is not shared across replicas or retained across restarts; expiry and signature checks still apply there. No client secrets or tokens are logged. Tests use an owned HTTP discovery/JWKS fixture and real signatures for successful logout and invalid-token/session-preservation controls.

In QQQ 4.1 RC1, discovery uses the configured provider base URL once. Successful logout clears the matching process-local access-token cache. When sessionStoreEnabled is true, QQQ first removes matching entries through the registered session-store provider; an eviction failure returns an error before deleting the session rows or recording the event as processed, so the same notification can be retried. This is sequential revocation in the current process and configured store; it does not promise cancellation of requests already in flight or eviction of another replica’s local token cache.

When the instance stores OAuth2 refresh tokens (see OAuth2RefreshTokens.adoc), QQQ deletes the matching sessions' stored refresh tokens before their session rows, so an ended session can’t be renewed. It doesn’t call the provider’s revocation endpoint, because the provider started the logout.

Schedulers and Scheduled Jobs

QQQ has the ability to automatically run various types of jobs on schedules, either defined in your instance’s meta-data, or optionally via data in your application, in a scheduledJob table.

Execution failures

Scheduled execution retains its existing log-based failure handling. Quartz listeners do not reliably receive failures from the process runner; serial variants continue after a logged failure. Exception propagation and missing-process diagnostics are deferred in #502. Scheduled saved reports still require an isolated authenticated owner session; see scheduled report execution.

Saved-report schedule lifecycle

With the saved-report and scheduled-job providers and a running QScheduleManager, explicitly configured reports synchronize linked jobs and parameters. Configure the schedulerName input default on the scheduledReportSyncToScheduledJob process’s preview step. Existing jobs retain their stored scheduler. Ordinary activation, cron/time-zone changes, deactivation and deletion remain supported.

Automatic scheduler selection and per-record synchronization warnings are deferred in #503. A source record may already be saved or deleted when a later synchronization or scheduler operation fails; inspect logs and linked job state. These operations do not provide database/scheduler atomicity, transactional scheduling, restart recovery or cancellation of running executions. The sample retains explicit-configuration lifecycle and actual local dispatch controls.

Schedulers and QSchedulerMetaData

2 types of schedulers are included in QQQ by default (though an application can define its own schedulers):

  • SimpleScheduler - is (as its name suggests) a simple class which uses java’s ScheduledExecutorService to run jobs on repeating intervals.

    • Cannot run cron schedules - only repeating intervals.

    • If multiple servers are running, each will potentially run the same job concurrently

    • Has no configurations, e.g., to limit the number of threads.

  • QuartzScheduler - uses the 3rd party Quartz Scheduler library to provide a much more capable, though admittedly more complex, scheduling solution.

    • Can run both cron schedules and repeating intervals.

    • By default, will not allow concurrent executions of the same job.

    • Supports multiple configurations, e.g., to limit the number of threads.

An application can define its own scheduler by providing a class which implements the QSchedulerInterface.

A QInstance can work with 0 or more schedulers, as defined by adding QSchedulerMetaData objects to the instance.

This meta-data class is abstract, and is extended by the 2 built-in schedulers (e.g., SimpleSchedulerMetaData and QuartzSchedulerMetaData). As such, these concrete subclasses are what you need to instantiate and add to your instance.

To configure a QuartzScheduler, you can add a Properties object to the QuartzSchedulerMetaData object. See Quartz’s documentation for available configuration properties.

Defining SchedulerMetaData
qInstance.addScheduler(new SimpleSchedulerMetaData().withName("mySimpleScheduler"));

qInstance.addScheduler(new QuartzSchedulerMetaData()
   .withProperties(myQuartzProperties)
   .withName("myQuartzScheduler"));

SchedulableTypes

The types of jobs which can be scheduled in a QQQ application are defined in the QInstance by instances of the SchedulableType meta-data class. These objects contain a name, along with a QCodeReference to the runner, which must be a class that implements the SchedulableRunner interface.

By default, (in the QInstanceEnricher), QQQ will make 3 SchedulableType options available:

  • PROCESS - Any Process defined in the QInstance can be scheduled.

  • QUEUE_PROCESSOR - A Queue defined in the QInstance, which requires polling (e.g., SQS), can be scheduled.

  • TABLE_AUTOMATIONS - A Table in the QInstance, with AutomationDetails referring to an AutomationProvider which requires polling, can be scheduled.

If an application only wants to use a subset of these SchedulableType options, or to add custom SchedulableType options, the QInstance will need to have 1 or more SchedulableType objects in it before the QInstanceEnricher runs.

User-defined Scheduled Jobs

To allow users to schedule jobs (rather than using programmer-defined schedules (in meta-data)), you can add a set of tables to your QInstance, using the ScheduledJobsMetaDataProvider class:

Adding the ScheduledJob tables and related meta-data to a QInstance
new ScheduledJobsMetaDataProvider().defineAll(
   qInstance, backendName, table -> tableEnricher(table));

This meta-data provider adds a "scheduledJob" and "scheduledJobParameter" table, along with some PossibleValueSources. These tables include post-action customizers, which manage (re-, un-) scheduling jobs based on changes made to records in this these tables.

Also, when QScheduleManager is started, it will query these tables,and will schedule jobs as defined therein.

You can use a mix of user-defined and meta-data defined scheduled jobs in your instance. However, if a ScheduledJob record references a process, queue, or table automation with a meta-data defined schedule, the ScheduledJob will NOT be started by ScheduleManager — rather, the meta-data definition will "win".

Example of inserting scheduled jobs records directly into an SQL backend
-- A process:
INSERT INTO scheduled_job (label, scheduler_name, cron_expression, cron_time_zone_id, repeat_seconds, type, is_active) VALUES
   ('myProcess', 'QuartzScheduler', null, null, 300, 'PROCESS', 1);
INSERT INTO scheduled_job_parameter (scheduled_job_id, `key`, value) VALUES
   ((SELECT id FROM scheduled_job WHERE label = 'myProcess'), 'processName', 'myProcess');

-- A table's insert & update automations:
INSERT INTO scheduled_job (label, scheduler_name, cron_expression, cron_time_zone_id, repeat_seconds, type, is_active) VALUES
   ('myTable.PENDING_INSERT_AUTOMATIONS', 'QuartzScheduler', null, null, 15, 'TABLE_AUTOMATIONS', 1),
   ('myTable.PENDING_UPDATE_AUTOMATIONS', 'QuartzScheduler', null, null, 15, 'TABLE_AUTOMATIONS', 1);
INSERT INTO scheduled_job_parameter (scheduled_job_id, `key`, value) VALUES
   ((SELECT id FROM scheduled_job WHERE label = 'myTable.PENDING_INSERT_AUTOMATIONS'), 'tableName', 'myTable'),
   ((SELECT id FROM scheduled_job WHERE label = 'myTable.PENDING_INSERT_AUTOMATIONS'), 'automationStatus', 'PENDING_INSERT_AUTOMATIONS'),
   ((SELECT id FROM scheduled_job WHERE label = 'myTable.PENDING_UPDATE_AUTOMATIONS'), 'tableName', 'myTable'),
   ((SELECT id FROM scheduled_job WHERE label = 'myTable.PENDING_UPDATE_AUTOMATIONS'), 'automationStatus', 'PENDING_UPDATE_AUTOMATIONS');

-- A queue processor:
INSERT INTO scheduled_job (label, scheduler_name, cron_expression, cron_time_zone_id, repeat_seconds, type, is_active) VALUES
   ('mySqsQueue', 'QuartzScheduler', null, null, 60, 'QUEUE_PROCESSOR', 1);
INSERT INTO scheduled_job_parameter (scheduled_job_id, `key`, value) VALUES
   ((SELECT id FROM scheduled_job WHERE label = 'mySqsQueue'), 'queueName', 'mySqsQueue');

Running Scheduled Jobs

In a server running QQQ, if you wish to start running scheduled jobs, you need to initialize the QScheduleManager singleton class, then call its start() method.

Note that internally, this class will check for a system property of qqq.scheduleManager.enabled or an environment variable of QQQ_SCHEDULE_MANAGER_ENABLED, and if the first value found is "false", then the scheduler will not actually run its jobs (but, in the case of the QuartzScheduler, it will be available for managing scheduled jobs).

The method QScheduleManager.initInstance requires 2 parameters: Your QInstance, and a Supplier<QSession> lambda, which returns the session that will be used for scheduled jobs when they are executed.

Starting the Schedule Manager service
QScheduleManager.initInstance(qInstance, () -> systemUserSession).start();

Examples

Attach a schedule in meta-data to a Process
QProcessMetaData myProcess = new QProcessMetaData()
   // ...
   .withSchedule(new QScheduleMetaData()
      .withSchedulerName("myScheduler")
      .withDescription("Run myProcess every five minutes")
      .withRepeatSeconds(300));

OAuth2 refresh tokens

OAuth2AuthenticationModule can keep a user’s session alive after its access token expires. If the provider returns a refresh token at login, QQQ stores it on the server. When the access token expires, or is about to, QQQ uses the refresh token to get a new one, so the user doesn’t have to log in again through the identity provider. Without this, a session ends when its first access token expires.

Enabling it

  1. Ask for a refresh token. Most providers issue one only when the offline_access scope is requested and allowed for the client. Next UI 1.0 requests offline_access by default. Add it to the OAuth2 meta-data scopes as well, for example openid profile email offline_access.

  2. Add the refresh-token table. Refresh tokens are kept in their own table, so existing userSession schemas don’t change. Defining the table in your instance is what turns storage on:

    qInstance.addTable(new UserSessionRefreshTokenMetaDataProducer(backendName).produce(qInstance));

    For an RDBMS backend, create the table with columns matching UserSessionRefreshToken: id, create_date, modify_date, session_uuid (VARCHAR(40), unique), refresh_token (TEXT), refresh_token_expires_at and session_started_at. For example (PostgreSQL):

    CREATE TABLE qqq_user_session_refresh_token
(
   id                       INTEGER GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
   create_date              TIMESTAMP,
   modify_date              TIMESTAMP,
   session_uuid             VARCHAR(40) NOT NULL UNIQUE,
   refresh_token            TEXT,
   refresh_token_expires_at TIMESTAMP,
   session_started_at       TIMESTAMP
);

    Name the table the same way your backend names the userSession table.

  3. Set an encryption key. Refresh tokens are stored encrypted, so the key is required whenever the table is defined and refresh tokens are enabled. Instance validation fails without it. See Encryption at rest.

Without the table, refresh tokens are discarded and sessions behave as they did before.

Encryption at rest

A refresh token can mint new access tokens for as long as the provider allows (Authentik’s default is 30 days). Anyone holding a copy of the table, such as a database dump, a replica or a backup in object storage, could impersonate its users. For that reason QQQ never stores a refresh token in plaintext:

  • Each token is encrypted with AES-256-GCM, using a random 96-bit nonce per value. The stored value has the form v1:<keyId>:<base64(nonce | ciphertext | tag)>.

  • The encryption is bound to the session: the session uuid and the userSession row’s userId are the GCM associated data. A value copied to another row fails to decrypt.

  • The key comes from configuration, never from the database. It is never serialized to clients or written to logs.

  • A value that fails to decrypt (wrong or missing key, a tampered value, or a value moved from another row) is treated as having no refresh token. QQQ deletes the session’s rows, the client gets the usual 401, and the user logs in again. The log entry records only the session uuid and a short reason, such as authenticationFailed or unknownKeyId.

Generating and setting the key

The key is 32 random bytes, base64-encoded:

openssl rand -base64 32

Keep it in your secret store and pass it in through the environment. Don’t commit it. ${env.X} and ${prop.X} references are resolved:

new OAuth2AuthenticationMetaData()
   // ...
   .withRefreshTokenEncryptionKey("${env.QQQ_OAUTH2_REFRESH_TOKEN_KEY}");

Every replica that shares the session tables must use the same key. If the key is lost, the stored refresh tokens can’t be used: sessions end when their current access tokens expire, and users log in again.

Rotating the key

Each stored value records the id of the key that encrypted it. To rotate:

  1. Generate a new key.

  2. Make the new key current under a new id, and keep the old key, under its old id, in previousRefreshTokenEncryptionKeys. If you never set an id, the old key’s id is default.

    .withRefreshTokenEncryptionKey("${env.QQQ_OAUTH2_REFRESH_TOKEN_KEY_2026}")
.withRefreshTokenEncryptionKeyId("2026")
.withPreviousRefreshTokenEncryptionKeys(Map.of("default", "${env.QQQ_OAUTH2_REFRESH_TOKEN_KEY}"))

  3. Deploy. Previous keys are used only to decrypt. A value is re-encrypted with the current key the next time it is written, which happens on its session’s next refresh.

  4. After the longest refresh-token lifetime has passed, remove the previous key. Any remaining values under that key can’t be decrypted, so their sessions end as described above.

Local development

For local development only, allowPlaintextRefreshTokens(true) lets the instance validate without a key. Refresh tokens are then stored as-is, and validation logs a loud warning. Don’t use it anywhere else. When a key is set, it is always used and this setting has no effect.

Table lock-down

UserSessionRefreshTokenMetaDataProducer hides the table, turns off auditing and hides the refreshToken field. It also sets the table’s permission rules to HAS_ACCESS_PERMISSION on a dedicated permission, qqq.internal.userSessionRefreshToken.noAccess.hasAccess. Don’t grant that permission to any role. Because the table is hidden, the permission isn’t listed among the instance’s available permissions. The v1 table API (query, get, count, export), reports and the UI therefore deny the table to every user, admins included. OAuth2AuthenticationModule reads and writes the table as the system user, which these rules don’t affect.

If your app exposes tables through qqq-middleware-api, don’t add API meta-data for this table. Its permission rules deny it there too.

Encrypting userSession.accessToken is not covered yet. Access tokens are shorter-lived, but they are still credentials, so keep database and backup access restricted.

Meta-data options

These are set on OAuth2AuthenticationMetaData:

  • refreshTokensEnabled - Boolean, default true. Store and use refresh tokens. Set it to false to keep the old behavior even when the table exists.

  • refreshTokenSkewSeconds - Integer, default 60. QQQ refreshes the access token when it has this many seconds or fewer left, so requests don’t fail at the exact moment of expiry. It must not be negative.

  • maxSessionLifetimeSeconds - Integer, optional. The longest a session can be kept alive by refreshing, counted from login. After that, the refresh token is revoked and deleted, and the session ends when its current access token expires. If not set, there is no limit beyond the provider’s own refresh-token lifetime.

  • refreshTokenTableName - String, optional. Use a table other than userSessionRefreshToken. When set, the table must exist in the instance.

  • refreshTokenEncryptionKey - String, required when refresh tokens are stored. A base64-encoded 32-byte AES key, or a ${env.X} / ${prop.X} reference to one. It is not serialized.

  • refreshTokenEncryptionKeyId - String, default default. The id stored with each value encrypted under the current key. It may contain letters, digits, _, . and -, up to 64 characters.

  • previousRefreshTokenEncryptionKeys - Map of key id to key, optional. Retired keys, used only to decrypt, for rotation. It is not serialized.

  • allowPlaintextRefreshTokens - Boolean, default false. For local development only: lets refresh tokens be stored unencrypted when no key is set, and logs a warning.

Behavior

  • When refreshes happen. Resuming a session by its uuid (the session cookie) and isSessionValid both check the stored access token. If it has expired or is inside the skew window, QQQ calls the provider’s token endpoint with grant_type=refresh_token. The endpoint comes from OIDC discovery on baseUrl. The new access token replaces the old one in userSession.

  • Rotation. If the provider returns a new refresh token, QQQ saves it before the new access token. If the provider returns none, QQQ keeps using the existing one.

  • Concurrent requests. Within one application process, requests for the same session are serialized, so only one of them calls the provider; the others use the token it stored. Across replicas, a replica whose refresh fails with invalid_grant first checks whether another replica already rotated the token, and if so uses that one.

  • Rejected refresh tokens. When the provider answers invalid_grant (for example, the token expired or was revoked), QQQ deletes the session’s userSession and refresh-token rows and removes it from the session store if one is enabled. The client gets the usual 401 and logs in again.

  • Temporary errors. Network errors and other provider errors keep the refresh token, so a later request can try again. Meanwhile, the session is invalid once its access token has expired.

  • Logout. Logging out deletes the stored refresh token. If the provider’s discovery document advertises a revocation_endpoint, QQQ revokes the token there first (RFC 7009); revocation is best effort. Back-channel logout (see BackChannelLogout.adoc) deletes the refresh tokens of the sessions it ends, without calling the provider, since the provider already ended them.

  • Secrecy. Refresh tokens stay on the server, encrypted (see Encryption at rest). They are not put on the QSession, in API responses or in cookies. The refreshToken field is hidden, so standard queries and gets leave it out. The table is hidden, not audited, and denied to every role. Log messages about refreshing, revoking and storing tokens never include the tokens, nor the descriptions in the provider’s refresh error responses.

Provider settings

  • Authentik: in the OAuth2/OpenID provider, add the offline_access scope mapping to the provider’s scopes. The access-token lifetime ("Access token validity") can then go back to a short value such as minutes=5 or hours=1. "Refresh token validity" controls how long an idle session can last.

  • Auth0: turn on Allow Offline Access in the API settings, and enable Refresh Token Rotation for the application. Auth0AuthenticationModule does not exchange authorization codes, so it doesn’t use this feature; it applies to OAuth2AuthenticationModule.

  • Keycloak: offline_access is a default optional client scope. Keycloak’s refresh_expires_in response field is recorded, and refresh tokens past that time are not used.

Subclasses

Subclasses that override createSession or isSessionValid and call super get refreshes without any changes, because refreshing happens in the access-token lookup that both methods use. A subclass that builds sessions without calling super (for example, for machine tokens) is not affected. Back-channel logout calls the public static method OAuth2AuthenticationModule.deleteStoredRefreshTokens(QInstance, Collection<String>), which is also available to custom logout flows.

Enterprise Service Bus (ESB)

QQQ publishes table and process events as CloudEvents 1.0 JSON messages over Jakarta Messaging (JMS). A process trigger consumes from a queue or a durable topic subscription. The broker stores and redelivers messages. Optionally, a transactional outbox makes table events survive broker outages, and idempotent triggers skip redelivered messages (see below); both are off by default.

Run the sample

The qqq-sample-project includes an embedded ActiveMQ Artemis broker. Start com.kingsrook.sampleapp.SampleJavalinServer to start both the HTTP server and ESB runtime. The sample publishes person INSERT, UPDATE, and DELETE events to the personEvents topic. Its syncPerson process subscribes to that topic and logs each received event’s ID and person ID. Stop the sample server to stop its consumers and broker.

The broker listens on 127.0.0.1:61616 by default. Set -Dqqq.sample.esb.port=<port> before startup to choose another local port. The sample broker is in memory, has authentication disabled, and is for local development only. Its data is lost on shutdown.

An application using its own broker adds qqq-esb and the client for the chosen broker, defines the metadata below, then calls QEsbRuntime.getInstance().start(qInstance) after creating its validated instance. Call stop() during shutdown. A broker outage does not block application boot; the runtime retries connecting.

Metadata

Define a QEsbProviderMetaData for each broker with a unique name, type, and url. Types are ACTIVEMQ_ARTEMIS and RABBITMQ. The Artemis URL uses tcp://host:port; the RabbitMQ URL is an AMQP URI including its virtual host. Optional username and password are connection credentials. Values can use ${env.VARIABLE} interpolation. Keep credentials outside source control.

Define each QEsbDestinationMetaData with a name, providerName, and type (QUEUE or TOPIC). destinationName optionally overrides the broker-side name; by default it equals name.

On a table, EsbTableMetaData contains EsbTablePublication entries with destinationName and events INSERT, UPDATE, or DELETE. On a process, EsbProcessMetaData can publish STARTED, COMPLETED, and FAILED, and can contain EsbTrigger entries. A trigger names its destination and can configure:

  • subscriptionName for a topic’s shared durable subscription; its default is <process>.<broker-side destination> and it must be unique per provider.

  • mode (SINGLE, the default, or BATCH), batchSize (100), and batchWaitMs (1000).

  • concurrency (1 consumer per node), maxAttempts (3), and retry delay, multiplier, and maximum delay (retryDelayMs, retryMultiplier, retryMaxDelayMs).

  • onDeadLetter (DEAD_LETTER_QUEUE by default or DISCARD), optional deadLetterDestinationName, optional runAsSessionSupplier, optional timeoutMs, and startPaused (false).

QQQ validates unknown providers, destinations, and processes at boot. In SINGLE mode a trigger passes one CloudEvent in esbMessages; BATCH passes a list. When the process’s tableName matches an event’s table, the process callback selects that event’s record key. Table messages include the current record on insert, current and old records on update, and the old record on delete.

Brokers and delivery

ActiveMQ Artemis and RabbitMQ 4.x are supported through JMS 3.x. ActiveMQ Classic is unsupported because it lacks the shared durable topic subscription required by triggers. For RabbitMQ, queues and durable subscription queues use quorum queues. A queue’s consumers compete for messages; each topic subscription receives its own copy, while consumers sharing that subscription divide its messages.

Table events publish after a successful transaction commits. A publish that failed because of a failover or a stale session is retried once on a fresh session; otherwise a failure is logged and counted but does not undo the table write, and the event is lost. Enable the transactional outbox (below) so it is not. Trigger success commits the JMS session; failure rolls it back for broker redelivery until maxAttempts, when the event is dead-lettered or discarded. Keep maxAttempts below the broker’s own redelivery limit (Artemis defaults to 10; RabbitMQ quorum queues default to 20). Broker high availability is configured on the broker itself; QQQ connects to it through the provider URL (see Single broker or HA, below).

Stopping the runtime interrupts active process runs. A run that completes successfully during the interrupt grace period is committed; otherwise, a stopped run’s messages roll back for redelivery after restart. This stopped-run rollback does not increment QQQ’s failure counter, but broker redelivery advances JMSXDeliveryCount, which QQQ uses for maxAttempts. Repeated shutdowns can therefore consume delivery attempts.

In batch mode, a retry rolls back the whole JMS transaction, including any dead-letter sends for unparseable messages. Those messages are redelivered with the batch and are dead-lettered or discarded, according to onDeadLetter, when a later transaction commits.

Delivery is at least once. Write triggered processes so a repeat delivery is safe. There is no ordering guarantee unless concurrency is 1, no duplicate filtering unless idempotent triggers are enabled, and no XA transaction across the broker and table backend. Avoid publishing an event that triggers the same process indefinitely. A rollback on a non-transactional table backend drops pending events even if its write persisted. If fetching the old record for DELETE fails, QQQ publishes no delete event.

Transactional outbox

With the outbox enabled, a table write’s events are written to an outbox table in the same backend transaction as the records, so they commit or roll back together. A relay thread on every node then publishes them. A broker outage delays events but never drops them.

qInstance.addTable(EsbOutboxTable.defineTable("esb_outbox", "rdbms"));   // same backend as the publishing tables
EsbInstanceMetaData.of(qInstance)
   .withOutbox(new EsbOutboxConfig().withEnabled(true));                  // tableName defaults to esb_outbox

  • Where the rows are written: in the write’s transaction when it is on the outbox table’s backend; after the commit when the transaction is on another backend (not atomic; avoid this); and right away, before the action returns, when there is no transaction. Writes that omit record change events write no rows.

  • If the outbox insert fails inside a transaction, the record write fails too (the listener is a required listener), so the caller rolls back. Without a transaction the records are already stored; the failure is logged and counted.

  • The relay claims each destination’s oldest PENDING rows with a lease (SELECT …​ FOR UPDATE SKIP LOCKED on PostgreSQL, then lease_owner and lease_expires_at). It sends them in one broker transaction in id order, then marks them SENT. Only one node sends a destination’s rows at a time, so order per destination is kept.

  • Each message carries the outbox row’s uuid as the JMS string property qqq_message_id. Directly published messages use the event id. Delivery is at least once: a send whose rows were not marked SENT (because of a crash, a lost database connection, or an expired lease) is sent again with the same id. Consumers dedupe on that id.

  • A failed send while the broker is connected counts an attempt on the head row. That row is retried alone after a doubling backoff (initialBackoffMs 1000 to maxBackoffMs 300000) and blocks its destination meanwhile. After maxAttempts (10) it becomes DEAD. A failed send while the broker is unreachable does not count, and the rows wait. To retry a DEAD row, set its status back to PENDING and attempts to 0.

  • SENT rows are purged after retentionDays (7), checked every purgeIntervalMs (1 h). DEAD rows are kept.

  • The core.esb.outbox health check is DOWN while any row is DEAD and DEGRADED when the oldest PENDING row is older than pendingAlertMs (5 min). It reports dead, pending and oldest-pending-age metrics.

  • Other settings: batchSize (100), pollIntervalMs (1000; a commit on the node wakes the relay sooner), and leaseMs (60000).

  • The relay is started and stopped by EsbRuntimeService, apart from the trigger runtime. It runs on a daemon thread and stops within 5 s at application stop. Running it on several nodes needs a backend with SKIP LOCKED (PostgreSQL). On other backends it relies on leases alone, which is safe for one node.

  • Process lifecycle events and EsbPublishAction still publish directly.

Idempotent triggers

Failovers, lost commits, and outbox re-sends redeliver messages. With dedupe enabled, each trigger records the message ids it processed in a processed-message table and skips a message it already processed. The message is still acknowledged, and the process does not run again.

qInstance.addTable(EsbProcessedMessageTable.defineTable("esb_processed_message", "rdbms"));
EsbInstanceMetaData.of(qInstance)
   .withDedupe(new EsbDedupeConfig().withEnabled(true));   // retentionDays defaults to 7

The key is the message’s qqq_message_id property, falling back to the CloudEvent id (ce_id). The trigger checks the table before running and records the ids after a successful run, before the JMS commit. A failed run is not recorded, so its retry runs. A failure between the run and the record can still repeat a run, so processes should stay idempotent. Records older than retentionDays are purged. A redelivery after that runs again.

The record is a shared table, not a per-node cache: after a failover the redelivery often reaches a different node, and a restarted node has an empty cache. Every message needs the database check either way, so no cache sits in front of it. The unique (trigger_name, message_id) index keeps the record correct when nodes race.

Host DDL (PostgreSQL, Liquibase formatted SQL)

Add only the tables you enable. Columns are snake_case and there are no foreign keys.

--liquibase formatted sql

--changeset qqq:esb-outbox-1
CREATE TABLE esb_outbox
(
   id               BIGSERIAL PRIMARY KEY,
   uuid             VARCHAR(36)  NOT NULL,
   destination_name VARCHAR(250) NOT NULL,
   event_type       VARCHAR(250),
   event_json       TEXT         NOT NULL,
   status           VARCHAR(20)  NOT NULL,
   attempts         INTEGER      NOT NULL DEFAULT 0,
   next_attempt_at  TIMESTAMP,
   lease_owner      VARCHAR(250),
   lease_expires_at TIMESTAMP,
   last_error       TEXT,
   created_at       TIMESTAMP    NOT NULL DEFAULT now(),
   sent_at          TIMESTAMP
);
CREATE UNIQUE INDEX esb_outbox_uuid_uk ON esb_outbox (uuid);
CREATE INDEX esb_outbox_pending_idx ON esb_outbox (destination_name, id) WHERE status = 'PENDING';
CREATE INDEX esb_outbox_status_sent_idx ON esb_outbox (status, sent_at);
--rollback DROP TABLE esb_outbox;

--changeset qqq:esb-processed-message-1
CREATE TABLE esb_processed_message
(
   id           BIGSERIAL PRIMARY KEY,
   trigger_name VARCHAR(250) NOT NULL,
   message_id   VARCHAR(250) NOT NULL,
   processed_at TIMESTAMP    NOT NULL DEFAULT now()
);
CREATE UNIQUE INDEX esb_processed_message_uk ON esb_processed_message (trigger_name, message_id);
CREATE INDEX esb_processed_message_processed_at_idx ON esb_processed_message (processed_at);
--rollback DROP TABLE esb_processed_message;

Single broker or HA

The mode is chosen by the provider url alone. The URL is passed to the Artemis client untouched.

Mode Provider url

Single broker

tcp://broker:61616

HA (live/backup pair or cluster)

(tcp://broker-a:61616,tcp://broker-b:61616)?ha=true&reconnectAttempts=-1&retryInterval=1000

  • Single broker: the client does not reconnect by itself. A lost connection reaches the JMS ExceptionListener, and EsbConnectionManager reconnects with backoff (1 s, doubling to 30 s). Then it fires its listeners, so triggers re-subscribe and the control channel listens again. Publishing sessions from the old connection are discarded.

  • HA: the client connects to the first broker that answers and fails over by itself, keeping the connection and re-attaching its sessions. It reports the failover to the ExceptionListener with code FAILOVER. The manager keeps the connection and only fires its listeners, so triggers rebuild their consumers on the new broker. A transacted commit cut by the failover throws; the trigger rolls back, and the message is redelivered (and skipped, with dedupe, if its run was recorded). A publish whose commit a failover rolled back (or left in doubt) is retried once on a fresh session; the outbox relay retries until the send succeeds. If the client gives up (a finite reconnectAttempts), the manager takes over as in single-broker mode.

  • Defaults applied unless the URL sets them: a 5 s connect timeout per connector and a 30 s callFailoverTimeout. The client’s own default of -1 would block a send or commit for as long as a failover lasts, which is forever when reconnectAttempts=-1 and every broker is down.

  • RabbitMQ: use its own clustering and a single AMQP URI (or a load balancer). The manager’s reconnect covers a node loss.

Visibility, management, and permissions

ESB endpoints under /qqq/v1/esb expose the overview, table and process publications, triggers, counters, and browsable queue or dead-letter messages. Counters are local to the node. Broker queue depth and consumer counts require the provider’s optional managementUrl and management credentials: Artemis uses Jolokia; RabbitMQ uses its management HTTP API.

The ESB overview requires esbView.hasAccess. A table’s ESB details require that table’s READ permission; a process’s details require process access. A user can browse a destination when they have ESB view access, READ on a table publishing there, or access to a process it triggers. Trigger pause, resume, restart, replay, and supported queue move actions require esbOperate.hasAccess; purge and delete require esbDelete.hasAccess. The server enforces these permissions. Artemis supports queue pause/resume, purge, delete, and move through management; RabbitMQ supports purge but not queue-level pause, selected delete, or move. Both support JMS queue browsing and QQQ trigger pause/resume/restart. Runtime trigger pause state is local and resets to startPaused on restart.

API server (OpenAPI)

See the endpoint specification API and the hosted framework documentation. Endpoint tags are optional.

Custom Application Code

QContext

The class QContext contains a collection of thread-local variables, to define the current context of the QQQ code that is currently running. For example, what QInstance (meta-data container) is being used, what QSession (user attributes) is active, etc.

Most of the time, main-line application code does not need to worry about setting up the QContext - although unit-test code can is a common exception to that rule. This is because all of QQQ’s entry-points into execution (e.g., web context handlers, CLI startup methods, schedulers, and multi-threaded executors) take care of initializing the context.

It is more common though, for application code to need to get data from the context, such as the current session or any piece of meta-data from the QInstance. The methods to access data from the QContext are fairly straightforward:

Examples

Get a QTableMetaData from the active QInstance
QTableMetaData myTable = QContext.getQInstance().getTable("myTable");
for(QFieldMetaData field : myTable.getFields().values())
{
   // ...
}
Get a security key value for the current user session
QSession           session   = QContext.getQSession();
List<Serializable> clientIds = session.getSecurityKeyValues("clientId");

QRecords

Almost all code inside a QQQ application will be dealing with Records (aka Tuples or Rows). That is: a collection of named values, representing a single Entity, Fact, or, Row from a QQQ Table.

The class that QQQ uses to work with records is called: QRecord.

Values

At its core, a QRecord is a wrapper around a Map<String, Serializable> values. These are the actual values for the fields in the table for the record. That is, direct representations of the values as they are stored in the QQQ Backend.

The keys in the values map are names from the QQQ Fields in the QQQ Table.

The values in values map are declared as Serializable (to help ensure the serializability of the QRecord as a whole). In practice, their types will be based on the QFieldType of the QQQ Field that they correspond to. That will typically be one of: String, Integer, Boolean, BigDecimal, Instant, LocalDate, LocalTime, or byte[]. Be aware that null values may be in the values map, especially/per if the backend/table support null. JSON serialization preserves explicit nulls, empty strings and empty arrays in this map. An absent field remains absent; display formatting and hidden/password rules are applied separately before serialization.

To work with the values map, the following methods are provided:

  • setValue(String fieldName, Serializable value) - Sets a value for the specified field in the record.

    • Overloaded as setValue(String fieldName, Object value) - For some cases where the value may not be known to be Serializable. In this overload, if the value is null or Serializable, the primary version of setValue is called. Otherwise, the value is passed through String::valueOf, and the result is stored.

    • Overloaded as setValue(QFieldMetaData field, Serializable value) - which simply defers to the primary version of setValue, passing field.getName() as the first parameter.

  • removeValue(String fieldName) - Remove the given field from the values map.

    • Note that in some situations this is an important distinction from having a null value in the map (See UpdateAction).

  • setValues(Map<String, Serializable> values) - Sets the full map of values.

  • getValues() - Returns the full map of values.

  • getValue(String fieldName) - Returns the value for the named field - possibly null - as a Serializable.

  • Several type-specific variations of getValueX(String fieldName), where internally, values will be not exactly type-cast, but effectively converted (if possible) to the requested type. These conversions are done using the ValueUtils.getValueAsX(Object) methods. These methods are generally the preferred/cleanest way to get record values in application code, when it is needed in a type-specific way .

    • getValueString(String fieldName)

    • getValueInteger(String fieldName)

    • getValueBoolean(String fieldName)

    • getValueBigDecimal(String fieldName)

    • getValueInstant(String fieldName)

    • getValueLocalDate(String fieldName)

    • getValueLocalTime(String fieldName)

    • getValueByteArray(String fieldName)

Display Values

In addition to the values map, a QRecord contains another map called displayValues, which only stores String values. That is to say, values for other types are stringified, based on their QQQ Field's type and displayFormat property. In addition, fields which have a possibleValueSource property will have their translated values set in the displayValues map.

By default, a QRecord will not have its displayValues populated. To populate displayValues, the QueryAction and GetAction classes take a property in their inputs called shouldGenerateDisplayValues, which must be set to true to generate displayValues. In addition, these two actions also have a property shouldTranslatePossibleValues in their inputs, which needs to be set to true if possible-value lookups are to be performed.

As an alternative to the shouldGenerateDisplayValues and shouldTranslatePossibleValues inputs to QueryAction and GetAction, one can directly call the QValueFormatter.setDisplayValuesInRecords and/or QPossibleValueTranslator.translatePossibleValuesInRecords methods. Or, for special cases, setDisplayValue(String fieldName, String displayValue) or setDisplayValues(Map<String, String> displayValues) can be called directly.

Backend Details

Sometimes a backend may want to place additional data in a QRecord that doesn’t exactly correspond to a field. To do this, the Map<String, Serializable> backendDetails member is used.

For example, an API backend may store the full JSON String that came from the API as a backend detail in a QRecord. Or fields that are marked as isHeavy, if the full (heavy) value of the field hasn’t been fetched, then the lengths of any such heavy fields may be stored in backendDetails.

QQQ’s Jackson serialization of a QRecord excludes the API provider’s raw jsonSourceObject backend-detail entry, including on associated records. That internal payload can contain fields omitted or masked in the record’s public values. Other details, including heavy-field lengths, remain serialized. The internal map, record copies and deserialization retain the payload; explicitly serializing the raw map bypasses the QRecord serialization boundary.

Copies used in write callbacks

Update/Delete old-record prefetch can retain row identity separately from visible field values. A hidden, heavy or masked primary key remains subject to its presentation rules. Use new QRecord(record) to preserve that identity when returning copied snapshots, including copies placed in immutable or reordered lists.

A captured record cannot be retargeted by changing its public key. During write prefetch, post-query replacements must identify rows in the native result; pre-delete replacements must identify original prefetched targets. A fresh replacement with a valid original key can work, but reconstruction from a sanitized map may lose the identity needed for matching and reject before DML.

The internal identity is absent from values, displayValues, backendDetails, QQQ JSON and Java serialization. Deserialization does not restore it, so a JSON or Java serialization round trip is not a substitute for the copy constructor in these callbacks. This bookkeeping does not change ordinary Query/Get field loading, masking or public serialization.

Errors and Warnings

A QRecord can carry QErrorMessage and QWarningMessage lists, accessed through getErrors()/getWarnings() and populated with addError()/addWarning(). Inspect the records returned by an action for row-level validation results; a returned record alone does not imply that a write succeeded. getErrorsAsString() and getWarningsAsString() provide text summaries. The action determines how errors and warnings affect processing.

Associated Records

The associatedRecords map holds named lists of related QRecords. Use withAssociatedRecords(name, records) or withAssociatedRecord(name, record) to populate it. Retrieval actions include associations only when requested and supported by the table metadata. This map is separate from flattened fields selected through a QueryJoin. See QRecord.

Recursive legacy HTTP forms

For legacy /data/{table} INSERT and UPDATE forms, send X-QQQ-Association-Format: record-v1 and one associations form field containing a JSON object. Each exact association name maps to an array of child envelopes. The child table is inferred from active metadata. A field named values belongs inside the envelope’s values object:

{"pets":[{"values":{"name":"Charlie","speciesId":1},"associatedRecords":{"notes":[{"values":{"note":"Copied note"}}]}}]}

The only envelope properties are values (required object) and associatedRecords (optional object). Scalar field values, null and flat scalar arrays are preserved. A BLOB uses {"base64":"AAH/gEEA"}; other object-shaped field values are rejected. Named empty arrays remain explicit replacements during UPDATE, while omitted groups remain untouched. The tree may contain at most 64 child-record levels. Unknown/inactive names, malformed JSON, invalid binary or unsupported format declarations return HTTP 400 before DML. The version header is for form requests, not JSON request bodies. Forms without it retain their flat child-record format.

These requests run ordinary INSERT/UPDATE validation, normalization, defaults and customizers. Supplied primary keys are data; a client preparing a copy must remove inherited keys and provide any required new manual keys. Inspect nested returned errors as well as parent errors. A successful earlier write can remain after a later native or business-validation failure without a supported caller transaction and rollback.

Custom fields

JsonPathStorage maps ordinary QFieldMetaData to a key in a shared backend JSON column. Runtime providers can publish these fields without rebuilding the application. The host owns schema migrations; refreshing metadata never runs DDL.

Storage declaration

new QFieldMetaData("priority", QFieldType.INTEGER)
   .withBackendStorage(new JsonPathStorage()
      .withColumnName("qqq_custom_fields")
      .withKey("priority"));

FieldBackendStorage is the extensible storage contract. Storage declarations are cloned with their fields. A storage declaration overrides the field’s backendName.

  • The JSON column must not also be declared as a normal table field, including a normal field whose backendName points to that column.

  • Keys match [a-z][a-z0-9_]{0,40}. Two logical fields cannot share a column/key pair.

  • Supported types: STRING, TEXT, INTEGER, LONG, DECIMAL, BOOLEAN, DATE and DATE_TIME.

  • Reads expose typed values under the logical field name. The backing column is internal.

  • Inserts collect non-null values into one object per JSON column.

  • Updates include only fields present in the update record. Null removes a key; an absent field leaves its key unchanged. Empty strings remain values.

Backend support

Backend

Support

PostgreSQL

JSONB insert, typed select, filters, ordering, aggregation/grouping and atomic key merges.

Memory

Nested maps, typed projection, filters, ordering and key merges; no schema needed.

H2

Unsupported for JsonPathStorage. Use memory for portable unit tests and PostgreSQL Testcontainers for SQL integration tests.

MySQL

Not implemented in this phase. Ordinary fields continue using the existing dialect.

MongoDB

Not implemented in this phase.

Unsupported RDBMS dialects reject JsonPathStorage operations explicitly. No test-only H2 aliases are installed. H2 has a simplified JSON accessor, but lacks the extraction and merge functions needed for this dialect implementation. This phase deliberately implements the complete storage contract only for PostgreSQL and memory. The qbit’s EAV fallback, field designer, schema availability checks and index management are separate work.

PostgreSQL

The host migration adds the column and, optionally, a containment index:

ALTER TABLE orders ADD COLUMN qqq_custom_fields jsonb NOT NULL DEFAULT '{}';
CREATE INDEX orders_custom_fields_gin ON orders
   USING gin (qqq_custom_fields jsonb_path_ops);

STRING and TEXT read with (column→>'key'). Other types append ::integer, ::bigint, ::numeric, ::boolean, ::date or ::timestamptz. The same expression is used for selection, range predicates, ordering, aggregates and group-by. Dates and instants are stored as ISO strings; numbers and booleans are native JSON scalar values.

Equality uses column @> ?::jsonb with one typed JSON object parameter. IN is an OR of these containment predicates, one parameter per alternative, allowing a GIN index. Case-insensitive or field-function comparisons use the typed expression instead. Ranges and sorting can benefit from host-managed expression indexes.

Updates use:

column = (COALESCE(column, '{}'::jsonb) || ?::jsonb) - ?::text[]

The first parameter contains changed non-null keys; the second lists changed null keys. The merge executes inside PostgreSQL’s row update, so concurrent updates of different keys survive. Updates to the same key have ordinary database last-writer semantics.

Runtime field overlay

Register a RuntimeMetaDataProviderInterface through RecordFeaturesMetaData.ofOrWithNew(instance).withRuntimeProviders(…​). Its contribution can include fields, virtual fields, section placement, search field names, feature overrides and a generation:

return new RuntimeMetaDataContribution()
   .withGeneration(revision)
   .withTables(Map.of("order", new RuntimeTableMetaDataContribution()
      .withFields(List.of(priorityField))
      .withSectionPlacements(Map.of("priority", "customFields"))
      .withSearchFields(List.of("priority"))));

Call RuntimeMetaDataOverlay.refresh(instance) after the provider’s data is committed. Each refresh gathers providers, clones affected tables, removes the overlay’s former fields, applies the new contributions, and runs table enrichment and validation. Code-defined names win clashes, which are logged. An invalid table keeps its previous published metadata while valid tables can advance. Removing a provider or a field removes its prior contribution.

New sections are appended after existing sections, labeled "Custom fields", at tier T2. Stored fields without explicit placement use customFields. Hidden and virtual fields are not automatically placed in base sections; virtual fields retain their existing table presentation rules. Search fields must satisfy the normal table search validation rules.

Contributed fields may carry AiFieldMetaData with a DataClassification (e.g. PHI); masking, audit and egress controls treat them like code fields.

Validated replacements publish through one copy-on-write table map under the instance lock. The instance’s startup validation state is preserved. Readers holding an older table object may finish using it; new metadata lookups see the replacement. Refresh increments metaDataGeneration (or adopts a higher provider generation). Frontend metadata and the v1 metadata endpoints read the current fields and generation. Consumers should refetch when the generation changes. Polling, after-commit registration and cross-pod notifications belong to the runtime provider/qbit integration.

QRecordEntities

While QRecords provide a flexible mechanism for passing around record data in a QQQ Application, they have one big disadvantage from the point-of-view of a Java Application: They do not provide a mechanism to ensure compile-time checks of field names or field types.

As such, an alternative mechanism exists, which allows records in a QQQ application to be worked with following a more familiar Java Bean (Entity Bean) like pattern. This mechanism is known as a QRecordEntity.

Specifically speaking, QRecordEntity is an abstract base class, whose purpose is to be the base class for entity-bean classes. Using reflection, QRecordEntity is able to convert objects back and forth from QRecord to specific entity-bean subtypes.

For example, the method QRecordEntity::toQRecord() converts a record entity object into a QRecord.

Inversely, QRecordEntity::populateFromQRecord(QRecord record) sets fields in a record entity object, based on the values in the supplied QRecord. It is conventional for a subclass of QRecordEntity to have both a no-arg (default) constructor, and a constructor that takes a QRecord as a parameter, and calls populateFromQRecord.

In addition to these constructors, a QRecordEntity subclass will generally contain:

  • A public static final String TABLE_NAME, used throughout the application as a constant reference to the name for the QQQ Table.

  • A series of private fields, corresponding to the fields in the table that the entity represents.

    • If these fields are annotated as @QField(), then the QQQ Table meta-data for the table that the entity represents can in part be inferred by QQQ, by using the method QTableMetaData::withFieldsFromEntity.

  • getX(), setX(), and withX() methods for all of the entity’s fields.

Examples

Example Definition of a QRecordEntity subclass: Person.java
/*******************************************************************************
 ** QRecordEntity for the person table.
 *******************************************************************************/
public class Person extends QRecordEntity
{
   public static final String TABLE_NAME = "person";

   @QField(isEditable = false)
   private Integer id;

   @QField()
   private String firstName;

   @QField()
   private String lastName;

   @QField()
   private Integer age;

   // all other fields

   /*******************************************************************************
    ** Default constructor
    *******************************************************************************/
   public Person()
   {
   }


   /*******************************************************************************
    ** Constructor that takes a QRecord
    *******************************************************************************/
   public Person(QRecord record)
   {
      populateFromQRecord(record);
   }


   /*******************************************************************************
    ** Custom method to concatenate firstName and lastName
    *******************************************************************************/
   public String getFullName()
   {
      //////////////////////////////////////////////////////////////////////
      // if there were more than an example, we could do some null-checks //
      // here to avoid silly output like "Bobby null" :)                  //
      //////////////////////////////////////////////////////////////////////
      return (firstName + " " + lastName);
   }

   // all getter, setter, and fluent setter (wither) methods
}

The core ORM actions and process-execution actions of QQQ work with QRecords. However, application engineers may want to apply patterns like the following example, to gain the compile-time safety of QRecordEntities:

Example Usage of a QRecordEntity
//////////////////////////////////////////////////////////////////////
// assume we're working with the "person" table & entity from above //
//////////////////////////////////////////////////////////////////////
List<QRecord> recordsToUpdate = new ArrayList<>();
for(QRecord record : inputRecordList)
{
   /////////////////////////////////////////////////////////////////////////////
   // call the constructor that copies values from the record into the entity //
   /////////////////////////////////////////////////////////////////////////////
   Person person = new Person(record);

   ////////////////////////////////////////////////
   // call a custom method defined in the entity //
   ////////////////////////////////////////////////
   LOG.info("Processing: " + person.getFullName());

   /////////////////////////////////////////////////////////////
   // age is an Integer, so no type-conversion is needed here //
   /////////////////////////////////////////////////////////////
   person.setAge(person.getAge() + 1);

   ///////////////////////////////////////////////////////////////////////////
   // to pass the updated records to UpdateAction, convert them to QRecords //
   ///////////////////////////////////////////////////////////////////////////
   recordsToUpdate.add(person.toQRecord());
}

Process Backend Steps

In many QQQ applications, much of the code that engineers write will take the form of Backend Steps for QQQ Processes. Such code is defined in classes which implement the interface BackendStep. This interface defines only a single method:

BackendStep.java
public interface BackendStep
{
   /*******************************************************************************
    ** Execute the backend step - using the request as input, and the result as output.
    **
    *******************************************************************************/
   void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException;
}

Process backend steps have access to state information - specifically, a list of records, and a map of name=value pairs - in the input & output objects. This state data is persisted by QQQ between steps (e.g., if a frontend step is presented to a user between backend steps).

RunBackendStepInput

All input data to the step is available in the RunBackendStepInput object. Key methods in this class are:

  • getRecords() - Returns the List of QRecords that are currently being acted on in the process.

  • getValues() - Returns a Map of String → Serializable; name=value pairs that are the state-data of the process.

    • Values can be added to this state from a process’s meta-data, from a screen, or from another backend step.

  • getValue(String fieldName) - Returns a specific value, by name, from the process’s state.

    • This method has several variations that return the value as a specific type, such as getValueString, getValueInteger, getValueBoolean…​

  • getAsyncJobCallback() - Accessor for an AsyncJobCallback object, which provides a way for the process backend step to communicate about its status or progress with a user running the process in a frontend. Provides methods:

    • updateStatus(String message) - Where general status messages can be given. For example, "Loading census data"

    • updateStatus(int current, int total) - For updating a progress meter. e.g., "47 of 1701" would be display by calling .updateStatus(47, 1701)

  • getFrontendStepBehavior() - Enum, indicating what should happen when a frontend step is encountered as the process’s next step to run. Possible values are:

    • BREAK - Indicates that the process’s execution should be suspended, so that the screen represented by the frontend step can be presented to a user. This would be the expected behavior if a process is being run by a user from a UI.

    • SKIP - Indicates that frontend steps should be skipped. This would be the expected behavior if a process is running from a scheduled job (without a user present to drive it), for example.

    • FAIL - Indicates that the process should end with an exception if a frontend step is encountered.

    • A backend step may want to act differently based on its frontendStepBehavior. For example, additional data may be looked up for displaying to a user if the behavior is BREAK.

  • getBasepullLastRunTime() - For Basepull processes, this is the Instant stored in the basepull table as the process’s last run time.

RunBackendStepOutput

All output from a process step should be placed in its RunBackendStepOutput object (and/or stored to a backend, as appropriate). Key methods in this class are:

  • addValue(String fieldName, Serializable value) - Adds a single named value to the process’s state, overwriting it the value if it already exists.

  • addRecord(QRecord record) - Add a QRecord to the process’s output.

  • addAuditSingleInput(AuditSingleInput auditSingleInput) - Add a new entry to the process’s list of audit inputs, to be stored at the completion of the process.

    • An AuditSingleInput object can most easily be built with the constructor: AuditSingleInput(String tableName, QRecord record, String auditMessage).

    • Additional audit details messages (sub-bullets that accompany the primary auditMessage) can be added to an AuditSingleInput via the addDetail(String message) method.

    • Note that at this time, the automatic storing of these audits is only provided by the execute step of a StreamedETLWithFrontendProcesses.

Example

Example of a BackendStep
/*******************************************************************************
 ** For the "person" table's "Add Age" process -
 ** For each input person record, add the specified yearsToAdd to their age.
 *******************************************************************************/
public class AddAge implements BackendStep
{
   /*******************************************************************************
    **
    *******************************************************************************/
   @Override
   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput)
   {
      /////////////////////////////////////////////////////////////////
      // get the yearsToAdd input field value from the process input //
      /////////////////////////////////////////////////////////////////
      Integer yearsToAdd = runBackendStepInput.getValueInteger("yearsToAdd");
      int totalYearsAdded = 0;

      ///////////////////////////////////////////////////
      // loop over the records passed into the process //
      ///////////////////////////////////////////////////
      for(QRecord record : runBackendStepInput.getRecords())
      {
         Integer age = record.getValueInteger("age");
         age += yearsToAdd;
         totalYearsAdded += yearsToAdd;

         ////////////////////////////////////////////////////////////////////////////////////////////
         // update the record with the new "age" value.                                            //
         // note that this update record object will implicitly be available to the process's next //
         // backend step, via the sharing of the processState object.                              //
         ////////////////////////////////////////////////////////////////////////////////////////////
         record.setValue("age", age);
      }

      /////////////////////////////////////////
      // set an output value for the process //
      /////////////////////////////////////////
      runBackendStepOutput.addValue("totalYearsAdded", totalYearsAdded);
   }
}

Backend Steps for StreamedETLWithFrontendProcesses

For StreamedETLWithFrontendProcess type processes, backend steps are defined a little bit differently than they are for other process types. In this type of process, the process meta-data defines 3 backend steps which are built-in to QQQ, and which do not have any custom application logic. These steps are:

  • StreamedETLPreviewStep

  • StreamedETLValidateStep

  • StreamedETLExecuteStep

For custom application logic to be implemented in a StreamedETLWithFrontendProcesses, an application engineer must define (up to) 3 backend step classes which are loaded by the steps listed above. These application-defined steps must extend specific classes (which themselves implement the BackendStep interface), to provide the needed logic of this style of process. These steps are:

  • Extract - a subclass of AbstractExtractStep - is responsible for Extracting records from the source table.

    • For this step, we can often use the QQQ-provided class ExtractViaQueryStep, or sometimes a subclass of it.

    • The Extract step is called before the Preview, Validate, and Result screens, though for the Preview screen, it is set to only extract a small number of records (10).

  • Transform - a subclass of AbstractTransformStep - is responsible for applying the majority of the business logic of the process. In ETL terminology, this is the "Transform" action - which means applying some type of logical transformation an input record (found by the Extract step) to generate an output record (stored by the Load step).

    • A Transform step’s runOnePage method will be called, potentially, multiple times, each time with a page of records in the runBackendStepInput parameter.

    • This method is responsible for adding records to the runBackendStepOutput, which will then be passed to the Load step.

    • This class is also responsible for implementing the method getProcessSummary, which provides the data to the Validate screen.

    • The runOnePage method will generally update ProcessSummaryLine objects to facilitate this functionality.

    • The Transform step is called before the Preview, Validate, and Result screens, consuming all records selected by the Extract step.

  • Load - a subclass of AbstractLoadStep - is responsible for the Load function of the ETL job. A quick word on terminology - this step is actually doing what we are more likely to think of as storing data - which feels like the opposite of “loading” - but we use the name Load to keep in line with the ETL naming convention…

    • The Load step is ONLY called before the Result screen is presented (possibly after Preview, if the user chose to skip validation, otherwise, after validation).

    • Similar to the Transform step, the Load step’s runOnePage method will be called potentially multiple times, with pages of records in its input.

    • As such, the Load step is generally the only step where data writes should occur.

      • e.g., a Transform step should not do any writes, as it will be called when the user is going to the Preview & Validate screens - e.g., before the user confirmed that they want to execute the action!

    • A common pattern is that the Load step just needs to insert or update the list of records output by the Transform step, in which case the QQQ-provided LoadViaInsertStep or LoadViaUpdateStep can be used, but custom use-cases can be built as well.

Another distinction between StreamedELTWithFrontendProcess steps and general QQQ process backend steps, is that the list of records in the input & output objects is NOT shared for StreamedELTWithFrontendProcess steps. The direct implication of this is, that a Transform step MUST explicitly call output.addRecord() for any records that it wants to pass along to the Load step.

Cancellation and transactions

Parent/child callback propagation, separate extraction inputs, preview callback preservation and new setup/transform/load/post-run/commit cancellation checks are deferred in #480. Existing preview code can replace the caller input’s callback with its extraction callback. Custom steps must manage cooperative cancellation and owned resource cleanup explicitly.

The existing transaction modes remain. Supported process/page transactions commit successful work, roll back their open transaction on the existing exception path, and close in finally. A cancellation flag alone does not establish the proposed exception/rollback boundary. Earlier page commits, auto-committed rows and custom external effects remain; operations already in progress are not undone. Preview and validation do not invoke the load step, and custom transforms should not write data.

Source examples

See AbstractTransformStep for runOnePage() and getProcessSummary(), and LoadViaInsertStep for writing the transformed records. The table-sync walkthrough demonstrates a concrete transform. Override the lifecycle methods required by your use case and keep writes in the execution/load phase.

Downloading generated files

A DOWNLOAD_FORM uses downloadFileName with either serverFilePath or storageTableName / storageReference. In 4.0, trusted backend code must explicitly authorize its output after generating the file. The built-in basic-report and saved-report steps do this automatically. Custom steps register local output with com.kingsrook.qqq.backend.core.actions.processes.ProcessFileDownload:

output.addValue("downloadFileName", "report.csv");
output.addValue("serverFilePath", ProcessFileDownload.register(generatedFile));

For a storage-backed output, close its output stream, call ProcessFileDownload.registerStorage(storageInput), and return the same storageInput.getTableName() and storageInput.getReference() as storageTableName and storageReference. Register only an output that the process has authorized for the caller. Never register an arbitrary path/reference copied from HTTP input or echoed process values. Table READ alone does not grant access to arbitrary storage objects.

Grants belong to the application instance, originating authentication session and selected backend variants. They expire after 30 minutes, are limited to 10,000 cached outputs per JVM, and are lost on restart or eviction. Downloads must reach the JVM that generated the grant. Applications using multiple nodes must route process and download requests to the same node. Authentication modules must preserve QSession.idReference across requests (or uuid when no reference is supplied); credential renewal can require regenerating the report. Grants do not delete output files or replace the application’s own cleanup policy. Local filesystem storage references must remain relative to the configured table directory, including when symbolic links are present.

One-time secrets in process output

A backend step can return a secret to the user exactly once using the shared com.kingsrook.qqq.backend.core.model.secrets.OneTimeSecret type:

runBackendStepOutput.addOneTimeSecret(new OneTimeSecret()
   .withLabel("Access token")
   .withValue(tokenKey)
   .withMasked("****" + tokenKey.substring(tokenKey.length() - 4))
   .withUsageText("Copy this token now. It will not be shown again."));

The first consumer is qbit-authentik’s Create Token process, which returns an access-token key. Streamed ETL transform and load steps can call the same API during runOnePage; the page secrets propagate to the enclosing backend step output. RunProcessAction gathers backend step secrets in execution order in RunProcessOutput.getOneTimeSecrets() for the current invocation only.

Secrets live in transient output fields, never in ProcessState, values, or records. The output getters are excluded from JSON serialization so process traces and lifecycle listener output JSON cannot expose them. OneTimeSecret.toString() omits the raw value. Only the masked strings are appended to the stored oneTimeSecretsMasked value (RunProcessAction.ONE_TIME_SECRETS_MASKED), allowing later screens to show placeholders. Supply safe masked text; do not copy the raw token into labels, masked text, state, records, summaries, exception messages, progress messages, or logs.

Synchronous v1 init/step and legacy process responses include an optional oneTimeSecrets: [{label, value, masked, usageText}] array and Cache-Control: no-store. The v1 API reuses the preferences PreferenceSecretV1 schema. If execution exceeds stepTimeoutMillis (3 seconds by default, or a zero timeout forces an async job), the response contains a jobUUID. The finished job’s secrets are then parked in memory (OneTimeSecretHandoff, keyed by process UUID, 5-minute time to live, never in process state) and returned, with Cache-Control: no-store, by the first status poll that sees the job complete; later polls never include them. A poll that reaches a different server instance, or arrives after the time to live, finds nothing: the secret is lost by design and only the masked state placeholders remain (rotate to get a new one).

Clients should display each value once, offer a copy action, and show usageText. Never cache, persist, log, or send the response to analytics. After dismissal, use the masked state placeholders to indicate that the secret was already shown.

Rendering Widgets

WidgetRenderer classes

In general, to fully implement a Widget, you must define its QWidgetMetaData, and supply a subclass of AbstractWidgetRenderer, to provide the data to the widget. (Note the "No Code" category of widgets, which are an exception to this generalization).

The only method required in a subclass of AbstractWidgetRenderer is:

public RenderWidgetOutput render(RenderWidgetInput input) throws QException

The fields available in RenderWidgetInput are:

  • InputSource inputSource - Defaults to QInputSource.SYSTEM for direct Java calls. Javalin and Slack widget entry points set QInputSource.USER. Custom renderers should carry this source into data actions and enforce the table permissions they require; client query parameters do not set it.

  • Map<String, String> queryParams - These are parameters supplied by the frontend, for example, if a user selected values from dropdowns to control a dimension of your widget, those name/value pairs would be in this map. Similarly, if your widget is being included on a record view screen, then the record’s primary key will be in this map.

  • QWidgetMetaDataInterface widgetMetaData - This is the meta-data for the widget being rendered. This can be useful in case you are using the same renderer class for multiple widgets.

The only field in RenderWidgetOutput is:

  • QWidgetData widgetData - This is a base class, with several attributes, and more importantly, several subclasses, specific to the type of widget that is being rendered.

Widget-Type Specific Rendering Details

Different widget types expect & require different types of values to be set in the RenderWidgetOutput by their renderers.

Pie Chart

The WidgetType.PIE_CHART requires an object of type ChartData. The fields on this type are:

  • chartData an instance of ChartData.Data, which has the following fields:

    • labels - List<String>, required - the labels for the slices of the pie.

    • datasets - List<Dataset> required - the data for each slice of the pie. For a Pie chart, only a single entry in this list is used (other chart types using ChartData may support more than 1 entry in this list). Fields in this object are:

      • label - String, required - a label to describe the dataset as a whole. e.g., "Orders" for a pie showing orders of different statuses.

      • data - List<Number>, required - the data points for each slice of the pie.

      • color - String - HTML color for the slice

      • urls - List<String> - Optional URLs for slices of the pie to link to when clicked.

      • backgroundColors - List<String> - Optional HTML color codes for each slice of the pie.

Pie chart widget example
// meta data
QWidgetMetaData widget = new QWidgetMetaData()
    .withName("pieChartExample")
    .withType(WidgetType.PIE_CHART.getType())
    .withGridColumns(4)
    .withIsCard(true)
    .withLabel("Pie Chart Example")
    .withCodeReference(new QCodeReference(PieChartExampleRenderer.class));
PieChartExampleRenderer implementation
public class PieChartExampleRenderer extends AbstractWidgetRenderer
{
   /*******************************************************************************
   ** main method of the widget renderer
   *******************************************************************************/
   @Override
   public RenderWidgetOutput render(RenderWidgetInput input) throws QException
   {
      List<String> labels = List.of("Apple", "Orange", "Banana", "Lime", "Blueberry");
      List<String> colors = List.of("#FF0000", "#FF8000", "#FFFF00", "#00FF00", "#0000FF");
      List<Number> data = List.of(100, 150, 75, 100, 200);

      ChartData chartData = new ChartData()
         .withChartData(new ChartData.Data()
            .withLabels(labels)
            .withDatasets(List.of(
               new ChartData.Data.Dataset()
                  .withLabel("Flavor")
                  .withData(data)
                  .withBackgroundColors(colors))));

      return (new RenderWidgetOutput(chartData));
   }
}
Composite

A WidgetType.COMPOSITE is built by using one or more smaller elements, known as Blocks. Note that Blocks can also be used in the data of some other widget types (specifically, within a cell of a Table-type widget, or (in the future?) as a header above a pie or bar chart).

A composite widget renderer must return data of type CompositeWidgetData, which has the following fields:

  • blocks - List<AbstractBlockWidgetData>, required - The blocks (1 or more) being composited together to make the widget. See below for details on the specific Block types.

  • styleOverrides - Map<String, Serializable> - Optional map of CSS attributes (named following javascriptStyleCamelCase) to apply to the <div> element that wraps the rendered blocks.

  • layout - Layout enum - Optional specifier for how the blocks should be laid out. e.g., predefined sets of CSS attributes to achieve specific layouts.

    • Note that some blocks are designed to work within composites with specific layouts. Look for matching names, such as Layout.BADGES_WRAPPER to go with NumberIconBadgeBlock.

Composite widget example - consisting of 3 Progress Bar Blocks, and one Divider Block
// meta data
QWidgetMetaData widget = new QWidgetMetaData()
    .withName("compositeExample")
    .withType(WidgetType.COMPOSITE.getType())
    .withGridColumns(4)
    .withIsCard(true)
    .withLabel("Composite Example")
    .withCodeReference(new QCodeReference(CompositeExampleRenderer.class));
CompositeExampleRenderer implementation
public class CompositeExampleRenderer extends AbstractWidgetRenderer
{
   @Override
   public RenderWidgetOutput render(RenderWidgetInput input) throws QException
   {
      CompositeWidgetData data = new CompositeWidgetData();

      data.addBlock(new ProgressBarBlockData()
         .withValues(new ProgressBarValues()
            .withHeading("Blocks")
            .withPercent(new BigDecimal("78.5"))));

      data.addBlock(new ProgressBarBlockData()
         .withValues(new ProgressBarValues()
            .withHeading("Progress")
            .withPercent(new BigDecimal(0))));

      data.addBlock(new DividerBlockData());

      data.addBlock(new ProgressBarBlockData()
         .withStyles(new ProgressBarStyles().withBarColor("#C0C000"))
         .withValues(new ProgressBarValues()
            .withHeading("Custom Color")
            .withPercent(new BigDecimal("75.3"))));

      return (new RenderWidgetOutput(data));
   }
}
Additional widget contracts

Use the matching data class when implementing another widget type. The frontend determines how each type is displayed.

Widget Data reference

Charts

ChartData

Child record list

ChildRecordListData

Statistics

StatisticsData

Parent widget

ParentWidgetData

Table

TableData

HTML

RawHTML

Divider

DividerWidgetData

Process

ProcessWidgetData

Stepper

StepperData

For the complete set, see WidgetType and the built-in renderers.

Block-type Specific Rendering Details

For Composite-type widgets (or other widgets which can include blocks), there are specific data classes required to be returned by the widget renderer.

Each block type defines a subclass of AbstractBlockWidgetData, which is a generic class with 3 type parameters:

  • V - an implementation of BlockValuesInterface - to define the type of values that the block uses.

  • S - an implementation of BlockSlotsInterface (expected to be an enum) - to define the "slots" in the block, that can have Tooltips and/or Links applied to them.

  • SX - an implementation of BlockStylesInterface - to define the types of style customizations that the block supports.

These type parameters are designed to ensure type-safety for the application developer, so that a block accepts its matching values, slot identifiers and style configuration.

Additional Tips

  • To make a Dashboard page (e.g., an App consisting of Widgets) with a parent widget use the parent widget’s label as the page’s label:

    • On the QAppMetaData that contains the Parent widget, call .withSupplementalMetaData(new MaterialDashboardAppMetaData().withShowAppLabelOnHomeScreen(false)).

    • In the Parent widget’s renderer, on the ParentWidgetData, call setLabel("My Label") and setIsLabelPageTitle(true).

Table Customizers

Register a QCodeReference under a supported TableCustomizers key on the table metadata. See customizer keys and the customizer contract.

QQQ Core Actions

QueryAction

The QueryAction is the basic action that is used to get records from a QQQ Table, generally according to a Filter. In SQL/RDBMS terms, it is analogous to a SELECT statement, where 0 or more records may be found and returned.

Examples

Basic Form
QueryInput input = new QueryInput();
input.setTableName("orders");
input.setFilter(new QQueryFilter()
   .withCriteria(new QFilterCriteria("total", GREATER_THAN, new BigDecimal("3.50")))
   .withOrderBy(new QFilterOrderBy("orderDate", false)));
QueryOutput output = new QueryAction().execute(input);
List<QRecord> records = output.getRecords();

Details

QueryAction, in general, can be called in two different modes:

  1. The default fetches records synchronously, performs the post-processing requested in the QueryInput, and returns them as a list in the QueryOutput.

  2. The alternative use-case is meant for larger operations, where one wouldn’t want all records matching a query in-memory. For this scenario, a RecordPipe object can be passed in to the QueryInput. This causes QueryAction to run its post-processing action on records as they are placed into the pipe, and to potentially block (per the pipe’s settings). This method of usage needs to be done on a separate thread from another thread which would be consuming records from the pipe. QQQ’s AsyncRecordPipeLoop class provides an implementation of doing such a dual-threaded job.

If the QQQ Table has a TableCustomizers.POST_QUERY_RECORD customizer (role postQueryRecord), it runs after records are fetched and before they leave the QueryAction through its QueryOutput or RecordPipe. The customizer’s returned list is used for subsequent formatting, association expansion and privacy processing; returning null fails the query. Direct and buffered delivery support changed list sizes. A plain single-record RecordPipe retains an existing limitation: removing the current record can fail, and expanding it can lose results (#491). Pipe termination and accepted-record accounting corrections are deferred in #492. A consumer already in progress can return records read before termination. A later error does not retract records already consumed; no new parent-cancellation guarantee is introduced.

When this customizer also serves Update/Delete old-record prefetch, its replacements must preserve valid native row identity; unrelated, unkeyed or retargeted captured records reject that write prefetch. Copy private snapshots with new QRecord(record); see QRecords. Ordinary public Query/Get replacement and privacy behavior is unchanged.

RDBMS and Memory queries use the active table metadata and the input source when applying READ rules to base and joined rows. A USER-specific personalizer can restrict either table. Unavailable personalized tables reject the request. For a LEFT join, child READ rules determine permitted matches before the parent receives an unmatched null child. Fields needed internally for joins or security do not become public selections merely because execution needs them.

MongoDB retains its existing BSON wrapper and civil-date/time conversion behavior. Declared-field Decimal128/Binary normalization and UTC civil-type conversion are deferred in #530. DATE_TIME still uses the existing Instant conversion; BSON datetime precision remains milliseconds.

Native write lookups

Backend QueryInterface.executeForDml(QueryInput) is the entry point for native lookups whose identities drive writes. Its default implementation delegates to execute. Common write prefetch, destructive-filter selection, stored-owner validation and Replace matching use this entry point; MongoDB also uses it for structural association discovery. Ordinary Query and uniqueness-conflict reads retain their existing path.

MongoDB materializes these lookups and requires each selected row to have a native BSON ObjectId, represented in QQQ by its hexadecimal String. Native BSON String keys are unsupported for these write lookups. A virtual projection that replaces _id, or decoding that changes the declared primary key from the selected ObjectId, rejects the lookup before its records reach common write callbacks or cascade processing. This check does not undo earlier writes or resolve concurrent changes and does not provide complete native String-key CRUD.

QueryInput

  • tableName - String, Required - Name of the table being queried against.

  • filter - QQueryFilter object - Specification for what records should be returned, based on QFilterCriteria objects, and how they should be sorted, based on QFilterOrderBy objects. If a filter is not given, then all rows in the table will be returned by the query.

  • transaction - QBackendTransaction object - Optional transaction object.

    • Behavior for this object is backend-dependant. In an RDBMS backend, this object is generally needed if you want your query to see data that may have been modified within the same transaction. The caller retains ownership of the transaction and decides when to commit, roll back and close it.

  • queryHints - EnumSet of QueryHint - Optional backend hints. MAY_USE_READ_ONLY_BACKEND permits configured replica routing; a supplied RDBMS transaction takes precedence. POTENTIALLY_LARGE_NUMBER_OF_RESULTS requests the available MySQL streaming optimization.

  • timeoutSeconds - Integer - Optional backend execution timeout. In RDBMS, null, zero and negative values disable the timeout; a positive value bounds the wait until result processing starts, not the time spent consuming a pipe. QueryAction.cancel() can cancel an assigned backend query; it does not latch cancellation requested before the backend delegate is assigned.

  • recordPipe - RecordPipe object - Optional pipe object that records are placed into, for asynchronous processing.

    • If a recordPipe is used, then records cannot be retrieved from the QueryOutput. Rather, such records must be read from the pipe’s consumeAvailableRecords() method.

    • A recordPipe should only be used when a QueryAction is running in a separate Thread from the record’s consumer.

  • shouldTranslatePossibleValues - boolean, default: false - Controls whether any fields in the table with a possibleValueSource assigned to them should have those possible values looked up (e.g., to provide text translations in the generated records' displayValues map).

    • For example, if running a query to present results to a user, this would generally need to be true. But if running a query to provide data as part of a process, then this can generally be left as false.

  • fieldsToTranslatePossibleValues - Set of String - When shouldTranslatePossibleValues is true, null translates all possible-value fields, a non-null set selects only those fields, and an empty set translates none. Selected joined fields use their qualified table or alias names.

  • shouldGenerateDisplayValues - boolean, default: false - Controls whether field level displayFormats should be used to populate the generated records' displayValues map.

    • For example, if running a query to present results to a user, this would generally need to be true. But if running a query to provide data as part of a process, then this can generally be left as false.

  • shouldFetchHeavyFields - boolean, default: false - Controls whether or not fields marked as isHeavy should be fetched & returned or not.

    • RDBMS and Memory omit unfetched heavy values and record their available lengths in backendDetails.heavyFieldLengths. Set this option to true to request the values; hidden-field and password options still apply.

  • shouldOmitHiddenFields - boolean, default: true - Controls whether or not fields marked as isHidden should be included in the result or not.

  • shouldMaskPasswords - boolean, default: true - Controls whether or not fields with type = PASSWORD should be masked, or if their actual values should be returned.

    • Hidden-field omission and password masking apply to selected joined fields under their qualified table or alias names, including display values. The REVEAL adornment retains its explicit password-unmasking behavior. Masking does not add a field that was absent from the returned projection.

  • queryJoins - List of QueryJoin objects - Optional list of tables to be joined with the main table being queried. See QueryJoin below for further details.

  • selectDistinct - boolean, default: false - Requests distinct selected rows. In RDBMS this applies to the entire selected projection, so different selected child values remain separate rows even when their parent is the same.

  • includeAssociations - boolean, default: false - Expands declared associated records. Piped association reads are buffered in batches, including the final partial batch.

  • associationNamesToInclude - Collection of String - When associations are enabled, null selects all declared groups and an empty collection selects none. Name descendant paths explicitly, such as pets and pets.notes. Descendants inherit read and privacy options; root field projections and query joins do not apply to their different tables.

  • fieldNamesToInclude - Set of String - Optional set of field names to be included in the records.

    • Fields from a queryJoin must be prefixed by the join table’s name or alias, and a period. Use unqualified names for fields from the table being queried, including virtual fields.

    • A null set here (default) means to include all fields from the table and any queryJoins set as select=true.

    • An empty set will cause an error, as well any unrecognized field names.

    • QueryAction will validate the set of field names, and throw an exception if any unrecognized names are given.

    • RDBMS and Memory apply this set to the backend projection. Read options may further omit or mask fields; a post-query customizer can change the returned records. Other backend modules may not implement field projection and can return all fields.

    • In Memory, a supported selectable virtual field such as animal.nameLength can be the only requested field when animal is a selected join. It does not require selecting a root or physical child field, and does not add its source fields to the returned projection.

QQueryFilter

A key component of QueryInput, a QQueryFilter defines both what records should be included in a query’s results (e.g., an SQL WHERE), as well as how those results should be sorted (SQL ORDER BY).

Memory retains active join context through Boolean subfilters. Supported virtual predicates and sorting read fields from their referenced table or alias, including local field arguments, while retaining each function’s null and sorting behavior. Evaluating an unselected virtual does not add it to Query output.

  • criteria - List of QFilterCriteria - Individual conditions or clauses to filter records. They are combined using the booleanOperator specified in the QQueryFilter. See below for further details.

  • orderBys - List of QFilterOrderBy - List of fields (and directions) to control the sorting of query results. In general, multiple orderBys can be given (depending on backend implementations).

  • booleanOperator - Enum of AND, OR, default: AND - Specifies the logical joining operator used among individual criteria.

  • subFilters - List of QQueryFilter - To build arbitrarily complex queries, with nested boolean logic, 0 or more subFilters may be provided.

    • Each subFilter can include its own additional subFilters.

    • Each subFilter can specify a different booleanOperator.

    • For example, consider the following QQueryFilter, that uses two subFilters, and a mix of booleanOperators

  • skip - Integer - Optional number of records to skip. RDBMS applies it only when a limit is supplied; skip-only correction is deferred in #529. Use deterministic ordering for repeatable pages.

  • limit - Integer - Optional maximum number of records. In RDBMS, zero returns no rows and null leaves the upper bound unrestricted. Negative-window behavior remains provider-specific; do not supply negative values.

  • subFilterSetOperator - Enum of UNION, UNION_ALL, INTERSECT, EXCEPT - Optional RDBMS set operation combining the subfilter result sets. UNION_ALL preserves duplicates; the other operations use SQL set semantics. Outer ordering and pagination apply to the combined result. Support depends on the native backend.

 queryInput.setFilter(new QQueryFilter()
    .withBooleanOperator(OR)
    .withSubFilters(List.of(
       new QQueryFilter().withBooleanOperator(AND)
          .withCriteria(new QFilterCriteria("firstName", EQUALS, "James"))
          .withCriteria(new QFilterCriteria("lastName", EQUALS, "Maes")),
       new QQueryFilter().withBooleanOperator(AND)
          .withCriteria(new QFilterCriteria("firstName", EQUALS, "Darin"))
          .withCriteria(new QFilterCriteria("lastName", EQUALS, "Kelkhoff"))
    )));

// which would generate the following WHERE clause in an RDBMS backend:
// WHERE (first_name='James' AND last_name='Maes') OR (first_name='Darin' AND last_name='Kelkhoff')

MongoDB Query, Count and Aggregate evaluate native-supported virtual predicates and inline functions on physical fields with their own arguments through flat or nested AND/OR filters. Inline filter functions do not replace physical values in Query results or Aggregate operands. READ locks continue to evaluate stored physical values independently of filter computation. An inline function applied to a virtual field is unsupported and rejected. Required filter computations need a native adapter; values supplied only by a Query customizer cannot filter native rows.

QFilterCriteria

  • fieldName - String, required - Reference to a field on the table being queried.

    • Or, in the case of a query with queryJoins, a qualified name of a field from a join-table (where the qualifier would be the joined table’s name or alias, followed by a dot)

      • For example: orderLine.sku or orderBillToCustomer.firstName

  • operator - Enum of QCriteriaOperator, required - Comparison operation to be applied to the field specified as fieldName and the values or otherFieldName.

    • e.g., EQUALS, NOT_IN, GREATER_THAN, BETWEEN, IS_BLANK, etc.

  • fieldFunction - FieldFunction, optional - Applies a supported function to this criterion’s field before comparison, using its own arguments. Function support depends on the backend.

  • values - List of values, conditional - Provides the value(s) that the field is compared against. The number of values (0, 1, 2, or more) required are based on the operator being used. If an otherFieldName is given, and the operator expects 1 value, then values is ignored, and otherFieldName is used.

  • otherFieldName - String, conditional - Specifies that the fieldName should be compared against another field in the records, rather than the values in the values property. Only used for operators that expect 1 value (e.g., EQUALS or LESS_THAN_OR_EQUALS - not IS_NOT_BLANK or IN).

QFilterCriteria definition examples:
// in-line, via constructors that take (List<Serializable> values) or (Serializable... values) as 3rd arg
new QFilterCriteria("id", IN, 1, 2, 3);
new QFilterCriteria("name", IS_BLANK);
new QFilterCriteria("orderNo", IN, orderNoList);
new QFilterCriteria("state", EQUALS, "MO");

// long-form, with fluent setters
new QFilterCriteria()
   .withFieldName("quantity")
   .withOperator(QCriteriaOperator.GREATER_THAN)
   .withValues(List.of(47));

// to use otherFieldName, long-form must be used
new QFilterCriteria()
   .withFieldName("firstName")
   .withOperator(QCriteriaOperator.EQUALS)
   .withOtherFieldName("lastName");

// using otherFieldName to build a criterion that looks at two fields from two different join tables
new QFilterCriteria()
   .withFieldName("billToCustomer.lastName")
   .withOperator(QCriteriaOperator.NOT_EQUALS)
   .withOtherFieldName("shipToCustomer.lastName");
QFilterOrderBy

  • fieldName - String, required - Reference to a field on the table being queried.

    • Or, in the case of a query with queryJoins, a qualified name of a field from a join-table (where the qualifier would be the joined table’s name or alias, followed by a dot)

  • isAscending - boolean, default: true - Specify if the sort is ascending or descending.

QFilterOrderBy definition examples:
// short-form, via constructors
new QFilterOrderBy("id"); // isAscending defaults to true.
new QFilterOrderBy("name", false);

// long-form, with fluent setters
new QFilterOrderBy()
   .withFieldName("birthDate")
   .withIsAscending(false);
QueryJoin

  • joinTable - String, required (though inferrable) - Name of the table that is being joined in to the existing query.

    • Will be inferred from joinMetaData, if joinTable is not set when joinMetaData gets set.

  • baseTableOrAlias - String, required (though inferrable) - Name of a table (or an alias) already defined in the query, to which the joinTable will be joined.

    • Will be inferred from joinMetaData, if baseTableOrAlias is not set when joinMetaData gets set (which will only use the leftTableName from the joinMetaData - never an alias).

  • joinMetaData - QJoinMetaData object - Optional specification of a QQQ Join in the current QInstance. If not set, will be looked up at runtime based on baseTableOrAlias and joinTable.

    • If set before baseTableOrAlias and joinTable, then they will be set based on the leftTable and rightTable in this object.

  • alias - String - Optional (unless multiple instances of the same table are being joined together, when it becomes required). Behavior based on SQL FROM clause aliases. If given, must be used as the part before the dot in field name specifications throughout the rest of the query input. The effective join name (the alias, or joinTable when no alias is supplied) must be unique among joins and must not equal the main table name. A self-join therefore needs a distinct alias.

  • select - boolean, default: false - Specify whether fields from the joinTable should be selected by the query. If true, selected values use the table-or-alias.field-name form. If false, the join can still affect row membership, filtering and ordering, but its fields are not part of the backend output projection.

  • type - Enum of INNER, LEFT, RIGHT, FULL, default: INNER - specifies the SQL-style type of join being performed.

Basic QueryJoin usage example:
// selecting from an "orderLine" table, joined to its corresponding (parent) "order" table
queryInput.withTableName("orderLine");
queryInput.withQueryJoin(new QueryJoin("order").withSelect(true));
QueryOutput queryOutput = new QueryAction().execute(queryInput);
queryOutput.getRecords().get(0).getValueBigDecimal("order.grandTotal");
"V" shaped query - selecting from one parent table, and two children joined to it:
// Requires the named tables and joins to be registered in the QInstance.
// selecting from an "order" table, and two children of it, orderLine and customer
queryInput.withTableName("order");
queryInput.withQueryJoin(new QueryJoin("orderLine").withSelect(true));
queryInput.withQueryJoin(new QueryJoin("customer").withSelect(true));
QueryOutput queryOutput = new QueryAction().execute(queryInput);
QRecord joinedRecord = queryOutput.getRecords().get(0);
joinedRecord.getValueString("orderNo");
joinedRecord.getValueString("orderLine.sku");
joinedRecord.getValueString("customer.firstName");
"Chain" shaped query - selecting from one parent table, a child table, and a grandchild:
// Requires the named tables and joins to be registered in the QInstance.
// selecting from an "order" table, with a "customer" child table, and an "address" sub-table
queryInput.withTableName("order");
queryInput.withQueryJoin(new QueryJoin("customer").withSelect(true));
queryInput.withQueryJoin(new QueryJoin("address").withBaseTableOrAlias("customer").withSelect(true));
QueryOutput queryOutput = new QueryAction().execute(queryInput);
QRecord joinedRecord = queryOutput.getRecords().get(0);
joinedRecord.getValueString("orderNo");
joinedRecord.getValueString("customer.firstName");
joinedRecord.getValueString("address.street1");
QueryJoin usage example where two tables have two different joins between them:
// Requires the named tables and joins to be registered in the QInstance.
// here there's a "fulfillmentPlan" table, which points at the order table (many-to-one,
// as an order's plan can change over time, and we keep old plans around).
// This join is named:  fulfillmentPlanJoinOrder
//
// The other join is "order" pointing at its current "fulfillmentPlan"
// This join is named:  orderJoinCurrentFulfillmentPlan

// to select an order along with its current fulfillment plan:
QueryInput currentPlanQuery = new QueryInput("order")
   .withQueryJoin(new QueryJoin(instance.getJoin("orderJoinCurrentFulfillmentPlan"))
      .withSelect(true));

// alternatively, select an order and all its fulfillment plans:
QueryInput allPlansQuery = new QueryInput("order")
   .withQueryJoin(new QueryJoin(instance.getJoin("fulfillmentPlanJoinOrder"))
      .withBaseTableOrAlias("order")
      .withJoinTable("fulfillmentPlan")
      .withSelect(true));
QueryJoin usage example for table with two joins to the same child table, selecting from both:
// given an "order" table with 2 foreign keys to a customer table (billToCustomerId and shipToCustomerId)
// Note, we must supply the JoinMetaData to the QueryJoin, to drive what fields to join on in each case.
// we must also define an alias for each of the QueryJoins
queryInput.withTableName("order");
queryInput.withQueryJoins(List.of(
   new QueryJoin(instance.getJoin("orderJoinShipToCustomer"))
       .withAlias("shipToCustomer")
       .withSelect(true),
   new QueryJoin(instance.getJoin("orderJoinBillToCustomer"))
       .withAlias("billToCustomer")
       .withSelect(true)));
QueryOutput queryOutput = new QueryAction().execute(queryInput);
QRecord record = queryOutput.getRecords().get(0);
String description = record.getValueString("billToCustomer.firstName")
   + " paid for an order, to be sent to "
   + record.getValueString("shipToCustomer.firstName");

Implicit joins depend on an unambiguous metadata path. Prefer an explicit QueryJoin when selecting fields or choosing between multiple relationships. The examples above require an initialized QContext, the named tables/joins, and a caller that handles QException. Check that records exist before accessing element zero.

QueryOutput

Physical and virtual fields follow the same privacy rules. Labels that use private fields are rebuilt from sanitized values, including on associated records. Explicit visibility options and REVEAL remain effective; masking does not add a field omitted by the backend’s projection. * records - List of QRecord - List of 0 or more records that match the query filter. * Note: If a *recordPipe was supplied to the QueryInput, then calling queryOutput.getRecords() will result in an IllegalStateException being thrown - as the records were placed into the pipe as they were fetched, and cannot all be accessed as a single list.

RDBMS qualified virtual-only projections retain their existing omission limitation (#531). This does not change the active metadata and private-field checks on returned values.

GetAction

The GetAction is essentially a subset of the QueryAction, only specifically meant to get just a single record from a QQQ Table. In SQL/RDBMS terms, it is analogous to a SELECT statement, where a single record may be found - or - it may not be found.

For all tables, GetAction can do a lookup by primary key. In addition, for tables that define a UniqueKey, name/value pairs (in the form of a Map) can be used as input for GetAction.

Examples

Basic form - by primary key
GetInput input = new GetInput();
input.setTableName("orders");
input.setPrimaryKey(1);
GetOutput output = new GetAction().execute(input);
QRecord record = output.getRecord();
Secondary form - by Unique Key
GetInput input = new GetInput();
input.setTableName("products");
input.setUniqueKey(Map.of("storeId", 1701, "sku", "ABCD"));
GetOutput output = new GetAction().execute(input);
QRecord record = output.getRecord();

GetInput

  • tableName - String, Required - Name of the table being queried against.

  • primaryKey - Serializable, Conditional - Value for the primary key field of the table being queried. Type should match the table’s primary key field’s type. If a primaryKey is not given, then a uniqueKey must be given.

  • uniqueKey - Map of String → Serializable, Conditional - Map of name-value pairs that define the record to be fetched. Keys in the map must be field names from the table being queried. Values in the map should be of types that correspond to the fields. If a primaryKey is not given, then a uniqueKey must be given. The map’s field names must exactly match a declared UniqueKey. Supplying both primaryKey and uniqueKey is rejected.

  • transaction - QBackendTransaction object - Optional transaction object.

    • Behavior for this object is backend-dependant. In an RDBMS backend, this object is generally needed if you want your query to see data that may have been modified within the same transaction.

  • queryHints - EnumSet of QueryHint - Optional backend query hints. Hint handling is backend-dependent. The existing Get-to-Query conversion does not carry these hints, so RDBMS Get retains primary routing. Get does not commit or close a caller-owned transaction.

  • shouldTranslatePossibleValues - boolean, default: false - Controls whether any fields in the table with a possibleValueSource assigned to them should have those possible values looked up (e.g., to provide text translations in the generated records' displayValues map).

    • For example, if getting a record to present to a user, this would generally need to be true. But if getting a record as part of a process, then this can generally be left as false.

  • shouldGenerateDisplayValues - boolean, default: false - Controls whether field level displayFormats should be used to populate the generated records' displayValues map.

    • For example, if getting a record to present to a user, this would generally need to be true. But if getting a record as part of a process, then this can generally be left as false.

  • shouldFetchHeavyFields - boolean, default: true - Controls whether or not fields marked as isHeavy should be fetched & returned or not.

  • shouldOmitHiddenFields - boolean, default: true - Controls whether or not fields marked as isHidden should be included in the result or not.

  • shouldMaskPasswords - boolean, default: true - Controls whether or not fields with type = PASSWORD should be masked, or if their actual values should be returned.

  • queryJoins - List of QueryJoin objects - Optional list of tables to be joined with the main table being queried. See QueryJoin under QueryAction for further details.

  • includeAssociations - boolean, default: false - Control whether or not records from tables defined as associations under the table being queried should be included in the result or not.

  • associationNamesToInclude - Collection of String - If includeAssociations is true, then this field can be used to limit which associated tables are included. If this field is null, then all associated tables are included. Otherwise, use exact association names, including parent-prefixed names such as pets.notes for nested groups.

Expanded Query/Get reads inherit heavy-field, hidden-field, password masking, possible-value and display-value options at each level. Root field selections and query joins are not applied to different child tables. Finite self-referencing trees terminate when no child records remain. Expansion beyond 64 child-record levels fails instead of returning a truncated tree; cyclic data therefore produces a bounded error.

Hidden-field removal and password masking cover physical and virtual fields, selected joined values, and their display values. Password masking preserves the backend’s projection instead of adding absent fields. Labels that depend on private fields are rebuilt from sanitized values, including labels supplied by a post-query customizer. Explicit read options and the REVEAL adornment retain their behavior. Native Get applies READ behaviors and privacy to the record returned by the customizer.

GetOutput

  • record - QRecord - The record that was specified by the input primaryKey or uniqueKey. Will be null if the record was not found.

CountAction

CountAction returns the size of a table’s selected population. Initialize QContext with the application and a valid session before constructing table inputs. A null filter selects all accessible records; an empty result returns zero. RDBMS and enumeration counts ignore skip and limit without changing the caller’s filter. Field names, including nested criteria and joined aliases, are validated against the effective metadata; filter value behaviors also apply.

CountOutput output = new CountAction().execute(new CountInput("person")
   .withFilter(new QQueryFilter(new QFilterCriteria("isEmployed", QCriteriaOperator.EQUALS, true))));
Integer employed = output.getCount();

RDBMS counts

An ordinary count returns matching SQL rows. Explicit joins can repeat the base record: the sample’s Person-to-Pet inner join has six rows and three distinct people. Set includeDistinctCount to true to return both count (matching rows) and distinctCount (distinct base primary keys, using their backend column mapping). Without that option, distinctCount is null. A security join that requires distinctness automatically deduplicates the default count; requesting both results still returns raw joined rows in count and unique base records in distinctCount.

Record security restricts the selected population, including multiple session keys, null-value policies and joined locks. User-sourced inputs use personalized metadata, so a field removed from personalized metadata cannot be used in a count filter. RDBMS execution retains active base/join mappings and READ locks, including USER-added locks; unavailable personalized tables reject explicitly. Removed physical keys remain private execution details when required for joins, READ policy or distinct counts, and remain unavailable in public filters. Core CountAction is also used by trusted application code: it validates the session and applies record security but does not itself require the middleware’s table READ permission.

Pass the caller’s RDBMSTransaction to see its pending changes. Count closes its own statement while leaving the caller responsible for commit, rollback and connection closure:

CountOutput pending = new CountAction().execute(new CountInput("person")
   .withTransaction(transaction));

QueryHint.MAY_USE_READ_ONLY_BACKEND permits the configured read-only backend, falling back to the primary when none is configured. An explicit RDBMS transaction takes precedence over this hint. Use a transaction belonging to the counted table’s backend.

timeoutSeconds is disabled for null, zero or negative values. A positive timeout applies after acquiring the connection and preparing the statement; it does not bound connection acquisition. An interrupted SQL count throws QUserFacingException with Count timed out.. To cancel an in-flight count from another thread, retain that invocation’s CountAction and call cancel(); a cancelled SQL operation reports Count was cancelled.. Cancellation before an active statement exists is best-effort. Use a separate action and thread-local QContext for each concurrent invocation.

MongoDB computed filters

MongoDB counts the population selected by native-supported virtual predicates or inline functions on physical fields, including nested AND/OR filters. Each criterion retains its own arguments. A required function without a native adapter is rejected; values supplied only by Query customizers cannot filter Count. These filters do not establish the RDBMS distinct-count, transaction or cancellation options for MongoDB.

Enumeration and HTTP

The enumeration backend counts all or filtered enum records, ignoring pagination. It does not implement the RDBMS distinct-count, transaction, timeout or cancellation options. Backend-specific options are not universal promises made by CountInput.

The sample’s legacy HTTP route accepts GET or POST /data/person/count. Supply a serialized QQueryFilter as the filter query or form parameter. queryJoins and includeDistinct=true are query parameters. The response contains count and, when requested and supported, distinctCount. The V1 route accepts POST /qqq/v1/table/person/count with JSON filter and joins properties.

Both HTTP versions require READ permission on the base table and on tables requested through explicit joins or implicitly through filters and ordering. A write grant alone does not authorize a count or a joined read. Denial returns HTTP 403 and an error without count values. Joins added only by the framework to enforce record-security locks do not require a separate public read grant; explicitly requesting those tables still does. Core trusted application actions retain their existing permission boundary.

The sample verifies these contracts in SampleCountContractTest, SampleQueryContractTest, SampleEnumerationContractTest and SampleJavalinServerTest. SampleDatabaseIT observes real blocked PostgreSQL counts for timeout/cancellation, verifies lock cleanup and performs a successful fresh count afterward. Other backend and middleware contracts remain separately tracked in the sample feature inventory.

AggregateAction

AggregateAction computes scalar or grouped values for a QQQ Table through its backend’s aggregate operation. Initialize QContext with the intended instance and session before constructing the input. This is a Java action API; it does not define a generic HTTP aggregate endpoint.

Example

Count canonical sample FieldLab rows by booleanValue, ordered by the group value.

Aggregate count = new Aggregate("id", AggregateOperator.COUNT);
GroupBy group = new GroupBy(QFieldType.BOOLEAN, "booleanValue");
AggregateInput input = new AggregateInput("fieldLab")
   .withAggregate(count)
   .withGroupBy(group)
   .withFilter(new QQueryFilter()
      .withOrderBy(new QFilterOrderByGroupBy(group)))
   .withLimit(20)
   .withInputSource(QInputSource.USER);
AggregateOutput output = new AggregateAction().execute(input);
for(AggregateResult result : output.getResults())
{
   Serializable groupValue = result.getGroupByValue(group);
   Serializable rowCount = result.getAggregateValue(count);
}

Descriptors and result values

Select the Aggregate and GroupBy descriptors needed for the computation. Use an empty aggregates list for group-only requests on RDBMS and MongoDB; null-list handling is backend-dependent. Each aggregate requires an available field name and an operator. The original operators are COUNT, COUNT_DISTINCT, SUM, MIN, MAX and AVG; analysis extensions are listed below. Each group requires an available field name and a result QFieldType; new GroupBy(fieldMetaData) derives both from the field. Use complete descriptors and non-negative limits. Unavailable fields fail common selection validation; malformed requests and invalid limits retain backend-specific behavior.

RDBMS and Memory COUNT(field) count non-null operands, and COUNT_DISTINCT(field) counts distinct non-null values; use a non-null primary key to count rows. Memory defaults count results to QFieldType.INTEGER. RDBMS COUNT may inherit the operand type unless it is temporal; COUNT_DISTINCT defaults to INTEGER. Set Aggregate.fieldType explicitly when a numeric result type is required. Their AVG of INTEGER/LONG defaults to DECIMAL; other results generally use the operand type unless overridden. Operator/type combinations, precision and conversion limits remain backend-dependent.

MongoDB retains native BSON and operand-based result conversion, including its existing Decimal128, binary, civil-time and AVG limitations. Use provider-specific result checks for these types. Memory grouping and COUNT_DISTINCT use Java value equality; decimal scale and array identity can distinguish otherwise equivalent values. Its DECIMAL AVG uses double arithmetic. Precision and conversion rules are backend-dependent.

RDBMS empty scalar input produces one result: counts are zero and SUM/MIN/MAX/AVG are null. Memory and MongoDB can produce no scalar row for empty input. Memory SUM uses zero for an all-null group. Grouped empty input produces no results. Null group values and empty strings remain distinct where the backend distinguishes them; Aggregate does not merge their labels. AggregateOutput.results contains AggregateResult objects with separate descriptor-keyed aggregateValues and groupByValues maps. Reuse matching descriptors to retrieve values and allow null results. These maps are not complete QRecords; Query READ behaviors, display formatting, label generation and possible-value translation are not applied to them.

Filtering, ordering and expressions

filter restricts source rows. RDBMS and MongoDB ignore Query-style skip/limit for aggregation. Memory applies those windows to source rows before grouping. RDBMS applies AggregateInput.limit to result rows; Memory and MongoDB retain their existing ignored-result-limit behavior. Without supported ordering, result order is not guaranteed. Use QFilterOrderByAggregate and QFilterOrderByGroupBy for supported provider ordering. RDBMS resolves typed aggregate ordering through selected positions or complete dialect expressions, including qualified and virtual operands. MongoDB supports its existing selected aggregate and typed group ordering; plain group and unselected aggregate sorting remain limited. Memory retains its own grouping and comparison rules.

Use unqualified names for selected base-table operands, including virtual fields. Explicit or supported inferred joins can supply tableOrAlias.field operands. Aggregate joins need not set select=true; join multiplicity affects counts and other aggregates. Selectable virtual fields require a supported field-function implementation. Memory evaluates base and joined virtual operands in their own table or alias scope, including local field arguments; an unmatched LEFT child supplies null local fields and each function retains its null behavior. RDBMS requires a native adapter, whose parameters are bound for expression occurrences in SELECT, GROUP BY and ORDER BY; native functions and ordering expressions remain dialect-dependent. MongoDB’s shared filter pipeline evaluates supported virtual predicates and inline functions on physical fields while keeping physical READ-owner values separate. Required filter computations reject missing native adapters; inline functions on virtual fields remain unsupported. Aggregate value/group computations retain their existing provider fallbacks. MongoDB has no aggregate join implementation; do not submit explicit joins, which the provider can ignore. GroupBy.formatString is trusted application-authored native formatting. RDBMS substitutes the escaped field expression through String.format; arbitrary format text is not converted into bind parameters. Do not accept SQL format strings from untrusted HTTP input. Memory formatting and other adapters are not a portable SQL-expression language.

Active metadata and private fields

Selection validation uses active personalized base/join metadata for the input source. RDBMS uses that metadata for native base/join mappings and READ/READ_AND_WRITE locks. Memory applies active READ rules through its shared read path, including supported INNER and LEFT joins; other adapters require separate provider validation. For locks on fields in the queried table, MongoDB Query, Count and Aggregate apply active READ and READ_AND_WRITE scopes and retain nested AND/OR groups; WRITE-only locks do not restrict these reads. USER uses personalized metadata; SYSTEM still applies its active table’s locks. A true all-access key satisfies its corresponding lock, including an OR alternative, without bypassing other required lock branches or the caller filter. Null session-key entries do not grant read access; access to null or missing lock values remains governed by the effective null-value policy and all-access key. USER-removed fields cannot become aggregate, group or ordering operands; filter fields retain their normal validation. Filter value behaviors apply before execution. Result values do not receive Query’s READ transformations. Available hidden/PASSWORD operands may be used by COUNT and COUNT_DISTINCT; cardinality is not password-masked. Counts can disclose null/distinct information and are not statistical anonymity. Aggregate returns computed map values directly. Hidden/PASSWORD field metadata does not establish a universal operand restriction or apply Query result masking. Callers must choose the operands they authorize for the result they expose. Public virtual expressions and trusted format strings remain application-authored. Table-permission grants belong to the calling entry point; invoking Aggregate does not itself create an HTTP READ grant.

Configured statistics and widgets

Column statistics run as USER for grouped values, scalar statistics and their preliminary row-limit count. They require READ on the base and requested joined tables, including joins used only by filters. An available private primary key may supply its internal count without being included in the returned records. ColumnStats presentation deliberately combines null and empty-string bins where applicable, independently of Aggregate’s native grouping semantics.

Javalin and Slack widget entry points mark RenderWidgetInput as USER. Built-in WidgetAggregate and Aggregate2DTableWidgetRenderer carry this source into Aggregate. For USER inputs, these built-ins require base-table READ; no-code aggregate filters also require READ on their joined tables. Two-dimensional grouping fields and labels use active metadata. Direct Java widget inputs default to SYSTEM; a query/form parameter named inputSource does not change the execution source. Custom renderers must propagate the input source and enforce their own required table permissions. No-code value-source failures retain the renderer’s existing missing-value behavior.

Two-dimensional tables retain their existing column naming: null and text-null values, or values matching the structural _row/_total keys, can collide. Choose column values that avoid these collisions.

AggregateInput

  • tableName - String, Required - Table with a supported native aggregate operation.

  • aggregates - List of Aggregate, default: null - Selected aggregate descriptors; optional for group-only requests.

  • groupBys - List of GroupBy, default: empty - Group descriptors; empty requests scalar aggregation when aggregates are selected.

  • filter - QQueryFilter, default: null - Source-row predicates and supported result ordering.

  • queryJoins - List of QueryJoin, default: null - Optional joins and aliases supported by the backend.

  • limit - Integer, default: null - Non-negative maximum result rows; null adds no Aggregate result limit.

  • inputSource - InputSource, inherited default: QInputSource.SYSTEM - Select USER explicitly for user-facing calls requiring USER metadata.

  • transaction - QBackendTransaction, default: null - Optional caller-owned transaction supported by the backend.

  • timeoutSeconds - Integer, default: null - Supported native timeout; null, zero and negative values disable the RDBMS timeout.

  • queryHints - EnumSet of QueryHint, default: empty - Includes MAY_USE_READ_ONLY_BACKEND where supported.

Transactions, cancellation and providers

RDBMS uses the supplied RDBMSTransaction connection, including its pending data, and never commits, rolls back or closes that caller transaction. Without that transaction, MAY_USE_READ_ONLY_BACKEND can select a configured read-only backend; otherwise routing falls back to the normal backend. A supplied transaction takes precedence over the hint. Positive RDBMS timeouts and cancel() on the executing AggregateAction request native statement cancellation; completion and error recovery depend on the driver/backend. Cancellation is not transaction rollback or an end-to-end deadline covering connection acquisition. Calling cancel() before the backend delegate is assigned does not queue a later cancellation. RDBMS cancels its timeout task when execution ends. MongoDB Aggregate retains its existing timeout-helper cleanup limitations; its timeout helper does not establish native cancellation, an end-to-end deadline or transaction rollback. Use a separate action instance per independent operation; these controls do not promise concurrent reuse of one action.

RDBMS, Memory and MongoDB provide aggregate implementations; support for the declared operators, types, functions, joins, nulls, ordering and limits differs by adapter/dialect. MongoDB COUNT currently counts input rows, and COUNT_DISTINCT is explicitly unsupported. Its result conversion does not establish the RDBMS/Memory count defaults above. A backend without an aggregate implementation rejects the operation. The common input/visibility checks do not certify all-provider parity, distributed transactions or portability of custom expressions.

Analysis operators

STDDEV_SAMP, STDDEV_POP, VAR_SAMP, VAR_POP, PERCENTILE_CONT, CORR and AVG_LENGTH return DECIMAL by default. COUNT_NULL, COUNT_BLANK, MIN_LENGTH and MAX_LENGTH return INTEGER. PERCENTILE_DISC preserves the operand type. An explicit Aggregate.fieldType still overrides result conversion. COUNT_BLANK counts nulls and strings whose SQL TRIM is empty; lengths exclude nulls and count characters. COUNT_BLANK, MIN_LENGTH, MAX_LENGTH and AVG_LENGTH require a STRING, TEXT or HTML operand; other field types are rejected before backend execution. Sample variance/deviation require two non-null values; population variants return zero for one value. Correlation uses only pairs with both values present and returns null for zero variance.

Percentiles require 0 ⇐ percentile ⇐ 1. Continuous percentiles interpolate at p * (n - 1) in sorted non-null values. Discrete percentiles return the first value with cumulative distribution at least p (the minimum at p = 0). Both percentile and otherFieldName participate in aggregate result-key equality.

Aggregate median = Aggregate.percentile("amount", 0.5);
Aggregate correlation = AggregateOperator.CORR.of("amount").withOtherFieldName("quantity");
AggregateOutput statistics = new AggregateAction().execute(new AggregateInput("orders")
   .withAggregate(median).withAggregate(correlation)
   .withAggregate(AggregateOperator.STDDEV_SAMP.of("amount"))
   .withAggregate(AggregateOperator.COUNT_NULL.of("amount")));

Bucketed grouping

GroupBy.bucket and formatString are mutually exclusive. Bucket descriptors participate in group result-key equality. DateBucketUnit supports YEAR, QUARTER, MONTH, WEEK (ISO Monday), DAY, HOUR, DAY_OF_WEEK (Monday = 1), and HOUR_OF_DAY (0–23). DATE_TIME uses ValueUtils.getSessionOrInstanceZoneId(); DATE is a civil date. Calendar bucket starts return LocalDate; HOUR returns an Instant at the session-local hour start; cyclic buckets return Integer. Numeric buckets return Integer: 0 below min, 1 through count inside the range, count+1 above max. Intervals are half-open except the final interval, which includes max: x == max returns count, including on PostgreSQL where native width_bucket would return count+1. Null operands remain a null group.

GroupBy month = new GroupBy(QFieldType.DATE_TIME, "createdAt")
   .withBucket(new DateBucket(DateBucketUnit.MONTH));
GroupBy histogram = new GroupBy(QFieldType.DECIMAL, "amount")
   .withBucket(new NumericBucket(BigDecimal.ZERO, new BigDecimal("100"), 10));
AggregateOutput buckets = new AggregateAction().execute(new AggregateInput("orders")
   .withGroupBy(histogram).withAggregate(AggregateOperator.COUNT.of("id")));

Capabilities and streamed fallback

AggregateInterface.getSupportedCapabilities(input) defaults to the empty set for existing backend implementations. Memory and PostgreSQL support all extensions. Base RDBMS identifies H2 from backend vendor metadata and supports all extensions there; MySQL supports all except PERCENTILE and CORRELATION. Other dialects must explicitly advertise supported features. MySQL date buckets with named zones require the database’s timezone tables to be populated. Date-bucket requests probe MySQL’s CONVERT_TZ once per backend metadata instance and zone, caching availability for that instance’s lifetime. If the conversion returns NULL, DATE_BUCKET is not advertised for that request, allowing the capability error or streamed fallback. After loading timezone tables, recreate the backend metadata (or restart) to clear the cached result. Fixed session offsets are bound as ISO offsets (including +00:00); PostgreSQL uses interval parameters to avoid POSIX text-zone sign reversal. Unsupported requested operators/buckets produce a QUserFacingException naming the missing capabilities unless fallback is enabled. Sampling alone never causes a capability error.

Set<AggregateCapability> capabilities = new QBackendModuleDispatcher()
   .getQBackendModule(input.getBackend()).getAggregateInterface().getSupportedCapabilities(input);
AggregateOutput output = new AggregateAction().execute(new AggregateInput("orders")
   .withAggregate(Aggregate.percentile("amount", 0.95))
   .withAllowStreamedFallback(true).withStreamedRowLimit(100_000));

Fallback evaluates the whole request using paged QueryAction reads with the same context, filter, joins, transaction and permission policy. Aggregate/group operands, including CORR’s second field, pass the existing core read-permission and field-lock checks first. The evaluator retains running state per group; exact percentiles and distinct counts retain values up to the row limit (default 1,000,000). rowsScanned counts consumed source rows. truncated is true when the limit is reached, even if it happens to equal the table size; label such results partial. Paging requires stable primary keys; callers needing a stable view during concurrent writes should supply a consistent-read transaction. The new memory path shares this evaluator without the fallback row cap. Legacy memory requests retain their existing behavior.

Sampling

samplePercent must be greater than zero and at most 100. PostgreSQL uses TABLESAMPLE SYSTEM (?) on the main table. Base H2/MySQL samples the main relation with RAND() < p/100 before joining; joined tables are not sampled independently. Memory uses every k-th main record in primary-key order, with k = max(1, round(100/p)); the requested percentage is approximate. An unsupported backend ignores sampling and returns sampled = false. Otherwise sampled = true and samplePercent records the requested percentage. A PostgreSQL block sample can be empty, especially for small tables. Counts are counts of sampled rows and are not automatically extrapolated.

AggregateOutput estimate = new AggregateAction().execute(new AggregateInput("orders")
   .withAggregate(AggregateOperator.COUNT.of("id"))
   .withSamplePercent(new BigDecimal("10")));
if(estimate.getSampled())
{
   // Label this result as an estimate from a sample.
}

Top-N

Top-N distinct values use the existing group, count, aggregate order and limit; request COUNT_DISTINCT separately for the total distinct count.

Aggregate count = AggregateOperator.COUNT.of("id");
AggregateInput topValues = new AggregateInput("orders")
   .withGroupBy(new GroupBy(QFieldType.STRING, "status")).withAggregate(count)
   .withFilter(new QQueryFilter().withOrderBy(new QFilterOrderByAggregate(count).withIsAscending(false)))
   .withLimit(10);

InsertAction

To insert (add, create) new records into any QQQ Table, the InsertAction is used. In SQL/RDBMS terms, it is analogous to a INSERT statement, where one or more records can be provided as input.

Examples

Canonical InsertAction invocation
InsertInput insertInput = new InsertInput();
insertInput.setTableName("person");
insertInput.setRecords(personRecordList);
InsertOutput insertOutput = new InsertAction().execute(insertInput);
List<QRecord> insertedPersonRecords = insertOutput.getRecords();

Details

InsertAction does several things beyond just inserting records into the specified table. A high-level flow of its internal logic is:

  1. For tables using an automation status field, set its value to PENDING_INSERT_AUTOMATIONS for all records that are going to be inserted.

  2. Perform the following validations, which include running the table’s PRE_INSERT_CUSTOMIZER, if one is defined, at the time that is specified by the customizer’s WhenToRun property (default is AFTER_ALL_VALIDATIONS):

    1. Ensure that default values specified in the table’s fields are present if needed.

    2. Apply per-field behaviors, as defined in QQQ Field meta-data, such as truncating strings that are longer than their specified max.

    3. Check for unique key violations to provide per-record errors and allow valid records to be stored while others fail. These preflight checks do not replace backend constraints, which also protect against concurrent writes and native value conversion.

    4. Validate that required fields (again, per QQQ Field meta-data) are set, generating per-record errors if they are not. Blank string inputs are missing values even for numeric, date or boolean fields. Zero, false and non-null binary values satisfy presence validation.

    5. Validate any security fields in the records - e.g., ensure that the user has permission to insert records with the values they are attempting to insert.

  3. Send the records to the table’s backend module to actually insert them into the backend storage.

  4. If the table has any associations defined, and if associated records are present, then recursively run InsertAction on the associated records.

    1. In particular, before these recursive InsertAction calls are made, values that were generated by the original insert may need to be propagated down into the associated records.

      • For example, if inserting order and lineItem records, where a QQQ Join exists between the two tables on order.id and lineItem.orderId, and order.id values were generated in the first InsertAction, then those values are propagated down into the associated lineItem.orderId fields.

  5. If the QInstance has an audit table, then based on the QQQ Table's audit rules, audits about the inserted records are created.

  6. If the table has a POST_INSERT_CUSTOMIZER, it is executed.

Overloads

InsertAction can be called in a few alternate forms, mostly just for convenience:

If inserting a single record, get that record back instead of the InsertOutput:
InsertInput insertInput = new InsertInput();
insertInput.setTableName("person");
insertInput.setRecords(List.of(personRecord));
QRecord insertedRecord = new InsertAction().executeForRecord(insertInput);

// or more compactly, using InsertInput.withRecord (instead of withRecords)
QRecord compactInsertedRecord = new InsertAction()
   .executeForRecord(new InsertInput("person").withRecord(personRecord));
Taking QRecordEntity objects as inputs instead of QRecords:
// insert a list of person entities:
InsertInput insertInput = new InsertInput("person").withRecordEntities(personList);
InsertOutput insertOutput = new InsertAction().execute(insertInput);

// or for a single person entity (also mapping the output record back to an entity):
Person insertedPerson = new Person(new InsertAction()
   .executeForRecord(new InsertInput("person").withRecordEntity(person)));

InsertInput

  • tableName - String, Required - Name of the table that records are to be inserted into.

  • records - List of QRecord - List of records to be inserted into the table. A null or empty list performs no backend insert and returns an empty record list after normal table/session validation.

Less common options

  • inputSource - InputSource object, default: QInputSource.SYSTEM - an indicator of what the source of the action is - generally, a SYSTEM triggered action, or a USER triggered action.

    • InsertAction will call the shouldValidateRequiredFields() method on this object to determine if it should validate that required fields on the records have values. Both QInputSource.SYSTEM and QInputSource.USER return true from this method, but application can define their own InputSource objects with different behavior.

    • In addition, this field can be used in pre- and post-insert customizers to drive further custom logic.

  • transaction - QBackendTransaction object - Optional caller-owned transaction, with backend-specific behavior. For RDBMS inserts, pass an RDBMSTransaction created from that table’s backend. Uniqueness preflight and writes use it. Insert does not commit, roll back or close it; the caller owns those decisions.

  • skipUniqueKeyCheck - boolean, default: false - control whether or not InsertAction should check for unique key violations before attempting to insert its records.

    • In a context where one has previously done this validation, or is okay letting the backend provide such checks, they may wish to avoid re-doing this work, and thus may set this property to true.

  • omitDmlAudit - boolean, default: false - control if the automatic DML audit that InsertAction generally performs should be omitted.

  • auditContext - String - optional message which can be included in DML audit messages, to give users more context about why the insert occurred.

InsertOutput

  • records - List of QRecord - Result records with details added based on the input action.

    • If there were warnings or errors, the corresponding field (warnings or errors) will be set in the records.

    • If the insert action generated any values (such as a serial id or a default value), those values will be in the record’s values map.

Inspect each returned record’s errors before treating it as inserted. Metadata validation can reject individual records while valid records are persisted. Native conversion or constraint failures may throw QException instead of returning records; earlier statements or associated writes may already have succeeded. Insert does not promise automatic batch or recursive atomicity. Use a supported caller-owned transaction and explicitly roll it back when the operation must be all-or-nothing.

UpdateAction

UpdateAction updates existing records in a QQQ Table using its configured backend. Initialize QContext with the intended instance and session before calling it. Each input record supplies the table’s declared primary key and the fields to change.

Example

Replace 1 with the primary key of an existing person record in the sample.

QRecord patch = new QRecord()
   .withValue("id", 1)
   .withValue("firstName", "Alex");
UpdateInput input = new UpdateInput("person").withRecord(patch);
UpdateOutput output = new UpdateAction().execute(input);
List<QRecord> results = output.getRecords();

Inspect each result’s errors and warnings; receiving a record does not prove that it was updated. executeForRecord(input) returns the first result and is intended for a single-record input. UpdateAction.executeForRecords(input) returns the result list directly. withRecordEntity and withRecordEntities convert QRecordEntity inputs to records.

Patch values and validation

Omitting an ordinary field retains its stored value; supplying null requests a null value. Field behaviors and customizers still apply: an omitted USER_ID field receives its dynamic default, and MODIFY_DATE updates automatically unless omitModifyDateUpdate is set. See QQQ Fields. Required-field validation checks supplied fields after field behaviors have run. Null and blank String operands are missing values, including strings supplied for numeric, date or boolean fields. Zero, false and non-null byte arrays, including empty, NUL and whitespace bytes, satisfy presence validation. Backend conversion and storage constraints can still reject an otherwise present value.

The declared primary key selects the row; it is not an ordinary field to replace. The native prefetch match uses typed identity, including decimal value and byte-array content equality. A separate ordinary field named id can be updated when the table declares a different primary key. These rules do not establish arbitrary physical primary-key support in every backend. MongoDB uses the configured logical primary key for its native ObjectId update selector; its prefetch and stored-owner reads follow the native write lookup restrictions. Declared unique keys are checked after the pre-update customizer; see QQQ Table for sparse-key and repeated-target rules.

Authorization and backend boundaries

Table permissions and record-lock scopes are separate; a WRITE-scoped lock does not itself restrict reads. Each HTTP operation retains its own table-permission checks; other applicable record locks still apply. See Permission Rules. Where the backend supports old-record prefetch, normal read visibility determines the visible targets. Old-record snapshots supplied to update customizers use the update input source and active personalized metadata; USER-removed fields stay absent. The framework preserves row identity when a snapshot’s primary key is hidden, omitted as heavy, or password-masked. This does not remove the requirement for a primary key in the submitted patch. See QRecords for copying these snapshots. Stored and proposed values must satisfy the applicable WRITE/READ_AND_WRITE lock tree. Stored-owner checks use private native values, including hidden/heavy/password fields, without merging them into public patches/results or the separate audit/customizer snapshot. Nested AND/OR branches retain their Boolean meaning. Trusted customizers must preserve validated authorization; the pre-update hook runs after ordinary security validation.

This path trusts the adapter’s native record representation; it does not guarantee complete projections from arbitrary providers. The API update adapter opts out of generic prefetch. Its provider contract and filesystem security require separate validation.

UpdateInput

  • tableName - String, Required - Name of the table to update.

  • records - List of QRecord - Patches with primary keys; use withRecord or withRecords. If records remains null or empty after validation and pre-update customizers, the backend update is skipped.

  • inputSource - InputSource, default: QInputSource.SYSTEM - Drives personalization and custom logic. Both standard SYSTEM and USER sources validate required fields; a custom source controls this through shouldValidateRequiredFields().

  • transaction - QBackendTransaction, default: null - Optional caller-owned transaction supported by the selected backend. Pass an RDBMSTransaction from the table’s backend for RDBMS operations; Update does not commit, roll back or close it.

  • omitModifyDateUpdate - boolean, default: false - Suppresses the automatic MODIFY_DATE field behavior.

  • omitTriggeringAutomations - boolean, default: false - Suppresses Update’s automation-status assignment; intended for specialized status-management operations.

  • omitRecordChangeEvents - boolean, default: false - For trusted system writes of derived counters or denormalized totals that are not business events. Suppresses optional record-change listeners; required integrity listeners still run. Record locks, field security, validation and permissions are unchanged. Also available on InsertInput and DeleteInput; does not require restore mode or change audit semantics.

  • omitDmlAudit - boolean, default: false - Suppresses the action’s DML audit.

  • auditContext - String, default: null - Optional context included in DML audit messages.

  • areAllValuesBeingUpdatedTheSame - Boolean, default: null - Trusted optimization hint for backends using the shared grouping helper. Null compares values automatically; an incorrect true value can produce an incorrect update.

  • flags - Set of ActionFlag, default: null - Optional flags interpreted by supporting action/customizer code.

Associations, results and transactions

Omitted association groups remain untouched; an explicit empty list requests removal of stored members. Replacement deletes omitted children before the parent update, then writes retained/new children after a successful parent update. Child operations retain their own validation and permissions. See QQQ Table and QRecords for association and HTTP form details.

UpdateOutput.records contains backend results, including per-record errors/warnings and applicable generated/default values. It is not a guaranteed complete reread of every stored field. Inspect nested errors as well as parent errors. Valid records may persist while other records fail validation; native failures can throw QException after earlier writes succeed. Update does not promise automatic batch or recursive rollback, distributed atomicity or concurrency isolation. Use a supported encompassing caller transaction and explicitly roll it back when all-or-nothing behavior is required.

DeleteAction

DeleteAction deletes selected records from a QQQ Table through its configured backend. Initialize QContext with the intended instance and session before constructing the input. The configured table and declared primary-key type determine the target records.

Example

Use a disposable sample record and replace 1 with its primary key. Deleting a person also traverses its declared pets and their notes associations.

DeleteInput input = new DeleteInput("person").withPrimaryKey(1);
DeleteOutput output = new DeleteAction().execute(input);
int deletedPeople = output.getDeletedRecordCount();
List<QRecord> errors = output.getRecordsWithErrors();
List<QRecord> warnings = output.getRecordsWithWarnings();

The count concerns the requested table, not the total number of deleted descendants. Error and warning lists are not input-aligned lists of successful records; handle null lists.

Selection and filter validation

Supply either nonempty primaryKeys or a queryFilter; supplying both rejects the request. Null or empty keys without a filter request no native deletion. Empty keys with a filter use that filter. An explicitly empty QQueryFilter selects all matching readable rows, subject to write validation and backend support. A zero-limit or unmatched filter selects no records. Use explicit criteria when limiting a destructive operation.

The common key-selection path validates filter structure and field names and rejects negative skip/limit before dispatch. Native query-specific operand and type checks still apply. Malformed criteria, including nested missing fields/operators, must reject rather than disappear and broaden the selection. Declared filter value behaviors apply to a private filter; USER-removed fields cannot be used for selection. Supported ordering, skip and limit select keys before DML. Presentation customizers do not choose those native targets.

Keys are converted to the declared type and deduplicated by typed identity before common validation and cascades. For prefetch-capable backends, explicit missing or READ-excluded keys produce NotFoundStatusMessage records. A hidden, heavy or password-masked key in the presentation snapshot does not by itself make an existing visible row missing; matching retains its native identity. A null key element is a missing target, not an absent selector. Invalid key conversion is an invocation error. Input key values and the original filter reference are restored on return or failure; list-object identity is not promised. MongoDB selection and prefetch follow the ObjectId requirements in the native write lookup contract; unsupported or disrupted selected keys reject before those records reach callbacks or cascades.

Authorization, personalization and customizers

Table permission grants and record-lock scopes are separate. Each HTTP operation retains its own permission checks. The active input source and personalized table metadata govern filter validation and presentation prefetch. USER-removed fields are excluded from that prefetch and its normal callback/error snapshots; SYSTEM retains its own metadata policy. The canonical association graph remains separate from presentation metadata.

For supported native prefetch, stored DELETE owners are checked with the applicable WRITE/READ_AND_WRITE lock tree. Private stored authorization values include hidden/heavy/password fields without treating presentation values as ownership. They are not merged into the separate presentation snapshot, public error records or audit/customizer data. Ordinary READ visibility still bounds the visible targets; a private owner read does not reveal a READ-denied record. The native adapter’s record representation remains trusted. API no-prefetch, filesystem security and other provider-specific contracts require separate validation.

PRE_DELETE_RECORD may return a replacement list, including an immutable list, and mark records with errors or warnings. Use new QRecord(record) when copying a private snapshot. A replacement must still identify an original prefetched target; a lost identity, unrelated target or retargeted captured key rejects before deletion. Valid fresh replacements with an original target key remain supported. See QRecords. Errors exclude their keys from native deletion; warnings permit deletion. Customizers remain trusted and must preserve authorization and privacy. performValidations(input, oldRecords, isPreview) runs validation and the pre-delete hook without invoking native deletion. Its supplied records must describe the intended visible targets; it is not a replacement for normal selection or a public authorization endpoint. Respect isPreview in customizers and avoid persistent side effects during preview. An absent old-record list returns null. POST_DELETE_RECORD runs after DML and auditing when old records are available. Its errors cannot undo deletion; a thrown exception becomes warnings. If any post-delete replacement has a missing, unrelated or retargeted captured identity, the returned list is not merged. Warnings are returned on the original successful snapshots, and deletedRecordCount is retained.

DeleteInput

  • tableName - String, Required - Name of an existing table with a supported delete operation.

  • primaryKeys - List of Serializable, default: null - Target identities; use withPrimaryKey or withPrimaryKeys.

  • queryFilter - QQueryFilter, default: null - Alternative selection; use withQueryFilter.

  • inputSource - InputSource, default: QInputSource.SYSTEM - Controls personalization and custom logic; HTTP callers retain their operation’s USER semantics.

  • transaction - QBackendTransaction, default: null - Optional supported caller-owned transaction. RDBMS uses the supplied RDBMSTransaction connection without committing, rolling back or closing it.

  • omitDmlAudit - boolean, default: false - Suppresses the action’s DML audit.

  • auditContext - String, default: null - Optional context for DML audit messages.

  • flags - Set of ActionFlag, default: null - Optional flags interpreted by supporting action/customizer code.

  • asyncJobCallback - AsyncJobCallback, inherited - A NonPersistedAsyncJobCallback is created on first access when none was supplied. Progress/cancellation behavior depends on the executing action/backend.

Associations, results and transaction boundaries

Cascade traversal uses the canonical declared association graph and complete native relationship membership. It does not depend on supplying child records or displaying their association groups in a UI. Declared join orientation and typed composite/non-primary-key tuples determine membership; a null tuple has no members. Descendants are deleted before ancestors. Cycles, rejected children or incomplete child deletion stop that parent mutation. Ancestor errors do not expose private descendant details. Input source, transaction, flags and audit options propagate to descendants. Unsupported structural discovery must fail before parent mutation; ordinary visibility-filtered queries cannot replace complete discovery.

DeleteOutput provides deletedRecordCount, recordsWithErrors and recordsWithWarnings. The root count may coexist with errors, including errors added after deletion. Empty result lists can be absent from JSON. For hidden or masked primary keys, generated not-found and native system-error diagnostics use generic messages rather than repeating the raw key. Custom application messages remain the customizer’s responsibility. Native failures may return per-key system errors or throw; an error does not prove the entire request left storage unchanged. Earlier pages, siblings or descendants may already have been deleted when a later operation fails. Delete does not create automatic batch/recursive rollback, distributed atomicity or concurrency isolation. Use a supported encompassing caller transaction and explicitly roll it back when all-or-nothing behavior is required.

ReplaceAction

ReplaceAction matches desired records by a caller-provided UniqueKey, inserts unmatched records, updates matches and optionally deletes omitted rows from a QQQ Table. Initialize QContext with the intended instance and session before constructing the input.

Example

This sample matches people by email and explicitly preserves omitted people.

QRecord desired = new QRecord()
   .withValue("firstName", "Alex")
   .withValue("lastName", "Example")
   .withValue("email", "[email protected]");
ReplaceInput input = new ReplaceInput()
   .withKey(new UniqueKey("email"))
   .withRecords(List.of(desired))
   .withPerformDeletes(false);
input.setTableName("person");
input.setInputSource(QInputSource.USER);
ReplaceOutput output = new ReplaceAction().execute(input);

Matching and privacy

Use distinct physical matching fields available in the active metadata; virtual or USER-removed fields cannot be matching keys. Supply an explicit record list with no null records, repeated record objects or existing record errors. Matching privately applies the fields' filter-value normalization and declared type conversion to desired key values, without running dynamic defaults, READ transformations or query customizers. Native matching retains applicable READ/READ_AND_WRITE locks and uses the input source, active metadata and transaction. Ordinary child write validation still applies. MongoDB matching uses the native write lookup contract, including refusal of selected native String keys or changed decoded primary keys.

Comparable duplicate desired tuples are rejected across the entire request, including page boundaries. Typed tuple/primary-key equality includes decimal value and binary content; ambiguous stored owners are rejected before child writes. Null components are distinct by default and do not match existing rows. allowNullKeyValuesToEqual=true enables null matching and duplicate detection for those tuples, including a single-column null lookup. This does not add a database uniqueness constraint or emulate arbitrary database collations/coercions. Native representation and READ enforcement remain the selected adapter’s responsibility.

Physical matching values are not copied into public records. Newly discovered hidden/heavy primary keys remain omitted, and unrevealed password keys remain masked; explicit caller-supplied keys retain their ordinary write semantics. Matched input records can receive an existing public key, so input objects are not immutable snapshots. Internal private identity is not serialized into QQQ JSON or Java serialization. Reusing or copying a masked/private record does not make that identity an explicit caller-supplied key or grant READ/WRITE authority.

Deletion scope

filter limits deletion only; matching and updates may affect rows outside its predicate. When deletion is enabled, retained keys are excluded before applying the original filter’s top-level ordering, skip and limit to omission candidates that satisfy its predicate. Ordinary Delete authorization still applies. For native-supported root UNION, UNION_ALL, INTERSECT or EXCEPT, retained keys are excluded from each operand; the root set operator and its ordering/window remain in effect. This preserves supported root set queries, without adding nested-set support or ordering by unselected fields. An explicitly empty desired list can remove the deletion scope; with no filter, Replace adds no predicate restricting that scope. Use performDeletes=false when omitted rows must remain.

INSERT callbacks and generated keys

With setPrimaryKeyInInsertedRecords=true, PRE_INSERT must retain one record per original INSERT candidate. Reordering is supported; use new QRecord(record) when replacing records so generated keys can reach the correct originals. Fresh reconstructed PRE_INSERT records are supported when this option is false. They cannot provide original-input correlation when it is true. POST_INSERT may reorder or copy records, but its result must contain exactly the native inserted primary-key set: missing, duplicate, unrelated or retargeted identities fail Replace. Use the QRecord copy constructor to retain private keys in callback snapshots; reconstruction from a sanitized values map can lose required identity. The native inserted rows determine which keys survive omission deletion. Generated-key publication uses input/copy identity rather than output position, occurs after successful child phases, and preserves private-key presentation.

Phase failures and transactions

Replace executes INSERT, then UPDATE, then optional DELETE, with ordinary validation, customizers and associations. Child record errors, including nested association errors, cause QException, stop later phases and prevent an owned transaction’s commit. Warnings alone do not fail a phase. Within Replace, recursive INSERT/UPDATE results are checked before POST callbacks can erase failed validation/PRE or native outcomes, and returned POST errors also fail the operation. Ordinary standalone INSERT/UPDATE keep their own result contracts. Child exceptions also fail Replace; failure does not return a partial ReplaceOutput.

  • Without a supplied transaction, Replace opens one through the backend, commits only after successful phases, rolls back on failure and closes it.

  • With a supplied transaction, Replace never commits, rolls back or closes it. Earlier successful work may remain pending; the caller decides whether to roll it back.

Rollback requires actual backend support; no-op or provider-specific transaction modes cannot guarantee restoration. A database rollback does not restore mutated Java input records or undo external customizer effects. These boundaries do not promise distributed atomicity, concurrency isolation or uniform provider behavior.

ReplaceInput

  • tableName - String, Required - Table whose native matching and child DML actions are used.

  • key - UniqueKey, Required - Distinct available physical fields used to match desired records.

  • records - List of QRecord, Required - Explicit desired records; an empty list can be destructive with deletes enabled.

  • filter - QQueryFilter, default: null - Deletion predicate and supported top-level ordering/window; retained keys are excluded before that window.

  • performDeletes - boolean, default: true - Enables deletion of omitted rows in scope.

  • allowNullKeyValuesToEqual - boolean, default: false - Allows null matching-key components to compare equal.

  • setPrimaryKeyInInsertedRecords - boolean, default: false - Publishes generated keys to the corresponding original input records after successful child phases, subject to callback identity and privacy rules above.

  • transaction - QBackendTransaction, default: null - Optional caller-owned transaction.

  • inputSource - InputSource, inherited default: QInputSource.SYSTEM - Propagated with active personalized metadata into matching and child actions; USER and SYSTEM retain their respective policies.

  • omitDmlAudit - boolean, default: false - Propagated to the child DML actions.

  • flags - Set of ActionFlag, default: null - Propagated to supporting child action/customizer code.

ReplaceOutput

  • insertOutput - InsertOutput - The completed INSERT phase’s result.

  • updateOutput - UpdateOutput - The completed UPDATE phase’s result.

  • deleteOutput - DeleteOutput - The completed DELETE phase’s result, or null when deletion is disabled.

Inspect child warnings and results using their normal contracts; there is no aggregate atomicity flag. Calling the core action does not create HTTP table-permission grants.

AuditAction

This action is implemented; its detailed reference is available in AuditAction source. Use the matching input/output classes and the current QContext.

QQQ Default Implementations

TableSyncProcess

The TableSyncProcess is designed to help solve the common use-case where you have a set of records in one table, you want to apply a transformation to them, and then you want them to result in records in a different table.

Sample Scenario

For example, you may be receiving an import file feed from a partner - say, a CSV file of order records - that you need to synchronize into your local database’s orders table.

High Level Steps

At a high-level, the steps of this task are:

  1. Get the records from the partner’s feed.

  2. Decide Insert/Update - e.g., if you already have any of these orders in your database table, in which case, they need updated, versus ones you don’t have, which need inserted.

  3. Map fields from the partner’s order record to your own fields.

  4. Store the necessary records in your database (insert or update).

What you need to tell QQQ

TableSyncProcess, as defined by QQQ, knows how to do everything for a job like this, other than the part that’s unique to your business. Those unique parts that you need to tell QQQ are:

  • What are the source & destination tables?

  • What are the criteria to identify source records to be processed?

  • What are the key fields are that "link" together records from the source & destination tables?

    • For the example above, imagine you have a "partnerOrderNo" field in your database’s order table - then you’d need to tell QQQ what field in the partner’s table provides the value for that field in your table.

  • How do fields / values from the source table map to fields & values in the destination table?

Detailed Steps

To get more specific, in a QQQ TableSyncProcess, here’s what happens for each of the high-level steps given above:

  • Getting records from the partner’s feed (done by QQQ):

    • Records from the source will be fetched via an Extract step, which runs a query to find the records that need processing.

    • Depending on your use-case, you may use an ExtractViaQueryStep (maybe with a Table Automation) or ExtractViaBasepullQueryStep (e.g., if you are polling a remote data source).

  • Deciding Insert/Update (done by QQQ):

    • Given a set of records from the source table (e.g., output of the Extract step mentioned above), get values from the "key" field in that table.

    • Do a lookup in the destination table, where its corresponding "key" field has the values extracted from the source records.

    • For each source record, if its "key" was found in the destination table, then plan an update to that existing corresponding destination record; else, plan to insert a new record in the destination table.

  • Mapping values (done by your custom Application code):

    • Specifically, mapping is done in a subclass of AbstractTableSyncTransformStep, in the populateRecordToStore method. Of particular interest are these two parameters to that method:

      • QRecord destinationRecord - as determined by the insert/update logic above, this will either be a new empty record (e.g., for inserting), or a fully populated record from the destination table (for updating).

      • QRecord sourceRecord - this is the record being processed, from the source table.

    • This method is responsible for setting values in the destinationRecord, including the matching foreign-key value for new records, and returning that record (unless it has decided that for some reason the record should not be stored, in which case it may return null).

  • Storing the records (done by QQQ):

    • This is typically done with the LoadViaInsertOrUpdateStep, though it is customizable if additional work is needed (e.g., via a subclass of LoadViaInsertOrUpdateStep, or a more custom subclass of AbstractLoadStep).

Bare-bones Example

For this example, let’s assume we’re setting up a partner-order-feed as described above, with the following details:

  • We have records from a partner in a table named "partnerOrderImport" (let’s assume the records may have been created using the QQQ FilesystemImporter process). These records have the following fields:

    • orderNo, date, city, state, postal, whseNo

  • We need to synchronize those records with table in our database named "order", with the following fields corresponding to those from the partner:

    • partnerOrderNo, orderDate, shipToCity, shipToState, shipToZipCode, warehouseId

  • The same conceptual order may appear in the "partnerOrderImport" multiple times, e.g., if they update some data on the order and re-transmit it to us. Meaning, we need to update our "order" records when we receive a new version an existing order.

To use TableSyncProcess for solving this use-case, we’ll need to create 2 things:

  1. A QProcessMetaData object, which we can create using the builder object provided by class TableSyncProcess. Note that this type of process is specialization of the standard QQQ StreamedETLWithFrontendProcess, as described elsewhere in this documentation.

  2. A subclass of AbstractTableSyncTransformStep, where we implement our mapping logic. Again, note that AbstractTableSyncTransformStep is a subclass of AbstractTransformStep, as used by StreamedETLWithFrontendProcess.

And to be good programmers, we’ll actually create a 3rd thing:

  1. A unit test for our Transform step.

Here are examples of these pieces of code. They require the named source and destination tables, an initialized QContext and an authenticated session. The test fixture must provide an isolated backend and a generated integer id on order.

Example of building process using the TableSyncProcess builder:
// the false argument below tells the build we are not a basepull-style process
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(false)

    // give our process a unique name within our QInstance
    .withName("partnerOrderToLocalOrderProcess")

    // tell the process to what class to use for transforming records from source to destination
    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)

    .getProcessMetaData();
Example implementation of an AbstractTableSyncTransformStep
public class PartnerOrderToOrderTransformStep extends AbstractTableSyncTransformStep
{
   @Override
   protected SyncProcessConfig getSyncProcessConfig()
   {
      return (new SyncProcessConfig(
         "partnerOrderImport", // source tableName
         "orderNo",            // source table key fieldName
         "order",              // destination tableName
         "partnerOrderNo"      // destination table foreign key fieldName
      ));
   }

   @Override
   public QRecord populateRecordToStore(RunBackendStepInput runBackendStepInput, QRecord destinationRecord, QRecord sourceRecord) throws QException
   {
      // map the matching key as well as the other values for new records
      destinationRecord.setValue("partnerOrderNo", sourceRecord.getValue("orderNo"));
      destinationRecord.setValue("orderDate", sourceRecord.getValue("date"));
      destinationRecord.setValue("shipToCity", sourceRecord.getValue("city"));
      destinationRecord.setValue("shipToState", sourceRecord.getValue("state"));
      destinationRecord.setValue("shipToZipCode", sourceRecord.getValue("postal"));
      return (destinationRecord);
   }

}
Example Unit Test for a transform step
   @Test
   void testTransformStep() throws QException
   {
      // insert 1 test order, that will be updated by the transform step
      Integer existingId = new InsertAction().execute(new InsertInput("order").withRecords(List.of(
         new QRecord().withValue("partnerOrderNo", 101).withValue("shipToState", "IL")
      ))).getRecords().get(0).getValueInteger("id");

      // set up input for the step - a list of 2 of the partner's orders
      RunBackendStepInput input = new RunBackendStepInput();
      input.setRecords(List.of(
         new QRecord().withValue("orderNo", 101).withValue("state", "NY"), // will update the order above
         new QRecord().withValue("orderNo", 102).withValue("state", "CA")  // will insert a new order
      ));
      RunBackendStepOutput output = new RunBackendStepOutput();

      // run the code under test - our transform step
      new PartnerOrderToOrderTransformStep().runOnePage(input, output);

      // Note that by just running the transform step, no records have been stored.
      // We can assert against the output of this step.

      assertEquals(existingId, output.getRecords().get(0).getValue("id"));
      assertEquals(101, output.getRecords().get(0).getValue("partnerOrderNo"));
      assertEquals("NY", output.getRecords().get(0).getValue("shipToState"));

      assertNull(output.getRecords().get(1).getValue("id"));
      assertEquals(102, output.getRecords().get(1).getValue("partnerOrderNo"));
      assertEquals("CA", output.getRecords().get(1).getValue("shipToState"));
   }

Pseudocode process flow

Now that we’ve seen the bare-bones example, let’s see an even more detailed breakdown of how a full TableSyncProcess works, by looking at its 3 "ETL" steps in pseudocode:

ExtractStep (Producer Thread)

  • Queries source table for records

    • Often based on Table Automations (e.g., for all newly inserted records) or Basepull pattern (polling for new/updated records).

TransformStep (Consumer Thread)

  • Receives pages of records from ExtractStep in the runOnePage method.

  • Makes sourceKeyList by getting sourceTableKeyField values from the records.

  • Calls initializeRecordLookupHelper(runBackendStepInput, sourceRecordList)

    • Calls getLookupsToPreLoad to control which lookups are performed.

  • Calls getExistingRecordsByForeignKey(runBackendStepInput, destinationTableForeignKeyField, destinationTableName, sourceKeyList);

    • Calls getExistingRecordQueryFilter(runBackendStepInput, sourceKeyList) as part of querying the destinationTable

    • Returns the output of buildExistingRecordsMap(destinationTableForeignKeyField, queryOutput.getRecords())

  • foreach input record (from sourceTable):

    • Calls getExistingRecord(existingRecordsByForeignKey, destinationForeignKeyField, sourceKeyValue)

    • if an existing record was returned (and if the syncConfig says performUpdates), this record is set as recordToStore

    • else if no existing record was returned (and if the syncConfig says performInserts), a new record is set as recordToStore

    • else continue the foreach.

    • call populateRecordToStore(runBackendStepInput, recordToStore, sourceRecord)

    • if a record is returned it is added to the process step output (to be stored in the LoadStep)

LoadStep (Consumer Thread)

  • Receives records from the output of the TransformStep.

  • Inserts and/or Updates destinationTable, with records returned by populateRecordToStore

Additional Process Configuration Examples

The following examples show how to use additional settings in the TableSyncProcess builder.

UI

While a TableSyncProcess will often run via a schedule and/or automation, we may also want to allow users to manually run it in a UI.

Making our process available for a UI
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(false)
    .withName("partnerOrderToLocalOrderProcess")
    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)

    // attach our process to its source table, to show up in UI
    .withTableName("partnerOrderImport")

    // add some fields to display on the review screen, in UI
    .withReviewStepRecordFields(List.of(
       new QFieldMetaData("clientId", QFieldType.STRING).withLabel("Client"),
       new QFieldMetaData("warehouseId", QFieldType.STRING).withLabel("Warehouse"),
       new QFieldMetaData("partnerOrderNo", QFieldType.STRING)))
    .getProcessMetaData();
Basepull

The previous example would work as a Table Automation (e.g., where the list of records identified in the Extract step were determined by the Automation system). However, a second common pattern is to use Basepull (e.g., if polling for updated records from a partner API endpoint).

Configuring our process as a Basepull
// the true argument below tells the build we ARE a basepull-style process
// this changes the default extract-step.
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(true)
    .withName("partnerOrderToLocalOrderProcess")
    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)

    // See Basepull documentation for details
    .withBasepullConfiguration(new BasepullConfiguration())

    // schedule our process to run automatically every minute
    .withSchedule(new QScheduleMetaData().withRepeatSeconds(60))

    .getProcessMetaData();

Additional Options in the Transform Step

Specifying to not perform Inserts or not perform Updates

We may have a scenario where we want our sync process to never update records if the key is already found in the destination table. We can configure this with an additional optional parameter to the SyncProcessConfig constructor:

Specifying to not do updates in a TableSyncProcess
   @Override
   protected SyncProcessConfig getSyncProcessConfig()
   {
      return (new SyncProcessConfig("partnerOrderImport", "orderNo", "order", "partnerOrderNo",
         true,  // performInserts
         false  // performUpdates
      ));
   }

Similarly, we may want to disallow inserts from a particular sync process. The performInserts argument to the SyncProcessConfig constructor lets us do that:

Specifying to not do inserts in a TableSyncProcess
   @Override
   protected SyncProcessConfig getSyncProcessConfig()
   {
      return (new SyncProcessConfig("partnerOrderImport", "orderNo", "order", "partnerOrderNo",
         false, // performInserts
         true   // performUpdates
      ));
   }
Customizing the query for existing records

In some cases, a specific Table Sync process may need to refine the query filter that is used to lookup existing records in the destination table (e.g. for determining insert vs. update).

For example, in our orders-from-a-partner scenario, if we have more than 1 partner sending us orders, where there could be overlapping orderNo values among them - we may have an additional field in our orders table to identify which partner an order came from. So then when we’re looking up orders by partnerOrderNo, we would need to also include the partnerId field in our query, so that we only update orders from the specific partner that we’re dealing with.

To do this (to customize the existing record query filter), we need can just override the method getExistingRecordQueryFilter. Generally we would start by calling the super version of the method, and then add to it additional criteria.

Customizing the query filter used to look for existing records
   /*******************************************************************************
    ** Define the query filter to find existing records.  e.g., for determining
    ** insert vs. update.  Subclasses may override this to customize the behavior,
    ** e.g., in case an additional field is needed in the query.
    *******************************************************************************/
   protected QQueryFilter getExistingRecordQueryFilter(RunBackendStepInput runBackendStepInput, List<Serializable> sourceKeyList)
   {
      QQueryFilter filter = super.getExistingRecordQueryFilter(runBackendStepInput, sourceKeyList);
      filter.addCriteria(new QFilterCriteria("partnerId", EQUALS, PARTNER_ID));
      return (filter);
   }
More efficient additional record lookups

It is a common use-case to need to map various ids from a partner’s system to ids in your own system. For the orders example, we might need to know what warehouse the order is shipping from. The customer may send their identifier for the warehouse, and we may need to map those identifiers to our own warehouse ids.

The QQQ-provided class RecordLookupHelper exists to help with performing lookups like this, and in particular, it can be used to execute one query to fetch a full table, storing records by a key field, then returning those records without performing additional queries.

AbstractTableSyncTransformStep has a protected recordLookupHelper member. If we override the method getLookupsToPreLoad(), then this object is populated by calling its preloadRecords method with each specified pair of tableNames and fieldNames.

Specifying tables to pre-load using a RecordLookupHelper
   /*******************************************************************************
    ** Specify a list of tableName/keyColumnName pairs to run through
    ** the preloadRecords method of the recordLookupHelper.
    *******************************************************************************/
   @Override
   protected List<Pair<String, String>> getLookupsToPreLoad()
   {
      return (List.of(
         Pair.of("warehouse", "partnerWarehouseNo")
      ));
   }

If we have preloaded some lookups, we can then use them in our populateRecordToStore method as follows:

Using the recordLookupHelper in populateRecordToStore
   // lookup warehouse with partnerWarehouseNo=whseNo from partner, and use our id in destination record
   String partnerWarehouseNo = sourceRecord.getValueString("whseNo");
   Integer warehouseId = recordLookupHelper.getRecordId("warehouse", "partnerWarehouseNo", partnerWarehouseNo, Integer.class);
   destinationRecord.setValue("warehouseId", warehouseId);
Additional override points

The following declarations list override signatures, not complete method implementations; provide method bodies when subclassing.

There are more methods which can be overridden in your AbstractTableSyncTransformStep subclass, to provide further customizations of behaviors, specifically in the area of dealing with existing records (e.g., the insert/update use-case).

Additional AbstractTableSyncTransformStep overrides
   /*******************************************************************************
    ** Run the existingRecordQueryFilter - to look in the destinationTable for
    ** any records that may need an update (rather than an insert).
    **
    ** Generally returns a Map, keyed by a Pair of the destinationTableForeignKeyField
    ** and the value in that field.  But, for more complex use-cases, one can override
    ** the buildExistingRecordsMap method, to make different keys (e.g., if there are
    ** two possible destinationTableForeignKeyFields).
    *******************************************************************************/
   protected Map<Pair<String, Serializable>, QRecord> getExistingRecordsByForeignKey
   (
      RunBackendStepInput runBackendStepInput,
      String destinationTableForeignKeyField,
      String destinationTableName,
      List<Serializable> sourceKeyList
   ) throws QException;


   /*******************************************************************************
    ** Overridable point where you can, for example, keys in the existingRecordsMap
    ** with different fieldNames from the destinationTable.
    **
    ** Note, if you're overriding this method, you'll likely also want & need to
    ** override getExistingRecord.
    *******************************************************************************/
   protected Map<Pair<String, Serializable>, QRecord> buildExistingRecordsMap
   (
      String destinationTableForeignKeyField,
      List<QRecord> existingRecordList
   ) throws QException;

   /*******************************************************************************
    ** Given the map of existingRecordsByForeignKey (as built by
    ** getExistingRecordsByForeignKey which calls buildExistingRecordsMap),
    ** get one record from that map, for a given key-value from a source record.
    **
    ** The destinationForeignKeyField is given as advice if needed (e.g., to see its type)
    *******************************************************************************/
   protected QRecord getExistingRecord
   (
      Map<Pair<String, Serializable>, QRecord> existingRecordsByForeignKey,
      QFieldMetaData destinationForeignKeyField,
      Serializable sourceKeyValue
   );

RenderTemplateAction

The RenderTemplateAction performs the job of taking a template - that is, a string of code, in a templating language, such as Velocity, and merging it with a set of data (known as a context), to produce some using-facing output, such as a String of HTML.

Actions run within the application’s current QContext; action inputs no longer accept a QInstance or session.

Examples

Canonical Form
RenderTemplateInput input = new RenderTemplateInput();
input.setCode("Hello, ${name}");
input.setTemplateType(TemplateType.VELOCITY);
input.setContext(Map.of("name", "Darin"));
RenderTemplateOutput output = new RenderTemplateAction().execute(input);
String result = output.getResult();
assertEquals("Hello, Darin", result);
Convenient Form
String result = RenderTemplateAction.renderVelocity(Map.of("name", "Darin"), "Hello, ${name}");
assertEquals("Hello, Darin", result);

RenderTemplateInput

  • code - String, Required - String of template code to be rendered, in the templating language specified by the type parameter.

  • type - Enum of VELOCITY, Required - Specifies the language of the template code.

  • context - Map of String → Object - Data to be made available to the template during rendering.

RenderTemplateOutput

  • result - String - Result of rendering the input template and context.

QQQ Utility Classes

RecordLookupHelper

RecordLookupHelper is a utility class that exists to help you…​ lookup records :)

OK, I’ll try to give a little more context:

Motivation 1: Performance

One of the most significant performance optimizations that the team behind QQQ has found time and time again, is to minimize the number of times you have to perform an I/O operation. To just say it more plainly: Make fewer calls to your database (or other backend).

This is part of why the DML actions in QQQ (InsertAction, UpdateAction, DeleteAction) are all written to work on multiple records: If you’ve got to insert 1,000 records, the performance difference between doing that as 1,000 SQL INSERT statements vs. just 1 statement cannot be overstated.

Similarly then, for looking up records: If we can do 1 round-trip to the database backend - that is - 1 query to fetch n records, then in almost all cases it will be significantly faster than doing n queries, one-by-one, for those n records.

The primary reason why RecordLookupHelper exists is to help you cut down on the number of times you have to make a round-trip to a backend data store to fetch records within a process.

This basically is version of caching, isn’t it? Take a set of data from "far away" (e.g., database), and bring it "closer" (local or instance variables), for faster access. So we may describe this as a "cache" through the rest of this document.

Motivation 2: Convenience

So, given that one wants to try to minimize the number of queries being executed to look up data in a QQQ processes, one can certainly do this "by-hand" in each process that they write.

Doing this kind of record caching in a QQQ Process BackendStep may be done as:

  • Adding a Map<Integer, QRecord> as a field in your class.

  • Setting up and running a QueryAction, with a filter based on the collection of the keys you need to look up, then iterating over (or streaming) the results into the map field.

  • Getting values out of the map when you need to use them (dealing with missing values as needed).

That’s not so bad, but, it does get a little verbose, especially if you’re going to have several such caches in your class.

As such, the second reason that RecordLookupHelper exists, is to be a reusable and convenient way to do this kind of optimization, by providing methods to perform the bulk query & map building operation described above, while also providing some convenient methods for accessing such data after it’s been fetched. In addition, a single instance of RecordLookupHelper can provide this service for multiple tables at once (e.g., so you don’t need to add a field to your class for each type of data that you’re trying to cache).

Use Cases

Preload records, then access them

Scenario:

  • We’re writing a process BackendStep that uses shipment records as input.

  • We need to know the order record associated with each shipment (via an orderId foreign key), for some business logic that isn’t germaine to the explanation of RecordLookupHelper.

  • We also to access some field on the shippingPartner record assigned to each shipment.

    • Note that here, the shipment table has a partnerCode field, which relates to the code unique-key in the shippingPartner table.

    • It’s also worth mentioning, we only have a handful of shippingPartner records in our database, and we never expect to have very many more than that.

Example of a process step using a RecordLookupHelper to preload records
public class MyShipmentProcessStep implements BackendStep
{
   // Add a new RecordLookupHelper field, which will "cache" both orders and shippingPartners
   private RecordLookupHelper recordLookupHelper = new RecordLookupHelper();

   @Override
   public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
   {
      // lookup the full shippingPartner table (because it's cheap & easy to do so)
      // use the partner's "code" as the key field (e.g,. they key in the helper's internal map).
      recordLookupHelper.preloadRecords("shippingPartner", "code");

      // get all of the orderIds from the input shipments
      List<Serializable> orderIds = input.getRecords().stream()
         .map(r -> r.getValue("orderId")).toList();

      // fetch the orders related to by these shipments
      recordLookupHelper.preloadRecords("order", "id", orderIds);

      for(QRecord shipment : input.getRecords())
      {
         // get someConfigField from the shippingPartner assigned to the shipment
         Boolean someConfig = recordLookupHelper.getRecordValue("shippingPartner", "someConfigField", "code", shipment.getValue("partnerCode"), Boolean.class);

         // get the order record assigned to the shipment
         QRecord order = recordLookupHelper.getRecordByKey("order", "id", shipment.getValue("orderId"));
      }
   }
}
Lazy fetching records

Scenario:

  • We have a BackendStep that is taking in purchaseOrderHeader records, from an API partner.

  • For each record, we need to make an API call to the partner to fetch the purchaseOrderLine records under that header.

    • In this contrived example, the partner’s API forces us to do these lookups order-by-order…​

  • Each purchaseOrderLine that we fetch will have a sku on it - a reference to our item table.

    • We need to look up each item to apply some business logic.

    • We assume there are very many item records in the backend database, so we don’t want to pre-load the full table. Also, we don’t know what sku values we will need until we fetch the purchaseOrderLine.

This is a situation where we can use RecordLookupHelper to lazily fetch the item records as we discover them, and it will take care of not re-fetching ones that it has already loaded.

Example of a process step using a RecordLookupHelper to lazy fetch records
public class MyPurchaseOrderProcessStep implements BackendStep
{
   // Add a new RecordLookupHelper field, which will "cache" lazy-loaded item records
   private RecordLookupHelper recordLookupHelper = new RecordLookupHelper();

   @Override
   public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
   {
      for(QRecord poHeader : input.getRecords())
      {
         // fetch the lines under the header
         Serializable poNo = poHeader.getValue("poNo");
         List<QRecord> poLines = new QueryAction().execute(new QueryInput("purchaseOrderLine")
            .withFilter(new QQueryFilter(new QFilterCriteria("poNo", EQUALS, poNo)))).getRecords();

         for(QRecord poLine : poLines)
         {
            // use recordLookupHelper to lazy-load item records by SKU.
            QRecord item = recordLookupHelper.getRecordByKey("item", "sku", poLine.getValue("sku"));

            // business logic related to item performed here.
         }
      }
   }
}

In this example, we will be doing exactly 1 query on the item table for each unique sku that is found across all of the poLine records we process. That is to say, if the same sku appears on only 1 poLine, or if it appears on 100 poLines, we will still only query once for that sku.

A slight tweak could be made to the example above, to make 1 item table lookup for each poHeader record:

Tweaked example doing 1 item lookup per poHeader
// continuing from above, after the List<QRecord> poLines has been built

// get all of the skus from the lines
List<Serializable> skus = poLines.stream().map(r -> r.getValue("sku")).toList();

// preload the items for the skus
recordLookupHelper.preloadRecords("item", "sku", new QQueryFilter(new QFilterCriteria("sku", IN, skus)));

for(QRecord poLine : poLines)
{
   // get the items from the helper
   QRecord item = recordLookupHelper.getRecordByKey("item", "sku", poLine.getValue("sku"));
}

In this example, we’ve made a trade-off: We will query the item table exactly 1 time for each poHeader that we process. However, if the same sku is on every PO that we process, we will end up fetching it multiple times.

This could end up being better or worse than the previous example, depending on the distribution of the data we are dealing with.

A further tweak, a hybrid approach, could potentially reap the benefits of both of these examples (at the tradeoff of, more code, more complexity):

Tweaked example doing 1 item lookup per poHeader, but only for not-previously-encountered skus
// Critically - we must tell our recordLookupHelper to NOT do any one-off lookups in this table
recordLookupHelper.setMayNotDoOneOffLookups("item", "sku");

// continuing from above, after the List<QRecord> poLines has been built

// get all of the skus from the lines
List<Serializable> skus = poLines.stream().map(r -> r.getValue("sku")).toList();

// determine which skus have not yet been loaded - e.g., they are not in the recordLookupHelper.
// this is why we needed to tell it above not to do one-off lookups; else it would lazy-load each sku here.
List<Serializable> skusToLoad = new ArrayList<>();
for(Serializable sku : skus)
{
   if(recordLookupHelper.getRecordByKey("item", "sku", sku) == null)
   {
      skusToLoad.add(sku);
   }
}

// preload the item records for any skus that are still needed
if(!skusToLoad.isEmpty())
{
   recordLookupHelper.preloadRecords("item", "sku",
      new QQueryFilter(new QFilterCriteria("sku", IN, skusToLoad)));
}

// continue as above

In this example, we will start by querying the item table once for each poHeader, but, if we eventually encounter a PO where all of its skus have already been loaded, then we may be able to avoid any item queries for such a PO.

Implementation Details

  • Internally, an instance of RecordLookupHelper maintains a number of Maps, with QQQ table names and field names as keys.

  • The accessing/lazy-fetching methods (e.g., any method whose name starts with getRecord) all begin by looking in these internal maps for the tableName and keyFieldName that they take as parameters.

    • If they find an entry in the maps, then it is used for producing a return value.

    • If they do not find an entry, then they will perform the a QueryAction, to try to fetch the requested record from the table’s backend.

      • Unless the setMayNotDoOneOffLookups method has been called for the (tableName, keyFieldName) pair.

Full API

Methods for accessing and lazy-fetching

  • getRecordByKey(String tableName, String keyFieldName, Serializable key)

Get a QRecord from tableName, where keyFieldName = key.

  • getRecordValue(String tableName, String requestedField, String keyFieldName, Serializable key)

Get the field requestedField from the record in tableName, where keyFieldName = key, as a Serializable. If the record is not found, null is returned.

  • getRecordValue(String tableName, String requestedField, String keyFieldName, Serializable key, Class<T> type)

Get the field requestedField from the record in tableName, where keyFieldName = key, as an instance of type. If the record is not found, null is returned.

  • getRecordId(String tableName, String keyFieldName, Serializable key)

Get the primary key of the record in tableName, where keyFieldName = key, as a Serializable. If the record is not found, null is returned.

  • getRecordId(String tableName, String keyFieldName, Serializable key, Class<T> type)

Get the primary key of the record in tableName, where keyFieldName = key, as an instance of type. If the record is not found, null is returned.

  • getRecordByUniqueKey(String tableName, Map<String, Serializable> uniqueKey)

Get a QRecord from tableName, where the record matches the field/value pairs in uniqueKey.

Note: this method does not use the same internal map as the rest of the class. As such, it does not take advantage of any data fetched via the preload methods. It is only used for caching lazy-fetches.

Methods for preloading

  • preloadRecords(String tableName, String keyFieldName)

Query for all records from tableName, storing them in an internal map keyed by the field keyFieldName.

  • preloadRecords(String tableName, String keyFieldName, QQueryFilter filter)

Query for records matching filter from tableName, storing them in an internal map keyed by the field keyFieldName.

  • preloadRecords(String tableName, String keyFieldName, List<Serializable> inList)

Query for records with the field keyFieldName having a value in inList from tableName, storing them in an internal map keyed by the field keyFieldName.

Config Methods

  • setMayNotDoOneOffLookups(String tableName, String fieldName)

For cases where you know that you have preloaded records for tableName, keyed by fieldName, and you know that some of the keys may not have been found, so you want to avoid doing a query when a missed key is found in one of the getRecord…​ methods, then if you call this method, an internal flag is set, which will prevent any such one-off lookups.

In other words, if this method has been called for a (tableName, fieldName) pair, then the getRecord…​ methods will only look in the internal map for records, and no queries will be performed to look for records.

Webhook signing and the SSRF-safe HTTP client

Two small utilities in qqq-backend-core for any code that sends HTTP requests to URLs that users typed in (webhook endpoints, alert channels, revalidation hooks):

  • com.kingsrook.qqq.backend.core.utils.webhooks: WebhookSecret, WebhookSigner, WebhookVerifier, WebhookVerificationException. These sign and verify requests in the Standard Webhooks format.

  • com.kingsrook.qqq.backend.core.utils.http: SafeHttpClient, SafeHttpPolicy, SafeHttpResponse, SafeHttpException, AddressRules. This client refuses to reach the internal network unless an allowlist says so.

They have no tables, threads or configuration of their own. qbit-webhooks is their first consumer; the Alerts and CDN qbits use them too.

Signing format

Every signed request carries three headers:

Header Value

webhook-id

The message (event) id. It stays the same on every retry of the same event, so receivers use it as the idempotency key.

webhook-timestamp

When this attempt was sent, as Unix seconds.

webhook-signature

v1,<base64 signature>. During a secret rotation there is one entry per active secret, separated by spaces: v1,<sig-new> v1,<sig-old>.

The signature is HMAC-SHA256, base64 encoded, over the bytes of:

<webhook-id>.<webhook-timestamp>.<exact request body>

Secrets are written whsec_<base64 key bytes>, and the HMAC key is the decoded bytes. A secret without the whsec_ prefix is used as raw UTF-8 bytes. Generated secrets are 32 random bytes; the shortest accepted is 16 bytes. WebhookSecret.toString() prints only a 12-character SHA-256 fingerprint, never the key.

Receivers must:

  1. Reject the request if any of the three headers is missing.

  2. Reject a timestamp more than 5 minutes from their own clock, either way. This stops replays of old requests.

  3. Compute the signature over the raw body bytes, before any JSON parsing.

  4. Accept the request if any v1 entry matches, comparing in constant time. Ignore entries with other version prefixes.

  5. Optionally, remember recent webhook-id values and ignore a repeat. Delivery is at-least-once.

Sending
Map<String, String> headers = WebhookSigner.headers(eventId, Instant.now(), bodyBytes, List.of(activeSecret));
// during rotation: List.of(newSecret, previousSecret)
Verifying in Java
WebhookVerifier verifier = new WebhookVerifier(List.of(WebhookSecret.parse(secretFromVault)));
try
{
   verifier.verify(requestHeaders, rawBodyBytes);   // header names are case-insensitive
}
catch(WebhookVerificationException e)
{
   // e.getReason(): MISSING_HEADERS, BAD_TIMESTAMP, TIMESTAMP_OUT_OF_TOLERANCE or NO_MATCHING_SIGNATURE
   respondWith(400);
}
Verifying in Node.js
const crypto = require('crypto');

function verifyWebhook(secret, headers, rawBody /* Buffer */) {
  const id = headers['webhook-id'];
  const ts = headers['webhook-timestamp'];
  const signatures = headers['webhook-signature'];
  if (!id || !ts || !signatures) throw new Error('missing webhook headers');
  if (Math.abs(Date.now() / 1000 - Number(ts)) > 300) throw new Error('timestamp outside tolerance');

  const key = secret.startsWith('whsec_') ? Buffer.from(secret.slice(6), 'base64') : Buffer.from(secret, 'utf8');
  const expected = crypto.createHmac('sha256', key).update(`${id}.${ts}.`).update(rawBody).digest();

  for (const entry of signatures.split(' ')) {
    const [version, signature] = entry.split(',');
    if (version !== 'v1' || !signature) continue;
    const given = Buffer.from(signature, 'base64');
    if (given.length === expected.length && crypto.timingSafeEqual(given, expected)) return true;
  }
  throw new Error('no matching signature');
}
Verifying in Python
import base64, hashlib, hmac, time

def verify_webhook(secret: str, headers: dict, raw_body: bytes) -> None:
    msg_id = headers.get("webhook-id")
    ts = headers.get("webhook-timestamp")
    signatures = headers.get("webhook-signature")
    if not (msg_id and ts and signatures):
        raise ValueError("missing webhook headers")
    if abs(time.time() - int(ts)) > 300:
        raise ValueError("timestamp outside tolerance")

    key = base64.b64decode(secret[6:]) if secret.startswith("whsec_") else secret.encode()
    digest = hmac.new(key, f"{msg_id}.{ts}.".encode() + raw_body, hashlib.sha256).digest()
    expected = base64.b64encode(digest).decode()

    for entry in signatures.split(" "):
        version, _, signature = entry.partition(",")
        if version == "v1" and hmac.compare_digest(signature, expected):
            return
    raise ValueError("no matching signature")

The reference vector from the Standard Webhooks libraries is checked by WebhookSigningTest. The secret whsec_MfKQ9r8GKYqrTwjUPD8ILPZIo2LaLaSw, id msg_p5jXN8AQM9LWM0D4loKWxJek, timestamp 1614265330 and body {"test": 2432232314} sign to v1,g0hM9SsE+OTPJTGt/tmIKtSyZlE3uFJELVlNIOLJ1OE=.

SSRF policy (SafeHttpClient)

SafeHttpPolicy policy = SafeHttpPolicy.defaults()                 // https only, no internal targets
   .withInternalAllowlist(List.of(".svc.cluster.local"))           // admin-chosen exceptions only
   .withConnectTimeout(Duration.ofSeconds(5))
   .withReadTimeout(Duration.ofSeconds(10))
   .withTotalTimeout(Duration.ofSeconds(30))
   .withMaxResponseBytes(64 * 1024);
try(SafeHttpClient client = new SafeHttpClient(policy))           // one pooled client per policy; share it
{
   SafeHttpResponse response = client.post(url, headers, body);    // any HTTP status is a response
}
catch(SafeHttpException e)
{
   // e.getReason(): INVALID_URL, BLOCKED_SCHEME, BLOCKED_ADDRESS (policy refusals: e.getReason().isPolicy()),
   // DNS_FAILURE, CONNECT_TIMEOUT, READ_TIMEOUT, TOTAL_TIMEOUT, TLS_FAILURE, CONNECTION_FAILED
}

What the client enforces:

  • Scheme and shape. Only https, plus http when the policy allows it. URLs with a user name or password, or with no host, are refused.

  • Blocked addresses. These ranges are refused by default:

    • "this network" 0.0.0.0/8

    • private: 10/8, 172.16/12 and 192.168/16

    • carrier-grade NAT 100.64/10

    • loopback: 127/8 and ::1

    • link-local: 169.254/16 and fe80::/10, which covers the cloud metadata endpoints

    • IETF protocol and documentation ranges

    • benchmarking 198.18/15

    • multicast, reserved and broadcast: 224/4 and 240/4

    • IPv6 unique-local fc00::/7, site-local fec0::/10, discard 100::/64 and documentation 2001:db8::/32

    An IPv4 address embedded in IPv6 is checked as IPv4: IPv4-mapped, IPv4-compatible, NAT64 64:ff9b::/96 and 6to4 2002::/16.

  • Resolve once, check all, pin. The host is resolved once per request. If any address it resolves to is blocked, the request is refused, so a mixed DNS answer does not slip through. The connection is then made only to the addresses that were checked; the client’s own DNS resolver refuses any host that was not checked. A DNS answer that changes between the check and the connect (DNS rebinding) is therefore never used.

  • No redirects. A 3xx is returned to the caller as the response and is never followed. A redirect to an internal address cannot be used to reach it.

  • TLS. TLS 1.2 or 1.3, the JVM trust store and standard host-name verification.

  • Timeouts and size. There are connect, read (idle) and total timeouts; the total timeout aborts a server that trickles bytes. At most maxResponseBytes of the body is read; the rest is dropped with the connection, and truncated() is true.

  • Allowlist (admin only). withInternalAllowlist re-opens specific internal targets. An entry can be a CIDR block (10.100.22.0/24), a single address, an exact host name, or a host suffix with a leading dot (.svc.cluster.local). The cloud metadata addresses (169.254.169.254, 100.100.100.200, fd00:ec2::254) and the unspecified addresses stay blocked even when an allowlist entry covers them. Only administrators should be able to set an allowlist.

  • Validation on save. checkUrl(url) applies the same checks without sending anything, so a URL can be rejected when it is saved. The send-time check still runs, because DNS can change in between.

Error messages never include request bodies, headers or secrets.

Migration Guides

Release Operations