Introduction

QQQ is a Low-code Application Framework for Engineers. Its purpose is to provide the basic structural elements of an application - things that every application needs - so that the engineers building an application on top of QQQ don’t need to worry about those pieces, and can instead focus on the unique needs of the application that they are building.

What makes QQQ special?

The scope of what QQQ provides is far-reaching, and most likely goes beyond what you may initially be thinking. That is to say, QQQ includes code all the way from the backend of an application, through its middleware layer, and including its frontend. For example, a common set of modules deployed in a QQQ application will provide:

  • Backend RDBMS/Database connectivity and access.

  • Frontend web UI (e.g., a React application)

  • A java web server acting as middleware between the frontend web UI and the backend

That is to say - as an engineer deploying a QQQ application - you do not need to write a single line of code that is concerned with any of those things.

  • You do not need to write code to connect to your database.

  • You do not write any web UI code.

  • You do not write any middleware code to tie together the frontend and backend.

Instead, QQQ includes all of these pieces. QQQ knows how to connect to databases (and actually, several other kinds of backend systems - but ignore that for now). Plus it knows how to do most of what an application needs to do with a database (single-record lookups, complex queries, joins, aggregates, bulk inserts, updates, deletes). QQQ also knows how to present the data from a database - in table views, or single-record views, or exports, or reports or widgets. And it knows how to present a powerful ad-hoc query interface to users, and how to show screens where users can create, update, and delete records. It also provides the connective tissue (middleware) between those backend layers where data is stored, and frontend layers were users interact with data.

What makes your application special?

I’ve said a lot about what does QQQ knows - but let’s dig a little deeper. What does QQQ know, and what does it not know? Well - what it doesn’t know is, it doesn’t know the special or unique aspects of the application that you are building.

So, what makes your application unique?

Is your application unique because it needs to have screens where users can search for records in a table?

No!

QQQ assumes (as does the author of this document) that all applications (at least of the nature that QQQ supports) need what we call Query Screens. So QQQ gives you a Query Screen for all of your tables - with zero code from you.

Is your application unique because you want users to be able to view, create, edit, and delete records from tables?

No!

QQQ assumes that all applications need these basic CRUD capabilities. So QQQ provides all of these UI screens - for view, create, edit, delete - along with the supporting middleware and backend code - all the way down to the SQL that selects & manages the data. You get it all for free - zero code.

Is your application unique because you have a fiz_bar table, with 47 columns, and a whoz_zat table, with 42 columns of its own, that joins to fiz_bar on the zizzy_ziz_id field?

Yes!

OK - we found some of it - what makes your application unique is the data that you’re working with. Your tables - their fields - the connection info for your database. QQQ doesn’t know those details. So - that’s your first job as a QQQ application engineer - to describe your data to QQQ.

For the example above - you need to tell QQQ that you have a fiz_bar table, and that you have a whoz_zat table - and you need to tell QQQ what the fields or columns in thsoe tables are. You can even tell QQQ how to join those tables (on that zizzy_ziz_id field). And then that’s it. Once QQQ has this QQQ Table meta data, it can then provide its Query screens, and full CRUD screens to your tables, in your database, with your fields.

And at the risk of repeating myself - you can do this (get a full Query & CRUD application) with zero lines of actual procedural code. You only need to supply meta-data (which, at the time of this writing, is done in Java, but it’s just creating objects - and in the future could be done in YAML files, for example).

Beyond Basics

Going beyond the basic wiring as described above, QQQ also provides some of the more advanced elements needed in a modern data-driven web application, including:

  • Authentication & Authorization

  • ad-hoc Query engine for access to data tables

  • Full CRUD (Create, Read, Update, Delete) capabilities

  • Multistep custom workflows ("Processes" in QQQ parlance)

  • Scheduled jobs

  • Enterprise Service Bus

So what do we mean by all of this? We said that basically every application needs, for example, Authentication & Authorization. Login screens. User & Role tables. Permissions. So, when it’s time for you to build a new application for your Big Tall Floor Lamp manufacturing business, do you need to start by writing a login screen? And a Permissions scheme? And throwing HTTP 401 errors? And managing user-role relationships? And then having a bug in the check permission logic on the Light Bulb Inventory Edit screen, so Jim is always keying in bad quantities, even though he isn’t supposed to have permission there?

No!

All of the (really important, even though application developers hate doing it) aspects of security - you don’t need to write ANY code for dealing with that. Just tell QQQ what Authentication provider you want to use (e.g., OAuth2 or Auth0), and - to paraphrase the old iMac ad - there’s no step 2. QQQ just does it.

QQQ can provide this type of application using a variety and/or combination of backend data storage types. And, whichever type of backend is used, QQQ gives a common interface (both user-facing and programmer-facing). Backend types include:

  • Relational Databases (RDBMS)

  • File Systems

  • Web APIs

  • NoSQL/Document Databases (Future)

In addition, out-of-the-box, QQQ also goes beyond these basics, delivering:

  • Bulk versions of all CRUD operations.

  • Automatically generated JSON APIs.

  • Auditing of data changes.

  • End-user (e.g., non-engineer) customization via dynamic scripting capabilities

todo say much more

QQQ Architecture

Like a house!

Developing a QQQ Application

In developing an application with QQQ, engineers will generally have to define two types of code:

  1. Meta Data - This is the code that you use to tell QQQ the shape of your application - your unique Tables, Processes, Apps, Reports, Widgets, etc. In general, this code is 100% declarative in nature (as opposed to procedural or functional). That is to say - it has no logic. It is just a definition of what exists - but it has no instructions, algorithms, or business logic.

    • In a future version of QQQ, we anticipate being able to define meta-data in a format such as YAML or JSON, or to even load it from a relational or document database. This speaks to the fact that this "code" is not executable code - but rather is a simple declaration of (meta) data.

    • A key function of QQQ then is to drive all of its layers of functionality - frontend UIs, middleware, and core backend actions (e.g., ORM operations) - based on this meta-data.

      • For example:

        1. You define the meta-data for a table in your application - including its fields and their data types, as well as what backend system the table exists within.

        2. Then, the QQQ Frontend Material Dashboard UI’s Query Screen loads that table’s meta-data, and uses it to control the screen that is presented. Including:

          • The data grid shown on the screen will have columns for each field in the table.

          • The Filter button in the Query Screen will present a menu listing all fields from the table for the user to build ad-hoc queries against the table. The data-types specified for the fields (in the meta-data) dictate what operators QQQ allows the user to use against fields (e.g., Strings offer "contains" vs Numbers offer "greater than").

          • Values for records from the table will be formatted for presentation based on the meta-data (such as a numeric field being shown with commas if it represents a quantity, or formatted as currency).

    • Other kinds of information that you tell QQQ about in the form of meta-data objects includes:

      • Details about the database you are using, and how to connect to it.

      • A database table’s name, fields, their types, its keys, and basic business rules (required fields, read-only fields, field lengths).

      • The specification of a custom workflow (process), including what screens are needed, with input & output values, and references to the custom application code for processing the data.

      • Details about a chart that summarizes data from a table for presentation as a dashboard widget.

      • The description of web API - its URL and authentication mechanism.

      • A table/path within a web API, and the fields returned in the JSON at that endpoint.

  2. Application code - to customize beyond what the QQQ framework does out-of-the box, and to provide application-specific business-logic. QQQ provides its programmers the same classes that it internally uses for record access, resulting in a unified application model. For example:

Lifecycle?

  • define meta data

    • enrichment

    • validation

  • start application

  • for dev - hotSwap

Meta Data Production

The first thing that an application built using QQQ needs to do is to define its meta data. This basically means the construction of a QInstance object, which is populated with the meta data objects defining the backend(s), tables, processes, possible-value sources, joins, authentication provider, etc, that make your application.

There are various styles that can be used for how you define your meta data, and for the most part they can be mixed and matched. They will be presented here based on the historical evolution of how they were added to QQQ, where we generally believe that better techniques have been added over time. So, you may wish to skip the earlier techniques, and jump straight to the end of this section. However, it can always be instructive to learn about the past, so, read at your own pace.

Omni-Provider

At the most basic level, the way to populate a QInstance is the simple and direct approach of creating one big class, possibly with just one big method, and just doing all the work directly inline there.

This may (clearly) violate several good engineering principles. However, it does have the benefit of being simple - if all of your meta-data is defined in one place, it can be pretty simple to find where that place is. So - especially in a small project, this technique may be worth continuing to consider.

Re: "doing all the work" as mentioned above - what work are we talking about? At a minimum, we need to construct the following meta-data objects, and pass them into our QInstance:

  • QAuthenticationMetaData - how (or if!) users will be authenticated to the application.

  • QBackendMeataData - a backend data store.

  • QTableMetaData - a table (within the backend).

  • QAppMetaData - an organizational unit to present the other elements in a UI.

Here’s what a single-method omni-provider could look like:

About the simplest possible single-file meta-data provider
public QInstance defineQInstance()
{
    QInstance qInstance = new QInstance();

    qInstance.setAuthentication(new QAuthenticationMetaData()
       .withName("anonymous")
       .withType(QAuthenticationType.FULLY_ANONYMOUS));

    qInstance.addBackend(new QBackendMetaData()
       .withBackendType(MemoryBackendModule.class)
       .withName("memoryBackend"));

    qInstance.addTable(new QTableMetaData()
        .withName("myTable")
        .withPrimaryKeyField("id")
        .withBackendName("memoryBackend")
        .withField(new QFieldMetaData("id", QFieldType.INTEGER)));

    qInstance.addApp(new QAppMetaData()
       .withName("myApp")
       .withSectionOfChildren(new QAppSection().withName("mySection"),
          qInstance.getTable("myTable")))

   return (qInstance);
}

Multi-method Omni-Provider

The next evolution of meta-data production comes by just applying some basic better-engineering principles, and splitting up from a single method that constructs all the things, to at least using unique methods to construct each thing, then calling those methods to add their results to the QInstance.

Multi-method omni- meta-data provider
public QInstance defineQInstance()
{
    QInstance qInstance = new QInstance();
    qInstance.setAuthentication(defineAuthenticationMetaData());
    qInstance.addBackend(defineBackendMetaData());
    qInstance.addTable(defineMyTableMetaData());
    qInstance.addApp(defineMyAppMetaData(qInstance));
    return qInstance;
}

public QAuthenticationMetaData defineAuthenticationMetaData()
{
    return new QAuthenticationMetaData()
       .withName("anonymous")
       .withType(QAuthenticationType.FULLY_ANONYMOUS);
}

public QBackendMetaData defineBackendMetaData()
{
    return new QBackendMetaData()
       .withBackendType(MemoryBackendModule.class)
       .withName("memoryBackend");
}

// implementations of defineMyTableMetaData() and defineMyAppMetaData(qInstance)
// left as an exercise for the reader

Multi-class Providers

Then the next logical evolution would be to put each of these single meta-data producing objects into its own class, along with calls to those classes. This gets us away from the "5000 line" single-class, and lets us stop using the word "omni":

Multi-class meta-data providers
public QInstance defineQInstance()
{
    QInstance qInstance = new QInstance();
    qInstance.setAuthentication(new AuthMetaDataProvider().defineAuthenticationMetaData());
    qInstance.addBackend(new BackendMetaDataProvider().defineBackendMetaData());
    qInstance.addTable(new MyTableMetaDataProvider().defineTableMetaData());
    qInstance.addApp(new MyAppMetaDataProvider().defineAppMetaData(qInstance));
    return qInstance;
}

public class AuthMetaDataProvider
{
    public QAuthenticationMetaData defineAuthenticationMetaData()
    {
        return new QAuthenticationMetaData()
           .withName("anonymous")
           .withType(QAuthenticationType.FULLY_ANONYMOUS);
    }
}

public class BackendMetaDataProvider
{
    public QBackendMetaData defineBackendMetaData()
    {
        return new QBackendMetaData()
           .withBackendType(MemoryBackendModule.class)
           .withName("memoryBackend");
    }
}

// implementations of MyTableMetaDataProvider and MyAppMetaDataProvider
// left as an exercise for the reader

MetaDataProducerInterface

As the size of your application grows, if you’re doing per-object meta-data providers, you may find it burdensome, when adding a new object to your instance, to have to write code for it in two places - that is - a new class to produce that meta-data object, AND a single line of code to add that object to your QInstance. As such, a mechanism exists to let you avoid that line-of-code for adding the object to the QInstance.

This mechanism involves adding the MetaDataProducerInterface to all of your classes that produce a meta-data object. This interface is generic, with a type parameter that will typically be the type of meta-data object you are producing, such as QTableMetaData, QProcessMetaData, or QWidgetMetaData, (technically, any class which implements TopLevelMetaData). Implementers of the interface are then required to override just one method: T produce(QInstance qInstance) throws QException;

Once you have your MetaDataProducerInterface classes defined, then there’s a one-time call needed to add all of the objects produced by these classes to your QInstance - as shown here:

Using MetaDataProducerInterface
public QInstance defineQInstance()
{
    QInstance qInstance = new QInstance();
    MetaDataProducerHelper.processAllMetaDataProducersInPackage(qInstance,
      "com.mydomain.myapplication");
    return qInstance;
}

public class AuthMetaDataProducer implements MetaDataProducerInterface<QAuthenticationMetaData>
{
    @Override
    public QAuthenticationMetaData produce(QInstance qInstance)
    {
        return new QAuthenticationMetaData()
           .withName("anonymous")
           .withType(QAuthenticationType.FULLY_ANONYMOUS);
    }
}

public class BackendMetaDataProducer implements MetaDataProducerInterface<QBackendMetaData>
{
    @Override
    public QBackendMetaData defineBackendMetaData()
    {
        return new QBackendMetaData()
           .withBackendType(MemoryBackendModule.class)
           .withName("memoryBackend");
    }
}

// implementations of MyTableMetaDataProvider and MyAppMetaDataProvider
// left as an exercise for the reader

MetaDataProducerMultiOutput

It is worth mentioning, that sometimes it might feel like a bridge that’s a bit too far, to make every single one of your meta-data objects require its own class. Some may argue that it’s best to do it that way - single responsibility principle, etc. But, if you’re producing, say, 5 widgets that are all related, and it’s only a handful of lines of code for each one, maybe you’d rather produce them all in the same class. Or maybe when you define a table, you’d like to define its joins and widgets at the same time.

This approach can be accomplished by making the type argument for your MetaDataProducerInterface be MetaDataProducerMultiOutput - a simple class that just wraps a list of other MetaDataProducerOutput objects.

Returning a MetaDataProducerMultiOutput
public class MyMultiProducer implements MetaDataProducerInterface<MetaDataProducerMultiOutput>
{
    @Override
    public MetaDataProducerMultiOutput produce(QInstance qInstance)
    {
       MetaDataProducerMultiOutput output = new MetaDataProducerMultiOutput();

       output.add(new QPossibleValueSource()...);
       output.add(new QJoinMetaData()...);
       output.add(new QJoinMetaData()...);
       output.add(new QWidgetMetaData()...);
       output.add(new QTableMetaData()...);

       return (output);
    }
}

Aside: TableMetaData with RecordEntities

At this point, let’s take a brief aside to dig deeper into the creation of a QTableMeta object. Tables, being probably the most important meta-data type in QQQ, have a lot of information that can be specified in their meta-data object.

At the same time, if you’re writing any custom code in your QQQ application (e.g., any processes or table customizers), where you’re working with records from tables, you may prefer being able to work with entity beans (e.g., java classes with typed getter & setter methods), rather than the default object type that QQQ’s ORM actions return, the QRecord, which carries all of its values in a Map (where you don’t get compile-time checks of field names or data types). QQQ has a mechanism for dealing with this - in the form of the QRecordEntity class.

So - if you want to build your application using entity beans (which is recommended, for the compile-time safety that they provide in custom code), you will be writing a QRecordEntity class for each of your tables, which will look like:

QRecordEntity example
public class MyTable extends QRecordEntity
{
   public static final String TABLE_NAME = "myTable";

   @QField(isEditable = false, isPrimaryKey = true)
   private Integer id;

   @QField()
   private String name;

   // no-arg constructor and constructor that takes a QRecord
   // getters & setters (and optional fluent setters)
}

The point of introducing this topic here and now is, that a QRecordEntity can be used to shortcut to defining some of the attributes in a QTableMetaData object. Specifically, in a MetaDataProducer<QTableMetaData> you may say:

QTableMetaDataProducer using a QRecordEntity
public QTableMetaData produce(QInstance qInstance) throws QExcpetion
{
  return new QTableMetaData()
     .withName(MyTable.TABLE_NAME)
     .withFieldsFromEntity(MyTable.class)
     .withBackendName("memoryBackend");
}

That withFieldsFromEntity call is one of the biggest benefits of this technique. It allows you to avoid defining all of the fields in you table in two places (the entity and the table meta-data).

MetaData Producing Annotations for Entities

If you are using QRecordEntity classes that correspond to your tables, then you can take advantage of some additional annotations on those classes, to produce more related meta-data objects associated with those tables. The point of this is to eliminate boilerplate, and simplify / speed up the process of getting a new table built and deployed in your application.

Furthermore, the case can be made that it is beneficial to keep the meta-data definition for a table as close as possible to the entity that corresponds to the table. This enables modifications to the table (e.g., adding a new field/column) to only require edits in one java source file, rather than necessarily requiring edits in two files.

@QMetaDataProducingEntity

This is an annotation meant to be placed on a QRecordEntity subclass, which you would like to be processed by an invocation of MetaDataProducerHelper, to automatically produce some meta-data objects.

This annotation supports:

  • Creating table meta-data for the corresponding record entity table. Enabled by setting produceTableMetaData=true.

    • One may customize the table meta data that is produced automatically by supplying a class that extends MetaDataCustomizerInterface in the annotation attribute tableMetaDataCustomizer.

    • In addition to (or as an alternative to) the per-table MetaDataCustomizerInterface that can be specified in @QMetaDataProducingEntity.tableMetaDataCustomzier, when an application calls MetaDataProducerHelper.processAllMetaDataProducersInPackage, an additional MetaDataCustomizerInterface can be given, to apply a common set of adjustments to all tales being generated by the call.

  • Making a possible-value-source out of the table. Enabled by setting producePossibleValueSource=true.

  • Processing child tables to create joins and childRecordList widgets

@ChildTable

This is an annotation used as a value that goes inside a @QMetadataProducingEntity annotation, to define child-tables, e.g., for producing joins and childRecordList widgets related to the table defined in the entity class.

@ChildJoin

This is an annotation used as a value inside a @ChildTable inside a @QMetadataProducingEntity annotation, to control the generation of a QJoinMetaData, as a ONE_TO_MANY type join from the table represented by the annotated entity, to the table referenced in the @ChildTable annotation.

@ChildRecordListWidget

This is an annotation used as a value that goes inside a @QMetadataProducingEntity annotation, to control the generation of a QWidgetMetaData - for a ChildRecordList widget.

QRecordEntity with meta-data producing annotations and a table MetaDataCustomizer
@QMetaDataProducingEntity(
   produceTableMetaData = true,
   tableMetaDataCustomizer = MyTable.TableMetaDataCustomizer.class,
   producePossibleValueSource = true,
   childTables = {
      @ChildTable(
         childTableEntityClass = MyChildTable.class,
         childJoin = @ChildJoin(enabled = true),
         childRecordListWidget = @ChildRecordListWidget(enabled = true, label = "Children"))
   }
)
public class MyTable extends QRecordEntity
{
   public static final String TABLE_NAME = "myTable";

   public static class TableMetaDataCustomizer implements MetaDataCustomizerInterface<QTableMetaData>
   {
      @Override
      public QTableMetaData customizeMetaData(QInstance qInstance, QTableMetaData table) throws QException
      {
         String childJoinName = QJoinMetaData.makeInferredJoinName(TABLE_NAME, MyChildTable.TABLE_NAME);

         table
            .withUniqueKey(new UniqueKey("name"))
            .withIcon(new QIcon().withName("table_bar"))
            .withRecordLabelFormat("%s")
            .withRecordLabelFields("name")

            .withSection(new QFieldSection("identity", new QIcon().withName("badge"), Tier.T1,
               List.of("id", "name")))
            // todo additional sections for other fields
            .withSection(new QFieldSection("children", new QIcon().withName("account_tree"), Tier.T2)
               .withWidgetName(childJoinName))

            .withExposedJoin(new ExposedJoin()
               .withLabel("Children")
               .withJoinPath(List.of(childJoinName))
               .withJoinTable(MyChildTable.TABLE_NAME));

         return (table);
      }
   }

   @QField(isEditable = false, isPrimaryKey = true)
   private Integer id;

   // remaining fields, constructors, getters & setters left as an exercise for the reader and/or the IDE
}

The class given in the example above, if processed by the MetaDataProducerHelper, would add the following meta-data objects to your QInstance:

  • A QTableMetaData named myTable, with all fields annotated as @QField from the QRecordEntity class, and with additional attributes as set in the TableMetaDataCustomizer inner class.

  • A QPossibleValueSource named myTable, of type TABLE, with myTable as its backing table.

  • A QJoinMetaData named myTableJoinMyChildTable, as a ONE_TO_MANY type, between those two tables.

  • A QWidgetMetaData named myTableJoinMyChildTable, as a CHILD_RECORD_LIST type, that will show a list of records from myChildTable as a widget, when viewing a record from myTable.

Other MetaData Producing Annotations

Similar to these annotations for a RecordEntity, a similar one exists for a PossibleValueEnum class, to automatically write the meta-data to use that enum as a possible value source in your application:

@QMetaDataProducingPossibleValueEnum

This is an annotation to go on a PossibleValueEnum class, which you would like to be processed by MetaDataProducerHelper, to automatically produce a PossibleValueSource meta-data based on the enum.

PossibleValueEnum with meta-data producing annotation
@QMetaDataProducingPossibleValueEnum(producePossibleValueSource = true)
public enum MyOptionsEnum implements PossibleValueEnum<Integer>
{
   // values and methods left as exercise for reader
}

The enum given in the example above, if processed by the MetaDataProducerHelper, would add the following meta-data object to your QInstance:

  • A QPossibleValueSource named MyOptionsEnum, of type ENUM, with MyOptionsEnum as its backing enum.

Meta Data Types

QInstance

An application in QQQ is defined as a set of Meta Data objects. These objects are all stored together in an object called a QInstance.

Currently, a QInstance must be programmatically constructed in java code - e.g., by constructing objects which get added to the QInstance, for example:

Adding meta-data for two tables and one process to a QInstance
QInstance qInstance = new QInstance();
qInstance.addTable(definePersonTable());
qInstance.addTable(defineHomeTable());
qInstance.addProcess(defineSendPersonHomeProcess());

It is on the QQQ roadmap to allow meta-data to be defined in a non-programmatic way, for example, in YAML or JSON files, or even from a dynamic data source (e.g. a document or relational database).

The middleware and/or frontends being used in an application will drive how the QInstance is connected to the running server/process. For example, using the qqq-middleware-javalin module, a the QJavalinImplementation class () has a constructor which takes a QInstance as an argument:

Starting a QQQ Javalin middleware server - passing a QInstance as a parameter to the new QJavalinImplementation
QJavalinImplementation qJavalinImplementation = new QJavalinImplementation(qInstance);
Javalin service = Javalin.create();
service.routes(qJavalinImplementation.getRoutes());
service.start();

QInstance Setup:

These are the methods that one is most likely to use when setting up (defining) a QInstance object:

  • add(TopLevelMetaDataInterface metaData) - Generic method that takes most of the meta-data subtypes that can be added to an instance, such as QBackendMetaData, QTableMetaData, QProcessMetaData, etc. There are also type-specific methods (e.g., addTable, addProcess, etc), which one can call instead - this would just be a matter of personal preference.

QInstance Usage:

Generally you will set up a QInstance in your application’s startup flow, and then place it in the server (e.g., javalin). But, if, during application-code runtime, you need access to any of the meta-data in the instance, you access it via the QContext object’s static getInstance() method. This can be useful, for example, to get a list of the defined tables in the application, or fields in a table, or details about a field, etc.

It is generally considered risky and/or not a good idea at all to modify the QInstance after it has been validated and a server is running. Future versions of QQQ may in fact restrict modifications to the instance after validation.

Backends

A key component of QQQ is its ability to connect to various backend data stores, while providing the same interfaces to those backends - both User Interfaces, and Programming Interfaces.

For example, out-of-the-box, QQQ can connect to:

  • RDBMS (Relational Database Management Systems, such as MySQL)

  • File Systems (Amazon S3 or local disk)

  • JSON Web APIs (using a custom mapping class per-API backend).

  • In-Memory data stores

All QQQ Tables in a QQQ instance must belong to a backend. As such, any instance using tables (which would be almost all instances) must define 1 or more backends.

QBackendMetaData

Backends are defined in a QQQ Instance in a QBackendMetaData object. These objects will have some common attributes, but many different attributes based on the type of backend being used.

QBackendMetaData Properties:

  • name - String, Required - Unique name for the backend within the QQQ Instance.

  • backendType - String, Required - Identifier for the backend type being defined.

    • This attribute is typically set in the constructor of a QBackendMetaData subclass, and as such does not need to be set when defining a backend meta data object.

  • enabledCapabilities and disabledCapability - Sets, containing Capability enum values. Basic rules that apply to all tables in the backend, describing what actions (such as Delete, or Count) are supported in the backend.

    • By default, QQQ assumes that a backend supports most capabilities, with one exception being QUERY_STATS.

    • TODO fully explain rules here

  • usesVariants - Boolean, default false - Control whether or not the backend supports the concept of "Variants".

    • Supporting variants means that tables within the backend can exist in alternative "variants" of the backend. For example, this might mean a sharded or multi-tenant database backend (perhaps a different server or database name per-client). Or this might mean using more than one set of credentials for connecting to an API backend - each of those credential sets would be a "variant".

    • A backend that uses variants requires additional properties to be set. TODO complete variant documentation

In a QQQ application, one will typically not create an instance of QBackendMetaData directly, but instead will create an instance of one of its subclasses, specific to the type of backend being used. The currently available list of such classes are:

RDBMSBackendMetaData

The meta data required for working with tables in an RDBMS (relational database) backend are defined in an instance of the RDBMSBackendMetaData class.

RDBMSBackendMetaData Properties:

  • vendor - String, Required - Database vendor. Currently supported values are: aurora, mysql, h2.

  • jdbcUrl - String, Optional - Full JDBC URL for connecting to the database.

    • If this property is provided, then following properties (which are the components of a JDBC URL) are ignored. In other words, you can either provide the jdbcUrl, or the individual components that make it up.

  • hostName - String, Conditionally Required - Hostname or ip address of the RDBMS server.

  • port - Integer, Conditionally Required - Port used to connect to the RDBMS server.

  • databaseName - String, Conditionally Required - Name of the database being connected to, within the RDBMS server.

  • username - String, Conditionally Required - Username for authenticating in the database server.

  • password - String, Conditionally Required - Password for authenticating in the database server.

Examples

/*******************************************************************************
** Full example of constructing an RDBMSBackendMetaData
*******************************************************************************/
public class ExampleDatabaseBackendMetaDataProducer extends MetaDataProducer<QBackendMetaData>
{
   public static final String NAME = "rdbmsBackend";

   /*******************************************************************************
    ** Produce the QBackendMetaData
    *******************************************************************************/
   @Override
   public QBackendMetaData produce(QInstance qInstance)
   {
      ///////////////////////////////////////////////////////////////////////
      // read settings from either a .env file or the system's environment //
      ///////////////////////////////////////////////////////////////////////
      QMetaDataVariableInterpreter interpreter = new QMetaDataVariableInterpreter();
      String vendor       = interpreter.interpret("${env.RDBMS_VENDOR}");
      String hostname     = interpreter.interpret("${env.RDBMS_HOSTNAME}");
      String port         = interpreter.interpret("${env.RDBMS_PORT}");
      String databaseName = interpreter.interpret("${env.RDBMS_DATABASE_NAME}");
      String username     = interpreter.interpret("${env.RDBMS_USERNAME}");
      String password     = interpreter.interpret("${env.RDBMS_PASSWORD}");

      return (new RDBMSBackendMetaData()
         .withName(NAME)
         .withVendor(vendor)
         .withHostName(hostname)
         .withPort(ValueUtils.getValueAsInteger(port))
         .withDatabaseName(databaseName)
         .withUsername(username)
         .withPassword(password)
         .withCapability(Capability.QUERY_STATS));
   }
}
S3BackendMetaData

The meta data required for working with tables in an Amazon S3 backend are defined in an instance of the S3BackendMetaData class.

S3BackendMetaData Properties:

  • bucketName - String, Required - Bucket name to connect to inside AWS S3.

  • accessKey - String, Required - Access key for connecting to S3 inside AWS S3.

  • secretKey - String, Required - Secret key for connecting to S3 inside AWS S3.

  • region - String, Required - AWS region containing the Bucket in S3.

  • basePath - String, Required - Base path to the files within the S3 bucket.

FilesystemBackendMetaData

The meta data required for working with tables in a (local) filesystem backend are defined in an instance of the FilesystemBackendMetaData class.

FilesystemBackendMetaData Properties:

  • basePath - String, Required - Base path to the backend’s files.

APIBackendMetaData

The meta data required for working with tables in a web API are defined in an instance of the APIBackendMetaData class.

QQQ provides a minimal, reasonable default implementation for working with web APIs, making assumptions such as using POST to insert records, and GET with a primary key in the URL to get a single record. However, in our experience, almost all APIs are implemented differently enough, that a layer of custom code is required. For example, query/search endpoints are almost always unique in how they take their search parameters, and how they wrap their output.

To deal with this, a QQQ API Backend can define a custom class in the actionUtil property of the backend meta-data, as a subclass of BaseAPIActionUtil, which ideally can override methods at the level where unique functionality is needed. For example, an application need not define the full method for executing a Query against an API backend (which would need to make the HTTP request (actually multiple, to deal with pagination)). Instead, one can just override the buildQueryStringForGet method, where the unique details of making the request are defined, and maybe the jsonObjectToRecord method, where records are mapped from the API’s response to a QRecord.

todo - full reference and examples for BaseAPIActionUtil

APIBackendMetaData Properties:

  • baseUrl - String, Required - Base URL for connecting to the API.

  • contentType - String, Required - value of Content-type header included in all requests to the API.

  • actionUtil - QCodeReference, Required - Reference to a class that extends BaseAPIActionUtil, where custom code for working with this API backend is defined.

  • customValues - Map of String → Serializable - Application-defined additional name=value pairs that can

  • authorizationType - Enum, Required - Specifies how authentication is provided for the API. The value here, dictates which other authentication-related properties are required. Possible values are:

    • API_KEY_HEADER - Uses the apiKey property in an HTTP header named API-Key. In the future, when needed, QQQ will add a property, tentatively named apiKeyHeaderName, to allow customization of this header name.

    • API_TOKEN - Uses the apiKey property in an Authroization header, as: "Token " + apiKey

    • BASIC_AUTH_API_KEY - Uses the apiKey property, Base64-encoded, in an Authroization header, as "Basic " + base64(apiKey)

    • BASIC_AUTH_USERNAME_PASSWORD - Combines the username and password properties, Base64-encoded, in an Authroization header, as "Basic " + base64(username + ":" + password)

    • OAUTH2 - Implements an OAuth2 client credentials token grant flow, using the properties clientId and clientSecret. By default, the id & secret are sent as query-string parameters to the API’s oauth/token endpoint. Alternatively, if the meta-data has a customValue of setCredentialsInHeader=true, then the id & secret are posted in an Authorization header (base-64 encoded, and concatenated with ":").

    • API_KEY_QUERY_PARAM - Uses the apiKey property as a query-string parameter, with its name taken from the apiKeyQueryParamName property.

    • CUSTOM - Has a no-op implementation at the QQQ layer. Assumes that an override of protected void handleCustomAuthorization(HttpRequestBase request) be implemented in the backend’s actionUtil class. This would be

  • apiKey - String, Conditional - See authorizationType above for details.

  • clientId - String, Conditional - See authorizationType above for details.

  • clientSecret - String, Conditional - See authorizationType above for details.

  • username - String, Conditional - See authorizationType above for details.

  • password - String, Conditional - See authorizationType above for details.

  • apiKeyQueryParamName - String, Conditional - See authorizationType above for details.

Apps

QQQ User Interfaces (e.g., Material Dashboard) generally organize their contents via Apps. Apps are a lightweight construct in QQQ - basically just containers for other objects.

Specifically, Apps can contain:

QAppMetaData

Apps are defined in a QQQ Instance in QAppMetaData objects. Apps must consist of either 1 or more QQQ Widgets, or 1 or more Sections, which are expected to contain 1 or more QQQ Tables, QQQ Processes, or QQQ Reports.

QAppMetaData Properties:

  • name - String, Required - Unique name for the app within the QQQ Instance.

  • label - String - User-facing label for the app, presented in User Interfaces. Inferred from name if not set.

  • permissionRules - QPermissionRules - Permissions to apply to the app. See Permission Rules for details.

  • children - List of QAppChildMetaData - Objects contained within the app. These can be QQQ Tables, QQQ Processes, QQQ Reports or other QQQ Apps.

    • See the example below for some common patterns for how these child-meta data objects are added to an App.

  • parentAppName - String - For an app which is a child of another app, the parent app’s name is referenced in this field.

    • Note that this is generally automatically set when the child is added to its parent, in the addChild method.

  • icon - QIcon - An icon to display in a UI for the app. See Icons.

  • widgets - List of String - A list of names of QQQ Widgets to include in the app.

  • sections - List of QAppSection - A list of QAppSection objects, to create organizational subdivisions within the app.

    • As shown in the example below, the method withSectionOfChildren can be used to fluently add a new QAppSection, along with its child objects, to both an app and a section all at once.

QAppSection

A QAppSection is an organizational subsection of a QQQ App.

  • name - String, Required - Unique name for the section within its app.

  • label - String - User-facing label for the section, presented in User Interfaces. Inferred from name if not set.

  • icon - QIcon - An icon to display in a UI for the section. See Icons.

  • tables - List of String - A list of names of QQQ Tables to include in the section.

  • processes - List of String - A list of names of QQQ Processes to include in the section.

  • reports - List of String - A list of names of QQQ Reports to include in the section.

Examples

/*******************************************************************************
** Full example of constructing a QAppMetaData object.
*******************************************************************************/
public class ExampleAppMetaDataProducer extends MetaDataProducer<QAppMetaData>
{

   /*******************************************************************************
    ** Produce the QAppMetaData
    *******************************************************************************/
   @Override
   public QAppMetaData produce(QInstance qInstance) throws QException
   {
      return (new QAppMetaData()
         .withName("sample")
         .withLabel("My Sample App")
         .withIcon(new QIcon().withName("thumb_up"))
         .withWidgets(List.of(
            UserWelcomeWidget.NAME,
            SystemHealthLineChartWidget.NAME))
         .withSectionOfChildren(new QAppSection().withName("peoplePlacesAndThings"),
            qInstance.getTable(People.TABLE_NAME),
            qInstance.getTable(Places.TABLE_NAME),
            qInstance.getTable(Things.TABLE_NAME),
            qInstance.getProcess(AssociatePeopleWithPlacesProcess.NAME))
         .withSectionOfChildren(new QAppSection().withName("math").withLabel("Mathematics"),
            qInstance.getProcess(ComputePiProcess.NAME),
            qInstance.getReport(PrimeNumbersReport.NAME),
            qInstance.getReport(PolygonReport.NAME)));
   }


   /*******************************************************************************
    ** Since this meta-data producer expects to find other meta-data objects in the
    ** QInstance, give it a sortOrder higher than the default (which we'll expect
    ** the other objects used).
    *******************************************************************************/
   @Override
   public int getSortOrder()
   {
      return (Integer.MAX_VALUE);
   }

}

Tables

One of the most common types of object in a QQQ Instance is the Table. In the most common use-case, a QQQ Table may be the in-app representation of a Database table. That is, it is a collection of records (or rows) of data, each of which has a set of fields (or columns).

QQQ also allows other types of data sources (QQQ Backends) to be used as tables, such as File systems, API’s, etc. All of these backend types present the same interfaces (both user-interfaces, and application programming interfaces), regardless of their backend type.

QTableMetaData

Tables are defined in a QQQ Instance in QTableMetaData objects. All tables must reference a QQQ Backend, a list of fields that define the shape of records in the table, and additional data to describe how to work with the table within its backend.

QTableMetaData Properties:

  • name - String, Required - Unique name for the table within the QQQ Instance.

  • label - String - User-facing label for the table, presented in User Interfaces. Inferred from name if not set.

  • backendName - String, Required - Name of a QQQ Backend in which this table’s data is managed.

  • fields - Map of String → QQQ Field, Required - The columns of data that make up all records in this table.

  • primaryKeyField - String, Conditional - Name of a QQQ Field that serves as the primary key (unique identifier) for records in this table.

    • Whether a primary key field is required or not depends on the backend type that the table belongs to.

  • uniqueKeys - List of UniqueKey - Definition of additional unique keys or constraints (from an RDBMS point of view) from the table. e.g., sets of columns which must have unique values for each record in the table. The properties of the UniqueKey object are:

    • fieldNames - List of String, Required - List of field names from this table.

    • label - String - Optional label to be shown to users with error messages (e.g., for violation of this unique key).

  • backendDetails - QTableBackendDetails or subclass - Additional data to configure the table within its QQQ Backend.

    • For example, for an RDBMS-type backend, the name of the table within the database.

    • vs. a FileSystem backend, this may be the sub-path where files for the table are stored.

    • todo - details on these

  • automationDetails - QTableAutomationDetails - Configuration of automated jobs that run against records in the table, e.g., upon insert or update.

  • customizers - Map of String → QCodeReference - References to custom code that are injected into standard table actions, that allow applications to customize certain parts of how the table works.

    • Allowed values for keys in this map come from the role property of the TableCustomizers enum.

    • Based on the key in this map, the QCodeReference used as the value must be of the appropriate java type, as specified in the expectedType property of the TableCustomizers enum value corresponding to the key.

    • Example:

// in defining a QTableMetaData, a customizer can be added as:
.withCustomizer(TableCustomizers.PRE_INSERT_RECORD, new QCodeReference(MyPreInsCustomizer.class))

// where MyPreInsCustomizer would be defined as:
public class MyPreInsCustomizer extends AbstractPreInsertCustomizer

  • isHidden - Boolean, default false - Option to hide the table from all User Interfaces.

  • parentAppName - String - Name of a QQQ App that this table exists within.

    • This field generally does not need to be set on the table when it is defined, but rather, is set when the table gets placed within an app.

  • icon - QIcon - Icon associated with this table in certain user interfaces. See Icons.

  • recordLabelFormat - String - Java Format String, used with recordLabelFields to produce a label shown for records from the table.

  • recordLabelFields - List of String, Conditional - Used with recordLabelFormat to provide values for any format specifiers in the format string. These strings must be field names within the table.

    • Example of using recordLabelFormat and recordLabelFields:

// given these fields in the table:
new QFieldMetaData("name", QFieldType.STRING)
new QFieldMetaData("birthDate", QFieldType.DATE)

// We can produce a record label such as "Darin Kelkhoff (1980-05-31)" via:
.withRecordLabelFormat("%s (%s)")
.withRecordLabelFields(List.of("name", "birthDate"))

  • sections - List of QFieldSection - Mechanism to organize fields within user interfaces, into logical sections. If any sections are present in the table meta data, then all fields in the table must be listed in exactly 1 section. If no sections are defined, then instance enrichment will define default sections.

  • associatedScripts - List of AssociatedScript - Definition of user-defined scripts that can be associated with records within the table.

  • enabledCapabilities and disabledCapabilities - Set of Capability enum values - Overrides from the backend level, for capabilities that this table does or does not possess.

  • associations - List of Association - tables whose records can be managed along with records from this table. See below for details.

  • recordSecurityLocks - List of RecordSecurityLock - locks that apply to records in the table - e.g., to control what users can or cannot access records in the table. See RecordSecurityLock below for details.

  • permissionRules - QPermissionRules object - define the permission/access rules for the table. See Permission Rules for details.

  • auditRules - QAuditRules object - define the audit rules for the table. See QAuditRules below for details.

  • cacheOf - CacheOf object - specify that this table serves as a "cache of" another table. See CacheOf object below for details.

  • exposedJoins - List of ExposedJoin objects - optional list of joined tables that are to be exposed in User Interfaces. See ExposedJoin object below for details.

todo: supplementalMetaData (API)

QFieldSection

When users view records from a QQQ Table in a UI, fields are organized on the screen based on the QFieldSection objects in the table’s meta-data.

QFieldSection Properties:

  • name - String, Required - unique identifier for the section within its table.

  • label - String - User-facing label for the section, presented in User Interfaces. Inferred from name if not set.

  • tier - enum - importance of the fields in section for the table. Different tiers may be presented differently in UI’s. Only a single T1 section is allowed per-table. Possible values are: T1, T2, and T3.

  • icon - QIcon - Icon associated with this section in certain user interfaces. See Icons.

  • isHidden - Boolean, default false - Option to hide the table from all User Interfaces.

  • gridColumns - Integer - Option to specify how many columns in a grid layout the section should use. For the Material-Dashboard frontend, this is a grid of 12.

  • fieldNames - List of String, Conditional - List of names of QQQ Fields from this table to be included in this section.

  • widgetName - String, Conditional - Name of a QQQ Widget to be displayed in this section.

    • Note that exactly one of fieldNames or widgetName must be used.

QTableAutomationDetails

Records in QQQ can have application-defined custom actions automatically asynchronously executed against them after they are inserted or updated. The configuration to enable this functionality is assigned to a table in a QTableAutomationDetails object.

QTableAutomationDetails Properties:

  • statusTracking - AutomationStatusTracking object, Required - define how QQQ should keep track, per record, its status (e.g., pending-insert-automations, running-update-automations, etc). Properties of AutomationStatusTracking object are:

    • type - enum, Required - what type of tracking is used for the table. Possible values are:

      • FIELD_IN_TABLE - specifies that the table has a field which stores an AutomationStatus id.

      • Additional types may be defined in the future, such as ONE_TO_ONE_TABLE or SHARED_TABLE.

    • fieldName - String, Conditional - for type=FIELD_IN_TABLE, this property specifies the name of the QQQ Field in the table that stores the AutomationStatus id.

  • providerName - String, Required - name of an Automation Provider within the QQQ Instance, which is responsible for running the automations on this table.

  • overrideBatchSize - Integer - optional control over how many records from the table are processed in a single batch/page. For tables with "slow" actions (e.g., one that may need to make an API call per-record), using a smaller batch size (say, 50) may be required to avoid timeout errors.

  • actions - List of TableAutomationAction - list of the actions to perform on new and updated records in the table. Properties are:

    • name - String, Required - unique identifier for the action within its table.

    • triggerEvent - enum, Required - indicate which event type (POST_INSERT, POST_UPDATE, or PRE_DELETE (which is not yet implemented)) the action applies to.

    • priority - Integer, default 500 - mechanism to control the order in which actions on a table are executed, if there are more than one. Actions with a smaller value for priority are executed first. Ties are broken in an undefined manner.

    • filter - QQueryFilter - optional filter that gets applied to records when they match the triggerEvent, to control which records have the action ran against them.

    • includeRecordAssociations - Boolean, default false - for tables that have associations, control whether or not a record’s associated records are loaded when records are fetched and passed into the action’s custom code.

    • values - Map of String → Serializable - optional application-defined map of name=value pairs that can be passed into the action’s custom code.

    • processName - String, Conditional - name of a QQQ Processes in the QQQ Instance which is executed as the custom-code of the action.

    • codeReference - QCodeReference, Conditional - reference to a class that extends RecordAutomationHandler, to be executed as the custom-code of the action.

      • Note, exactly one of processName or codeReference must be provided.

Association

An Association is a way to define a relationship between tables, that facilitates, for example, a parent record having a list of its child records included in it when it is returned from a Query. Similarly, associated records can automatically be inserted/updated/deleted if they are included in a parent record when it is stored.

Association Properties:

  • name - String, Required - unique name for the association within this table. Used as the key in the associatedRecords map within QRecord objects for this table.

  • associatedTableName - String, Required - name of a QQQ Table, which is the associated table.

  • joinName - String, Required - name of a QQQ Join in the instance, which defines how the tables are joined.

RecordSecurityLock

A RecordSecurityLock is the mechanism through which users can be allowed or denied access to read and/or write records, based on values in the record, and values in the user’s session. Record security locks must correspond to a Security Key Type.

For example:

  • An instance may have a security key type called clientId.

  • Users may have 1 or more clientId values in their Session, or, they may have an "All Clients" key in their session (e.g., for internal/admin users).

  • For some tables, it may be required to limit visibility to records based on a user’s clientId key. To do this. a RecordSecurityLock would be applied to the table, specifying the clientId field corresponds to the clientId security key.

  • With these settings in place, QQQ will prevent users from viewing records from this table that do not have a matching key, and will similarly prevent users from writing records with an invalid key value.

    • For example, in an RDBMS backend, all SELECT statements generated against such a table will have an implicit filter, such as AND client_id = ? based on the user’s security key values.

RecordSecurityLock Properties:

  • securityKeyType - String, Required - name of a Security Key Type in the Instance.

  • fieldName - String, Required - name of a QQQ Field in this table (or a joined table, if joinNameChain is set), where the value for the lock is stored.

  • joinNameChain - List of String - if the lock value is not stored in this table, but rather comes from a joined table, then this property defines the path of joins from this table to the table with the lock field.

  • nullValueBehavior - enum, default: DENY - control how records with a null value in the lock field should behave. Possible values are:

    • DENY - deny all users access to a record with a null value in the lock field (unless the user has an all-access key - see Security Key Type)

    • ALLOW - allow all users access to a record with a null value in the lock field.

    • ALLOW_WRITE_ONLY - allow all users to write records with null in the lock field, but deny reads on records with null in the lock field (also excepted by all-access keys).

  • lockScope - enum, default: READ_AND_WRITE - control what types of operations the lock applies to. Possible values are:

    • READ_AND_WRITE - control both reading and writing records based on the user having an appropriate security key.

    • WRITE - allow all users to read the record, but limit writes to users with an appropriate security key.

QAuditRules

The audit rules on a table define the level of detail that is automatically stored in the audit table (if any) for DML actions (Insert, Update, Delete).

QAuditRules Properties:

  • auditLevel - enum, Required - level of details that are audited. Possible values are:

    • NONE - no automatic audits are stored for the table.

    • RECORD - only record-level audits are stored for the table (e.g., a message such as "record was edited", but without field-level details)

    • FIELD - full field-level audits are stored (e.g., including all old & new values as audit details).

CacheOf

One QQQ Table can be defined as a "cache of" another QQQ Table by assigning a CacheOf object to the table which will function as the cache. Note, at this time, only limited use-cases are supported.

CacheOf Properties:

  • sourceTable - String, Required - name of the other QQQ Table that is the source of data in this cache.

  • expirationSeconds - Integer - optional number of seconds that a cached record is allowed to exist before it is considered expired, and must be re-fetched from the source table.

  • cachedDateFieldName - String, Conditional - used with expirationSeconds to define the field in this table that is used for storing the timestamp for when the record was cached.

  • useCases - List of CacheUseCase - what caching use-cases are to be implemented.

Properties of CacheUseCase are:

  • type - Enum, Required - the type of use-case. Possible values are:

    • PRIMARY_KEY_TO_PRIMARY_KEY - the primary key in the cache table equals the primary key in the source table.

    • UNIQUE_KEY_TO_PRIMARY_KEY - a unique key in the cache table equals the primary key in the source table.

    • UNIQUE_KEY_TO_UNIQUE_KEY - a unique key in the cache table equals a unique key in the source table.

  • cacheSourceMisses - Boolean, default false - whether or not, if a "miss" happens in the source, if that fact gets cached.

  • cacheUniqueKey - UniqueKey, conditional - define the fields in the cache table that define the unique key being used as the cache key.

  • sourceUniqueKey - UniqueKey, conditional - define the fields in the source table that define the unique key being used as the cache key.

  • doCopySourcePrimaryKeyToCache - Boolean, default false - specify whether or not the value of the primary key in the source table should be copied into records built in the cache table.

  • excludeRecordsMatching - List of QQueryFilter - optional filter to be applied to records before they are cached. If a record matches the filter, then it will not be cached.

ExposedJoin

Query screens in QQQ applications can potentially allow users to both display fields from joined tables, and filter by fields from joined tables, for any QQQ Join explicitly defined as an Exposed Join.

The reasoning why not all joins are implicitly exposed is that in many applications, the full join-graph can sometimes be overwhelming, surprisingly broad, and not necessarily practically useful. This could be subject to change in the future, e.g., given a UI that allowed users to more explicitly add additional join tables…​

ExposedJoin Properties:

  • label - String, Required - how the joined table should be presented in the UI.

  • joinTable - String, Required - name of the QQQ Table that is joined to this table, and is being exposed as a join in the UI.

  • joinPath - List of String, Required - names of 1 or more QQQ Joins that describe how to get from this table to the join table.

AssociatedScript

A QQQ Table can have end-user defined Script records associated with individual records in the table by use of the associatedScripts property of the table’s meta-data.

The "types" of these scripts (e.g., how they are used in an application) are wholly application-designed & managed. QQQ provides the mechanism for UI’s to present and manage such scripts (e.g., the Developer Mode screen in the Material Dashboard), as well as an interface to load & execute such scripts RunAssociatedScriptAction).

AssociatedScript Properties:

  • fieldName - String, Required - name of a QQQ Field in the table which stores the id of the associated script record.

  • scriptTypeId - Serializable (typically Integer), Required - primary key value from the "scriptType" table in the instance, to designate the type of the Script.

  • scriptTester - QCodeReference - reference to a class which implements TestScriptActionInterface, that can be used by UI’s for running an associated script to test it.

Supplemental Meta Data

QQQ Frontend Material Dashboard

When running a QQQ application with the QQQ Frontend Material Dashboard module (QFMD), there are various pieces of supplemental meta-data which can be assigned to a Table, to modify some behaviors for the table in this UI.

Default Quick Filter Field Names

QFMD’s table query has a "Basic" mode, which will always display a subset of the table’s fields as quick-access filters. By default, the "Tier 1" fields on a table (e.g., fields in a Section that is marked as T1) will be used for this purpose.

However, you can customize which fields are shown as the default quick-filter fields, by providing a list of field names in a MaterialDashboardTableMetaData object, placed in the table’s supplementalMetaData.

table.withSupplementalMetaData(new MaterialDashboardTableMetaData()
    .withDefaultQuickFilterFieldNames(List.of("id", "warehouseId", "statusId", "orderDate")));
Go To Field Names

QFMD has a feature where a table’s query screen can include a "Go To" button, which a user can hit to open a modal popup, into which the user can enter a record’s identifier, to be brought directly to the record matching that identifier.

To use this feature, the table must have a List of GotoFieldNames set in its MaterialDashboardTableMetaData object in the table’s supplementalMetaData.

Each entry in this list is actually a list of fields, e.g., to account for a multi-value unique-key.

table.withSupplementalMetaData(new MaterialDashboardTableMetaData()
    .withGotoFieldNames(List.of(
        List.of("id"),
        List.of("partnerName", "partnerOrderId"))));

Processes

Besides QQQ Tables, the other most common type of object in a QQQ Instance is the Process. Processes are "custom" actions (e.g., defined by the application developers, rather than QQQ) that users and/or the system can execute against records. Processes generally are made up of two types of sub-objects:

  • Screens - i.e., User Interfaces (e.g., for gathering input and/or showing output).

  • BackendSteps - Java classes (of type BackendStep) that execute the logic of the process.

QProcessMetaData

Processes are defined in a QQQ Instance in a QProcessMetaData object. In addition to directly building a QProcessMetaData object setting its properties directly, there are a few common process patterns that provide Builder objects for ease-of-use. See StreamedETLWithFrontendProcess below for a common example

QProcessMetaData Properties:

  • name - String, Required - Unique name for the process within the QQQ Instance.

  • label - String - User-facing label for the process, presented in User Interfaces. Inferred from name if not set.

  • icon - QIcon - Icon associated with this process in certain user interfaces. See Icons.

  • tableName - String - Name of a QQQ Table that the process is associated with in User Interfaces (e.g., Action menu).

  • isHidden - Boolean, default false - Option to hide the process from all User Interfaces.

  • basepullConfiguration - BasepullConfiguration - config for the common "Basepull" pattern, of identifying records with a timestamp greater than the last time the process was ran. See below for details.

  • permissionRules - QPermissionRules object - define the permission/access rules for the process. See Permission Rules for details.

  • steps and stepList - Map of String → QStepMetaData and List of QStepMetaData - Defines the screens and backend code that makes up the process.

    • stepList is the list of steps in the order that they will be executed (that is to say - this is the default order of execution - but it can be customized - see How to customize a Linear process flow for details).

    • steps is a map, including all steps from stepList, but which may also include steps which can used by the process if its backend steps make the decision to do so, at run-time (e.g., using How to customize a Linear process flow).

    • A process’s steps are normally defined in one of two was:

      • 1) by a single call to .withStepList(List<QStepMetaData>), which internally adds each step into the steps map.

      • 2) by multiple calls to .addStep(QStepMetaData), which adds a step to both the stepList and steps map.

    • If a process also needs optional steps (for a How to customize a Linear process flow), they should be added by a call to .addOptionalStep(QStepMetaData), which only places them in the steps map.

  • stepFlow - enum, default LINEAR - specifies the the flow-control logic between steps. Possible values are:

    • LINEAR - steps are executed in-order, through the stepList. A backend step can customize the nextStepName or re-order the stepList, if needed. In a frontend step, a user may be given the option to go back to a previous step as well.

    • STATE_MACHINE - steps are executed as a Fine State Machine, starting with the first step in stepList, but then proceeding based on the nextStepName specified by the previous step. Thus allowing much more flexible flows.

  • schedule - QScheduleMetaData - set up the process to run automatically on the specified schedule. See below for details.

  • minInputRecords - Integer - not used…​

  • maxInputRecords - Integer - not used…​

todo: supplementalMetaData (API)

QStepMetaData

This is the base class for the two types of steps in a process - screens and backend code. There are some shared attributes of both of them, defined here.

QStepMetaData Properties:

  • name - String, Required - Unique name for the step within the process.

  • label - String - User-facing label for the step, presented in User Interfaces. Inferred from name if not set.

  • stepType - String - Deprecated.

QFrontendStepMetaData

For processes with a user-interface, they must define one or more "screens" in the form of QFrontendStepMetaData objects.

QFrontendStepMetaData Properties:

  • components - List of QFrontendComponentMetaData - a list of components to be rendered on the screen.

  • formFields - List of String - list of field names used by the screen as form-inputs.

  • viewFields - List of String - list of field names used by the screen as visible outputs.

  • recordListFields - List of String - list of field names used by the screen in a record listing.

  • format - Optional String - directive for a frontend to use specialized formatting for the display of the process.

    • Consult frontend documentation for supported values and their meanings.

  • backStepName - Optional String - For processes using LINEAR flow, if this value is given, then the frontend should offer a control that the user can take (e.g., a button) to move back to an earlier step in the process.

QFrontendComponentMetaData

A screen in a process may consist of multiple "components" - e.g., help text, and a form, and a list of records. Each of these components are defined in a QFrontendComponentMetaData.

QFrontendComponentMetaData Properties:

  • type - enum, Required - The type of component to display. Each component type works with different keys in the values map. Possible values for type are:

    • EDIT_FORM - Displays a list of fields for editing, similar to a record edit screen. Requires that formFields be populated in the step.

    • VIEW_FORM - Displays a list of fields for viewing (not editing), similar to a record view screen. Requires that viewFields be populated in the step.

    • HELP_TEXT - Block of help text to be display on the screen. Requires an entry in the component’s values map named "text".

    • HTML - Block of custom HTML, generated by the process backend. Expects a process value named html.

    • DOWNLOAD_FORM - Presentation of a link to download a file generated by the process. Expects process values named downloadFileName and serverFilePath.

    • GOOGLE_DRIVE_SELECT_FOLDER - Special form that presents a UI from Google Drive, where the user can select a folder (e.g., as a target for uploading files in a subsequent backend step).

    • BULK_EDIT_FORM - For use by the standard QQQ Bulk Edit process.

    • BULK_LOAD_FILE_MAPPING_FORM, BULK_LOAD_VALUE_MAPPING_FORM, or BULK_LOAD_PROFILE_FORM - For use by the standard QQQ Bulk Load process.

    • VALIDATION_REVIEW_SCREEN - For use by the QQQ Streamed ETL With Frontend process family of processes. Displays a component prompting the user to run full validation or to skip it, or, if full validation has been ran, then showing the results of that validation.

    • PROCESS_SUMMARY_RESULTS - For use by the QQQ Streamed ETL With Frontend process family of processes. Displays the summary results of running the process.

    • WIDGET - Render a QQQ Widget. Requires that widgetName be given as a value for the component.

    • RECORD_LIST - Deprecated. Showed a grid with a list of records as populated by the process.

  • values - Map of String → Serializable - Key=value pairs, with different expectations based on the component’s type. See above for details.

QBackendStepMetaData

Process Backend Steps are where custom (at this time, Java, but in the future, potentially, from any supported language) code is executed, to provide the logic of the process. QQQ comes with several common backend steps, such as for extracting lists of records, storing lists of records, etc.

QBackendStepMetaData Properties:

  • code - QCodeReference, Required - Reference to the code to be executed for the step. The referenced code must implement the BackendStep interface.

  • inputMetaData - QFunctionInputMetaData - Definition of the data that the backend step expects and/or requires. Sub-properties are:

    • fieldList - List of QQQ Fields - Optional list of fields used by the process step. In general, a process does not have to specify the fields that its steps use. It can be used, however, for example, to cause a defaultValue to be applied to a field if it isn’t specified in the process’s input. It can also be used to cause the process to throw an error, if a field is marked as isRequired, but a value is not present.

    • recordListMetaData - RecordListMetaData object - Not used at this time.

QStateMachineStep

Processes that use flow = STATE_MACHINE should use process steps of type QStateMachineStep.

A common pattern seen in state-machine processes, is that they will present a frontend-step to a user, then always run a given backend-step in response to that screen which the user submitted. Inside that backend-step, custom application logic will determine the next state to go to, which is typically another frontend-step (which would then submit data to its corresponding backend-step, and continue the FSM).

To help facilitate this pattern, factory methods exist on QStateMachineStep, for constructing the commonly-expected types of state-machine steps:

  • frontendThenBackend(name, frontendStep, backendStep) - for the frontend-then-backend pattern described above.

  • backendOnly(name, backendStep) - for a state that only has a backend step. This might be useful as a “reset” step, to run before restarting a state-loop.

  • frontendOnly(name, frontendStep) - for a state that only has a frontend step, which would always be followed by another state, which must be specified as the defaultNextStepName on the QStateMachineStep.

BasepullConfiguration

A "Basepull" process is a common pattern where an application needs to perform some action on all new (or updated) records from a particular data source. To implement this pattern, one needs to store the timestamp of when the action runs, then query the source for records where a date-time field is after that timestamp.

QQQ helps facilitate this pattern by automatically retrieving and updating that timestamp field, and by building a default query filter based on that timestamp.

This is done by adding a BasepullConfiguration object to a process’s meta-data.

BasepullConfiguration Properties:

  • tableName - String, Required - Name of a QQQ Table in the QQQ Instance where the basepull timestamps are stored.

  • keyField - String, Required - Name of a QQQ Field in the basepull table that stores a unique identifier for the process.

  • keyValue - String - Optional value to be stored in the keyField of the basepull table as the unique identifier for the process. If not set, then the process’s name is used.

  • lastRunTimeFieldName - String, Required - Name of a QQQ Field in the basepull table that stores the last-run time for the process.

  • hoursBackForInitialTimestamp - Integer - Optional number of hours to go back in time (from now) for the first time that the process is executed (i.e., if there is no timestamp stored in the basepull table).

  • timestampField - String, Required - Name of a QQQ Field in the table being queried against the last-run timestamp.

QScheduleMetaData

QQQ can automatically run processes using an internal scheduler, if they are configured with a QScheduleMetaData object.

QScheduleMetaData Properties

  • repeatSeconds - Integer, Conditional - How often the process should be executed, in seconds.

  • repeatMillis - Integer, Conditional - How often the process should be executed, in milliseconds. Mutually exclusive with repeatSeconds.

  • initialDelaySeconds - Integer, Conditional - How long between when the scheduler starts and the process should first run, in seconds.

  • initialDelayMillis - Integer, Conditional - How long between when the scheduler starts and the process should first run, in milliseconds. Mutually exclusive with initialDelaySeconds.

  • variantRunStrategory - enum, Conditional - For processes than run against QQQ Tables that use a QQQ Backend with Variants enabled, this property controls if the various instances of the process should run in PARALLEL or in SERIAL.

  • variantBackend - enum, Conditional - For processes than run against QQQ Tables that use a QQQ Backend with Variants enabled, this property specifies the name of the QQQ Backend.

StreamedETLWithFrontendProcess

A common pattern for QQQ processes to exhibit is called the "Streamed ETL With Frontend" process pattern. This pattern is to do an "Extract, Transform, Load" job on a potentially large set of records.

The records are Streamed through the process’s steps, meaning, QQQ runs multiple threads - a producer, which is selecting records, and a consumer, which is processing records. As such, in general, an unlimited number of records can be processed by a process, without worrying about exhausting server resources (e.g., OutOfMemory).

These processes also have a standard user-interface for displaying a summary of what the process will do (and has done), with a small number of records as a preview. The goal of the summary is to give the user the big-picture of what the process will do (e.g., X records will be inserted; Y records will be updated), along with a small view of some details on the records that will be stored (e.g., on record A field B will be set to C).

This type of process uses 3 backend steps, and 2 frontend steps, as follows:

  • preview (backend) - does just a little work (limited # of rows), to give the user a preview of what the final result will be - e.g., some data to seed the review screen.

  • review (frontend) - a review screen, which after the preview step does not have a full process summary, but can generally tell the user how many records are input to the process, and can show a preview of a small number of the records.

  • validate (backend) - optionally (per input on review screen), does like the preview step, but does it for all records from the extract step.

  • review (frontend) - a second view of the review screen, if the validate step was executed. Now that the full validation was performed, a full process summary can be shown, along with a some preview records.

  • execute (backend) - processes all the rows - does all the work - stores data in the backend.

  • result (frontend) - a result screen, showing a "past-tense" version of the process summary.

These backend steps are defined within QQQ, meaning they themselves do not execute any application-defined custom code. Instead, these steps use the following secondary backend steps:

  • Extract - Fetch the rows to be processed. Used by preview (but only for a limited number of rows), validate (without limit), and execute (without limit).

  • Transform - Do whatever transformation is needed to the rows. Done on preview, validate, and execute. Always works with a page of records at a time. Since it is called on the preview & validate steps, it should NOT ever store any data (unless it does a specific check to confirm that it is being used on an execute step).

  • Load - Store the records into the backend, as appropriate. Only called by the execute step. Always works with a page of records at a time.

The meta-data for a StreamedETLWithFrontendProcess uses several input fields on its steps. As such, it can be somewhat clumsy and error-prone to fully define a StreamedETLWithFrontendProcess. To improve this programmer-interface, an inner Builder class exists within StreamedETLWithFrontendProcess (generated by a call to StreamedETLWithFrontendProcess.processMetaDataBuilder()).

StreamedETLWithFrontendProcess.Builder methods:

  • withName(String name) - Set the name for the process.

  • withLabel(String label) - Set the label for the process.

  • withIcon(QIcon icon) - Set an Icon to be display with the process in the UI.

  • withExtractStepClass(Class<? extends AbstractExtractStep>) - Define the Extract step for the process. If no special extraction logic is needed, ExtractViaQuery.class is often a reasonable default. In other cases, ExtractViaQuery can be a reasonable class to extend for a custom extract step.

  • withTransformStepClass(Class<? extends AbstractTransformStep>) - Define the Transform step for the process. If no transformation logic is needed, NoopTransformStep.class can be used (though this is not very common).

  • withLoadStepClass(Class<? extends AbstractLoadStep>) - Define the Load step for the process. Several standard implementations exist, such as: LoadViaInsertStep.class, LoadViaUpdateStep.class, and LoadViaDeleteStep.class.

  • withTableName(String tableName) - Specify the name of the QQQ Table that the process should be associated with in the UI.

  • withSourceTable(String sourceTable) - Specify the name of the QQQ Table to be used as the source of records for the process.

  • withDestinationTable(String destinationTable) - Specify the name of the QQQ Table to be used as the destination for records from the process.

  • withSupportsFullValidation(Boolean supportsFullValidation) - By default, all StreamedETLWithFrontendProcesses do allow the user to choose to run the full validation step. However, in case cases it may not make sense to do so - so this method can be used to turn off that option.

  • withDoFullValidation(Boolean doFullValidation) - By default, all StreamedETLWithFrontendProcesses will prompt the user if they want to run the full validation step or not. However, in case cases you may want to enforce that the validation step always be executed. Calling this method will remove the option from the user, and always run a full validation.

  • withTransactionLevelAutoCommit(), withTransactionLevelPage(), and withTransactionLevelProcess() - Change the transaction-level used by the process. By default, these processes are ran with a single transaction for all pages of their execute step. But for some cases, doing page-level transactions can reduce long-transactions and locking within the system.

  • withPreviewMessage(String previewMessage) - Sets the message shown on the validation review screen(s) above the preview records.

  • withReviewStepRecordFields(List<QFieldMetaData> fieldList) -

  • withFields(List<QFieldMetaData> fieldList) - Adds additional input fields to the preview step of the process.

  • withBasepullConfiguration(BasepullConfiguration basepullConfiguration) - Add a BasepullConfiguration to the process.

  • withSchedule(QScheduleMetaData schedule) - Add a QScheduleMetaData to the process.

How to customize a Linear process flow

As referenced in the definition of the QProcessMetaData Properties, by default, (with flow = LINEAR) a process will execute each of its steps in-order, as defined in the stepList property. However, a Backend Step can customize this flow as follows:

  • RunBackendStepOutput.setOverrideLastStepName(String stepName)

    • QQQ’s RunProcessAction keeps track of which step it "last" ran, e.g., to tell it which one to run next. However, if a step sets the OverrideLastStepName property in its output object, then the step named in that property becomes the effective "last" step, thus determining which step comes next.

  • RunBackendStepOutput.updateStepList(List<String> stepNameList)

    • Calling this method changes the process’s runtime definition of steps to be executed. Thus allowing a completely custom flow. It should be noted, that the "last" step name (as tracked by QQQ within RunProcessAction) does need to be found in the new stepNameList - otherwise, the framework will not know where you were, for figuring out where to go next.

Example of a defining process that can use a customized linear flow:
// for a case like this, it would be recommended to define all step names in constants:
public final static String STEP_START = "start";
public final static String STEP_A     = "a";
public final static String STEP_B     = "b";
public final static String STEP_C     = "c";
public final static String STEP_1     = "1";
public final static String STEP_2     = "2";
public final static String STEP_3     = "3";
public final static String STEP_END   = "end";

// also, to define the possible flows (lists of steps) in constants as well:
public final static List<String> LETTERS_STEP_LIST = List.of(
  STEP_START, STEP_A, STEP_B, STEP_C, STEP_END);

public final static List<String> NUMBERS_STEP_LIST = List.of(
  STEP_START, STEP_1, STEP_2, STEP_3, STEP_END);

// when we define the process's meta-data, we only give a "skeleton" stepList -
// we must at least have our starting step, and we may want at least one frontend step
// for the UI to show some placeholder(s):
QProcessMetaData process = new QProcessMetaData()
   .withName(PROCESS_NAME)
   .withStepList(List.of(
      new QBackendStepMetaData().withName(STEP_START)
         .withCode(new QCodeReference(/*...*/)),
      new QFrontendStepMetaData()
         .withName(STEP_END)
   ));

// the additional steps get added via `addOptionalStep`, which only puts them in
// the process's stepMap, not its stepList!
process.addOptionalStep(new QFrontendStepMetaData().withName(STEP_A));
process.addOptionalStep(new QBackendStepMetaData().withName(STEP_B)
   .withCode(new QCodeReference(/*...*/)));
process.addOptionalStep(new QFrontendStepMetaData().withName(STEP_C));

process.addOptionalStep(new QBackendStepMetaData().withName(STEP_1)
   .withCode(new QCodeReference(/*...*/)));
process.addOptionalStep(new QFrontendStepMetaData().withName(STEP_2));
process.addOptionalStep(new QBackendStepMetaData().withName(STEP_3)
   .withCode(new QCodeReference(/*...*/)));
Example of a process backend step adjusting the process’s runtime flow:
/***************************************************************************
** look at the value named "which". if it's "letters", then make the process
** go through the stepList consisting of letters; else, update the step list
** to be the "numbers" steps.
**
** Also - if the "skipSomeSteps" value is give as true, then set the
** overrideLastStepName to skip again (in the letters case, skip past A, B
** and C; in the numbers case, skip past 1 and 2).
***************************************************************************/
public static class StartStep implements BackendStep
{
  @Override
  public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException
  {
     Boolean skipSomeSteps = runBackendStepInput.getValueBoolean("skipSomeSteps");

     if(runBackendStepInput.getValueString("which").equals("letters"))
     {
        runBackendStepOutput.updateStepList(LETTERS_STEP_LIST);
        if(BooleanUtils.isTrue(skipSomeSteps))
        {
           runBackendStepOutput.setOverrideLastStepName(STEP_C);
        }
     }
     else
     {
        runBackendStepOutput.updateStepList(NUMBERS_STEP_LIST);
        if(BooleanUtils.isTrue(skipSomeSteps))
        {
           runBackendStepOutput.setOverrideLastStepName(STEP_2);
        }
     }
  }
}
How to allow a process to go back

The simplest option to allow a process to present a "Back" button to users, thus allowing them to move backward through a process (e.g., from a review screen back to an earlier input screen), is to set the property backStepName on a QFrontendStepMetaData.

If the step that is executed after the user hits "Back" is a backend step, then within that step, runBackendStepInput.getIsStepBack() will return true (but ONLY within that first step after the user hits "Back"). It may be necessary within individual processes to be aware that the user has chosen to go back, to reset certain values in the process’s state.

Alternatively, if a frontend step’s "Back" behavior needs to be dynamic (e.g., sometimes not available, or sometimes targeting different steps in the process), then in a backend step that runs before the frontend step, a call to runBackendStepOutput.getProcessState().setBackStepName() can be made, to customize the value which would otherwise come from the QFrontendStepMetaData.

Widgets

Widgets are the most customizable UI components in QQQ. They can be used either on App Home Screens (e.g., as Dashboard screens), or they can be included into Record View screens.

QQQ defines several types of widgets, such as charts (pie, bar, line), numeric displays, application-populated tables, or even fully custom HTML.

QWidgetMetaData

A Widget is defined in a QQQ Instance in a QWidgetMetaData object.

QWidgetMetaData Properties:

  • name - String, Required - Unique name for the widget within the QQQ Instance.

  • type - String, Required - Specifies the UI & data type for the widget.

  • label - String - User-facing header or title for a widget.

  • tooltip - String - Text contents to be placed in a tooltip associated with the widget’s label in the UI.

    • Values should come from the WidgetType enum’s getType() method (e.g., WidgetType.BAR_CHART.getType())

  • gridColumns - Integer - for a desktop-sized screen, in a 12-based grid, how many columns the widget should consume.

  • codeReference - QCodeReference, Required - Reference to the custom code, a subclass of AbstractWidgetRenderer, which is responsible for loading data to render the widget.

  • footerHTML - String - HTML String, which, if present, will be displayed in the footer of the widget (not supported by all widget types).

  • isCard - boolean, default false TODO

  • showReloadButton - boolean, default true TODO

  • showExportButton - boolean, default false TODO

  • dropdowns - TODO

  • storeDropdownSelections - boolean TODO

  • icons - Map<String, QIcon> TODO

  • defaultValues - Map<String, Serializable> TODO

There are also some subclasses of QWidgetMetaData, for some specific widget types:

ParentWidgetMetaData Properties:

  • title - String TODO - how does this differ from label?

  • childWidgetNameList - List<String>, Required

  • childProcessNameList - List<String> TODO appears unused - check, and delete

  • laytoutType - enum of GRID or TABS, default GRID

QNoCodeWidgetMetaData Properties:

  • values - List<AbstractWidgetValueSource> TODO

  • outputs - List<AbstractWidgetOutput> TODO

TODO - Examples

Fields

QQQ Fields define

QFieldMetaData

QFieldMetaData Properties:

  • name - String, Required - Unique name for the field within its container (table, process, etc).

  • label - String - User-facing label for the field, presented in User Interfaces.

  • type - enum of QFieldType, Required - Data type for values in the field.

  • backendName - String - Name of the field within its backend.

    • For example, in an RDBMS-backed table, a field’s name may be written in camel case, but its backendName written with underscores.

  • isRequired - boolean, default false - Indicator that a value is required in this field.

  • isEditable - boolean, default true - Indicator that users may edit values in this field.

  • displayFormat - String, default %s - Java Format Specifier string, used to format values in the field for display in user interfaces. Used to set values in the displayValues map within a QRecord.

    • Recommended values for displayFormat come from the DisplayFormat interface, such as DisplayFormat.CURRENCY, DisplayFormat.COMMAS, or DisplayFormat.DECIMAL2_COMMAS.

  • defaultValue - Value to use for the field if no other value is given. Type is based on the field’s type.

  • possibleValueSourceName - String - Reference to a {link-pvs} to be used for this field. Values in this field should correspond to ids from the referenced Possible Value Source.

  • maxLength - Integer - Maximum length (number of characters) allowed for values in this field. Only applicable for fields with type=STRING. Needs to be used with a FieldBehavior of type ValueTooLongBehavior.

Field Behaviors

Additional behaviors can be attached to fields through the use of the behaviors attribute, which is a Set of 0 or more instances of implementations of the FieldBehavior interface. Note that in some cases, these instances may be enum constants, but other times may be regular Objects.

QQQ provides a set of common field behaviors. Applications can also define their own field behaviors by implementing the FieldBehavior interface, and attaching instances of their custom behavior classes to fields.

ValueTooLongBehavior

Used on String fields. Requires the field to have a maxLength set. Depending on the chosen instance of this enum, before a record is Inserted or Updated, if the value in the field is longer than the maxLength, then one of the following actions can occur:

  • TRUNCATE - The value will be simply truncated to the maxLength.

  • TRUNCATE_ELLIPSIS - The value will be truncated to 3 characters less than the maxLength, and three periods (an ellipsis) will be placed at the end.

  • ERROR - An error will be reported, and the record will not be inserted or updated.

  • PASS_THROUGH - Nothing will happen. This is the same as not having a ValueTooLongBehavior on the field.

Examples of using ValueTooLongBehavior
   new QFieldMetaData("sku", QFieldType.STRING)
      .withMaxLength(40),
      .withBehavior(ValueTooLongBehavior.ERROR),

   new QFieldMetaData("reason", QFieldType.STRING)
      .withMaxLength(250),
      .withBehavior(ValueTooLongBehavior.TRUNCATE_ELLIPSIS),
ValueRangeBehavior

Used on Numeric fields. Specifies min and/or max allowed values for the field. For each of min and max, the following attributes can be set:

  • minValue / maxValue - the number that is the limit.

  • minAllowEqualTo / maxAllowEqualTo - boolean (default true). Controls if < (>) or ≤ (≥).

  • minBehavior / maxBehavior - enum of ERROR (default) or CLIP.

    • If ERROR, then a value not within the range causes an error, and the value does not get stored.

    • else if CLIP, then a value not within the range gets "clipped" to either be the min/max (if allowEqualTo), or to the min/max plus/minus the clipAmount

  • minClipAmount / maxClipAmount - Default 1. Used when behavior is CLIP (only applies when not allowEqualTo).

Examples of using ValueRangeBehavior
    new QFieldMetaData("noOfShoes", QFieldType.INTEGER)
      .withBehavior(new ValueRangeBehavior().withMinValue(0));

    new QFieldMetaData("price", QFieldType.BIG_DECIMAL)
      .withBehavior(new ValueRangeBehavior()
         // set the min value to be >= 0, and an error if an input is < 0.
         .withMinValue(BigDecimal.ZERO)
         .withMinAllowEqualTo(true)
         .withMinBehavior(ERROR)
         // set the max value to be < 100 - but effectively, clip larger values to 99.99
         // here we use the .withMax() method that takes 4 params vs. calling 4 .withMax*() methods.
         .withMax(new BigDecimal("100.00"), false, CLIP, new BigDecimal("0.01"))
    );
DynamicDefaultValueBehavior

Used to set a dynamic default value to a field when it is being inserted or updated. For example, instead of having a hard-coded defaultValue specified in the field meta-data, and instead of having to add, for example, a pre-insert custom action.

  • CREATE_DATE - On inserts, sets the field’s value to the current time.

  • MODIFY_DATE - On inserts and updates, sets the field’s value to the current time.

  • USER_ID - On inserts and updates, sets the field’s value to the current user’s id (but only if the value is currently null).

Note that the QInstanceEnricher will, by default, add the CREATE_DATE and MODIFY_DATE DynamicDefaultValueBehavior options to any fields named "createDate" or "modifyDate". This behavior can be disabled by setting the configAddDynamicDefaultValuesToFieldsNamedCreateDateAndModifyDate property on the QInstanceEnricher instance used by the application to false.

Examples of using DynamicDefaultValueBehavior
   new QFieldMetaData("createDate", QFieldType.DATE_TIME)
      .withBehavior(DynamicDefaultValueBehavior.CREATE_DATE),

   new QFieldMetaData("modifyDate", QFieldType.DATE_TIME)
      .withBehavior(DynamicDefaultValueBehavior.MODIFY_DATE),

   new QFieldMetaData("createdByUserId", QFieldType.STRING)
      .withBehavior(DynamicDefaultValueBehavior.USER_ID),
DateTimeDisplayValueBehavior

By default, in QQQ, fields of type DATE_TIME are stored in UTC, and their values in a QRecord is a java Instant instance, which is always UTC. However, frontends will prefer to display date-time values in the user’s local Time Zone whenever possible.

Using DateTimeDisplayValueBehavior allows a DATE_TIME field to be displayed in a different Time Zone. An example use-case for this would be displaying airplane flight times, where you would want a flight from California to New York to display Pacific Time for its departure time, and Eastern Time for its arrival.

An instance of DateTimeDisplayValueBehavior can be configured to either use a hard-coded time ZoneId (for example, to always show users UTC, or a business’s home-office time zone). Or, it can be set up to get the time zone to use from another field in the table.

Examples of using DateTimeDisplayValueBehavior
new QTableMetaData().withName("flights").withFields(List.of(
   ...
   new QFieldMetaData("departureTimeZoneId", QFieldType.STRING),
   new QFieldMetaData("arrivaTimeZoneId", QFieldType.STRING),

   new QFieldMetaData("departureTime", QFieldType.DATE_TIME)
      .withBehavior(new DateTimeDisplayValueBehavior()
         .withZoneIdFromFieldName("departureTimeZoneId")),

   new QFieldMetaData("arrivalTime", QFieldType.DATE_TIME)
      .withBehavior(new DateTimeDisplayValueBehavior()
         .withZoneIdFromFieldName("arrivalTimeZoneId"))

   new QFieldMetaData("ticketSaleStartDateTime", QFieldType.DATE_TIME)
      .withBehavior(new DateTimeDisplayValueBehavior()
         .withDefaultZoneId("UTC"))
CaseChangeBehavior

A field can be made to always go through a toUpperCase or toLowerCase transformation, both before it is stored in a backend, and after it is read from a backend, by adding a CaseChangeBehavior to it:

Examples of using CaseChangeBehavior
new QTableMetaData().withName("item").withFields(List.of(

   new QFieldMetaData("sku", QFieldType.STRING)
      .withBehavior(CaseChangeBehavior.TO_UPPER_CASE)),

   new QFieldMetaData("username", QFieldType.STRING)
      .withBehavior(CaseChangeBehavior.TO_LOWER_CASE)),

Possible Value Sources

TODO

QPossibleValueSource

A Possible Value Source is defined in a QQQ Instance in a QPossibleValueSource object.

TODO

QPossibleValueSource Properties:

  • name - String, Required - Unique name for the possible value source within the QQQ Instance.

TODO

Joins

A QJoinMetaData is a meta-data object that tells QQQ, essentially “it is possible for these 2 tables to join, here’s how to do it”.

Joins can be used then, in an application, in a number of possible ways:

  • In a QQQ Table, we can specify joins to be “exposed”, e.g., made available to users on a query screen

  • Also in a Table, as part of an “Association”, which sets up one table as a “parent” of another, such that you can store (and fetch) the child-records at the same time as the parent

    • A common use-case here may be an order & lineItem table - such that QQQ can generate an API uses to allow you to post an order and its lines in a single request, and they get stored all together

  • In defining the security field (record lock) on a table, sometimes, it isn’t a field directly on the table, but instead comes from a joined table (possibly even more than just 1 table away).

    • For example, maybe a lineItem table, doesn’t have a clientId, but needs secured by that field

      • so its recordLock can specify a “joinNameChain” that describes how to get from lineItem to order.clientId

  • The QueryAction can take (through its QueryInput object) zero or more QueryJoin objects, which must make a reference (implicitly or explicitly) to a QJoinMetaData. See the section on QueryJoins for more details.

QJoinMetaData

Joins are defined in a QQQ Instance in a QJoinMetaData object.

In this object, we have the concept of a "leftTable" and a "rightTable". There isn’t generally anything special about which table is on the "left" and which is on the "right". But the remaining pieces of the QJoinMetaData do all need to line-up with these sides.

For example:

  • The Type (one-to-one, one-to-many, many-to-one) - where the leftTable comes first, and rightTable comes second (e.g., a one-to-many means 1-row in leftTable has many-rows in rightTable associated with it)

  • In a JoinOn object, the 1st field name given is from the leftTable; the second fieldName from the rightTable.

QJoinMetaData Properties:

  • name - String, Required - Unique name for the join within the QQQ Instance.

    • One convention is to name joins based on (leftTable + "Join" + rightTable).

    • If you do not wish to define join names yourself, the method withInferredName() can be called (which defers to public static String makeInferredJoinName(String leftTable, String rightTable)), to create a name for the join following the (leftTable + "Join" + rightTable) convention.

  • leftTable - String, Required - Name of a QQQ Table in the QInstance.

  • rightTable - String, Required - Name of a QQQ Table in the QInstance.

  • type - enum, Required - cardinality between the two tables in the join.

    • e.g., ONE_TO_ONE, ONE_TO_MANY (indicating 1 record in the left table may join to many records in the right table), or MANY_TO_ONE (vice-versa).

    • Note that there is no MANY_TO_MANY option, as a many-to-many is built as multiple QJoinMetaData’s going through the intermediary (intersection) table.

  • joinOns - List<JoinOn>, Required - fields used to join the tables. Note: In the 2-arg JoinOn constructor, the leftTable’s field comes first. Alternatively, the no-arg constructor can be used along with .withLeftField().withRightField()

  • orderBys - List<QFilterOrderBy> - Optional list of order-by objects, used in some framework-generated queries using the join. The field names are assumed to come from the rightTable.

TODO what else do we need here?

Security Key Types

In QQQ, record-level security is provided by using a lock & key metaphor.

The use-case being handled here is:

  • A user has full permission on a table (query, insert, update, and delete).

  • However, they should only be allowed to read a sub-set of the rows in the table.

    • e.g., maybe it’s a multi-tenant system, or the table has user-specific records.

The lock & key metaphor is realized by the user being associated with one or more "Keys" (as values in their session), and records in tables being associated with one or more "Locks" (as values in fields). A user is only allowed to access records where the user’s key(s) match the record’s security lock(s).

For a practical example, picture a multi-tenant Order Management System,where all orders are assigned to a "client". Users (customers) should only be able to see orders associated with the client which that user works for.

In this scenario, the order table would have a "lock" on its clientId field. Customer-users would have a clientId key in their session. When the QQQ backend did a search for records (e.g., an SQL query) it would implicitly (without any code being written by the application developer) filter the table to only allow the user to see records with their clientId.

To implement this scenario, the application would define the following pieces of meta-data:

  • At the QQQ-Instance level, a SecurityKeyType, to define a domain of possible locks & keys within an application.

    • An application can define multiple Security Key Types. For example, maybe clientId and userId as key types.

  • At the per-table level, a RecordSecurityLock, which references a security key type, and how that key type should be applied to the table.

    • For example, what field stores the clientId value in the order table.

  • Finally, when a user’s session is constructed via a QQQ Authentication provider, security key values are set, based on data from the authentication backend.

Additional Scenarios

All Access Key

A "super-user" may be allowed to access all records in a table regardless of their record locks, if the Security Key Type specifies an allAccessKeyName, and if the user has a key in their session with that key name, and a value of true. Going back to the lock & key metaphor, this can be thought of as a "skeleton key", that can unlock any record lock (of the security key’s type).

Null Value Behaviors

In a record security lock, different behaviors can be defined for handling rows with a null key value.

For example:

  • Sometimes orders may be loaded into the OMS system described above, where the application doesn’t yet know what client the order belongs to. In this case, the application may need to ensure that such records, with a null value in clientId are hidden from customer-users, to avoid potentially leaking a different client’s data.

    • This can be accomplished with a record security lock on the order table, with a nullValueBehavior of DENY.

    • Furthermore, if internal/admin users should be given access to such records, then the security key type can be configured with a nullValueBehaviorKeyName (e.g., "clientIdNullValueBehavior"), which can be set per-user to allow access to records, overriding the table lock’s specified nullValueBehavior.

      • This could also be done by giving internal/admin users an allAccessKey, but sometimes that is not what is required.

  • Maybe a warehouse locations table is assigned a clientId once inventory for a client is placed in the location, at which point in time, only the client’s users should be allowed to see the record. But, if no client has been assigned to the location, and clientId is null, then you may want to allow any user to see such records.

    • This can be accomplished with a record security lock on the Warehouse Locations table, with a nullValueBehavior of ALLOW.

QSecurityKeyType

A Security Key Type is defined in a QQQ Instance in a QSecurityKeyType object.

QSecurityKeyType Properties:

  • name - String, Required - Unique name for this security key type within the QQQ Instance.

  • allAccessKeyName - String - Optional name of the all-access security key associated with this key type.

  • nullValueBehaviorKeyName - String - Optional name of the null-value-behavior overriding security key associated with this key type.

    • Note, name, allAccessKeyName, and nullValueBehaviorKeyName are all checked against each other for uniqueness. A QInstanceValidationException will be thrown if any name collisions occur.

  • possibleValueSourceName - String - Optional reference to a possible value source from which value for the key can come.

Reports

QQQ can generate reports based on QQQ Tables defined within a QQQ Instance. Users can run reports, providing input values. Alternatively, application code can run reports as needed, supplying input values.

QReportMetaData

Reports are defined in a QQQ Instance with a QReportMetaData object. Reports are defined in terms of their sources of data (QReportDataSource), and their view(s) of that data (QReportView).

QReportMetaData Properties:

  • name - String, Required - Unique name for the report within the QQQ Instance.

  • label - String - User-facing label for the report, presented in User Interfaces. Inferred from name if not set.

  • processName - String - Name of a QQQ Process used to run the report in a User Interface.

  • inputFields - List of QQQ Field - Optional list of fields used as input to the report.

    • The values in these fields can be used via the syntax ${input.NAME}, where NAME is the name attribute of the inputField.

    • For example:

// given this inputField:
new QFieldMetaData("storeId", QFieldType.INTEGER)

// its run-time value can be accessed, e.g., in a query filter under a data source:
new QFilterCriteria("storeId", QCriteriaOperator.EQUALS, List.of("${input.storeId}"))

// or in a report view's title or field formulas:
.withTitleFields(List.of("${input.storeId}"))
new QReportField().withName("storeId").withFormula("${input.storeId}")

  • dataSources - List of QReportDataSource, Required - Definitions of the sources of data for the report. At least one is required.

QReportDataSource

Data sources for QQQ Reports can either reference QQQ Tables within the QQQ Instance, or they can provide custom code in the form of a CodeReference to a Supplier, for use cases such as a static data tab in an Excel report.

QReportDataSource Properties:

  • name - String, Required - Unique name for the data source within its containing Report.

  • sourceTable - String, Conditional - Reference to a QQQ Table in the QQQ Instance, which the data source queries data from.

  • queryFilter - QQueryFilter - If a sourceTable is defined, then the filter specified here is used to filter and sort the records queried from that table when generating the report.

  • staticDataSupplier - QCodeReference, Conditional - Reference to custom code which can be used to supply the data for the data source, as an alternative to querying a sourceTable.

    • Must be a JAVA code type

    • Must be a REPORT_STATIC_DATA_SUPPLIER code usage.

    • The referenced class must implement the interface: Supplier<List<List<Serializable>>>.

QReportView

Report Views control how the source data for a report is organized and presented to the user in the output report file. If a DataSource describes the rows for a report (e.g., what table provides what records), then a View may be thought of as describing the columns in the report. A single report can have multiple views, specifically, for the use-case where an Excel file is being generated, in which case each View creates a tab or sheet within the xlsx file.

QReportView Properties:

  • name - String, Required - Unique name for the view within its containing Report.

  • label - String - Used as a sheet (tab) label in Excel formatted reports.

  • type - enum of TABLE, SUMMARY, PIVOT. Required - Defines the type of view being defined.

    • TABLE views are a simple listing of the records from the data source.

    • SUMMARY views are essentially pre-computed Pivot Tables. That is to say, the aggregation done by a Pivot Table in a spreadsheet file is done by QQQ while generating the report. In this way, a non-spreadsheet report (e.g., PDF or CSV) can have summarized data, as though it were a Pivot Table in a live spreadsheet.

    • PIVOT views produce actual Pivot Tables, and are only supported in Excel files (and are not supported at the time of this writing).

  • dataSourceName - String, Required - Reference to a DataSource within the report, that is used to provide the rows for the view.

  • varianceDataSourceName - String - Optional reference to a second DataSource within the report, that is used in SUMMARY type views for computing variances.

    • For example, given a Data Source with a filter that selects all sales records for a given year, a Variance Data Source may have a filter that selects the previous year, for doing comparissons.

  • pivotFields - List of String, Conditional - For SUMMARY or PIVOT type views, specify the field(s) used as pivot rows.

    • For example, in a summary view of orders, you may "pivot" on the customerId field, to produce one row per-customer, with aggregate data for that customer.

  • titleFormat - String - Java Format String, used with titleFields (if given), to produce a title row, e.g., first row in the view (before any rows from the data source).

  • titleFields - List of String, Conditional - Used with titleFormat, to provide values for any format specifiers in the format string. Syntax to reference a field (e.g., from a report input field) is: ${input.NAME}, where NAME is the name attribute of the inputField.

    • Example of using titleFormat and titleFields:

// given these inputFields:
new QFieldMetaData("startDate", QFieldType.DATE)
new QFieldMetaData("endDate", QFieldType.DATE)

// a view can have a title row like this:
.withTitleFormat("Weekly Sales Report - %s - %s")
.withTitleFields(List.of("${input.startDate}", "${input.endDate}"))

  • includeHeaderRow - boolean, default true - Indication that first row of the view should be the column labels.

    • If true, then header row is put in the view.

    • If false, then no header row is put in the view.

  • includeTotalRow - boolean, default false - Indication that a totals row should be added to the view. All numeric columns are summed to produce values in the totals row.

    • If true, then totals row is put in the view.

    • If false, then no totals row is put in the view.

  • includePivotSubTotals - boolean, default false - For a SUMMARY or PIVOT type view, if there are more than 1 pivotFields being used, this field is an indication that each higher-level pivot should include sub-totals.

    • TODO - provide example

  • columns - List of QReportField, required - Definition of the columns to appear in the view. See section on QReportField for details.

  • orderByFields - List of QFilterOrderBy, optional - For a SUMMARY or PIVOT type view, how to sort the rows.

  • recordTransformStep - QCodeReference, subclass of AbstractTransformStep - Custom code reference that can be used to transform records after they are queried from the data source, and before they are placed into the view. Can be used to transform or customize values, or to look up additional values to add to the report.

    • TODO - provide example

  • viewCustomizer - QCodeReference, implementation of interface Function<QReportView, QReportView> - Custom code reference that can be used to customize the report view, at runtime. Can be used, for example, to dynamically define the report’s columns.

    • TODO - provide example

QReportField

Report Fields define the fields (AKA columns) of data that appear in a report view. These fields can either be direct references to fields from the report’s data sources, or values computed using formula defined in the QReportField.

QReportField Properties:

  • name - String, required - Unique identifier for the field within its ReportView. In general, will be a reference to a field from the ReportView’s DataSource unless a *formula is given (for SUMMARY type views), the field is marked as isVirtual, or the field is marked as showPossibleValueLabel).

  • label - String - Optional text label to identify the field, for example, in a header row. If not given, may be derived from field, where possible.

  • type - QFieldType

  • formula - String, conditional - Required for SUMMARY type views. Defines the formula to be used for computing the value in this field.

    • For example:

.withName("reportEndDate").withFormula("${input.endDate}")

.withName("count").withFormula("${pivot.count.id}")

.withName("percentOfTotal").withFormula("=DIVIDE(${pivot.count.id},${total.count.id})")

.withName("sumCost").withFormula("${pivot.sum.cost}")

.withName("sumCharge").withFormula("${pivot.sum.charge}")

.withName("profit").withFormula("=MINUS(${pivot.sum.charge},${pivot.sum.cost})")

.withName("totalCost").withFormula("=DIVIDE_SCALE(${pivot.sum.cost},${pivot.count.id},2)")

.withName("revenuePer").withFormula("=DIVIDE_SCALE(${pivot.sum.charge},${pivot.count.id},2)")

.withName("marginPer").withFormula("=MINUS(DIVIDE_SCALE(${pivot.sum.charge},${pivot.count.id},2),DIVIDE_SCALE(${pivot.sum.cost},${pivot.count.id},2))")

.withName("thisWeekMargin").withFormula("=SCALE(DIVIDE(${thisRow.profit},${pivot.sum.charge}),2)")

.withName("previousWeekProfit").withFormula("=MINUS(${variancePivot.sum.charge},${variancePivot.sum.cost})")

.withName("previousWeekMargin").withFormula("=SCALE(DIVIDE(${thisRow.previousWeekProfit},${variancePivot.sum.charge}),2)")

.withName("marginThisVsPrevious").withFormula("=SCALE(MINUS(${thisRow.margin},${thisRow.marginPrevious}),3)")

.withName("exception").withFormula("""
   =IF(LT(${thisRow.margin},0),Negative Margin,IF(LT(${thisRow.marginThisVsPrevious},0),Margin Decreased,""))""")

  • displayFormat String

  • isVirtual Boolean, default false - (needs reviewed - may only be required for report views using a data source with a staticDataSupplier)

  • showPossibleValueLabel Boolean, default false - To show a translated value for a Possible Value field (e.g., a name or other value meaningful to a user, instead of a foreign key).

  • sourceFieldName String - Used for the scenario where a possibleValue field is included in a report both as the foreign key (raw, id value), and the translated "label" value. In that case, the field marked with showPossibleValueLabel = true should be given a different name, and should use sourceFieldName to indicate the field that has the id value.

    • For example:

// this field would have the "raw" warehouseId values
// e.g., integers - foreign keys to a warehouse table.  Generally useful for machines to know.
new QReportField("warehouseId")
   .withLabel("Warehouse Id"),

// this field would have the translated values from the warehouse PossibleValueSource
// for example, maybe the name field from the warehouse table.  A string, useful for humans to read.
new QReportField("warehouseName")
   .withSourceFieldName("warehouseId")
   .withShowPossibleValueLabel(true)
   .withLabel("Warehouse Name"),

Icons

TODO

QIcon

Icons are defined in a QQQ Instance in a QIcon object.

TODO

QIcon Properties:

  • name - String - Name of an icon from the Material UI Icon set

    • Note that icon names from the link above need to be converted from CamelCase to underscore_case…​

  • path - String - Path to a file served under the application’s web root, to be used as the icon.

Either name or path must be specified. If both are given, then name is used.

Permission Rules

TODO

PermissionRule

TODO

PermissionRule Properties:

TODO

Services

QQQ Middleware: Javalin web server

QQQ provides a standard implementation of a middleware layer - that is - code that exists between the QQQ backend and user interface. This implementation is a web server built using the Javalin framework, packaged and deployed in the qqq-middleware-javalin maven module

The de facto way to create a QQQ application server is to write a class which uses an instance of one of the subclasses of QApplicationJavalinServer.

For example, if your application metadata is defined in a directory of yaml files, your server class could be implemented as:

ConfigFileBasedQQQApplication usage example
public static void main(String[] args)
{
   try
   {
      String path = "src/main/resources/metadata";
      ConfigFilesBasedQQQApplication application = new ConfigFilesBasedQQQApplication(path);
      QApplicationJavalinServer javalinServer = new QApplicationJavalinServer(application);
      javalinServer.start();
   }
   catch(Exception e)
   {
      LOG.error("Failed to start javalin server.  See stack trace for details.", e);
   }
}

A similar class exists if your metadata is produced by a package of Java MetaDataProducer objects: MetaDataProducerBasedQQQApplication.

QApplicationJavalinServer

This class provides the bridge between your QQQ Application (e.g., your metadata) and the QQQ Middleware layer served by a Javalin web server. It has several properties to control behaviors:

  • Integer port - (default 8000) - port to use for serving HTTP.

  • boolean serveFrontendMaterialDashboard - (default true) whether to serve the javascript frontend provided in the maven artifact qqq-frontend-material-dashboard.

  • boolean serveLegacyUnversionedMiddlewareAPI - (default true) whether to serve a version the original implementation of the QQQ middleware, which current version of qqq-frontend-material-dashboard are compatible with.

  • List<AbstractMiddlewareVersion> middlewareVersionList - (default contains MiddlewareVersionV1) - list of newer, formally versioned implementations of the QQQ middleware interface to be served.

  • Consumer<Javalin> javalinConfigurationCustomizer - (default null) - optional hook to customize the javalin service object before it is started.

  • List<QJavalinRouteProviderInterface> additionalRouteProviders - (default null) - list of fully custom implementations of QJavalinRouteProviderInterface, to add additional endpoints to the javalin server.

    • Note, you may first want to consider using JavalinRouteProviderMetaData instead - see below.

  • QJavalinMetaData javalinMetaData - (default null) - optional alternative place to define JavalinMetaData (vs. defining it in the QInstance). Note that if it is set in both places, the one in the QApplicationJavalinServer is used.

JavalinMetaData

Certain behaviors of a QQQ Javalin server are configured in a declarative manner by adding a QJavalinMetaData object to the supplementalMetaData in your QInstance (or, as mentioned above, by setting it directly on the QApplicationJavalinServer):

  • List<JavalinRouteProviderMetaData> routeProviders - (default null) optional list of custom route providers to add to the Javalin server. See below for details.

  • String uploadedFileArchiveTableName - (default null) - reference to a QQQ Table in your application instance, needed to support the Bulk Load process, as well as any other processes which need to accept an uploaded file as input.

  • boolean loggerDisabled - (default false)

  • Function<QJavalinAccessLogger.LogEntry, Boolean> logFilter - (default null)

  • boolean queryWithoutLimitAllowed - (default false)

  • Integer queryWithoutLimitDefault - (default 1000)

  • Level queryWithoutLimitLogLevel - (default INFO)

JavalinRouteProviderMetaData

This type of metadata allows you to add additional http route providers to your Javalin instance, e.g., for serving static files or for running custom code from your application (in the form of QQQ Processes) to respond to HTTP requests.

  • String hostedPath - (required)

  • String fileSystemPath - (required for a static router)

  • String processName - required for a dynamic, process-based router. Must be a process name within the QQQ Instance. See below for additional details

  • List<String> methods - required list of HTTP methods (verbs) that are served by the route provider

  • QCodeReference routeAuthenticator - Optional reference to a class that implements `RouteAuthenticatorInterface, to provide security authentication to all requests handled by the route provider.

    • A default implementation is provided as SimpleRouteAuthenticator, which requires that a user session be present to access paths served by the route provider.

Process-based route provider processes

If you define a JavalinRouteProviderMetaData with a processName (e.g., to serve dynamic HTTP responses from your javalin server), the process that you implement will be called to respond to any HTTP requests received by the javalin server which match the hostedPath and methods that are specified in the metadata.

The QQQ javalin server will marshal request data from the javalin context into the process’s payload, conforming to the shape of the ProcessBasedRouterPayload class. Similarly, the http response will be built by taking values from the process’s output/state conforming to the fields in that class. As such, it is recommended to use a ProcessBasedRouterPayload instance, as show in this example:

Process-based router usage example (including ProcessBasedRouterPayload)
public class MyDynamicSiteProcessStep implements BackendStep
{
   @Override
   public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException
   {
      ProcessBasedRouterPayload payload = input.getProcessPayload(ProcessBasedRouterPayload.class);
      String path = payload.getPath();
      payload.setResponseString("You requested: " + path);
      output.setProcessPayload(payload);
   }
}

Schedulers and Scheduled Jobs

QQQ has the ability to automatically run various types of jobs on schedules, either defined in your instance’s meta-data, or optionally via data in your application, in a scheduledJob table.

Schedulers and QSchedulerMetaData

2 types of schedulers are included in QQQ by default (though an application can define its own schedulers):

  • SimpleScheduler - is (as its name suggests) a simple class which uses java’s ScheduledExecutorService to run jobs on repeating intervals.

    • Cannot run cron schedules - only repeating intervals.

    • If multiple servers are running, each will potentially run the same job concurrently

    • Has no configurations, e.g., to limit the number of threads.

  • QuartzScheduler - uses the 3rd party Quartz Scheduler library to provide a much more capable, though admittedly more complex, scheduling solution.

    • Can run both cron schedules and repeating intervals.

    • By default, will not allow concurrent executions of the same job.

    • Supports multiple configurations, e.g., to limit the number of threads.

An application can define its own scheduler by providing a class which implements the QSchedulerInterface.

A QInstance can work with 0 or more schedulers, as defined by adding QSchedulerMetaData objects to the instance.

This meta-data class is abstract, and is extended by the 2 built-in schedulers (e.g., SimpleSchedulerMetaData and QuartzSchedulerMetaData). As such, these concrete subclasses are what you need to instantiate and add to your instance.

To configure a QuartzScheduler, you can add a Properties object to the QuartzSchedulerMetaData object. See Quartz’s documentation for available configuration properties.

Defining SchedulerMetaData
qInstance.addScheduler(new SimpleSchedulerMetaData().withName("mySimpleScheduler"));

qInstance.addScheduler(new QuartzSchedulerMetaData()
   .withName("myQuartzScheduler")
   .withProperties(myQuartzProperties);

SchedulableTypes

The types of jobs which can be scheduled in a QQQ application are defined in the QInstance by instances of the SchedulableType meta-data class. These objects contain a name, along with a QCodeReference to the runner, which must be a class that implements the SchedulableRunner interface.

By default, (in the QInstanceEnricher), QQQ will make 3 SchedulableType options available:

  • PROCESS - Any Process defined in the QInstance can be scheduled.

  • QUEUE_PROCESSOR - A Queue defined in the QInstance, which requires polling (e.g., SQS), can be scheduled.

  • TABLE_AUTOMATIONS - A Table in the QInstance, with AutomationDetails referring to an AutomationProvider which requires polling, can be scheduled.

If an application only wants to use a subset of these SchedulableType options, or to add custom SchedulableType options, the QInstance will need to have 1 or more SchedulableType objects in it before the QInstanceEnricher runs.

User-defined Scheduled Jobs

To allow users to schedule jobs (rather than using programmer-defined schedules (in meta-data)), you can add a set of tables to your QInstance, using the ScheduledJobsMetaDataProvider class:

Adding the ScheduledJob tables and related meta-data to a QInstance
new ScheduledJobsMetaDataProvider().defineAll(
   qInstance, backendName, table -> tableEnricher(table));

This meta-data provider adds a "scheduledJob" and "scheduledJobParameter" table, along with some PossibleValueSources. These tables include post-action customizers, which manage (re-, un-) scheduling jobs based on changes made to records in this these tables.

Also, when QScheduleManager is started, it will query these tables,and will schedule jobs as defined therein.

You can use a mix of user-defined and meta-data defined scheduled jobs in your instance. However, if a ScheduledJob record references a process, queue, or table automation with a meta-data defined schedule, the ScheduledJob will NOT be started by ScheduleManager — rather, the meta-data definition will "win".

Example of inserting scheduled jobs records directly into an SQL backend
-- A process:
INSERT INTO scheduled_job (label, scheduler_name, cron_expression, cron_time_zone_id, repeat_seconds, type, is_active) VALUES
   ('myProcess', 'QuartzScheduler', null, null, 300, 'PROCESS', 1);
INSERT INTO scheduled_job_parameter (scheduled_job_id, `key`, value) VALUES
   ((SELECT id FROM scheduled_job WHERE label = 'myProcess'), 'processName', 'myProcess');

-- A table's insert & update automations:
INSERT INTO scheduled_job (label, scheduler_name, cron_expression, cron_time_zone_id, repeat_seconds, type, is_active) VALUES
   ('myTable.PENDING_INSERT_AUTOMATIONS', 'QuartzScheduler', null, null, 15, 'TABLE_AUTOMATIONS', 1),
   ('myTable.PENDING_UPDATE_AUTOMATIONS', 'QuartzScheduler', null, null, 15, 'TABLE_AUTOMATIONS', 1);
INSERT INTO scheduled_job_parameter (scheduled_job_id, `key`, value) VALUES
   ((SELECT id FROM scheduled_job WHERE label = 'myTable.PENDING_INSERT_AUTOMATIONS'), 'tableName', 'myTable'),
   ((SELECT id FROM scheduled_job WHERE label = 'myTable.PENDING_INSERT_AUTOMATIONS'), 'automationStatus', 'PENDING_INSERT_AUTOMATIONS'),
   ((SELECT id FROM scheduled_job WHERE label = 'myTable.PENDING_UPDATE_AUTOMATIONS'), 'tableName', 'myTable'),
   ((SELECT id FROM scheduled_job WHERE label = 'myTable.PENDING_UPDATE_AUTOMATIONS'), 'automationStatus', 'PENDING_UPDATE_AUTOMATIONS');

-- A queue processor:
INSERT INTO scheduled_job (label, scheduler_name, cron_expression, cron_time_zone_id, repeat_seconds, type, is_active) VALUES
   ('mySqsQueue', 'QuartzScheduler', null, null, 60, 'QUEUE_PROCESSOR', 1);
INSERT INTO scheduled_job_parameter (scheduled_job_id, `key`, value) VALUES
   ((SELECT id FROM scheduled_job WHERE label = 'mySqsQueue'), 'queueName', 'mySqsQueue');

Running Scheduled Jobs

In a server running QQQ, if you wish to start running scheduled jobs, you need to initialize the QScheduleManger singleton class, then call its start() method.

Note that internally, this class will check for a system property of qqq.scheduleManager.enabled or an environment variable of QQQ_SCHEDULE_MANAGER_ENABLED, and if the first value found is "false", then the scheduler will not actually run its jobs (but, in the case of the QuartzSchdeuler, it will be available for managing scheduled jobs).

The method QScheduleManager.initInstance requires 2 parameters: Your QInstance, and a Supplier<QSession> lambda, which returns the session that will be used for scheduled jobs when they are executed.

Starting the Schedule Manager service
QScheduleManager.initInstance(qInstance, () -> systemUserSession).start();

Examples

Attach a schedule in meta-data to a Process
QProcessMetaData myProcess = new QProcessMetaData()
   // ...
   .withSchedule(new QScheduleMetaData()
      .withSchedulerName("myScheduler")
      .withDescription("Run myProcess every five minutes")
      .withRepeatSeconds(300))

API server (OpenAPI)

todo

Custom Application Code

QContext

The class QContext contains a collection of thread-local variables, to define the current context of the QQQ code that is currently running. For example, what QInstance (meta-data container) is being used, what QSession (user attributes) is active, etc.

Most of the time, main-line application code does not need to worry about setting up the QContext - although unit-test code can is a common exception to that rule. This is because all of QQQ’s entry-points into execution (e.g., web context handlers, CLI startup methods, schedulers, and multi-threaded executors) take care of initializing the context.

It is more common though, for application code to need to get data from the context, such as the current session or any piece of meta-data from the QInstance. The methods to access data from the QContext are fairly straightforward:

Examples

Get a QTableMetaData from the active QInstance
QTableMeataData myTable = QContext.getQInstance().getTable("myTable");
for(QFieldMeataData field : myTable.getFields().values())
{
   // ...
}
Get a security key value for the current user session
QSession           session   = QContext.getQSession();
List<Serializable> clientIds = session.getSecurityKeyValues("clientId");

QRecords

Almost all code inside a QQQ application will be dealing with Records (aka Tuples or Rows). That is: a collection of named values, representing a single Entity, Fact, or, Row from a QQQ Table.

The class that QQQ uses to work with records is called: QRecord.

Values

At its core, a QRecord is a wrapper around a Map<String, Serializable> values. These are the actual values for the fields in the table for the record. That is, direct representations of the values as they are stored in the QQQ Backend.

The keys in the values map are names from the QQQ Fields in the QQQ Table.

The values in values map are declared as Serializable (to help ensure the serializability of the QRecord as a whole). In practice, their types will be based on the QFieldType of the QQQ Field that they correspond to. That will typically be one of: String, Integer, Boolean, BigDecimal, Instant, LocalDate, LocalTime, or byte[]. Be aware that null values may be in the values map, especially/per if the backend/table support null.

To work with the values map, the following methods are provided:

  • setValue(String fieldName, Serializable value) - Sets a value for the specified field in the record.

    • Overloaded as setValue(String fieldName, Object value) - For some cases where the value may not be known to be Serializable. In this overload, if the value is null or Serializable, the primary version of setValue is called. Otherwise, the value is passed through String::valueOf, and the result is stored.

    • Overloaded as setValue(QFieldMetaData field, Serializable value) - which simply defers to the primary version of setValue, passing field.getName() as the first parameter.

  • removeValue(String fieldName) - Remove the given field from the values map.

    • Note that in some situations this is an important distinction from having a null value in the map (See [UpdateAction)]).

  • setValues(Map<String, Serializable> values) - Sets the full map of values.

  • getValues() - Returns the full map of values.

  • getValue(String fieldName) - Returns the value for the named field - possibly null - as a Serializable.

  • Several type-specific variations of getValueX(String fieldName), where internally, values will be not exactly type-cast, but effectively converted (if possible) to the requested type. These conversions are done using the ValueUtils.getValueAsX(Object) methods. These methods are generally the preferred/cleanest way to get record values in application code, when it is needed in a type-specific way .

    • getValueString(String fieldName)

    • getValueInteger(String fieldName)

    • getValueBoolean(String fieldName)

    • getValueBigDecimal(String fieldName)

    • getValueInstant(String fieldName)

    • getValueLocalDate(String fieldName)

    • getValueLocalTime(String fieldName)

    • getValueByteArray(String fieldName)

Display Values

In addition to the values map, a QRecord contains another map called displayValues, which only stores String values. That is to say, values for other types are stringified, based on their QQQ Field's type and displayFormat property. In addition, fields which have a possibleValueSource property will have their translated values set in the displayValues map.

By default, a QRecord will not have its displayValues populated. To populate displayValues, the QueryAction and GetAction classes take a property in their inputs called shouldGenerateDisplayValues, which must be set to true to generate displayValues. In addition, these two actions also have a property shouldTranslatePossibleValues in their inputs, which needs to be set to true if possible-value lookups are to be performed.

As an alternative to the shouldGenerateDisplayValues and shouldTranslatePossibleValues inputs to QueryAction and GetAction, one can directly call the QValueFormatter.setDisplayValuesInRecords and/or qPossibleValueTranslator.translatePossibleValuesInRecords methods. Or, for special cases, setDisplayValue(String fieldName, String displayValue) or setDisplayValues(Map<String, String> displayValues) can be called directly.

Backend Details

Sometimes a backend may want to place additional data in a QRecord that doesn’t exactly correspond to a field. To do this, the Map<String, Serializable> backendDetails member is used.

For example, an API backend may store the full JSON String that came from the API as a backend detail in a QRecord. Or fields that are marked as isHeavy, if the full (heavy) value of the field hasn’t been fetched, then the lengths of any such heavy fields may be stored in backendDetails.

Errors and Warnings

todo

Associated Records

todo

QRecordEntities

While QRecords provide a flexible mechanism for passing around record data in a QQQ Application, they have one big disadvantage from the point-of-view of a Java Application: They do not provide a mechanism to ensure compile-time checks of field names or field types.

As such, an alternative mechanism exists, which allows records in a QQQ application to be worked with following a more familiar Java Bean (Entity Bean) like pattern. This mechanism is known as a QRecordEntity.

Specifically speaking, QRecordEntity is an abstract base class, whose purpose is to be the base class for entity-bean classes. Using reflection, QRecordEntity is able to convert objects back and forth from QRecord to specific entity-bean subtypes.

For example, the method QRecordEntity::toQRecord() converts a record entity object into a QRecord.

Inversely, QRecordEntity::populateFromQRecord(QRecord record) sets fields in a record entity object, based on the values in the supplied QRecord. It is conventional for a subclass of QRecordEntity to have both a no-arg (default) constructor, and a constructor that takes a QRecord as a parameter, and calls populateFromQRecord.

In addition to these constructors, a QRecordEntity subclass will generally contain:

  • A public static final String TABLE_NAME, used throughout the application as a constant reference to the name for the QQQ Table.

  • A series of private fields, corresponding to the fields in the table that the entity represents.

    • If these fields are annotated as @QField(), then the QQQ Table meta-data for the table that the entity represents can in part be inferred by QQQ, by using the method QTableMetaData::withFieldsFromEntity.

  • getX(), setX(), and withX() methods for all of the entity’s fields.

Examples

Example Definition of a QRecordEntity subclass: Person.java
/*******************************************************************************
 ** QRecordEntity for the person table.
 *******************************************************************************/
public class Person extends QRecordEntity
{
   public static final String TABLE_NAME = "person";

   @QField(isEditable = false)
   private Integer id;

   @QField()
   private String firstName;

   @QField()
   private String lastName;

   @QField()
   private Integer age;

   // all other fields

   /*******************************************************************************
    ** Default constructor
    *******************************************************************************/
   public Person()
   {
   }


   /*******************************************************************************
    ** Constructor that takes a QRecord
    *******************************************************************************/
   public Person(QRecord record)
   {
      populateFromQRecord(record);
   }


   /*******************************************************************************
    ** Custom method to concatenate firstName and lastName
    *******************************************************************************/
   public String getFullName()
   {
      //////////////////////////////////////////////////////////////////////
      // if there were more than an example, we could do some null-checks //
      // here to avoid silly output like "Bobby null" :)                  //
      //////////////////////////////////////////////////////////////////////
      return (firstName + " " + lastName);
   }

   // all getter, setter, and fluent setter (wither) methods
}

The core ORM actions and process-execution actions of QQQ work with QRecords. However, application engineers may want to apply patterns like the following example, to gain the compile-time safety of QRecordEntities:

Example Usage of a QRecordEntity
//////////////////////////////////////////////////////////////////////
// assume we're working with the "person" table & entity from above //
//////////////////////////////////////////////////////////////////////
List<QRecord> recordsToUpdate = new ArrayList<>();
for(QRecord record : inputRecordList)
{
   /////////////////////////////////////////////////////////////////////////////
   // call the constructor that copies values from the record into the entity //
   /////////////////////////////////////////////////////////////////////////////
   Person person = new Person(record);

   ////////////////////////////////////////////////
   // call a custom method defined in the entity //
   ////////////////////////////////////////////////
   LOG.info("Processing: " + person.getFullName());

   /////////////////////////////////////////////////////////////
   // age is an Integer, so no type-conversion is needed here //
   /////////////////////////////////////////////////////////////
   person.setAge(person.getAge() + 1);

   ///////////////////////////////////////////////////////////////////////////
   // to pass the updated records to UpdateAction, convert them to QRecords //
   ///////////////////////////////////////////////////////////////////////////
   recordsToUpdate.add(person.toQRecord());
}

Process Backend Steps

In many QQQ applications, much of the code that engineers write will take the form of Backend Steps for QQQ Processes. Such code is defined in classes which implement the interface BackendStep. This interface defines only a single method:

BackendStep.java
public interface BackendStep
{
   /*******************************************************************************
    ** Execute the backend step - using the request as input, and the result as output.
    **
    *******************************************************************************/
   void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput) throws QException;
}

Process backend steps have access to state information - specifically, a list of records, and a map of name=value pairs - in the input & output objects. This state data is persisted by QQQ between steps (e.g., if a frontend step is presented to a user between backend steps).

RunBackendStepInput

All input data to the step is available in the RunBackendStepInput object. Key methods in this class are:

  • getRecords() - Returns the List of QRecords that are currently being acted on in the process.

  • getValues() - Returns a Map of String → Serializable; name=value pairs that are the state-data of the process.

    • Values can be added to this state from a process’s meta-data, from a screen, or from another backend step.

  • getValue(String fieldName) - Returns a specific value, by name, from the process’s state.

    • This method has several variations that return the value as a specific type, such as getValueString, getValueInteger, getValueBoolean…​

  • getAsyncJobCallback() - Accessor for an AsyncJobCallback object, which provides a way for the process backend step to communicate about its status or progress with a user running the process in a frontend. Provides methods:

    • updateStatus(String message) - Where general status messages can be given. For example, "Loading census data"

    • updateStatus(int current, int total) - For updating a progress meter. e.g., "47 of 1701" would be display by calling .updateStatus(47, 1701)

  • getFrontendStepBehavior() - Enum, indicating what should happen when a frontend step is encountered as the process’s next step to run. Possible values are:

    • BREAK - Indicates that the process’s execution should be suspended, so that the screen represented by the frontend step can be presented to a user. This would be the expected behavior if a process is being run by a user from a UI.

    • SKIP - Indicates that frontend steps should be skipped. This would be the expected behavior if a process is running from a scheduled job (without a user present to drive it), for example.

    • FAIL - Indicates that the process should end with an exception if a frontend step is encountered.

    • A backend step may want to act differently based on its frontendStepBehavior. For example, additional data may be looked up for displaying to a user if the behavior is BREAK.

  • getBasepullLastRunTime() - For Basepull processes, this is the Instant stored in the basepull table as the process’s last run time.

RunBackendStepOutput

All output from a process step should be placed in its RunBackendStepOutput object (and/or stored to a backend, as appropriate). Key methods in this class are:

  • addValue(String fieldName, Serializable value) - Adds a single named value to the process’s state, overwriting it the value if it already exists.

  • addRecord(QRecord record) - Add a [QRecord] to the process’s output.

  • addAuditSingleInput(AuditSingleInput auditSingleInput) - Add a new entry to the process’s list of audit inputs, to be stored at the completion of the process.

    • An AuditSingleInput object can most easily be built with the constructor: AuditSingleInput(String tableName, QRecord record, String auditMessage).

    • Additional audit details messages (sub-bullets that accompany the primary auditMessage) can be added to an AuditSingleInput via the addDetail(String message) method.

    • Note that at this time, the automatic storing of these audits is only provided by the execute step of a StreamedETLWithFrontendProcesses.

Example

Example of a BackendStep
/*******************************************************************************
 ** For the "person" table's "Add Age" process -
 ** For each input person record, add the specified yearsToAdd to their age.
 *******************************************************************************/
public class AddAge implements BackendStep
{
   /*******************************************************************************
    **
    *******************************************************************************/
   @Override
   public void run(RunBackendStepInput runBackendStepInput, RunBackendStepOutput runBackendStepOutput)
   {
      /////////////////////////////////////////////////////////////////
      // get the yearsToAdd input field value from the process input //
      /////////////////////////////////////////////////////////////////
      Integer yearsToAdd = runBackendStepInput.getValueInteger("yearsToAdd");
      int totalYearsAdded = 0;

      ///////////////////////////////////////////////////
      // loop over the records passed into the process //
      ///////////////////////////////////////////////////
      for(QRecord record : runBackendStepInput.getRecords())
      {
         Integer age = record.getValueInteger("age");
         age += yearsToAdd;
         totalYearsAdded += yearsToAdd;

         ////////////////////////////////////////////////////////////////////////////////////////////
         // update the record with the new "age" value.                                            //
         // note that this update record object will implicitly be available to the process's next //
         // backend step, via the sharing of the processState object.                              //
         ////////////////////////////////////////////////////////////////////////////////////////////
         record.setValue("age", age);
      }

      /////////////////////////////////////////
      // set an output value for the process //
      /////////////////////////////////////////
      runBackendStepOutput.addValue("totalYearsAdded", totalYearsAdded);
   }
}

Backend Steps for StreamedETLWithFrontendProcesses

For StreamedETLWithFrontendProcess type processes, backend steps are defined a little bit differently than they are for other process types. In this type of process, the process meta-data defines 3 backend steps which are built-in to QQQ, and which do not have any custom application logic. These steps are:

  • StreamedETLPreviewStep

  • StreamedETLValidateStep

  • StreamedETLExecuteStep

For custom application logic to be implemented in a StreamedETLWithFrontendProcesses, an application engineer must define (up to) 3 backend step classes which are loaded by the steps listed above. These application-defined steps must extend specific classes (which themselves implement the BackendStep interface), to provide the needed logic of this style of process. These steps are:

  • Extract - a subclass of AbstractExtractStep - is responsible for Extracting records from the source table.

    • For this step, we can often use the QQQ-provided class ExtractViaQueryStep, or sometimes a subclass of it.

    • The Extract step is called before the Preview, Validate, and Result screens, though for the Preview screen, it is set to only extract a small number of records (10).

  • Transform - a subclass of AbstractTransformStep - is responsible for applying the majority of the business logic of the process. In ETL terminology, this is the "Transform" action - which means applying some type of logical transformation an input record (found by the Extract step) to generate an output record (stored by the Load step).

    • A Transform step’s runOnePage method will be called, potentially, multiple times, each time with a page of records in the runBackendStepInput parameter.

    • This method is responsible for adding records to the runBackendStepOutput, which will then be passed to the Load step.

    • This class is also responsible for implementing the method getProcessSummary, which provides the data to the Validate screen.

    • The runOnePage method will generally update ProcessSummaryLine objects to facilitate this functionality.

    • The Transform step is called before the Preview, Validate, and Result screens, consuming all records selected by the Extract step.

  • Load - a subclass of AbstractLoadStep - is responsible for the Load function of the ETL job. A quick word on terminology - this step is actually doing what we are more likely to think of as storing data - which feels like the opposite of “loading” - but we use the name Load to keep in line with the ETL naming convention…

    • The Load step is ONLY called before the Result screen is presented (possibly after Preview, if the user chose to skip validation, otherwise, after validation).

    • Similar to the Transform step, the Load step’s runOnePage method will be called potentially multiple times, with pages of records in its input.

    • As such, the Load step is generally the only step where data writes should occur.

      • e.g., a Transform step should not do any writes, as it will be called when the user is going to the Preview & Validate screens - e.g., before the user confirmed that they want to execute the action!

    • A common pattern is that the Load step just needs to insert or update the list of records output by the Transform step, in which case the QQQ-provided LoadViaInsertStep or LoadViaUpdateStep can be used, but custom use-cases can be built as well.

Another distinction between StreamedELTWithFrontendProcess steps and general QQQ process backend steps, is that the list of records in the input & output objects is NOT shared for StreamedELTWithFrontendProcess steps. The direct implication of this is, that a Transform step MUST explicitly call output.addRecord() for any records that it wants to pass along to the Load step.

Example
Examples of a Transform and Load step for a StreamedELTWithFrontendProcess
   // todo!

todo: more details on these 3 specialized types of process steps (e.g., method to overload, when stuff like pre-action is called; how summaries work).

Rendering Widgets

WidgetRenderer classes

In general, to fully implement a Widget, you must define its QWidgetMetaData, and supply a subclass of AbstractWidgetRenderer, to provide the data to the widget. (Note the "No Code" category of widgets, which are an exception to this generalization).

The only method required in a subclass of AbstractWidgetRenderer is:

public RenderWidgetOutput render(RenderWidgetInput input) throws QException

The fields available in RenderWidgetInput are:

  • Map<String, String> queryParams - These are parameters supplied by the frontend, for example, if a user selected values from dropdowns to control a dimension of your widget, those name/value pairs would be in this map. Similarly, if your widget is being included on a record view screen, then the record’s primary key will be in this map.

  • QWidgetMetaDataInterface widgetMetaData - This is the meta-data for the widget being rendered. This can be useful in case you are using the same renderer class for multiple widgets.

The only field in RenderWidgetOutput is:

  • QWidgetData widgetData - This is a base class, with several attributes, and more importantly, several subclasses, specific to the type of widget that is being rendered.

Widget-Type Specific Rendering Details

Different widget types expect & require different types of values to be set in the RenderWidgetOutput by their renderers.

Pie Chart

The WidgetType.PIE_CHART requires an object of type ChartData. The fields on this type are:

  • chartData an instance of ChartData.Data, which has the following fields:

    • labels - List<String>, required - the labels for the slices of the pie.

    • datasets - List<Dataset> required - the data for each slice of the pie. For a Pie chart, only a single entry in this list is used (other chart types using ChartData may support more than 1 entry in this list). Fields in this object are:

      • label - String, required - a label to describe the dataset as a whole. e.g., "Orders" for a pie showing orders of different statuses.

      • data - List<Number>, required - the data points for each slice of the pie.

      • color - String - HTML color for the slice

      • urls - List<String> - Optional URLs for slices of the pie to link to when clicked.

      • backgroundColors - List<String> - Optional HTML color codes for each slice of the pie.

Pie chart widget example
// meta data
new QWidgetMetaData()
    .withName("pieChartExample")
    .withType(WidgetType.PIE_CHART.getType())
    .withGridColumns(4)
    .withIsCard(true)
    .withLabel("Pie Chart Example")
    .withCodeReference(new QCodeReference(PieChartExampleRenderer.class));

// renderer
private List<String> labels = new ArrayList<>();
private List<String> colors = new ArrayList<>();
private List<Number> data   = new ArrayList<>();

/*******************************************************************************
** helper method - to add values for a slice to the lists
*******************************************************************************/
private void addSlice(String label, String color, Number datum)
{
    labels.add(label);
    colors.add(color);
    data.add(datum);
}

/*******************************************************************************
** main method of the widget renderer
*******************************************************************************/
@Override
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
   addSlice("Apple", "#FF0000", 100);
   addSlice("Orange", "#FF8000", 150);
   addSlice("Banana", "#FFFF00", 75);
   addSlice("Lime", "#00FF00", 100);
   addSlice("Blueberry", "#0000FF", 200);

   ChartData chartData = new ChartData()
      .withChartData(new ChartData.Data()
         .withLabels(labels)
         .withDatasets(List.of(
            new ChartData.Data.Dataset()
               .withLabel("Flavor")
               .withData(data)
               .withBackgroundColors(colors)
               .withUrls(urls))));

   return (new RenderWidgetOutput(chartData));
}
Bar Chart

todo

Stacked Bar Chart

todo

Horizontal Bar Chart

todo

Child Record List

todo

Line Chart

todo

Small Line Chart

todo

Statistics

todo

Parent Widget

todo

Composite

A WidgetType.COMPOSITE is built by using one or more smaller elements, known as Blocks. Note that Blocks can also be used in the data of some other widget types (specifically, within a cell of a Table-type widget, or (in the future?) as a header above a pie or bar chart).

A composite widget renderer must return data of type CompositeWidgetData, which has the following fields:

  • blocks - List<AbstractBlockWidgetData>, required - The blocks (1 or more) being composited together to make the widget. See below for details on the specific Block types.

  • styleOverrides - Map<String, Serializable> - Optional map of CSS attributes (named following javascriptStyleCamelCase) to apply to the <div> element that wraps the rendered blocks.

  • layout - Layout enum - Optional specifier for how the blocks should be laid out. e.g., predefined sets of CSS attributes to achieve specific layouts.

    • Note that some blocks are designed to work within composites with specific layouts. Look for matching names, such as Layout.BADGES_WRAPPER to go with NumberIconBadgeBlock.

Composite widget example - consisting of 3 Progress Bar Blocks, and one Divider Block
// meta data
new QWidgetMetaData()
    .withName("compositeExample")
    .withType(WidgetType.COMPOSITE.getType())
    .withGridColumns(4)
    .withIsCard(true)
    .withLabel("Composite Example")
    .withCodeReference(new QCodeReference(CompositeExampleRenderer.class));

// renderer
public RenderWidgetOutput render(RenderWidgetInput input) throws QException
{
   CompositeWidgetData data = new CompositeWidgetData();

   data.addBlock(new ProgressBarBlockData()
      .withValues(new ProgressBarValues()
         .withHeading("Blocks")
         .withPercent(new BigDecimal("78.5"))));

   data.addBlock(new ProgressBarBlockData()
      .withValues(new ProgressBarValues()
         .withHeading("Progress")
         .withPercent(new BigDecimal(0))));

   data.addBlock(new DividerBlockData());

   data.addBlock(new ProgressBarBlockData()
      .withStyles(new ProgressBarStyles().withBarColor("#C0C000"))
      .withValues(new ProgressBarValues()
         .withHeading("Custom Color")
         .withPercent(new BigDecimal("75.3"))));

   return (new RenderWidgetOutput(data));
}
Table

todo

HTML

todo

Divider

todo

Process

todo

Stepper

todo

Data Bag Viewer

todo

Script Viewer

todo

Block-type Specific Rendering Details

For Composite-type widgets (or other widgets which can include blocks), there are specific data classes required to be returned by the widget renderer.

Each block type defines a subclass of AbstractBlockWidgetData, which is a generic class with 3 type parameters:

  • V - an implementation of BlockValuesInterface - to define the type of values that the block uses.

  • S - an implementation of BlockSlotsInterface (expected to be an enum) - to define the "slots" in the block, that can have Tooltips and/or Links applied to them.

  • SX - an implementation of BlockStylesInterface - to define the types of style customizations that the block supports.

These type parameters are designed to ensure type-safety for the application developer, to ensure that only

Additional Tips

  • To make a Dashboard page (e.g., an App consisting of Widgets) with a parent widget use the parent widget’s label as the page’s label:

    • On the QAppMetaData that contains the Parent widget, call .withSupplementalMetaData(new MaterialDashboardAppMetaData().withShowAppLabelOnHomeScreen(false)).

    • In the Parent widget’s renderer, on the ParentWidgetData, call setLabel("My Label") and setIsLabelPageTitle(true).

Table Customizers

todo

QQQ Core Actions

QueryAction

The QueryAction is the basic action that is used to get records from a QQQ Table, generally according to a Filter. In SQL/RDBMS terms, it is analogous to a SELECT statement, where 0 or more records may be found and returned.

Examples

Basic Form
QueryInput input = new QueryInput();
input.setTableName("orders");
input.setFilter(new QQueryFilter()
   .withCriteria(new QFilterCriteria("total", GREATER_THAN, new BigDecimal("3.50")))
   .withOrderBy(new QFilterOrderBy("orderDate", false)));
QueryOutput output = new QueryAction.execute(input);
List<QRecord> records = output.getRecords();

Details

QueryAction, in general, can be called in two different modes:

  1. The most common use-case case, and default, fetches all records synchronously, does any post-processing (as requested in the QueryInput), and returns all records as a list in the QueryOutput).

  2. The alternative use-case is meant for larger operations, where one wouldn’t want all records matching a query in-memory. For this scenario, a RecordPipe object can be passed in to the QueryInput. This causes QueryAction to run its post-processing action on records as they are placed into the pipe, and to potentially block (per the pipe’s settings). This method of usage needs to be done on a separate thread from another thread which would be consuming records from the pipe. QQQ’s AsyncRecordPipeLoop class provides an implementation of doing such a dual-threaded job.

If the QQQ Table has a POST_QUERY_CUSTOMIZER defined, then after records are fetched from the backend, that code is executed on the records before they leave the QueryAction (either through its QueryOutput or RecordPipe).

QueryInput

  • table - String, Required - Name of the table being queried against.

  • filter - QQueryFilter object - Specification for what records should be returned, based on QFilterCriteria objects, and how they should be sorted, based on QFilterOrderBy objects. If a filter is not given, then all rows in the table will be returned by the query.

  • transaction - QBackendTransaction object - Optional transaction object.

    • Behavior for this object is backend-dependant. In an RDBMS backend, this object is generally needed if you want your query to see data that may have been modified within the same transaction.

  • recordPipe - RecordPipe object - Optional pipe object that records are placed into, for asynchronous processing.

    • If a recordPipe is used, then records cannot be retrieved from the QueryOutput. Rather, such records must be read from the pipe’s consumeAvailableRecords() method.

    • A recordPipe should only be used when a QueryAction is running in a separate Thread from the record’s consumer.

  • shouldTranslatePossibleValues - boolean, default: false - Controls whether any fields in the table with a possibleValueSource assigned to them should have those possible values looked up (e.g., to provide text translations in the generated records' displayValues map).

    • For example, if running a query to present results to a user, this would generally need to be true. But if running a query to provide data as part of a process, then this can generally be left as false.

  • shouldGenerateDisplayValues - boolean, default: false - Controls whether field level displayFormats should be used to populate the generated records' displayValues map.

    • For example, if running a query to present results to a user, this would generally need to be true. But if running a query to provide data as part of a process, then this can generally be left as false.

  • shouldFetchHeavyFields - boolean, default: true - Controls whether or not fields marked as isHeavy should be fetched & returned or not.

  • shouldOmitHiddenFields - boolean, default: true - Controls whether or not fields marked as isHidden should be included in the result or not.

  • shouldMaskPassword - boolean, default: true - Controls whether or not fields with type = PASSWORD should be masked, or if their actual values should be returned.

  • queryJoins - List of QueryJoin objects - Optional list of tables to be joined with the main table being queried. See QueryJoin below for further details.

  • fieldNamesToInclude - Set of String - Optional set of field names to be included in the records.

    • Fields from a queryJoin must be prefixed by the join table’s name or alias, and a period. Field names from the table being queried should not have any sort of prefix.

    • A null set here (default) means to include all fields from the table and any queryJoins set as select=true.

    • An empty set will cause an error, as well any unrecognized field names.

    • QueryAction will validate the set of field names, and throw an exception if any unrecognized names are given.

    • Note that this is an optional feature, which some backend modules may not implement. Meaning, they would always return all fields.

QQueryFilter

A key component of QueryInput, a QQueryFilter defines both what records should be included in a query’s results (e.g., an SQL WHERE), as well as how those results should be sorted (SQL ORDER BY).

  • criteria - List of QFilterCriteria - Individual conditions or clauses to filter records. They are combined using the booleanOperator specified in the QQueryFilter. See below for further details.

  • orderBys - List of QFilterOrderBy - List of fields (and directions) to control the sorting of query results. In general, multiple orderBys can be given (depending on backend implementations).

  • booleanOperator - Enum of AND, OR, default: AND - Specifies the logical joining operator used among individual criteria.

  • subFilters - List of QQueryFilter - To build arbitrarily complex queries, with nested boolean logic, 0 or more subFilters may be provided.

    • Each subFilter can include its own additional subFilters.

    • Each subFilter can specify a different booleanOperator.

    • For example, consider the following QQueryFilter, that uses two subFilters, and a mix of booleanOperators

  • skip - Integer - Optional number of records to be skipped at the beginning of the result set. e.g., for implementing pagination.

  • limit - Integer - Optional maximum number of records to be returned by the query.

 queryInput.setFilter(new QQueryFilter()
    .withBooleanOperator(OR)
    .withSubFilters(List.of(
       new QQueryFilter().withBooleanOperator(AND)
          .withCriteria(new QFilterCriteria("firstName", EQUALS, "James"))
          .withCriteria(new QFilterCriteria("lastName", EQUALS, "Maes")),
       new QQueryFilter().withBooleanOperator(AND)
          .withCriteria(new QFilterCriteria("firstName", EQUALS, "Darin"))
          .withCriteria(new QFilterCriteria("lastName", EQUALS, "Kelkhoff"))
    )));

// which would generate the following WHERE clause in an RDBMS backend:
// WHERE (first_name='James' AND last_name='Maes') OR (first_name='Darin' AND last_name='Kelkhoff')
QFilterCriteria

  • fieldName - String, required - Reference to a field on the table being queried.

    • Or, in the case of a query with queryJoins, a qualified name of a field from a join-table (where the qualifier would be the joined table’s name or alias, followed by a dot)

      • For example: orderLine.sku or orderBillToCustomer.firstName

  • operator - Enum of QCriteriaOperator, required - Comparison operation to be applied to the field specified as fieldName and the values or otherFieldName.

    • e.g., EQUALS, NOT_IN, GREATER_THAN, BETWEEN, IS_BLANK, etc.

  • values - List of values, conditional - Provides the value(s) that the field is compared against. The number of values (0, 1, 2, or more) required are based on the operator being used. If an otherFieldName is given, and the operator expects 1 value, then values is ignored, and otherFieldName is used.

  • otherFieldName - String, conditional - Specifies that the fieldName should be compared against another field in the records, rather than the values in the values property. Only used for operators that expect 1 value (e.g., EQUALS or LESS_THAN_OR_EQUALS - not IS_NOT_BLANK or IN).

QFilterCriteria definition examples:
// in-line, via constructors that take (List<Serializable> values) or (Serializable... values) as 3rd arg
new QFilterCriteria("id", IN, 1, 2, 3)
new QFilterCriteria("name", IS_BLANK)
new QFilterCriteria("orderNo", IN, orderNoList)
new QFilterCriteria("state", EQUALS, "MO")

// long-form, with fluent setters
new QFilterCriteria()
   .withFieldName("quantity")
   .withOpeartor(QCriteriaOperator.GREATER_THAN)
   .withValues(List.of(47));

// to use otherFieldName, long-form must be used
new QFilterCriteria()
   .withFieldName("firstName")
   .withOpeartor(QCriteriaOperator.EQUALS)
   .withOtherFieldName("lastName");

// using otherFieldName to build a criterion that looks at two fields from two different join tables
new QFilterCriteria()
   .withFieldName("billToCustomer.lastName")
   .withOpeartor(QCriteriaOperator.NOT_EQUALS)
   .withOtherFieldName("shipToCustomer.lastName");
QFilterOrderBy

  • fieldName - String, required - Reference to a field on the table being queried.

    • Or, in the case of a query with queryJoins, a qualified name of a field from a join-table (where the qualifier would be the joined table’s name or alias, followed by a dot)

  • isAscending - boolean, default: true - Specify if the sort is ascending or descending.

QFilterOrderBy definition examples:
// short-form, via constructors
new QFilterOrderBy("id") // isAscending defaults to true.
new QFilterOrderBy("name", false)

// long-form, with fluent setters
new QFilterOrderBy()
   .withFieldName("birthDate")
   .withIsAscending(false);
QueryJoin

  • joinTable - String, required (though inferrable) - Name of the table that is being joined in to the existing query.

    • Will be inferred from joinMetaData, if joinTable is not set when joinMetaData gets set.

  • baseTableOrAlias - String, required (though inferrable) - Name of a table (or an alias) already defined in the query, to which the joinTable will be joined.

    • Will be inferred from joinMetaData, if baseTableOrAlias is not set when joinMetaData gets set (which will only use the leftTableName from the joinMetaData - never an alias).

  • joinMetaData - QJoinMetaData object - Optional specification of a QQQ Join in the current QInstance. If not set, will be looked up at runtime based on baseTableOrAlias and joinTable.

    • If set before baseTableOrAlias and joinTable, then they will be set based on the leftTable and rightTable in this object.

  • alias - String - Optional (unless multiple instances of the same table are being joined together, when it becomes required). Behavior based on SQL FROM clause aliases. If given, must be used as the part before the dot in field name specifications throughout the rest of the query input.

  • select - boolean, default: false - Specify whether fields from the joinTable should be selected by the query. If true, then the QRecord objects returned by this query will have values with corresponding to the (table-or-alias . field-name) form.

  • type - Enum of INNER, LEFT, RIGHT, FULL, default: INNER - specifies the SQL-style type of join being performed.

Basic QueryJoin usage example:
// selecting from an "orderLine" table, joined to its corresponding (parent) "order" table
queryInput.withTableName("orderLine");
queryInput.withQueryJoin(new QueryJoin("order").withSelect(true));
...
queryOutput.getRecords().get(0).getValueBigDecimal("order.grandTotal");
"V" shaped query - selecting from one parent table, and two children joined to it:
// TODO this needs verified for accuracy, though is a reasonable starting point as-is
// selecting from an "order" table, and two children of it, orderLine and customer
queryInput.withTableName("order");
queryInput.withQueryJoin(new QueryJoin("orderLine").withSelect(true));
queryInput.withQueryJoin(new QueryJoin("customer").withSelect(true));
...
QRecord joinedRecord = queryOutput.getRecords().get(0);
joinedRecord.getValueString("orderNo");
joinedRecord.getValueString("orderLine.sku");
joinedRecord.getValueString("customer.firstName");
"Chain" shaped query - selecting from one parent table, a child table, and a grandchild:
// TODO this needs verified for accuracy, though is a reasonable starting point as-is
// selecting from an "order" table, with a "customer" child table, and an "address" sub-table
queryInput.withTableName("order");
queryInput.withQueryJoin(new QueryJoin("customer").withSelect(true));
queryInput.withQueryJoin(new QueryJoin("address").withSelect(true));
...
QRecord joinedRecord = queryOutput.getRecords().get(0);
joinedRecord.getValueString("orderNo");
joinedRecord.getValueString("customer.firstName");
joinedRecord.getValueString("address.street1");
QueryJoin usage example where two tables have two different joins between them:
// TODO this needs verified for accuracy, though is a reasonable starting point as-is
// here there's a "fulfillmentPlan" table, which points at the order table (many-to-one,
// as an order's plan can change over time, and we keep old plans around).
// This join is named:  fulfillmentPlanJoinOrder
//
// The other join is "order" pointing at its current "fulfillmentPlan"
// This join is named:  orderJoinCurrentFulfillmentPlan

// to select an order along with its current fulfillment plan:
queryInput.withTableName("order");
queryInput.withQueryJoin(new QueryJoin(instance.getJoin("orderJoinCurrentFulfillmentPlan"))
   .withSelect(true));

// to select an order, and all fulfillment plans for an order (1 or more records):
queryInput.withTableName("order");
queryInput.withQueryJoin(new QueryJoin(instance.getJoin("fulfillmentPlanJoinOrder"))
   .withSelect(true));
QueryJoin usage example for table with two joins to the same child table, selecting from both:
// given an "order" table with 2 foreign keys to a customer table (billToCustomerId and shipToCustomerId)
// Note, we must supply the JoinMetaData to the QueryJoin, to drive what fields to join on in each case.
// we must also define an alias for each of the QueryJoins
queryInput.withTableName("order");
queryInput.withQueryJoins(List.of(
   new QueryJoin(instance.getJoin("orderJoinShipToCustomer")
       .withAlias("shipToCustomer")
       .withSelect(true)),
   new QueryJoin(instance.getJoin("orderJoinBillToCustomer")
       .withAlias("billToCustomer")
       .withSelect(true))));
...
record.getValueString("billToCustomer.firstName")
   + " paid for an order, to be sent to "
   + record.getValueString("shipToCustomer.firstName")
Implicit QueryJoin, where unambiguous and required by QQueryFilter
// TODO finish and verify
queryInput.withTableName("order");

QueryOutput

  • records - List of QRecord - List of 0 or more records that match the query filter.

    • Note: If a recordPipe was supplied to the QueryInput, then calling queryOutput.getRecords() will result in an IllegalStateException being thrown - as the records were placed into the pipe as they were fetched, and cannot all be accessed as a single list.

GetAction

The GetAction is essentially a subset of the QueryAction, only specifically meant to get just a single record from a QQQ Table. In SQL/RDBMS terms, it is analogous to a SELECT statement, where a single record may be found - or - it may not be found.

For all tables, GetAction can do a lookup by primary key. In addition, for tables that define a UniqueKey, name/value pairs (in the form of a Map) can be used as input for GetAction.

Examples

Basic form - by primary key
GetInput input = new GetInput();
input.setTableName("orders");
input.setPrimaryKey(1);
GetOutput output = new GetAction.execute(input);
QRecord record = output.getRecord();
Secondary form - by Unique Key
GetInput input = new GetInput();
input.setTableName("products");
input.setUniqueKey(Map.of("storeId", 1701, "sku", "ABCD"));
GetOutput output = new GetAction.execute(input);
QRecord record = output.getRecord();

GetInput

  • table - String, Required - Name of the table being queried against.

  • primaryKey - Serializable, Conditional - Value for the primary key field of the table being queried. Type should match the table’s primary key field’s type. If a primaryKey is not given, then a uniqueKey must be given.

  • uniqueKey - Map of String → Serializable, Conditional - Map of name-value pairs that define the record to be fetcheed. Keys in the map must be field names from the table being queried. Values in the map should should be of types that correspond to the fields. If a primaryKey is not given, then a uniqueKey must be given. If both primaryKey and uniqueKey are given, then uniqueKey is ignored.

  • transaction - QBackendTransaction object - Optional transaction object.

    • Behavior for this object is backend-dependant. In an RDBMS backend, this object is generally needed if you want your query to see data that may have been modified within the same transaction.

  • shouldTranslatePossibleValues - boolean, default: false - Controls whether any fields in the table with a possibleValueSource assigned to them should have those possible values looked up (e.g., to provide text translations in the generated records' displayValues map).

    • For example, if getting a record to present to a user, this would generally need to be true. But if getting a record as part of a process, then this can generally be left as false.

  • shouldGenerateDisplayValues - boolean, default: false - Controls whether field level displayFormats should be used to populate the generated records' displayValues map.

    • For example, if getting a record to present to a user, this would generally need to be true. But if getting a record as part of a process, then this can generally be left as false.

  • shouldFetchHeavyFields - boolean, default: true - Controls whether or not fields marked as isHeavy should be fetched & returned or not.

  • shouldOmitHiddenFields - boolean, default: true - Controls whether or not fields marked as isHidden should be included in the result or not.

  • shouldMaskPassword - boolean, default: true - Controls whether or not fields with type = PASSWORD should be masked, or if their actual values should be returned.

  • queryJoins - List of QueryJoin objects - Optional list of tables to be joined with the main table being queried. See QueryJoin under QueryAction for further details.

  • includeAssociations - boolean, default: false - Control whether or not records from tables defined as associations under the table being queried should be included in the result or not.

  • associationNamesToInclude - Collection of String - If includeAssociations is true, then this field can be used to limit which associated tables are included. If this field is null, then all associated tables are included. Otherwise, a table is only included if its name is in this collection.

GetOutput

  • record - QRecord - The record that was specified by the input primaryKey or uniqueKey. Will be null if the record was not found.

CountAction

todo

AggregateAction

todo

InsertAction

To insert (add, create) new records into any QQQ Table, the InsertAction is used. In SQL/RDBMS terms, it is analogous to a INSERT statement, where one or more records can be provided as input.

Examples

Canonical InsertAction invocation
InsertInput insertInput = new InsertInput();
insertInput.setTableName("person");
insertInput.setRecords(personRecordList);
InsertOutput insertOutput = new InsertAction().execute(insertInput);
List<QRecord> insertedPersonRecords = insertOutput.getRecords();

Details

InsertAction does several things beyond just inserting records into the specified table. A high-level flow of its internal logic is:

  1. For tables using an automation status field, set its value to PENDING_INSERT_AUTOMATIONS for all records that are going to be inserted.

  2. Perform the following validations, which include running the table’s PRE_INSERT_CUSTOMIZER, if one is defined, at the time that is specified by the customizer’s WhenToRun property (default is AFTER_ALL_VALIDATIONS):

    1. Ensure that default values specified in the table’s fields are present if needed.

    2. Apply per-field behaviors, as defined in QQQ Field meta-data, such as truncating strings that are longer than their specified max.

    3. Check for unique key violations (done here instead of in the backend, to provide better error messaging, and to allow a subset of records to be stored while some fail). We might want to make an input control in the future to specify that either the full input set should succeed or fail…​

    4. Validate that required fields (again, per QQQ Field meta-data) are set, generating per-record errors if they are not.

    5. Validate any security fields in the records - e.g., ensure that the user has permission to insert records with the values they are attempting to insert.

  3. Send the records to the table’s backend module to actually insert them into the backend storage.

  4. If the table has any associations defined, and if associated records are present, then recursively run InsertAction on the associated records.

    1. In particular, before these recursive InsertAction calls are made, values that were generated by the original insert may need to be propagated down into the associated records.

      • For example, if inserting order and lineItem records, where a QQQ Join exists between the two tables on order.id and lineItem.orderId, and order.id values were generated in the first InsertAction, then those values are propagated down into the associated lineItem.orderId fields.

  5. If the QInstance has an audit table, then based on the QQQ Table's audit rules, audits about the inserted records are created.

  6. If the table has a POST_INSERT_CUSTOMIZER, it is executed.

Overloads

InsertAction can be called in a few alternate forms, mostly just for convenience:

If inserting a single record, get that record back instead of the InsertOutput:
InsertInput insertInput = new InsertInput();
insertInput.setTableName("person");
insertInput.setRecords(List.of(personRecord));
QRecord insertedRecord = new InsertAction().executeForRecord(insertInput);

// or more compactly, using InsertInput.withRecord (instead of withRecords)
QRecord insertedRecord = new InsertAction()
   .executeForRecord(new InsertInput("person").withRecord(personRecord));
Taking QRecordEntity objects as inputs instead of QRecords:
// insert a list of person entities:
InsertInput insertInput = new InsertInput("person").withRecordEntities(personList);
InsertOutput insertOutput = new InsertAction().execute(insertInput);

// or for a single person entity (also mapping the output record back to an entity):
Person insertedPerson = new Person(new InsertAction()
   .executeForRecord(new InsertInput("person").withRecordEntity(person)));

InsertInput

  • table - String, Required - Name of the table that records are to be inserted into.

  • records - List of QRecord, Required - List of records to be inserted into the table. If the list is empty, the insert action does a noop.

Less common options

  • inputSource - InputSource object, default: QInputSource.SYSTEM - an indicator of what the source of the action is - generally, a SYSTEM triggered action, or a USER triggered action.

    • InsertAction will call the shouldValidateRequiredFields() method on this object to determine if it should validate that required fields on the records have values. Both QInputSource.SYSTEM and QInputSource.USER return true from this method, but application can define their own InputSource objects with different behavior.

    • In addition, this field can be used in pre- and post-insert customizers to drive further custom logic.

  • skipUniqueKeyCheck - boolean, default: false - control whether or not InsertAction should check for unique key violations before attempting to insert its records.

    • In a context where one has previously done this validation, or is okay letting the backend provide such checks, they may wish to avoid re-doing this work, and thus may set this property to true.

  • omitDmlAudit - boolean, default: false - control if the automatic DML audit that InsertAction generally performs should be omitted.

  • auditContext - String - optional message which can be included in DML audit messages, to give users more context about why the insert occurred.

InsertOutput

  • records - List of QRecord - Copy of the input list of records, with details added based on the results of the input action.

    • If there were warnings or errors, the corresponding field (warnings or errors) will be set in the records.

    • If the insert action generated any values (such as a serial id or a default value), those values will be in the record’s fields map.

UpdateAction

todo

DeleteAction

todo

AuditAction

todo

QQQ Default Implementations

TableSyncProcess

The TableSyncProcess is designed to help solve the common use-case where you have a set of records in one table, you want to apply a transformation to them, and then you want them to result in records in a different table.

Sample Scenario

For example, you may be receiving an import file feed from a partner - say, a CSV file of order records - that you need to synchronize into your local database’s orders table.

High Level Steps

At a high-level, the steps of this task are:

  1. Get the records from the partner’s feed.

  2. Decide Insert/Update - e.g., if you already have any of these orders in your database table, in which case, they need updated, versus ones you don’t have, which need inserted.

  3. Map fields from the partner’s order record to your own fields.

  4. Store the necessary records in your database (insert or update).

What you need to tell QQQ

TableSyncProcess, as defined by QQQ, knows how to do everything for a job like this, other than the part that’s unique to your business. Those unique parts that you need to tell QQQ are:

  • What are the source & destination tables?

  • What are the criteria to identify source records to be processed?

  • What are the key fields are that "link" together records from the source & destination tables?

    • For the example above, imagine you have a "partnerOrderNo" field in your database’s order table - then you’d need to tell QQQ what field in the partner’s table provides the value for that field in your table.

  • How do fields / values from the source table map to fields & values in the destination table?

Detailed Steps

To get more specific, in a QQQ TableSyncProcess, here’s what happens for each of the high-level steps given above:

  • Getting records from the partner’s feed (done by QQQ):

    • Records from the source will be fetched via an Extract step, which runs a query to find the records that need processing.

    • Depending on your use-case, you may use an ExtractViaQueryStep (maybe with a Table Automation) or ExtractViaBasepullQueryStep (e.g., if you are polling a remote data source).

  • Deciding Insert/Update (done by QQQ):

    • Given a set of records from the source table (e.g., output of the Extract step mentioned above), get values from the "key" field in that table.

    • Do a lookup in the destination table, where its corresponding "key" field has the values extracted from the source records.

    • For each source record, if its "key" was found in the destination table, then plan an update to that existing corresponding destination record; else, plan to insert a new record in the destination table.

  • Mapping values (done by your custom Application code):

    • Specifically, mapping is done in a subclass of AbstractTableSyncTransformStep, in the populateRecordToStore method. Of particular interest are these two parameters to that method:

      • QRecord destinationRecord - as determined by the insert/update logic above, this will either be a new empty record (e.g., for inserting), or a fully populated record from the destination table (for updating).

      • QRecord sourceRecord - this is the record being processed, from the source table.

    • This method is responsible for setting values in the destinationRecord, and returning that record (unless it has decided that for some reason the record should not be stored, in which case it may return null).

  • Storing the records (done by QQQ):

    • This is typically done with the LoadViaInsertOrUpdateStep, though it is customizable if additional work is needed (e.g., via a subclass of LoadViaInsertOrUpdateStep, or a more custom subclass of AbstractLoadStep).

Bare-bones Example

For this example, let’s assume we’re setting up a partner-order-feed as described above, with the following details:

  • We have records from a partner in a table named "partnerOrderImport" (let’s assume the records may have been created using the QQQ FilesystemImporter process). These records have the following fields:

    • orderNo, date, city, state, postal, whseNo

  • We need to synchronize those records with table in our database named "order", with the following fields corresponding to those from the partner:

    • partnerOrderNo, orderDate, shipToCity, shipToState, shipToZipCode, warehouseId

  • The same conceptual order may appear in the "partnerOrderImport" multiple times, e.g., if they update some data on the order and re-transmit it to us. Meaning, we need to update our "order" records when we receive a new version an existing order.

To use TableSyncProcess for solving this use-case, we’ll need to create 2 things:

  1. A QProcessMetaData object, which we can create using the builder object provided by class TableSyncProcess. Note that this type of process is specialization of the standard QQQ StreamedETLWithFrontendProcess, as described elsewhere in this documentation.

  2. A subclass of AbstractTableSyncTransformStep, where we implement our mapping logic. Again, note that AbstractTableSyncTransformStep is a subclass of AbstractTransformStep, as used by StreamedETLWithFrontendProcess.

And to be good programmers, we’ll actually create a 3rd thing:

  1. A unit test for our Transform step.

Here are examples of these pieces of code:

Example of building process using the TableSyncProcess builder:
// the false argument below tells the build we are not a basepull-style process
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(false)

    // give our process a unique name within our QInstance
    .withName("partnerOrderToLocalOrderProcess")

    // tell the process to what class to use for transforming records from source to destination
    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)

    .getProcessMetaData();
Example implementation of an AbstractTableSyncTransformStep
public class PartnerOrderToOrderTransformStep extends AbstractTableSyncTransformStep
{
   @Override
   protected SyncProcessConfig getSyncProcessConfig()
   {
      return (new SyncProcessConfig(
         "partnerOrderImport", // source tableName
         "orderNo",            // source table key fieldName
         "order",              // destination tableName
         "partnerOrderNo"      // destination table foreign key fieldName
      ));
   }

   @Override
   public QRecord populateRecordToStore(RunBackendStepInput runBackendStepInput, QRecord destinationRecord, QRecord sourceRecord) throws QException
   {
      // map simple values from source table to destination table
      destinationRecord.setValue("orderDate", sourceRecord.get("date"));
      destinationRecord.setValue("shipToAddressCity", sourceRecord.get("city"));
      destinationRecord.setValue("shipToAddressState", sourceRecord.get("state"));
      destinationRecord.setValue("shipToAddressZipCode", sourceRecord.get("postal"));
      return (destinationRecord);
   }

}
Example Unit Test for a transform step
   @Test
   void testTransformStep()
   {
      // insert 1 test order, that will be updated by the transform step
      Integer existingId = new InsertAction().execute(new InsertInput("order").withRecords(List.of(
         new QRecord().withValue("partnerOrderNo", 101).withValue("shipToState", "IL")
      ))).getRecords().get(0).getValueInteger("id");

      // set up input for the step - a list of 2 of the partner's orders
      RunBackendStepInput input = new RunBackendStepInput();
      input.setRecords(List.of(
         new QRecord().withValue("orderNo", 101).withValue("state", "NY"), // will update the order above
         new QRecord().withValue("orderNo", 102).withValue("state", "CA")  // will insert a new order
      ));
      RunBackendStepOutput output = new RunBackendStepOutput();

      // run the code under test - our transform step
      new PartnerOrderToOrderTransformStep().run(input, output);

      // Note that by just running the transform step, no records have been stored.
      // We can assert against the output of this step.

      assertEquals(existingId, output.getRecords().get(0).getValue("id"));
      assertEquals(101, output.getRecords().get(0).getValue("partnerOrderNo"));
      assertEquals("NY", output.getRecords().get(0).getValue("shipToState"));

      assertNull(output.getRecords().get(1).getValue("id"));
      assertEquals(102, output.getRecords().get(1).getValue("partnerOrderNo"));
      assertEquals("CA", output.getRecords().get(1).getValue("shipToState"));
   }

   @Test
   void testFullProcess()
   {
      // todo! :)
   }

Pseudocode process flow

Now that we’ve seen the bare-bones example, let’s see an even more detailed breakdown of how a full TableSyncProcess works, by looking at its 3 "ETL" steps in pseudocode:

ExtractStep (Producer Thread)

  • Queries source table for records

    • Often based on Table Automations (e.g., for all newly inserted records) or Basepull pattern (polling for new/updated records).

TransformStep (Consumer Thread)

  • Receives pages of records from ExtractStep in the run method.

  • Makes sourceKeyList by getting sourceTableKeyField values from the records.

  • Calls initializeRecordLookupHelper(runBackendStepInput, sourceRecordList)

    • Calls getLookupsToPreLoad to control which lookups are performed.

  • Calls getExistingRecordsByForeignKey(runBackendStepInput, destinationTableForeignKeyField, destinationTableName, sourceKeyList);

    • Calls getExistingRecordQueryFilter(runBackendStepInput, sourceKeyList) as part of querying the destinationTable

    • Returns the output of buildExistingRecordsMap(destinationTableForeignKeyField, queryOutput.getRecords())

  • foreach input record (from sourceTable):

    • Calls getExistingRecord(existingRecordsByForeignKey, destinationForeignKeyField, sourceKeyValue)

    • if an existing record was returned (and if the syncConfig says performUpdates), this record is set as recordToStore

    • else if no existing record was returned (and if the syncConfig says performInserts), a new record is set as recordToStore

    • else continue the foreach.

    • call populateRecordToStore(runBackendStepInput, recordToStore, sourceRecord)

    • if a record is returned it is added to the process step output (to be stored in the LoadStep)

LoadStep (Consumer Thread)

  • Receives records from the output of the TransformStep.

  • Inserts and/or Updates destinationTable, with records returned by populateRecordToStore

Additional Process Configuration Examples

The following examples show how to use additional settings in the TableSyncProcess builder.

UI

While a TableSyncProcess will often run via a schedule and/or automation, we may also want to allow users to manually run it in a UI.

Making our process available for a UI
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(true)
    .withName("partnerOrderToLocalOrderProcess")
    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)

    // attach our process to its source table, to show up in UI
    .withTableName("partnerOrderImport")

    // add some fields to display on the review screen, in UI
    .withReviewStepRecordFields(List.of(
       new QFieldMetaData("clientId", QFieldType.STRING).withLabel("Client"),
       new QFieldMetaData("warehouseId", QFieldType.STRING).withLabel("Warehouse"),
       new QFieldMetaData("partnerOrderNo", QFieldType.STRING)))
    .getProcessMetaData();
Basepull

The previous example would work as a Table Automation (e.g., where the list of records identified in the Extract step were determined by the Automation system). However, a second common pattern is to use Basepull (e.g., if polling for updated records from a partner API endpoint).

Configuring our process as a Basepull
// the true argument below tells the build we ARE a basepull-style process
// this changes the default extract-step.
QProcessMetaData processMetaData = TableSyncProcess.processMetaDataBuilder(false)
    .withName("partnerOrderToLocalOrderProcess")
    .withSyncTransformStepClass(PartnerOrderToOrderTransformStep.class)

    // See Basepull documentation for details
    .withBasepullConfiguration(new BasepullConfiguration())

    // schedule our process to run automatically every minute
    .withSchedule(new QScheduleMetaData().withRepeatSeconds(60))

    .getProcessMetaData();

Additional Options in the Transform Step

Specifying to not perform Inserts or not perform Updates

We may have a scenario where we want our sync process to never update records if the key is already found in the destination table. We can configure this with an additional optional parameter to the SyncProcessConfig constructor:

Specifying to not do updates in a TableSyncProcess
   @Override
   protected SyncProcessConfig getSyncProcessConfig()
   {
      return (new SyncProcessConfig("partnerOrderImport", "orderNo", "order", "partnerOrderNo",
         true,  // performInserts
         false  // performUpdates
      ));
   }

Similarly, we may want to disallow inserts from a particular sync process. The performInserts argument to the SyncProcessConfig constructor lets us do that:

Specifying to not do inserts in a TableSyncProcess
   @Override
   protected SyncProcessConfig getSyncProcessConfig()
   {
      return (new SyncProcessConfig("partnerOrderImport", "orderNo", "order", "partnerOrderNo",
         false, // performInserts
         true   // performUpdates
      ));
   }
Customizing the query for existing records

In some cases, a specific Table Sync process may need to refine the query filter that is used to lookup existing records in the destination table (e.g. for determining insert vs. update).

For example, in our orders-from-a-partner scenario, if we have more than 1 partner sending us orders, where there could be overlapping orderNo values among them - we may have an additional field in our orders table to identify which partner an order came from. So then when we’re looking up orders by partnerOrderNo, we would need to also include the partnerId field in our query, so that we only update orders from the specific partner that we’re dealing with.

To do this (to customize the existing record query filter), we need can just override the method getExistingRecordQueryFilter. Generally we would start by calling the super version of the method, and then add to it additional criteria.

Customizing the query filter used to look for existing records
   /*******************************************************************************
    ** Define the query filter to find existing records.  e.g., for determining
    ** insert vs. update.  Subclasses may override this to customize the behavior,
    ** e.g., in case an additional field is needed in the query.
    *******************************************************************************/
   protected QQueryFilter getExistingRecordQueryFilter(RunBackendStepInput runBackendStepInput, List<Serializable> sourceKeyList)
   {
      QQueryFilter filter = super.getExistingRecordQueryFilter(runBackendStepInput, sourceKeyList);
      filter.addCriteria(new QFilterCriteria("partnerId", EQUALS, PARTNER_ID));
      return (filter);
   }
More efficient additional record lookups

It is a common use-case to need to map various ids from a partner’s system to ids in your own system. For the orders example, we might need to know what warehouse the order is shipping from. The customer may send their identifier for the warehouse, and we may need to map those identifiers to our own warehouse ids.

The QQQ-provided class RecordLookupHelper exists to help with performing lookups like this, and in particular, it can be used to execute one query to fetch a full table, storing records by a key field, then returning those records without performing additional queries.

AbstractTableSyncTransformStep has a protected recordLookupHelper member. If we override the method getLookupsToPreLoad(), then this object is populated by calling its preloadRecords method with each specified pair of tableNames and fieldNames.

Specifying tables to pre-load using a RecordLookupHelper
   /*******************************************************************************
    ** Specify a list of tableName/keyColumnName pairs to run through
    ** the preloadRecords method of the recordLookupHelper.
    *******************************************************************************/
   @Override
   protected List<Pair<String, String>> getLookupsToPreLoad()
   {
      return (List.of(
         Pair.of("warehouse", "partnerWarehouseNo")
      ));
   }

If we have preloaded some lookups, we can then use them in our populateRecordToStore method as follows:

Using the recordLookupHelper in populateRecordToStore
   // lookup warehouse with partnerWarehouseNo=whseNo from partner, and use our id in destination record
   String partnerWarehouseNo = sourceRecord.getValue("whseNo");
   Integer warehouseId = recordLookupHelper.getRecordId("warehouse", "partnerWarehouseNo", whseNo, Integer.class);
   destinationRecord.setValue("warehouseId", warehouseId);
Additional override points

There are more methods which can be overridden in your AbstractTableSyncTransformStep subclass, to provide further customizations of behaviors, specifically in the area of dealing with existing records (e.g., the insert/update use-case).

Additional AbstractTableSyncTransformStep overrides
   /*******************************************************************************
    ** Run the existingRecordQueryFilter - to look in the destinationTable for
    ** any records that may need an update (rather than an insert).
    **
    ** Generally returns a Map, keyed by a Pair of the destinationTableForeignKeyField
    ** and the value in that field.  But, for more complex use-cases, one can override
    ** the buildExistingRecordsMap method, to make different keys (e.g., if there are
    ** two possible destinationTableForeignKeyFields).
    *******************************************************************************/
   protected Map<Pair<String, Serializable>, QRecord> getExistingRecordsByForeignKey
   (
      RunBackendStepInput runBackendStepInput,
      String destinationTableForeignKeyField,
      String destinationTableName,
      List<Serializable> sourceKeyList
   ) throws QException;


   /*******************************************************************************
    ** Overridable point where you can, for example, keys in the existingRecordsMap
    ** with different fieldNames from the destinationTable.
    **
    ** Note, if you're overriding this method, you'll likely also want & need to
    ** override getExistingRecord.
    *******************************************************************************/
   protected Map<Pair<String, Serializable>, QRecord> buildExistingRecordsMap
   (
      String destinationTableForeignKeyField,
      List<QRecord> existingRecordList
   );

   /*******************************************************************************
    ** Given the map of existingRecordsByForeignKey (as built by
    ** getExistingRecordsByForeignKey which calls buildExistingRecordsMap),
    ** get one record from that map, for a given key-value from a source record.
    **
    ** The destinationForeignKeyField is given as advice if needed (e.g., to see its type)
    *******************************************************************************/
   protected QRecord getExistingRecord
   (
      Map<Pair<String, Serializable>, QRecord> existingRecordsByForeignKey,
      QFieldMetaData destinationForeignKeyField,
      Serializable sourceKeyValue
   );

QQQ Utility Classes

RecordLookupHelper

RecordLookupHelper is a utility class that exists to help you…​ lookup records :)

OK, I’ll try to give a little more context:

Motivation 1: Performance

One of the most significant performance optimizations that the team behind QQQ has found time and time again, is to minimize the number of times you have to perform an I/O operation. To just say it more plainly: Make fewer calls to your database (or other backend).

This is part of why the DML actions in QQQ (InsertAction, UpdateAction, DeleteAction) are all written to work on multiple records: If you’ve got to insert 1,000 records, the performance difference between doing that as 1,000 SQL INSERT statements vs. just 1 statement cannot be overstated.

Similarly then, for looking up records: If we can do 1 round-trip to the database backend - that is - 1 query to fetch n records, then in almost all cases it will be significantly faster than doing n queries, one-by-one, for those n records.

The primary reason why RecordLookupHelper exists is to help you cut down on the number of times you have to make a round-trip to a backend data store to fetch records within a process.

This basically is version of caching, isn’t it? Take a set of data from "far away" (e.g., database), and bring it "closer" (local or instance variables), for faster access. So we may describe this as a "cache" through the rest of this document.

Motivation 2: Convenience

So, given that one wants to try to minimize the number of queries being executed to look up data in a QQQ processes, one can certainly do this "by-hand" in each process that they write.

Doing this kind of record caching in a QQQ Process BackendStep may be done as:

  • Adding a Map<Integer, QRecord> as a field in your class.

  • Setting up and running a QueryAction, with a filter based on the collection of the keys you need to look up, then iterating over (or streaming) the results into the map field.

  • Getting values out of the map when you need to use them (dealing with missing values as needed).

That’s not so bad, but, it does get a little verbose, especially if you’re going to have several such caches in your class.

As such, the second reason that RecordLookupHelper exists, is to be a reusable and convenient way to do this kind of optimization, by providing methods to perform the bulk query & map building operation described above, while also providing some convenient methods for accessing such data after it’s been fetched. In addition, a single instance of RecordLookupHelper can provide this service for multiple tables at once (e.g., so you don’t need to add a field to your class for each type of data that you’re trying to cache).

Use Cases

Preload records, then access them

Scenario:

  • We’re writing a process BackendStep that uses shipment records as input.

  • We need to know the order record associated with each shipment (via an orderId foreign key), for some business logic that isn’t germaine to the explanation of RecordLookupHelper.

  • We also to access some field on the shippingPartner record assigned to each shipment.

    • Note that here, the shipment table has a partnerCode field, which relates to the code unique-key in the shippingPartner table.

    • It’s also worth mentioning, we only have a handful of shippingPartner records in our database, and we never expect to have very many more than that.

Example of a process step using a RecordLookupHelper to preload records
public class MyShipmentProcessStep implements BackendStep
{
   // Add a new RecordLookupHelper field, which will "cache" both orders and shippingPartners
   private RecordLookupHelper recordLookupHelper = new RecordLookupHelper();

   @Override
   public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException;
   {
      // lookup the full shippingPartner table (because it's cheap & easy to do so)
      // use the partner's "code" as the key field (e.g,. they key in the helper's internal map).
      recordLookupHelper.preloadRecords("shippingPartner", "code");

      // get all of the orderIds from the input shipments
      List<Serializable> orderIds = input.getRecords().stream()
         .map(r -> r.getValue("id")).toList();

      // fetch the orders related to by these shipments
      recordLookupHelper.preloadRecords("order", "id", orderIds);

      for(QRecord shipment : input.getRecords())
      {
         // get someConfigField from the shippingPartner assigned to the shipment
         Boolean someConfig = recordLookupHelper.getRecordValue("shippingPartner", "someConfigField", "code", shipment.getValue("partnerCode"));

         // get the order record assigned to the shipment
         QRecord order = recordLookupHelper.getRecordByKey("order", "id", shipment.getValue("orderId"));
      }
   }
}
Lazy fetching records

Scenario:

  • We have a BackendStep that is taking in purchaseOrderHeader records, from an API partner.

  • For each record, we need to make an API call to the partner to fetch the purchaseOrderLine records under that header.

    • In this contrived example, the partner’s API forces us to do these lookups order-by-order…​

  • Each purchaseOrderLine that we fetch will have a sku on it - a reference to our item table.

    • We need to look up each item to apply some business logic.

    • We assume there are very many item records in the backend database, so we don’t want to pre-load the full table. Also, we don’t know what sku values we will need until we fetch the purchaseOrderLine.

This is a situation where we can use RecordLookupHelper to lazily fetch the item records as we discover them, and it will take care of not re-fetching ones that it has already loaded.

Example of a process step using a RecordLookupHelper to lazy fetch records
public class MyPurchaseOrderProcessStep implements BackendStep
{
   // Add a new RecordLookupHelper field, which will "cache" lazy-loaded item records
   private RecordLookupHelper recordLookupHelper = new RecordLookupHelper();

   @Override
   public void run(RunBackendStepInput input, RunBackendStepOutput output) throws QException;
   {
      for(QRecord poHeader : input.getRecords())
      {
         // fetch the lines under the header
         Serializable poNo = poHeader.getValue("poNo");
         List<QRecord> poLines = new QueryAction().execute(new QueryInput("purchaseOrderLine")
            .withFilter(new QQueryFilter(new QFilterCriteria("poNo", EQUALS, poNo))));

         for(QRecord poLine : poLines)
         {
            // use recordLookupHelper to lazy-load item records by SKU.
            QRecord item = recordLookupHelper.getRecordByKey("item", "sku", poLine.getValue("sku"));

            // business logic related to item performed here.
         }
      }
   }
}

In this example, we will be doing exactly 1 query on the item table for each unique sku that is found across all of the poLine records we process. That is to say, if the same sku appears on only 1 poLine, or if it appears on 100 poLines, we will still only query once for that sku.

A slight tweak could be made to the example above, to make 1 item table lookup for each poHeader record:

Tweaked example doing 1 item lookup per poLine
// continuing from above, after the List<QRecord> poLines has been built

// get all of the skus from the lines
List<Serializable> skus = poLines.stream().map(r -> r.getValue("sku")).toList();

// preload the items for the skus
recordLookupHelper.preloadRecords("item", "sku", new QQueryFilter(new QFilterCriteria("sku", IN, skus)));

for(QRecord poLine : poLines)
{
   // get the items from the helper
   QRecord item = recordLookupHelper.getRecordByKey("item", "sku", poLine.getValue("sku"));

In this example, we’ve made a trade-off: We will query the item table exactly 1 time for each poHeader that we process. However, if the same sku is on every PO that we process, we will end up fetching it multiple times.

This could end up being better or worse than the previous example, depending on the distribution of the data we are dealing with.

A further tweak, a hybrid approach, could potentially reap the benefits of both of these examples (at the tradeoff of, more code, more complexity):

Tweaked example doing 1 item lookup per poLine, but only for not-previously-encountered skus
// Critically - we must tell our recordLookupHelper to NOT do any one-off lookups in this table
recordLookupHelper.setMayNotDoOneOffLookups("item", "sku");

// continuing from above, after the List<QRecord> poLines has been built

// get all of the skus from the lines
List<Serializable> skus = poLines.stream().map(r -> r.getValue("sku")).toList();

// determine which skus have not yet been loaded - e.g., they are not in the recordLookupHelper.
// this is why we needed to tell it above not to do one-off lookups; else it would lazy-load each sku here.
List<Serializable> skusToLoad = new ArrayList<>();
for(Serializable sku : skus)
{
   if(recordLookupHelper.getRecordByKey("item", "sku", sku) == null)
   {
      skusToLoad.add(sku);
   }
}

// preload the item records for any skus that are still needed
if(!skusToLoad.isEmpty())
{
   recordLookupHelper.preloadRecords("item", "sku",
      new QQueryFilter(new QFilterCriteria("sku", IN, skusToLoad)));
}

// continue as above

In this example, we will start by querying the item table once for each poHeader, but, if we eventually encounter a PO where all of its skus have already been loaded, then we may be able to avoid any item queries for such a PO.

Implementation Details

  • Internally, an instance of RecordLookupHelper maintains a number of Maps, with QQQ table names and field names as keys.

  • The accessing/lazy-fetching methods (e.g., any method whose name starts with getRecord) all begin by looking in these internal maps for the tableName and keyFieldName that they take as parameters.

    • If they find an entry in the maps, then it is used for producing a return value.

    • If they do not find an entry, then they will perform the a QueryAction, to try to fetch the requested record from the table’s backend.

      • Unless the setMayNotDoOneOffLookups method has been called for the (tableName, keyFieldName) pair.

Full API

Methods for accessing and lazy-fetching

  • getRecordByKey(String tableName, String keyFieldName, Serializable key)

Get a QRecord from tableName, where keyFieldName = key.

  • getRecordValue(String tableName, String requestedField, String keyFieldName, Serializable key)

Get the field requestedField from the record in tableName, where keyFieldName = key, as a Serializable. If the record is not found, null is returned.

  • getRecordValue(String tableName, String requestedField, String keyFieldName, Serializable key, Class<T> type)

Get the field requestedField from the record in tableName, where keyFieldName = key, as an instance of type. If the record is not found, null is returned.

  • getRecordId(String tableName, String keyFieldName, Serializable key)

Get the primary key of the record in tableName, where keyFieldName = key, as a Serializable. If the record is not found, null is returned.

  • getRecordId(String tableName, String keyFieldName, Serializable key, Class<T> type)

Get the primary key of the record in tableName, where keyFieldName = key, as an instance of type. If the record is not found, null is returned.

  • getRecordByUniqueKey(String tableName, Map<String, Serializable> uniqueKey)

Get a QRecord from tableName, where the record matches the field/value pairs in uniqueKey.

Note: this method does not use the same internal map as the rest of the class. As such, it does not take advantage of any data fetched via the preload methods. It is only used for caching lazy-fetches.

Methods for preloading

  • preloadRecords(String tableName, String keyFieldName)

Query for all records from tableName, storing them in an internal map keyed by the field keyFieldName.

  • preloadRecords(String tableName, String keyFieldName, QQueryFilter filter)

Query for records matching filter from tableName, storing them in an internal map keyed by the field keyFieldName.

  • preloadRecords(String tableName, String keyFieldName, List<Serializable> inList)

Query for records with the field keyFieldName having a value in inList from tableName, storing them in an internal map keyed by the field keyFieldName.

Config Methods

  • setMayNotDoOneOffLookups(String tableName, String fieldName)

For cases where you know that you have preloaded records for tableName, keyed by fieldName, and you know that some of the keys may not have been found, so you want to avoid doing a query when a missed key is found in one of the getRecord…​ methods, then if you call this method, an internal flag is set, which will prevent any such one-off lookups.

In other words, if this method has been called for a (tableName, fieldName) pair, then the getRecord…​ methods will only look in the internal map for records, and no queries will be performed to look for records.