Skip to content

Configuration Filters

Jump to bottom
Thomas Trutt edited this page Apr 14, 2025 · 1 revision

Filters are defined in YAMl files located in the config directory and have been implemented to allow for end users to check multiple aspects of the data returned. The filters are ran in order and there are filters for both Charges and Refunds. The details of how these filters are implemented are described below.

filters:
  charge_filters:
    - ...
  credit_filters:
    - ...
  process_filter:
    - ...

Filter example

vars:
  run_env: "TEST"
  trans_active: true

filters:
  charge_filters:
    - name: "FineType"
      error_message: "Wrong Fee Fine Type"
      load: "FeeFineTypes"
      flatten: true
      filter_field: "feeFineId"
      field_transform: "NONE"
      filter_operator: "IN_FILE"
      filter_value: ""
      log_error: false

  credit_filters:
    - name: "PaymentType"
      error_message: "Wrong Transfer Account"
      load: false
      flatten: false
      filter_field: "transferAccount"
      field_transform: "NONE"
      filter_operator: "EQUALS"
      filter_value: "ENV|TRANS_ACCOUNT_NAME"
      log_error: false
  process_filter:

Filters are required to be in order, as they will be processed in the order listed. If you do not wish to use a filter, you can just leave the filters array empty.

vars Fields

The vars section allows you to set variables that can be used in the jobs section. This allows you to set a variable once and use it in multiple jobs.

Fields

name

  • The name of the filter. This will be used in the count data returned by the backend system

For example a name of FineType will be used in the count data as failedFineType: 0 or passedFineType: 0. Theses values are exposed to the template system as {{ summary.failedFineType }} and {{ summary.passedFineType }}.

error_message

  • Error message that will be added to the error log data.

This data will be recorded to the json as field errorCode and can be accessed in the template system as {{ errorCode }}

load

  • (options) false or <<NAME>> of the JSON file to be imported
    • false will not load any file
    • <<NAME>> will load the file from the data directory. The file must be a JSON file and must be in the dataSets directory.

flatten

  • (options) true or false
    • Should the JSON file be flattened. This will extract the UUID field from the array and place it in a new array. this is used to match based on the In function, or with files containing match patterns.

For example the following JSON file:

[
    {
        "uuid": "1234",
        "ownerID": "9879",
        "description": "Fine 1"
    },
    {
        "uuid": "5678",
        "ownerID": "8767",
        "description": "Fine 2"
    }
]

Would result in a filter array of

[
    "1234": {
        "ownerID": "9879",
        "description": "Fine 1"
    },
    "5678": {
        "ownerID": "8767",
        "description": "Fine 2"
    }
]

filter_field

  • The field in the charges record that should be compared to by the filter.

field_transform

  • (options) NONE or COUNT
    • NONE - No transformation is applied to the field
    • COUNT - The number of characters in the field is counted. This is useful for checking the length of a field.

filter_operator

  • (options) EQUALS, NOT_EQUAL, ONE_OF, NULL_OR_ONE_OF, LONGER_THAN, SHORTER_THAN or IN_FILE
    • EQUALS - The filter_field must be equal to the filter_value
    • NOT_EQUAL - The filter_field must not be equal to the filter_value
    • ONE_OF - The filter_field must be one of the values in the filter_value
    • NULL_OR_ONE_OF - the filter_field must be null or one of the values in the filter_value
    • LONGER_THAN - The filter_field must be longer than the filter_value; this is useful for counting the number of charters in a field for example to ensure that an external ID is the correct length.
    • SHORTER_THAN - The filter_field must be shorter than the filter_value; this is useful for counting the number of charters in a field for example to ensure that an external ID is the correct length.
    • IN_FILE - Is the field value in the injected file from the load field

filter_value

  • The value to compare to the field.
  • To use a environmental variable you can use the following format: ENV|<<ENVIRONMENTAL_VARIABLE_NAME>> for example ENV|TRANS_ACCOUNT_NAME. This will try to match the value in the TRANS_ACCOUNT_NAME environmental variable to the field value.

log_error

  • (options) true or false
    • true - Any records that fo match the filter will be included in the error_data section of the formatted data.
    • false - Any records that fo match the filter will be not included in the error_data section of the formatted data.

Practical examples

Check for Fee Fine Types in a JSON File.

In this example we are loading the FeeFineTypes.json file and flattening it. We are then comparing the feeFineId field in the charge data to the uuid field in the FeeFineTypes.json file. If the feeFineId is not in the FeeFineTypes.json file, then the error message Wrong Fee Fine Type will be added to the error log data.

FeeFineTypes.json

[
    {
        "uuid": "1234",
        "ownerID": "9879",
        "description": "Fine 1"
    },
    {
        "uuid": "5678",
        "ownerID": "8767",
        "description": "Fine 2"
    }
]

.env Filter expanded for clarity

filters:
   charge_filters:
       - name: "FineType"
           error_message: "Wrong Fee Fine Type"
           load: "FeeFineTypes"
           flatten: true
           filter_field: "feeFineId"
           field_transform: "NONE"
           filter_operator: "IN_FILE"
           filter_value: ""
           log_error: true

Check a Custom field in the user record

In this example we want to check if the custom User field bursar is set to true, as in there bursar account is active, or if it has not been set at all. When it comes to custom user fields a value is not entered until the user record is edited via the UI, even if a default value is set. This means that the field will be null until the user record is edited. This is why we are using the NULL_OR_ONE_OF operator. Since ww are not loading a file, the load field is set to false and the flatten field is also set to false.

filters:
    charge_filters:
        -   name: "BursarActive"
            error_message: "Exports suspended for user"
            load: false
            flatten: false
            filter_field: "patron.customFields.bursar"
            field_transform: "NONE"
            filter_operator: "NULL_OR_ONE_OF"
            filter_value: "['opt_0']"
            log_error: true

Check the length of a field

For our institution, the external ID for a user should be 10 characters long. This filter will check that the patron.externalSystemId field is 10 characters long. If it is not, then the error message The external ID is not valid will be added to the error log data. The field_transform is set to COUNT to count the number of characters in the field. the filter_operator is set to SHORTER_THAN to check that the field is shorter than the filter_value of 10 to ensure or value meets what we expect.

filters:
    charge_filters:
        -   name: "BadExternalID"
            error_message: "The external ID is not valid"
            load: false
            flatten: false
            filter_field: "patron.externalSystemId"
            field_transform: "COUNT"
            filter_operator: "SHORTER_THAN"
            filter_value: 10
            log_error: true

Clone this wiki locally