output_validators not effecting package_extras not in database

[mackeynichols]

CKAN Version

2.10.8

Describe the Bug

When an extra in the package_extra database table is not populated/does not exist, output_validators won't run on that extra field.

Replicating the bug

Input:
Call <our_ckan>/api/3/action/package_show?id=id123 with the following body

• package id123 doesnt have a value for "collection_method" in the package_extra database table
• package id123 is of dataset_type: "dataset"

Expected Output:

• "collection_method" field is shown as collection_method: null

Actual Output:

• "collection_method" field is not shown in package_show response

Configurations

I am using this YAML to configure scheming:

scheming_version: 2
dataset_type: dataset
about: A reimplementation of the default CKAN dataset schema
about_url: http://github.com/ckan/ckanext-scheming

dataset_fields:
  - field_name: title
    label: Title
    required: true

  - field_name: url
    label: URL
    preset: dataset_slug
    form_placeholder: eg. my-dataset
    required: true

  - field_name: collection_method
    label: Collection Method
    output_validators: default_to_none

resource_fields: 
  - field_name: title
    label: Title
    required: true

  - field_name: url
    label: URL
    preset: dataset_slug
    form_placeholder: eg. my-dataset
    required: true

It references this custom validator:

def default_to_none(value):
    if value:
        return value
    else:
        return None

I have been refreshing the SOLR index before and after making the package_show call.

[mackeynichols]

To be specific, when I say:

package id123 doesnt have a value for "collection_method" in the package_extra database table

I mean that there is no record with key "collection_method" in the database. The output_validators work if there is a record with key "collection_method" even if the value is empty

[mackeynichols]

After some digging, I think this behavior is from CKAN core's /usr/lib/ckan/default/src/ckan/ckan/lib/navl/dictization_functions.py 's _validate() function, which receives a schema with a reference to a key, but will remove it if that key isn't populated; references to collection_method, or any key without data in the CKAN database package_extra table is removed after _validate()'s "#before run" on line 343

Ill look to see if there's a better way to add the behavior Im referring to in ckanext-scheming without having to touch CKAN core; this _validate() function looks like it used for lots of things beside validating package and resource metadata