Condition filter limited to metadata keys returned by metadataKeys endpoint

[Junjiequan]

Summary

Condition filters on the frontend currently work only for metadata keys returned by the metadataKeys endpoint. Since this endpoint is limited (e.g. 100 or 1000 keys depending on configuration), users cannot filter on metadata keys that are not included in the returned list.

Steps to Reproduce

1. Open dataset page and add a condition
2. Type metadata keys that exists in datasets but is not included in the list due to the limit.
3. Notice that the filter cannot be applied

Current Behaviour

Condition filters only allow selecting metadata keys present in the limited metadataKeys response. Metadata keys outside this list cannot be queried, even if they exist in the dataset.

Expected Behaviour

Users should be able to filter on any valid or invalid metadata key, regardless of whether it appears in the limited metadataKeys list.

Details

• The metadataKeys endpoint returns a capped number of keys based on configuration.
• User can only select from the capped list to apply the filter
• Proposed solutions:
  • Allow user to apply any arbitrary metadata keys to the filter.