Required controlledVocabulary metadata marked as valid while empty

[stevenferey]

Bug description

In Dataverse (v6.8), when a previously optional metadata field becomes required for a dataset, or when a dataset is created via the API without all required metadata (dataverse.api.allow-incomplete-metadata=true), the system displays an Incomplete metadata label and normally prevents DRAFT form validation or dataset publication.

However, if the user uploads a file, Dataverse inserts a record for the missing required controlledVocabulary metadata into the datasetfieldvalue table.

Consequence: required controlledVocabulary metadata fields (e.g. subject) are treated as valid in the dataset metadata form even though their value is empty.

Steps to reproduce

1. Create a dataset via API with dataverse.api.allow-incomplete-metadata=true without including all required metadata.
2. Upload a file to the dataset.
3. Open the dataset metadata edit form.
4. Notice that required controlledVocabulary fields appear valid although no value is set.

Expected behavior

Required controlledVocabulary fields should remain invalid until a valid value is provided.

Affected version

Dataverse v6.8

Are you thinking about creating a pull request for this issue?

No fix is currently planned by the team.

[pdurbin]

@stevenferey thanks for the bug report. Out of curiosity, are you able to reproduce this in the SPA at https://beta.dataverse.org/spa/ ?

@ErykKul since you added the "incomplete metadata" concept, I'm just tagging you as a heads up. 😄

[ErykKul]

I assigned it to me; I will investigate it and see if I can reproduce it (and do a PR). We did not notice that behavior yet, but it is certainly something relevant to KU Leuven. If someone wants to join, you are welcome to do so.

[stevenferey]

@stevenferey thanks for the bug report. Out of curiosity, are you able to reproduce this in the SPA at https://beta.dataverse.org/spa/ ?

@pdurbin, no because the application server must have the feature enabled in order to create a dataset with incomplete mandatory metadata (and my institutional information is blocking account creation)
dataverse.api.allow-incomplete-metadata=true

[ErykKul]

I made a PR that should fix it. I could reproduce the problem with a unit test; I could not reproduce it with the combination of API/UI edits. I think that the test would have to be very specific for that, where the only missing value is the controlled vocabulary value? Or specific citation block? To be sure, the code would have to be tested in the specific conditions where it is reproducible.

@stevenferey ,
What value is exactly inserted to datasetfieldvalue table? The code did already check for blank and "N/A" values. What I added is the check if the specific value is in the possible values list. Do you upload the file with UI or API to reproduce it (I tested both ways without any problem, however, I tested on v. 6.7.1)? Is it only the 6.8 that has this problem? The important part is also the reindex after any changes to the metadata block. Meanwhile, I will also try to test the 6.8 specifically to see if I can reproduce the problem there (before the fix, and than with the fix, if I can reproduce the problem)

[ErykKul]

I have now tested with 6.8, I copied a dataset from the demo dataverse with only the subject left out, and added files via UI (JSF version) and API, the subject is still missing and the dataset is invalid:

Only the subject is missing:

Setting the subject manually in UI makes the dataset valid.

Tested with v. 6.8 build 1994-92d1ec8 (the official released version)

There must be very specific about this bug, or I am doing something wrong when trying to reproduce it. Let me know if it still persists.

[ErykKul]

I have discovered that I can save the dataset after removing the subject, making it incomplete. This should be not possible in the UI. It might be the cause of the problem. I will investigate it in more detail.

[ErykKul]

Removing the subject and saving was a bug; I improved the validation issues detection for the JSF, you cannot save without setting all mandatory fields now, including the controlled vocabulary values (it already worked for other types of values). PR is updated now. I tested the fix in UI.

[stevenwinship]

I have been trying to QA this pr. I created the dataset by setting dataverse.api.allow-incomplete-metadata=true and adding qp
/api/dataverses/" + dataverseAlias + "/datasets?doNotValidate=true

The create worked
I then uploaded a file to the dataset which added the row to the db table datasetfieldvalue
7 0 "N/A" 10

At this point I cannot publish and continue to be notified that there is a missing field.

Long story short I cannot reproduce this bug. (FYI I am using the current develop branch to reproduce)