Add publication and versioning #5

MyPyDavid · 2024-11-04T18:37:12Z

this PR adds the publication workflow with file upload to Zenodo. The export format and view can be selected in the form.
the record id is stored in the project.value with a configurable attribute (eg. https://rdmorganiser.github.io/terms/project/metadata/publication/zenodo/concept_record_id)
- setting keys: publish_record_id_attribute_prefix and publish_record_id_attribute_key
this record_id is used to create new versions of the zenodo record (an existing publication will be updated with newer versions).
added support for InvenioRDM API
- setting keys: zenodo_url (the instance url), zenodo_auth_scope (need to set the scope to user:email )

refactoring

moved classes into metadata, forms.py and methods into utils.py

MyPyDavid · 2024-11-14T10:13:25Z

Hi @cpfaff , do you want to check out this branch maybe?
Ive included your changes but also did a lot of refactoring again, hope that it's clear enough, we could discuss it in a meeting.

MyPyDavid · 2024-12-11T13:09:45Z

maybe it would be better to try and merge this PR before adding other parts for the REST API to it.
I've asked @jochenklar for the Review

MyPyDavid · 2025-06-05T14:09:05Z

can we merge this? @jochenklar

jochenklar

Hey, I finally got some time to look at the plugin, sorry for the long wait. I guess it works (almost), but I have some remarks regarding the architecture.

rdmo_zenodo/exports/base.py

rdmo_zenodo/templates/plugins/exports_zenodo.html

jochenklar · 2025-07-24T07:55:10Z

rdmo_zenodo/exports/metadata.py

+from rdmo.projects.exports import Export
+
+
+class ZenodoMetadataExport(Export):


The name is a bit misleading since this is not an export plugin, or at least it is not sopposed to be used independently, right? I also think it should not inherit Export. Maybe ZenodoMetadataBuilder, since this is how it is used. It could also be a mixin for the export plugins or part of the base class, depends on why exaclty this pattern was chosen.

yes it is more like a mixin and does not need Export, it is only using these self.get_text and self.get_values methods. I think the base class was getting too big that's why I placed it in a new one.

I've now refactored this Metadata stuff into a metadata package and some builder classes. These should have no external dependencies besides the settings, the project or snapshot related things are provided by the caller of these builders in the get_post_data method.
It has some duplication with all these field names but maybe its cleaner and clearer now?

rdmo_zenodo/exports/metadata.py

jochenklar · 2025-07-24T07:59:41Z

rdmo_zenodo/exports/metadata.py

+            'metadata': self._filter_empty_values(metadata)
+        }
+
+    def _filter_empty_values(self, metadata: Dict[str, Any]) -> Dict[str, Any]:


I don't think this is more readable than:

a = settings.A if a: metadata['a'] = a

but ok...

rdmo_zenodo/exports/utils.py

jochenklar · 2025-07-24T08:10:06Z

rdmo_zenodo/exports/utils.py

@@ -0,0 +1,83 @@
+from django.http import HttpResponse


There are some missing newlines between the methods in this file.

jochenklar · 2025-07-24T08:11:04Z

rdmo_zenodo/exports/utils.py

+
+def render_project_views(project, snapshot, attachments_format, view=None):
+
+    if view is None:


Was this for debugging, if yes, it should be removed.

Ok it seems I need the view for the plugin to work. Maybe this should be a template whithin the plugin?

ok, so that the user can customize with a custom template? Maybe the view should be selected in the initial form as well by the user to make it easier to customize?

Ive added a selection for the view and export_format to the form.

rdmo_zenodo/exports/publish_snapshot.py

rdmo_zenodo/exports/utils.py

MyPyDavid · 2025-07-28T15:37:33Z

rdmo_zenodo/exports/publish.py

+
+            record_versions_url = self.validate_record_id_from_project_value_at_zenodo()
+            # TODO, currently the authentication can get stuck when trying out the dataset export
+            # first and this one afterwards, a 403 needs to be handled in the Export class.


I still have one problem with the authentication sometimes. When I first export the dataset and afterwards try to publish the snapshot then I get a 403 on the post call (and the Oauth Error template with None as message) . Could this be because of the different callback urls that are set for each export provider and that the token is then invalid? I can only get out of it by closing the browser.
Should these plugins not just have a single authentication and callback uri? Or does it need to handled in the https://github.com/rdmorganiser/rdmo/blob/79917de8dfdb8be0c988bc72f1352b307ce03cb1/rdmo/services/providers.py#L62 ?

Maybe this issue zenodo/zenodo#2168 is related somehow?

Now I'm getting another authentication error ERROR: callback error: b'{"error": "invalid_grant"}' (400) after clicking "Authorize application" on the https://sandbox.zenodo.org/oauth/authorize website. Maybe it's similar to the issue reported in zenodo/zenodo-rdm#878.
After clicking around in the debug tools,and browsing back and forth, I now get another error WARNING: post error: b'{"status": 400, "message": "Invalid value e."}' (400). It seems that I had a mistake in my metadata after all and was sending e.g. 'languages': [{'id': 'e'}] ).
After fixing that the request works again.

I still have one problem with the authentication sometimes. When I first export the dataset and afterwards try to publish the snapshot then I get a 403 on the post call (and the Oauth Error template with None as message) . Could this be because of the different callback urls that are set for each export provider and that the token is then invalid? I can only get out of it by closing the browser. Should these plugins not just have a single authentication and callback uri? Or does it need to handled in the https://github.com/rdmorganiser/rdmo/blob/79917de8dfdb8be0c988bc72f1352b307ce03cb1/rdmo/services/providers.py#L62 ?

I've added a post_with_retry method that pops the access_token and re-tries the self.post(...) after it failed with an "OAuth error". The user just needs to re-authenticate but it seems to work.
I think an extra callback post_failure should be added to https://github.com/rdmorganiser/rdmo/blob/79917de8dfdb8be0c988bc72f1352b307ce03cb1/rdmo/services/providers.py#L67 in parallel to post_success so that it can be implemented and adapted by a plugin. Or was there a problem with storing/retreiving the access token?

jochenklar · 2025-08-01T14:29:09Z

There is still a problem. For some reason, the stored zenodoid is one lower than the actual id.

jochenklar · 2025-08-01T14:31:26Z

Also, I am not sure creating the attribute in the Plugin is a good idea. In any case the URI prefix should not be hardcoded.

MyPyDavid · 2025-08-05T13:00:19Z

There is still a problem. For some reason, the stored zenodoid is one lower than the actual id.

This is not a problem but actually a feature. After initially creating a record and uploading a file, Zenodo returns a conceptrecid (https://developers.zenodo.org/#quickstart-upload) or in InvenioRDM the internal Concept version PID (https://inveniordm.docs.cern.ch/reference/metadata/#internal-pids-id-pid-parentid).

From https://inveniordm.docs.cern.ch/operate/customize/dois/#parent-or-concept-dois:

By default InvenioRDM will create two DOIs when an initial record is published, and create one DOI each time a new version of the record is published. The first DOI is the version DOI, which represents the specific record that is published. The second DOI is the parent DOI, which represents the concept of the record and will always resolve to the latest version. This feature has been implemented in Zenodo for many years, and the concept DOI enables researchers to cite something that won't change when they make changes to their records.

So that concept id is used when publishing another rdmo snapshot to the previously created record and thereby creating a new version in that record.

* Collects metadata for the project snapshot * Creates snapshot questions based document * Creates zenodo deposit, adds metadata, adds the file and publishes the deposit.

Refactored the Zenodo Export Provider to improve code quality, reduce duplication, and enhance error handling.

Signed-off-by: David Wallace <[email protected]>

MyPyDavid · 2025-08-06T08:56:40Z

Also, I am not sure creating the attribute in the Plugin is a good idea. In any case the URI prefix should not be hardcoded.

yes, Ive made it configurable. I've also tested against the sandbox https://inveniordm.web.cern.ch/ instance and made it work with a configurable setting for the zenodo_auth_scope.

Now in the readme:

    'zenodo_url': 'https://zenodo.org',  # optional, default https://zenodo.org , or your own InvenioRDM instance url
    'zenodo_auth_scope': 'deposit:write',  # optional, default 'deposit:write' or 'user:email' for InvenioRDM
    'publish_record_id_attribute_prefix': 'https://rdmorganiser.github.io/terms',  # optional, default is shown here
    'publish_record_id_attribute_key': 'project/metadata/publication/zenodo/concept_record_id',  # optional, default is shown here

Now I've rebased and it's ready I think!

Signed-off-by: David Wallace <[email protected]>

jochenklar

Sorry, we need to talk about this again. The current classes in metadata prevent reuse and customization.

MyPyDavid · 2025-09-22T09:01:40Z

rdmo_zenodo/exports/metadata/model.py

+
+
+@dataclass
+class ZenodoMetadata:


This model can be kept, maybe add a function get_metadata instead of get_post_data

MyPyDavid · 2025-10-13T12:32:22Z

Sorry, we need to talk about this again. The current classes in metadata prevent reuse and customization.

I went further down the metadata rabbit hole and tried to implement the complete schemas for the payload (incl metadata) send to the APIs with attrs classes. Is it customizable enough like this??

MyPyDavid · 2025-10-13T12:33:36Z

rdmo_zenodo/exports/metadata/invenio.py

+    identifier: str | None = None
+
+@attrs.define
+class InvenioMetadataV6:


this is the Invenio metadata

MyPyDavid requested a review from cpfaff November 4, 2024 18:37

MyPyDavid requested review from jochenklar and removed request for cpfaff December 11, 2024 13:07

MyPyDavid mentioned this pull request Jan 7, 2025

Hackathon24 Zenodo Publish Project and Snapshot #1

Merged

jochenklar requested changes Jul 24, 2025

View reviewed changes

MyPyDavid self-assigned this Jul 24, 2025

MyPyDavid commented Jul 28, 2025

View reviewed changes

MyPyDavid requested a review from jochenklar July 28, 2025 15:38

jochenklar approved these changes Aug 1, 2025

View reviewed changes

ctpfaff and others added 15 commits August 6, 2025 10:48

Publish project snapshots to zenodo

d40251e

* Collects metadata for the project snapshot * Creates snapshot questions based document * Creates zenodo deposit, adds metadata, adds the file and publishes the deposit.

Enhance Zenodo Export Provider

a8a37b4

Refactored the Zenodo Export Provider to improve code quality, reduce duplication, and enhance error handling.

refactor: split exports and add publish provider

877ce86

Signed-off-by: David Wallace <[email protected]>

docs: fix url for zenodo-publish

0c67ec7

Signed-off-by: David Wallace <[email protected]>

fix: add class attribute for RDMO_PLUGIN_KEY

19aaf8d

Signed-off-by: David Wallace <[email protected]>

feat,fix: add RDMO_PLUGIN_KEY and methods for pdf from snapshot upload

3d04c95

Signed-off-by: David Wallace <[email protected]>

feat: add functions for zenodo api upload

0c5d4a8

Signed-off-by: David Wallace <[email protected]>

build: autoupdate pre-commit

571a980

Signed-off-by: David Wallace <[email protected]>

chore: remove breakpoint

ae73bae

Signed-off-by: David Wallace <[email protected]>

refactor: rename to export_dataset

99ea1d3

Signed-off-by: David Wallace <[email protected]>

build: update ruff.lint config

3bfaef7

Signed-off-by: David Wallace <[email protected]>

feat: add zenodo urls to base

f326385

Signed-off-by: David Wallace <[email protected]>

feat: add ZenodoMetadataExport class in metadata.py

c7eef4e

Signed-off-by: David Wallace <[email protected]>

feat: add authorized json headers and rename to records url

84f1c1b

Signed-off-by: David Wallace <[email protected]>

refactor: move forms to forms.py

a452b5f

Signed-off-by: David Wallace <[email protected]>

MyPyDavid added 13 commits August 6, 2025 10:48

use plugin.key instead

82beed1

Signed-off-by: David Wallace <[email protected]>

refactor metadata into package with builder classes

a641b2c

Signed-off-by: David Wallace <[email protected]>

add view and export format to snapshot form

f216d40

Signed-off-by: David Wallace <[email protected]>

add publication type and define resource and upload types

9c289ad

Signed-off-by: David Wallace <[email protected]>

use metadata builder in get post data

ac51f25

Signed-off-by: David Wallace <[email protected]>

add view and format selection and use metadata builder in get post data

bcf4776

Signed-off-by: David Wallace <[email protected]>

clean up utils and handle exceptions

bb10309

Signed-off-by: David Wallace <[email protected]>

revert exports template and add one for publish

e94879f

Signed-off-by: David Wallace <[email protected]>

docs: add InvenioRDM sanbdox

b2a96ce

Signed-off-by: David Wallace <[email protected]>

add config for url, attribute and scope and support for InvenioRDM

dfcaf59

Signed-off-by: David Wallace <[email protected]>

rename metadata builder classes

17e8c30

Signed-off-by: David Wallace <[email protected]>

set default for zenodo_url and add to readme config

012693c

Signed-off-by: David Wallace <[email protected]>

style(docs): reorder config

627f8f5

Signed-off-by: David Wallace <[email protected]>

MyPyDavid force-pushed the add-publication-and-versioning branch from d6acdf2 to 627f8f5 Compare August 6, 2025 08:54

docs: update config comments

ee58804

Signed-off-by: David Wallace <[email protected]>

MyPyDavid requested a review from jochenklar August 6, 2025 08:57

use zenodo_record_id_uri as setting instead of prefix and key

53de9d5

Signed-off-by: David Wallace <[email protected]>

jochenklar requested changes Sep 2, 2025

View reviewed changes

MyPyDavid commented Sep 22, 2025

View reviewed changes

MyPyDavid added 7 commits October 9, 2025 11:00

re-implement metadata for zenodo and invenio

53dfc25

fix export to zenodo

91a701f

docs: update README

5d4aa90

update Provider classes and use get_metadata

59377ff

build: add attrs and cattrs to deps

bcfafd7

fix metadata schemas

e2d0744

handle auth error and refactor get_metadata

cd99917

MyPyDavid commented Oct 13, 2025

View reviewed changes

MyPyDavid requested a review from jochenklar October 13, 2025 12:34

		from rdmo.projects.exports import Export


		class ZenodoMetadataExport(Export):


		def render_project_views(project, snapshot, attachments_format, view=None):

		if view is None:

Add publication and versioning #5

Are you sure you want to change the base?

Add publication and versioning #5

Uh oh!

Conversation

MyPyDavid commented Nov 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

refactoring

Uh oh!

MyPyDavid commented Nov 14, 2024

Uh oh!

MyPyDavid commented Dec 11, 2024

Uh oh!

MyPyDavid commented Jun 5, 2025

Uh oh!

jochenklar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

MyPyDavid Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jochenklar commented Aug 1, 2025

Uh oh!

jochenklar commented Aug 1, 2025

Uh oh!

MyPyDavid commented Aug 5, 2025

Uh oh!

MyPyDavid commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jochenklar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MyPyDavid commented Oct 13, 2025

Uh oh!

MyPyDavid Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

MyPyDavid commented Nov 4, 2024 •

edited

Loading

MyPyDavid Jul 28, 2025 •

edited

Loading

MyPyDavid Oct 10, 2025 •

edited

Loading

MyPyDavid commented Aug 6, 2025 •

edited

Loading

MyPyDavid Oct 13, 2025 •

edited

Loading