A new and improved DPG data schema

[ricardomiron]

Hi there,

We are currently in the process of improving how we collect, store, and show DPG data in all our products, including the DPG nomination portal, DPG Registry, and public API. This entails an evolution of our current data schema, with a more structured way of collecting the data and some additional information that was not previously collected and we want your feedback!

Why are we doing this?

• It will make it easier for product owners to apply for DPG Recognition by having less ambiguity.
• It will help us with a new, soon-to-launched, DPG Registry and DPG Profile page.
• It will allow for more ways people can use the registry and API, including new filters and categorization options.
• It will provide valuable information that stakeholders ask frequently.
• Add compatibility with publiccode.yml standard

Where can I look at the new schema?
This is a first draft, some fields may change, get removed, or be added. It follows the Frictionless Data Table Schema specification, but it will ultimately be translated into a simpler JSON Schema.

👉 Take a look here.

What is purposely not included in this schema, but stored in the system:

• Any PII data that might be collected.
• Logs of the review process or status changes.
• Other metadata related to internal processes or systems.

[siddharthvipul]

Hi Ricardo, this is a fantastic move. At UNICEF Office of Innovation - where we support many internally developed solutions become DPG, and fund start ups to develop Open Source solutions (with idea of them becoming DPG). It's going to be incredibly valuable to get status of each solution supported through API.
For example, Vrapeutic (https://app.digitalpublicgoods.net/a/10127) - A UNICEF Venture Fund supported solution isn't DPG anymore. Having insight into status through API (and perhaps why it failed) would help us perhaps nudge the solutions, and even if not that, get us better insight.
I see this is not included in the API, but status change would be incredibly valuable to us.
as of now the api for this solution (https://api.digitalpublicgoods.net/api/dpg/10127) returns nothing as I see it. Would you have a recommendation, or another way to get automated status?

Thank you for this! again, it's great!

[ricardomiron]

Hi @siddharthvipul, there is a field on the metadata called status, and expiryDate that would allow users of the API and registry to know if any given solution is expired and when it expired, or if it was marked as ineligible, or is active DPG. With this, some automations can be made to get notified if a DPG status changes and we'll also explore the possibility of creating a webhook for pushing updates as well on status changes.

We currently only show the product owner the reason why they might have failed any particle indicator, we have now included a public field decisionNotes that should also allow anyone to get a summary of the decision and status of the solution. As part of these enhancements, we're also making some changes to the renewal process that will hopefully require less effort for product owners to renew their status every year and have less dropouts from the registry.

[eduffus]

Here are a few fields where I think there could be some improvements to cater for all DPG types:
solutionType vs category - I don't think the distinction is clear and this has been difficult to fill in the past. Category options feel more obvious.
privacyComplianceEvidence - the implementation of some configurable solutions (like OpenCRVS) will only be compliant with local privacy laws if implemented with these in mind. Can we modify this field to reflect this complexity?
openStandards - can we be more explicit about the types of open standards that are really relevant here? What makes a specification a standard? Does it need to be ISO certified? If I publish my API spec does that make it an open standard?
bestPractices - this is another lose term which could do with some extra focus. Who gets to define what the recognised best practices are?
dataPrivacySecurity - as above - the implementation of some configurable solutions (like OpenCRVS) will only be compliant with local data security laws if implemented with these in mind. Can we modify this field to reflect this complexity e.g. has the solution been penetration tested + evidence? What certification did the tester have?
developmentStatus - requires definitions
maintenanceType - multiple response types should be allowed

[ricardomiron]

Hi @eduffus , we're trying to make this new schema compatible with publiccode.yml so there are some things we took directly from that standard.

• solutionType vs category - these are distinct in a crucial way, where category can only be one of the main types of DPG (software, data, content, AI). On the other hand, solutionType can be considered as "sub-categories" (web, mobile, desktop, etc), exemplified here and directly taken from publiccode "softwareType" but expanded to all categories of DPGs.
• privacyComplianceEvidence - We have made some changes based on the suggestion of privacy experts, we're going to keep applicable laws only to those that are directly compatible "by design" and have more options on compliance evidence (privacy policy, terms of use, data retention policy, etc). We have now also added:
  • privacyConsiderations - this is a predefined checklist of best practices and considerations to have on the privacy policy or other relevant documentation to the solution.
  • privacyNotes - this field allows applicants to add context to the other relevant practices, configurable options, etc.
• openStandards - we agree there should be more meaningful open standards listed and not every single spec, because of the diversity of how a standard or specification might be relevant from one solution to the other, which varies a lot depending on the type of solution, its purpose, category, etc. we'll try to manage that complexity through documentation and suggestions both on the nomination form and DPG Wiki. But we don't have the capacity to specify, validate, or require a subset of standards given this complexity and sectorial knowledge needed.
• bestPractices - similar to open standards, we might suggest how to include only the most relevant ones and provide suggestions. The main idea is that the implement or whoever is looking at this data can make a decision based on this information regarding if these practices are "good enough" for their use case, but we're open to exploring something like a predefined checklist with relevant best practices such a those that make software easier to deploy (i.e. Digital Square Shelf Readiness).
• dataPrivacySecurity - this indicator focuses more on the technical aspects of privacy and security, whereas privacyCompliance focuses more on the legal aspects. We have changed this field so it is now an open text field that would allow you to put any context or additional information.
• developmentStatus - 👍
• maintenanceType - we have changed this to an array to allow multiple responses.

Thanks for the great feedback!

[jeremi]

I agree with @eduffus 's comments. I'd like to add few more comments:

isBasedOn

The current definition of isBasedOn seems unclear. Some points to consider:

• Clarification needed on the scope: Should it include all frameworks and technologies used, or only major foundational elements?
• For complex projects like OpenSPP, listing multiple dependencies (Odoo, ODK, MOSIP, IDPASS, OpenG2P, python, formio, postgresql...) could be appropriate, but may lead to an extensive list.
• It might be helpful to differentiate between "core dependencies" and "integrations/plugins" to provide a clearer picture of the DPG's foundation.

privacyComplianceEvidence

The current approach to privacy compliance seems problematic to me for DPGs. I was always uncomfortable answering it.

1. Compliance vs. Compatibility:

   • DPGs themselves can't be "compliant" with regulations like GDPR, as compliance depends on implementation.
   • Maybe you could change "compliant" to "compatible" or "designed to support compliance with".

2. Focus on Implementation Support:

   • DPGs typically don't host information directly; the implementation does.
   • The DPG's privacy policy, terms of use, and data retention policy are less relevant than the tools it provides.
   • Suggestion: Shift focus to capabilities that support privacy-compliant implementations.
   • Possible fields:
     • privacyFeatures: ["Data minimization tools", "Consent management", "Data portability support"]
     • complianceToolkit: ["Sample privacy policy templates", "Data flow documentation guides"]

These changes would better reflect the role of DPGs in enabling privacy-compliant implementations rather than being compliant themselves.