Allow specifying additional GDPR data in yaml files (#27)

* feat: Allow specifying additional GDPR data in yaml files

* docs: Update and improve README

---------

Co-authored-by: Philippe Czaja <philippe.czaja-ext@rio.cloud>

Author: PhilippeCzajaTNG

README.md

@@ -4,7 +4,7 @@
 ![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/rio-cloud/gradle-gdpr-documentation-plugin/build-and-deploy.yaml)
 
 Gradle plugin to generate data classification documentation (needed for the GDPR documentation) for your project based
-on annotations on data classes.
+on annotations on data classes and/or from configuration files.
 
 ## Disclaimer
 
@@ -13,18 +13,136 @@ on annotations on data classes.
 > make sure to classify the data according to your own requirements. RIO is not responsible for your documentation.
 
 > [!NOTE]
-> RIO maintains this repository for their internal documentation. If you need different / additional functionality, please fork the project.
+> RIO maintains this repository for their internal documentation. If you need different / additional functionality,
+> please fork the project.
 
 ## Usage
 
-See [example project](./test).
+### Classify your data
+
+There are two ways to classify data:
+
+1. Annotate your data classes with the provided annotations
+2. Provide configuration files
+
+You can also combine both approaches. In case of conflicts, the configuration files will have precedence over the
+annotations.
+
+See [example project](./test) for examples of both approaches.
+
+#### Annotations
+
+You can annotate your data classes with the following annotations (defined
+in [GdprData](./core/src/main/kotlin/cloud/rio/gdprdoc/annotations/GdprDoc.kt)), describing the data flow and purpose:
+
+- `@Incoming` to mark incoming data (e.g. from a REST API)
+- `@Outgoing` to mark outgoing data (e.g. as a response to an API call from another service)
+- `@Persisted` to mark persisted data (e.g. in a database)
+- `@ReadModel` to mark read models (e.g. in a CQRS setup)
+
+You can also use multiple of these annotations on a single class. Note that `@ReadModel` will automatically classify the
+data both as incoming and as persisted.
+
+You can document the PII level of each field in the class with the `@Field` annotation which accepts the following enum
+values as parameter:
+
+- `PII`
+- `PSEUDONYM`
+- `NON_PII`
+
+#### Configuration files
+
+You can also provide one or multiple configuration files in yaml format to classify your data.
+
+Each configuration file consists of a list of classes, containing the fully qualified class name, one or multiple blocks
+describing the data flow and purpose of the data (analogous to the annotations), and a list of fields with their PII
+level.
+
+The following example illustrates the structure of the configuration file:
+
+```yaml
+classes:
+  - className: cloud.rio.example.adapter.restclient.IncomingDTO
+    incoming:
+      whatToDo: Forward via API
+      whereFrom: Some external service
+      links:
+        - cloud.rio.example.adapter.rest.OutgoingDTO
+    fields:
+      - name: id
+        level: PSEUDONYM
+      - name: name
+        level: PII
+      - name: description
+        level: NON_PII
+  - className: cloud.rio.example.adapter.rest.OutgoingDTO
+    outgoing:
+      sharedWith: Exposed via API
+      why: Display in frontend
+      links:
+      - cloud.rio.example.adapter.restclient.IncomingDTO
+    fields:
+      ...
+  - className: cloud.rio.example.adapter.db.PersistedEntity
+    persisted:
+      retention: 6 months
+      responsibleForDeletion: Automatic deletion job
+      links: []
+    fields:
+      ...
+  - className: cloud.rio.example.adapter.readmodel.ReadModel
+    readModel:
+      whatToDo: Persist in DB
+      whereFrom: Some external service
+      retention: 6 months
+      responsibleForDeletion: Automatic deletion job
+      links: []
+    fields:
+      ...
+```
+
+### Apply and configure the plugin
+
+To use the plugin, add the plugin to the plugins block of your `build.gradle.kts` file and add the core dependency to
+the compile time classpath:
+
+```kotlin
+plugins {
+    id("cloud.rio.gdprdoc") version "2.0.1"
+}
+
+dependencies {
+    compileOnly("cloud.rio.gdprdoc:core:2.0.1")
+}
+```
+
+You can configure the documentation generation task to change the output file name and location, and to specify the
+configuration files to use, for example in the following way:
+
+```kotlin
+tasks {
+    generateGdprDocumentation {
+        markdownReport = file("docs/gdpr/gdpr-documentation.md")
+        additionalGdprDataFiles.setFrom(
+            fileTree("src/main/resources") { include("**/gdpr-documentation.yaml") },
+        )
+    }
+}
+```
+
+By default, the output will be written to `build/reports/gdpr-documentation.md`, and no configuration files will be
+used.
+
+### Generate the documentation
 
 Generate the documentation by running:
+
 ```
 ./gradlew generateGdprDocumentation
 ```
-You find the documentation in `build/reports/gdpr-documentation.md`. It currently needs to be manually
-copied to `docs/gdpr-documentation.md`
+
+You find the documentation in `build/reports/gdpr-documentation.md` unless you configured a different location (see
+above).
 
 Make sure to enable PlantUML in your markdown renderer in your IDE to see the Data Flow Diagram.
 Backstage also supports PlantUML, so it should work there without additional setup.
@@ -34,54 +152,61 @@ Backstage also supports PlantUML, so it should work there without additional set
 ### CI/CD pipeline
 
 This plugin uses GitHub actions to build and deploy the plugin to the Gradle Plugin Portal.
-The workflow is defined in `.github/workflows/build-and-deploy.yaml`and triggered on every push 
+The workflow is defined in `.github/workflows/build-and-deploy.yaml`and triggered on every push
 to the `main` branch and on every pull request.
 
 ### Dependabot
-This repository uses [dependabot](https://dependabot.com/) to keep dependencies up to date. 
+
+This repository uses [dependabot](https://dependabot.com/) to keep dependencies up to date.
 Dependabot is configured in `.github/dependabot.yaml`.
 
 ### Release process
 
-The workflow uses the [release-please-action from Google](https://github.com/googleapis/release-please-action). 
+The workflow uses the [release-please-action from Google](https://github.com/googleapis/release-please-action).
 
-> Release Please assumes you are using Conventional Commit messages.
+> _release-please_ assumes you are using Conventional Commit messages.
 >
 > The most important prefixes you should have in mind are:
 >
 > fix: which represents bug fixes, and correlates to a SemVer patch.
 >
 > feat: which represents a new feature, and correlates to a SemVer minor.
 >
-> feat!:, or fix!:, refactor!:, etc., which represent a breaking change (indicated by the !) and will result in a SemVer major.
+> feat!:, or fix!:, refactor!:, etc., which represent a breaking change (indicated by the !) and will result in a SemVer
+> major.
 
-When release-please detects one or more conventional commits, it will create or update a pull request. 
+When release-please detects one or more conventional commits, it will create or update a pull request.
 Once the pull request is merged, the workflow will create a new release and deploy the plugin.
 
 ### Manually trigger the release process
 
 All commits not following the conventional commit format will be ignored by the release-please-action.
 Examples include:
+
 - updating documentation
 - merge dependabot updates
 - ...
 
 To trigger the action simply create an empty commit following the conventional commit format, e.g.:
+
 ```
 git commit --allow-empty -m "fix: prepare next release. Update dependencies"
 git push
 ```
 
 ### Secret management
-The required secrets for the GitHub actions are stored in the repository settings under "Secrets and variables" -> "Actions".
-Currently, they are not managed as code. 
+
+The required secrets for the GitHub actions are stored in the repository settings under "Secrets and variables" -> "
+Actions".
+Currently, they are not managed as code.
 
 ### Build and test the plugin
+
 ```
 ./gradlew clean build
 ```
 
-### Build and the the example project
+### Build and test the example project
 
 ```
 cd test

plugin/build.gradle.kts

@@ -17,6 +17,7 @@
 
 plugins {
     kotlin("jvm")
+    kotlin("plugin.serialization") version "1.9.0"
     id("com.gradle.plugin-publish") version "1.3.1"
     id("com.gradleup.shadow") version "8.3.9"
     `java-gradle-plugin`
@@ -28,6 +29,7 @@ repositories {
 
 dependencies {
     compileOnly(gradleApi())
+    implementation("com.charleskorn.kaml:kaml:0.92.0")
     implementation(kotlin("stdlib"))
     implementation("io.github.classgraph:classgraph:4.8.162")
     implementation(project(":core"))
@@ -48,8 +50,9 @@ gradlePlugin {
             id = "cloud.rio.gdprdoc"
             implementationClass = "cloud.rio.gdprdoc.GdprDocumentationPlugin"
             displayName = "RIO GDPR documentation plugin"
-            description = "Gradle plugin to generate data classification documentation (needed for the GDPR documentation) for your project based on annotations on data classes"
-            tags.set(listOf("gdpr", "documentation","rio", "rio.cloud"))
+            description =
+                "Gradle plugin to generate data classification documentation (needed for the GDPR documentation) for your project based on annotations on data classes"
+            tags.set(listOf("gdpr", "documentation", "rio", "rio.cloud"))
         }
     }
 }

plugin/src/main/kotlin/cloud/rio/gdprdoc/GenerateGdprDocumentationTask.kt

@@ -16,6 +16,8 @@
 
 package cloud.rio.gdprdoc
 
+import cloud.rio.gdprdoc.additionalgdprdata.AdditionalGdprDataLoader
+import cloud.rio.gdprdoc.additionalgdprdata.AdditionalGdprDataMapper
 import cloud.rio.gdprdoc.annotations.GdprData
 import cloud.rio.gdprdoc.report.GdprDataItem
 import cloud.rio.gdprdoc.report.GdprItemId
@@ -31,6 +33,7 @@ import org.gradle.api.provider.Property
 import org.gradle.api.tasks.Classpath
 import org.gradle.api.tasks.Input
 import org.gradle.api.tasks.InputFiles
+import org.gradle.api.tasks.Optional
 import org.gradle.api.tasks.OutputFile
 import org.gradle.api.tasks.TaskAction
 import org.gradle.internal.extensions.stdlib.uncheckedCast
@@ -51,6 +54,10 @@ abstract class GenerateGdprDocumentationTask : DefaultTask() {
     @get:OutputFile
     abstract val markdownReport: RegularFileProperty
 
+    @get:InputFiles
+    @get:Optional
+    abstract val additionalGdprDataFiles: ConfigurableFileCollection
+
     @TaskAction
     fun process() {
 
@@ -90,6 +97,26 @@ abstract class GenerateGdprDocumentationTask : DefaultTask() {
             }
         }
 
+        val additionalGdprDataLoader = AdditionalGdprDataLoader()
+        val additionalGdprDataMapper = AdditionalGdprDataMapper(classPathFiles, logger)
+        additionalGdprDataFiles.files.forEach {
+            logger.lifecycle("Loading additional GDPR data from file: ${it.absolutePath}")
+            val additionalGdprData = try {
+                additionalGdprDataLoader.loadAdditionalGdprDataFromYamlFile(it)
+            } catch (e: Exception) {
+                logger.warn("Cannot read additional GDPR data from file ${it.absolutePath}: ${e.message}")
+                return@forEach
+            }
+            val (additionalGdprDataItems, additionalLinks) = additionalGdprDataMapper.mapToGdprDataItems(
+                additionalGdprData
+            )
+            additionalGdprDataItems.forEach { newItem ->
+                gdprDataItems.removeIf { existingItem -> existingItem.id == newItem.id }
+                gdprDataItems.add(newItem)
+            }
+            linkTargetClassesByItemId.putAll(additionalLinks)
+        }
+
         val gdprItemLinks = linkTargetClassesByItemId.flatMap { (source, targetClasses) ->
             targetClasses.flatMap { createLinks(sourceId = source, targetClassName = it, items = gdprDataItems) }
         }.toSet()

plugin/src/main/kotlin/cloud/rio/gdprdoc/additionalgdprdata/AdditionalGdprData.kt

@@ -0,0 +1,73 @@
+/*
+ *  Copyright 2025 TB Digital Services GmbH
+ *
+ *  Licensed under the Apache License, Version 2.0 (the "License");
+ *  you may not use this file except in compliance with the License.
+ *  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ *  Unless required by applicable law or agreed to in writing, software
+ *  distributed under the License is distributed on an "AS IS" BASIS,
+ *  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  See the License for the specific language governing permissions and
+ *  limitations under the License.
+ */
+
+package cloud.rio.gdprdoc.additionalgdprdata
+
+import cloud.rio.gdprdoc.annotations.READ_MODEL_DEFAULT_RESPONSIBLE_FOR_DELETION
+import cloud.rio.gdprdoc.annotations.READ_MODEL_DEFAULT_RETENTION
+import cloud.rio.gdprdoc.model.PiiLevel
+import kotlinx.serialization.Serializable
+
+@Serializable
+data class AdditionalGdprData(
+    val classes: List<AdditionalGdprDataItem>,
+)
+
+@Serializable
+data class AdditionalGdprDataItem(
+    val className: String,
+    val outgoing: Outgoing? = null,
+    val incoming: Incoming? = null,
+    val persisted: Persisted? = null,
+    val readModel: ReadModel? = null,
+    val fields: List<Field>,
+)
+
+@Serializable
+data class Outgoing(
+    val sharedWith: String,
+    val why: String,
+    val links: List<String> = emptyList(),
+)
+
+@Serializable
+data class Incoming(
+    val whereFrom: String,
+    val whatToDo: String,
+    val links: List<String> = emptyList(),
+)
+
+@Serializable
+data class Persisted(
+    val retention: String,
+    val responsibleForDeletion: String,
+    val links: List<String> = emptyList(),
+)
+
+@Serializable
+data class ReadModel(
+    val whereFrom: String,
+    val whatToDo: String,
+    val retention: String = READ_MODEL_DEFAULT_RETENTION,
+    val responsibleForDeletion: String = READ_MODEL_DEFAULT_RESPONSIBLE_FOR_DELETION,
+    val links: List<String> = emptyList(),
+)
+
+@Serializable
+data class Field(
+    val name: String,
+    val level: PiiLevel,
+)