From 228418581aefc3342e5023f8c5151768b59b1fc1 Mon Sep 17 00:00:00 2001 From: "andrei.kislitsyn" Date: Mon, 14 Jul 2025 17:05:04 +0400 Subject: [PATCH 1/5] data schemas topics fix --- docs/StardustDocs/d.tree | 22 ++-- docs/StardustDocs/topics/concepts/schemas.md | 46 -------- .../DataSchemaGenerationGradle.md | 0 .../DataSchemaGenerationMethods.md | 0 docs/StardustDocs/topics/schemas/schemas.md | 110 ++++++++++++++++++ .../{concepts => schemas}/schemasCustom.md | 0 .../schemasExternalJupyter.md | 0 .../{concepts => schemas}/schemasGradle.md | 0 .../schemasImportOpenApiGradle.md | 0 .../schemasImportOpenApiJupyter.md | 0 .../schemasImportSqlGradle.md | 0 .../schemasInheritance.md | 0 .../{concepts => schemas}/schemasJupyter.md | 0 13 files changed, 121 insertions(+), 57 deletions(-) delete mode 100644 docs/StardustDocs/topics/concepts/schemas.md rename docs/StardustDocs/topics/{concepts => schemas}/DataSchemaGenerationGradle.md (100%) rename docs/StardustDocs/topics/{concepts => schemas}/DataSchemaGenerationMethods.md (100%) create mode 100644 docs/StardustDocs/topics/schemas/schemas.md rename docs/StardustDocs/topics/{concepts => schemas}/schemasCustom.md (100%) rename docs/StardustDocs/topics/{concepts => schemas}/schemasExternalJupyter.md (100%) rename docs/StardustDocs/topics/{concepts => schemas}/schemasGradle.md (100%) rename docs/StardustDocs/topics/{concepts => schemas}/schemasImportOpenApiGradle.md (100%) rename docs/StardustDocs/topics/{concepts => schemas}/schemasImportOpenApiJupyter.md (100%) rename docs/StardustDocs/topics/{concepts => schemas}/schemasImportSqlGradle.md (100%) rename docs/StardustDocs/topics/{concepts => schemas}/schemasInheritance.md (100%) rename docs/StardustDocs/topics/{concepts => schemas}/schemasJupyter.md (100%) diff --git a/docs/StardustDocs/d.tree b/docs/StardustDocs/d.tree index 42a51a1b76..a859fa33d3 100644 --- a/docs/StardustDocs/d.tree +++ b/docs/StardustDocs/d.tree @@ -32,19 +32,19 @@ - - - - - - - - - - - + + + + + + + + + + + diff --git a/docs/StardustDocs/topics/concepts/schemas.md b/docs/StardustDocs/topics/concepts/schemas.md deleted file mode 100644 index 7a921bd08f..0000000000 --- a/docs/StardustDocs/topics/concepts/schemas.md +++ /dev/null @@ -1,46 +0,0 @@ -[//]: # (title: Working with Data Schemas) - - - -The Kotlin DataFrame library provides typed data access via [generation of extension properties](extensionPropertiesApi.md) for -type `DataFrame`, where -`T` is a marker class that represents `DataSchema` of [`DataFrame`](DataFrame.md). - -Schema of [`DataFrame`](DataFrame.md) is a mapping from column names to column types of [`DataFrame`](DataFrame.md). -It ignores order of columns in [`DataFrame`](DataFrame.md), but tracks column hierarchy. - -In Jupyter environment compile-time [`DataFrame`](DataFrame.md) schema is synchronized with real-time data after every cell execution. - -In IDEA projects, you can use the [Gradle plugin](schemasGradle.md#configuration) to extract schema from the dataset -and generate extension properties. - - -## Popular use cases with Data Schemas - -Here's a list of the most popular use cases with Data Schemas. - -* [**Data Schemas in Gradle projects**](schemasGradle.md)
- If you are developing a server application and building it with Gradle. - -* [**DataSchema workflow in Jupyter**](schemasJupyter.md)
- If you prefer Notebooks. - -* [**Schema inheritance**](schemasInheritance.md)
- It's worth knowing how to reuse Data Schemas generated earlier. - -* [**Custom Data Schemas**](schemasCustom.md)
- Sometimes it is necessary to create your own scheme. - -* [**Use external Data Schemas in Jupyter**](schemasExternalJupyter.md)
- Sometimes it is convenient to extract reusable code from Jupyter Notebook into the Kotlin JVM library. - Schema interfaces should also be extracted if this code uses Custom Data Schemas. - -* [**Schema Definitions from SQL Databases in Gradle Project**](schemasImportSqlGradle.md)
- When you need to take data from the SQL database. - -* [**Import OpenAPI 3.0.0 Schemas (Experimental) in Gradle Project**](schemasImportOpenApiGradle.md)
- When you need to take data from the endpoint with OpenAPI Schema. - -* [**Import Data Schemas, e.g. from OpenAPI 3.0.0 (Experimental), in Jupyter**](schemasImportOpenApiJupyter.md)
- Similar to [importing OpenAPI Data Schemas in Gradle projects](schemasImportOpenApiGradle.md), - you can also do this in Jupyter Notebooks. diff --git a/docs/StardustDocs/topics/concepts/DataSchemaGenerationGradle.md b/docs/StardustDocs/topics/schemas/DataSchemaGenerationGradle.md similarity index 100% rename from docs/StardustDocs/topics/concepts/DataSchemaGenerationGradle.md rename to docs/StardustDocs/topics/schemas/DataSchemaGenerationGradle.md diff --git a/docs/StardustDocs/topics/concepts/DataSchemaGenerationMethods.md b/docs/StardustDocs/topics/schemas/DataSchemaGenerationMethods.md similarity index 100% rename from docs/StardustDocs/topics/concepts/DataSchemaGenerationMethods.md rename to docs/StardustDocs/topics/schemas/DataSchemaGenerationMethods.md diff --git a/docs/StardustDocs/topics/schemas/schemas.md b/docs/StardustDocs/topics/schemas/schemas.md new file mode 100644 index 0000000000..ae81ae4703 --- /dev/null +++ b/docs/StardustDocs/topics/schemas/schemas.md @@ -0,0 +1,110 @@ +[//]: # (title: Data Schemas) + +The Kotlin DataFrame library provides typed data access via +[generation of extension properties](extensionPropertiesApi.md) for type +[`DataFrame`](DataFrame.md) (as well as [`DataRow`](DataRow.md)), where +`T` is a marker class that represents `DataSchema` of [`DataFrame`](DataFrame.md). + +Schema of [`DataFrame`](DataFrame.md) is a mapping from column names to column types of [`DataFrame`](DataFrame.md). +Data schema can be interpreted as a Kotlin interface or class. If the dataframe is hierarchical - contains +[column group](DataColumn.md#columngroup) or [column of dataframes](DataColumn.md#framecolumn), data schema +takes it into account and there is a separate class for each column group or inner `DataFrame`. + +For example, consider a simple hierarchical dataframe from +. + +This dataframe consists of two columns: `name`, which is a `String` column, and `info`, +which is a [**column group**](DataColumn.md#columngroup) containing two nested +[value columns](DataColumn.md#valuecolumn) — +`age` of type `Int`, and `height` of type `Double`. + + + + + + + + + + + + + + + + + + + + + + + + + +
nameinfo
ageheight
Alice23175.5
Bob27160.2
+ +Data schema corresponding to this dataframe can be represented like this : + +```kotlin +// Data schema of the "info" column group +@DataSchema +data class Info( + val age: Int, + val height: Float +) + +// Data schema of the entire dataframe +@DataSchema +data class Person( + val info: Info, + val name: String +) +``` + +[Extension properties](extensionPropertiesApi.md) for the `DataFrame` +are generated according to this schema and can be used for accessing columns and usage in operations: + +```kotlin +// Assuming `df` has type DataFrame + +// Get "age" column from "info" group +df.info.age + +// Select "name" and "height" columns +df.select { name and info.height } + +// Filter rows by age value +df.filter { age >= 18} +``` + + +## Popular use cases with Data Schemas + +Here's a list of the most popular use cases with Data Schemas. + +* [**Data Schemas in Gradle projects**](schemasGradle.md)
+ If you are developing a server application and building it with Gradle. + +* [**DataSchema workflow in Jupyter**](schemasJupyter.md)
+ If you prefer Notebooks. + +* [**Schema inheritance**](schemasInheritance.md)
+ It's worth knowing how to reuse Data Schemas generated earlier. + +* [**Custom Data Schemas**](schemasCustom.md)
+ Sometimes it is necessary to create your own scheme. + +* [**Use external Data Schemas in Jupyter**](schemasExternalJupyter.md)
+ Sometimes it is convenient to extract reusable code from Jupyter Notebook into the Kotlin JVM library. + Schema interfaces should also be extracted if this code uses Custom Data Schemas. + +* [**Schema Definitions from SQL Databases in Gradle Project**](schemasImportSqlGradle.md)
+ When you need to take data from the SQL database. + +* [**Import OpenAPI 3.0.0 Schemas (Experimental) in Gradle Project**](schemasImportOpenApiGradle.md)
+ When you need to take data from the endpoint with OpenAPI Schema. + +* [**Import Data Schemas, e.g. from OpenAPI 3.0.0 (Experimental), in Jupyter**](schemasImportOpenApiJupyter.md)
+ Similar to [importing OpenAPI Data Schemas in Gradle projects](schemasImportOpenApiGradle.md), + you can also do this in Jupyter Notebooks. diff --git a/docs/StardustDocs/topics/concepts/schemasCustom.md b/docs/StardustDocs/topics/schemas/schemasCustom.md similarity index 100% rename from docs/StardustDocs/topics/concepts/schemasCustom.md rename to docs/StardustDocs/topics/schemas/schemasCustom.md diff --git a/docs/StardustDocs/topics/concepts/schemasExternalJupyter.md b/docs/StardustDocs/topics/schemas/schemasExternalJupyter.md similarity index 100% rename from docs/StardustDocs/topics/concepts/schemasExternalJupyter.md rename to docs/StardustDocs/topics/schemas/schemasExternalJupyter.md diff --git a/docs/StardustDocs/topics/concepts/schemasGradle.md b/docs/StardustDocs/topics/schemas/schemasGradle.md similarity index 100% rename from docs/StardustDocs/topics/concepts/schemasGradle.md rename to docs/StardustDocs/topics/schemas/schemasGradle.md diff --git a/docs/StardustDocs/topics/concepts/schemasImportOpenApiGradle.md b/docs/StardustDocs/topics/schemas/schemasImportOpenApiGradle.md similarity index 100% rename from docs/StardustDocs/topics/concepts/schemasImportOpenApiGradle.md rename to docs/StardustDocs/topics/schemas/schemasImportOpenApiGradle.md diff --git a/docs/StardustDocs/topics/concepts/schemasImportOpenApiJupyter.md b/docs/StardustDocs/topics/schemas/schemasImportOpenApiJupyter.md similarity index 100% rename from docs/StardustDocs/topics/concepts/schemasImportOpenApiJupyter.md rename to docs/StardustDocs/topics/schemas/schemasImportOpenApiJupyter.md diff --git a/docs/StardustDocs/topics/concepts/schemasImportSqlGradle.md b/docs/StardustDocs/topics/schemas/schemasImportSqlGradle.md similarity index 100% rename from docs/StardustDocs/topics/concepts/schemasImportSqlGradle.md rename to docs/StardustDocs/topics/schemas/schemasImportSqlGradle.md diff --git a/docs/StardustDocs/topics/concepts/schemasInheritance.md b/docs/StardustDocs/topics/schemas/schemasInheritance.md similarity index 100% rename from docs/StardustDocs/topics/concepts/schemasInheritance.md rename to docs/StardustDocs/topics/schemas/schemasInheritance.md diff --git a/docs/StardustDocs/topics/concepts/schemasJupyter.md b/docs/StardustDocs/topics/schemas/schemasJupyter.md similarity index 100% rename from docs/StardustDocs/topics/concepts/schemasJupyter.md rename to docs/StardustDocs/topics/schemas/schemasJupyter.md From 9a7fb0fb4427c3b531db2aec139855e06737a208 Mon Sep 17 00:00:00 2001 From: "andrei.kislitsyn" Date: Tue, 22 Jul 2025 18:18:02 +0400 Subject: [PATCH 2/5] data schemas docs restructure --- docs/StardustDocs/d.tree | 16 +- docs/StardustDocs/topics/FAQ.md | 21 +- .../topics/gettingStarted/Modules.md | 2 +- .../topics/readSqlFromCustomDatabase.md | 2 +- .../Data-Schemas-In-Kotlin-Notebook.md | 180 ++++++++++++++++++ .../schemas/DataSchemaGenerationMethods.md | 4 +- .../topics/schemas/Migration-From-Plugins.md | 44 +++++ .../Gradle-Plugin.md} | 2 +- .../schemas/{ => gradle}/schemasGradle.md | 2 +- .../schemasImportOpenApiGradle.md | 0 .../{ => gradle}/schemasImportSqlGradle.md | 0 docs/StardustDocs/topics/schemas/schemas.md | 114 +++++++---- .../topics/schemas/schemasCustom.md | 80 -------- .../topics/schemas/schemasExternalJupyter.md | 50 ----- .../schemas/schemasImportOpenApiJupyter.md | 4 +- .../topics/schemas/schemasJupyter.md | 58 ------ 16 files changed, 324 insertions(+), 255 deletions(-) create mode 100644 docs/StardustDocs/topics/schemas/Data-Schemas-In-Kotlin-Notebook.md create mode 100644 docs/StardustDocs/topics/schemas/Migration-From-Plugins.md rename docs/StardustDocs/topics/schemas/{DataSchemaGenerationGradle.md => gradle/Gradle-Plugin.md} (99%) rename docs/StardustDocs/topics/schemas/{ => gradle}/schemasGradle.md (97%) rename docs/StardustDocs/topics/schemas/{ => gradle}/schemasImportOpenApiGradle.md (100%) rename docs/StardustDocs/topics/schemas/{ => gradle}/schemasImportSqlGradle.md (100%) delete mode 100644 docs/StardustDocs/topics/schemas/schemasCustom.md delete mode 100644 docs/StardustDocs/topics/schemas/schemasExternalJupyter.md delete mode 100644 docs/StardustDocs/topics/schemas/schemasJupyter.md diff --git a/docs/StardustDocs/d.tree b/docs/StardustDocs/d.tree index 54225b918a..f2ef4fd112 100644 --- a/docs/StardustDocs/d.tree +++ b/docs/StardustDocs/d.tree @@ -35,17 +35,16 @@
- - + - - - - + - + + + + + - @@ -194,4 +193,5 @@