Skip to content

Commit 5c6e8f8

Browse files
committed
Merge branch 'master' into jetbrains-junie-issue-1417-run-9adb9248-1a5b-4505-8012-2c3061ca05cf
# Conflicts: # core/src/main/kotlin/org/jetbrains/kotlinx/dataframe/util/deprecationMessages.kt
2 parents 56e7a6d + 281902d commit 5c6e8f8

File tree

210 files changed

+619004
-22549
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

210 files changed

+619004
-22549
lines changed

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,3 +9,6 @@
99
build
1010
.ipynb_checkpoints
1111
local.properties
12+
13+
# macOS
14+
.DS_Store

README.md

Lines changed: 66 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,45 @@
11
# Kotlin DataFrame: typesafe in-memory structured data processing for JVM
2+
23
[![JetBrains incubator project](https://jb.gg/badges/incubator.svg)](https://confluence.jetbrains.com/display/ALL/JetBrains+on+GitHub)
34
[![Kotlin component beta stability](https://img.shields.io/badge/project-beta-kotlin.svg?colorA=555555&colorB=DB3683&label=&logo=kotlin&logoColor=ffffff&logoWidth=10)](https://kotlinlang.org/docs/components-stability.html)
45
[![Kotlin](https://img.shields.io/badge/kotlin-2.0.20-blue.svg?logo=kotlin)](http://kotlinlang.org)
5-
[![Dynamic XML Badge](https://img.shields.io/badge/dynamic/xml?url=https%3A%2F%2Frepo1.maven.org%2Fmaven2%2Forg%2Fjetbrains%2Fkotlinx%2Fdataframe%2Fmaven-metadata.xml&query=%2F%2Fversion%5Bnot%28contains%28text%28%29%2C%22dev%22%29%29%20and%20not%28text%28%29%3D%221727%22%29%20%5D%5Blast%28%29%5D&label=Release%20version)](https://search.maven.org/artifact/org.jetbrains.kotlinx/dataframe)
6+
[![Dynamic XML Badge](https://img.shields.io/badge/dynamic/xml?url=https%3A%2F%2Frepo1.maven.org%2Fmaven2%2Forg%2Fjetbrains%2Fkotlinx%2Fdataframe%2Fmaven-metadata.xml&query=%2F%2Fversion%5Bnot%28contains%28text%28%29%2C%27dev%27%29%29+and+not%28contains%28text%28%29%2C%27n%27%29%29%5D%5Blast%28%29%5D&label=Release%20version)](https://search.maven.org/artifact/org.jetbrains.kotlinx/dataframe)
67
[![Dynamic XML Badge](https://img.shields.io/badge/dynamic/xml?url=https%3A%2F%2Frepo1.maven.org%2Fmaven2%2Forg%2Fjetbrains%2Fkotlinx%2Fdataframe%2Fmaven-metadata.xml&query=%2F%2Fversion%5Bcontains%28text%28%29%2C%22dev%22%29%5D%5Blast%28%29%5D&label=Dev%20version&color=yellow
78
)](https://search.maven.org/artifact/org.jetbrains.kotlinx/dataframe)
89
[![GitHub License](https://img.shields.io/badge/license-Apache%20License%202.0-blue.svg?style=flat)](http://www.apache.org/licenses/LICENSE-2.0)
910
[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/Kotlin/dataframe/HEAD)
1011

11-
Kotlin DataFrame aims to reconcile Kotlin's static typing with the dynamic nature of data by utilizing both the full power of the Kotlin language and the opportunities provided by intermittent code execution in Jupyter notebooks and REPL.
12+
Kotlin DataFrame aims to reconcile Kotlin's static typing with the dynamic nature of data by utilizing both the full
13+
power of the Kotlin language and the opportunities provided by intermittent code execution in Jupyter notebooks and
14+
REPL.
1215

1316
* **Hierarchical** — represents hierarchical data structures, such as JSON or a tree of JVM objects.
1417
* **Functional** — the data processing pipeline is organized in a chain of `DataFrame` transformation operations.
15-
* **Immutable** — every operation returns a new instance of `DataFrame` reusing underlying storage wherever it's possible.
18+
* **Immutable** — every operation returns a new instance of `DataFrame` reusing underlying storage wherever it's
19+
possible.
1620
* **Readable** — data transformation operations are defined in DSL close to natural language.
1721
* **Practical** — provides simple solutions for common problems and the ability to perform complex tasks.
1822
* **Minimalistic** — simple, yet powerful data model of three column kinds.
19-
* **Interoperable** — convertable with Kotlin data classes and collections. This also means conversion to/from other libraries' data structures is usually quite straightforward!
23+
* **Interoperable** — convertable with Kotlin data classes and collections. This also means conversion to/from other
24+
libraries' data structures is usually quite straightforward!
2025
* **Generic** — can store objects of any type, not only numbers or strings.
21-
* **Typesafe** — on-the-fly [generation of extension properties](https://kotlin.github.io/dataframe/extensionpropertiesapi.html) for type safe data access with Kotlin-style care for null safety.
22-
* **Polymorphic** — type compatibility derives from column schema compatibility. You can define a function that requires a special subset of columns in a dataframe but doesn't care about other columns.
26+
* **Typesafe**
27+
on-the-fly [generation of extension properties](https://kotlin.github.io/dataframe/extensionpropertiesapi.html) for
28+
type safe data access with Kotlin-style care for null safety.
29+
* **Polymorphic** — type compatibility derives from column schema compatibility. You can define a function that requires
30+
a special subset of columns in a dataframe but doesn't care about other columns.
2331
In notebooks this works out-of-the-box. In ordinary projects this requires casting (for now).
2432

25-
Integrates with [Kotlin Notebook](https://kotlinlang.org/docs/kotlin-notebook-overview.html).
26-
Inspired by [krangl](https://github.com/holgerbrandl/krangl), Kotlin Collections and [pandas](https://pandas.pydata.org/)
33+
Integrates with [Kotlin Notebook](https://kotlinlang.org/docs/kotlin-notebook-overview.html).
34+
Inspired by [krangl](https://github.com/holgerbrandl/krangl), Kotlin Collections
35+
and [pandas](https://pandas.pydata.org/)
2736

2837
## 🚀 Quickstart
2938

3039
Looking for a fast and simple way to learn the basics?
3140
Get started in minutes with our [Quickstart Guide](https://kotlin.github.io/dataframe/quickstart.html).
3241

33-
It walks you through the core features of Kotlin DataFrame with minimal setup and clear examples
42+
It walks you through the core features of Kotlin DataFrame with minimal setup and clear examples
3443
— perfect for getting up to speed in just a few minutes.
3544

3645
[![quickstart_preview](docs/StardustDocs/images/guides/quickstart_preview.png)](https://kotlin.github.io/dataframe/quickstart.html)
@@ -54,7 +63,7 @@ You could find the following articles there:
5463

5564
### What's new
5665

57-
1.0.0-Beta2: [Release notes](https://github.com/Kotlin/dataframe/releases/tag/v1.0.0-Beta2)
66+
1.0.0-Beta3: [Release notes](https://github.com/Kotlin/dataframe/releases/tag/v1.0.0-Beta3)
5867

5968
Check out this [notebook with new features](examples/notebooks/feature_overviews/0.15/new_features.ipynb) in v0.15.
6069

@@ -66,7 +75,7 @@ Check out this [notebook with new features](examples/notebooks/feature_overviews
6675
### Kotlin Notebook
6776

6877
You can use Kotlin DataFrame in [Kotlin Notebook](https://kotlinlang.org/docs/kotlin-notebook-overview.html),
69-
or other interactive environment with [Kotlin Jupyter Kernel](https://github.com/Kotlin/kotlin-jupyter) support,
78+
or other interactive environment with [Kotlin Jupyter Kernel](https://github.com/Kotlin/kotlin-jupyter) support,
7079
such as [Datalore](https://datalore.jetbrains.com/),
7180
and [Jupyter Notebook](https://jupyter.org/).
7281

@@ -90,7 +99,7 @@ Or manually specify the version:
9099
%use dataframe($dataframe_version)
91100
```
92101

93-
Refer to the
102+
Refer to the
94103
[Get started with Kotlin DataFrame in Kotlin Notebook](https://kotlin.github.io/dataframe/gettingstartedkotlinnotebook.html)
95104
for details.
96105

@@ -100,7 +109,7 @@ Add dependencies in the build.gradle.kts script:
100109

101110
```kotlin
102111
dependencies {
103-
implementation("org.jetbrains.kotlinx:dataframe:1.0.0-Beta2")
112+
implementation("org.jetbrains.kotlinx:dataframe:1.0.0-Beta3")
104113
}
105114
```
106115

@@ -115,7 +124,7 @@ repositories {
115124
Refer to the
116125
[Get started with Kotlin DataFrame on Gradle](https://kotlin.github.io/dataframe/gettingstartedgradle.html)
117126
for details.
118-
Also, check out the [custom setup page](https://kotlin.github.io/dataframe/gettingstartedgradleadvanced.html)
127+
Also, check out the [custom setup page](https://kotlin.github.io/dataframe/gettingstartedgradleadvanced.html)
119128
if you don't need some formats as dependencies,
120129
for Groovy, and for configurations specific to Android projects.
121130

@@ -124,32 +133,32 @@ for Groovy, and for configurations specific to Android projects.
124133
This example of Kotlin DataFrame code with
125134
the [Compiler Plugin](https://kotlin.github.io/dataframe/compiler-plugin.html) enabled.
126135
See [the full project](https://github.com/Kotlin/dataframe/tree/master/examples/kotlin-dataframe-plugin-example).
127-
See also
136+
See also
128137
[this example in Kotlin Notebook](https://github.com/Kotlin/dataframe/tree/master/examples/notebooks/readme_example.ipynb).
129138

130139
```kotlin
131140
val df = DataFrame
132-
// Read DataFrame from the CSV file.
133-
.readCsv("https://raw.githubusercontent.com/Kotlin/dataframe/master/data/jetbrains_repositories.csv")
134-
// And convert it to match the `Repositories` schema.
135-
.convertTo<Repositories>()
141+
// Read DataFrame from the CSV file.
142+
.readCsv("https://raw.githubusercontent.com/Kotlin/dataframe/master/data/jetbrains_repositories.csv")
143+
// And convert it to match the `Repositories` schema.
144+
.convertTo<Repositories>()
136145

137146
// Update the DataFrame.
138147
val reposUpdated = repos
139-
// Rename columns to CamelCase.
140-
.renameToCamelCase()
141-
// Rename "stargazersCount" column to "stars".
142-
.rename { stargazersCount }.into("stars")
143-
// Filter by the number of stars:
144-
.filter { stars > 50 }
145-
// Convert values in the "topic" column (which were `String` initially)
146-
// to the list of topics.
147-
.convert { topics }.with {
148-
val inner = it.removeSurrounding("[", "]")
148+
// Rename columns to CamelCase.
149+
.renameToCamelCase()
150+
// Rename "stargazersCount" column to "stars".
151+
.rename { stargazersCount }.into("stars")
152+
// Filter by the number of stars:
153+
.filter { stars > 50 }
154+
// Convert values in the "topic" column (which were `String` initially)
155+
// to the list of topics.
156+
.convert { topics }.with {
157+
val inner = it.removeSurrounding("[", "]")
149158
if (inner.isEmpty()) emptyList() else inner.split(',').map(String::trim)
150-
}
151-
// Add a new column with the number of topics.
152-
.add("topicCount") { topics.size }
159+
}
160+
// Add a new column with the number of topics.
161+
.add("topicCount") { topics.size }
153162

154163
// Write the updated DataFrame to a CSV file.
155164
reposUpdated.writeCsv("jetbrains_repositories_new.csv")
@@ -158,39 +167,45 @@ reposUpdated.writeCsv("jetbrains_repositories_new.csv")
158167
Explore [**more examples here**](https://kotlin.github.io/dataframe/guides-and-examples.html).
159168

160169
## Data model
170+
161171
* `DataFrame` is a list of columns with equal sizes and distinct names.
162172
* `DataColumn` is a named list of values. Can be one of three kinds:
163-
* `ValueColumn` — contains data
164-
* `ColumnGroup` — contains columns
165-
* `FrameColumn` — contains dataframes
173+
* `ValueColumn` — contains data
174+
* `ColumnGroup` — contains columns
175+
* `FrameColumn` — contains dataframes
166176

167177
## Visualizations
168178

169-
[Kandy](https://kotlin.github.io/kandy/welcome.html) plotting library provides seamless visualizations
179+
[Kandy](https://kotlin.github.io/kandy/welcome.html) plotting library provides seamless visualizations
170180
for your dataframes.
171181

172182
![kandy_preview](docs/StardustDocs/images/guides/kandy_gallery_preview.png)
173183

174184
## Kotlin, Kotlin Jupyter, Arrow, and JDK versions
175185

176-
This table shows the mapping between main library component versions and minimum supported Java versions.
177-
178-
| Kotlin DataFrame Version | Minimum Java Version | Kotlin Version | Kotlin Jupyter Version | Apache Arrow version |
179-
|--------------------------|----------------------|----------------|------------------------|----------------------|
180-
| 0.10.0 | 8 | 1.8.20 | 0.11.0-358 | 11.0.0 |
181-
| 0.10.1 | 8 | 1.8.20 | 0.11.0-358 | 11.0.0 |
182-
| 0.11.0 | 8 | 1.8.20 | 0.11.0-358 | 11.0.0 |
183-
| 0.11.1 | 8 | 1.8.20 | 0.11.0-358 | 11.0.0 |
184-
| 0.12.0 | 8 | 1.9.0 | 0.11.0-358 | 11.0.0 |
185-
| 0.12.1 | 8 | 1.9.0 | 0.11.0-358 | 11.0.0 |
186-
| 0.13.1 | 8 | 1.9.22 | 0.12.0-139 | 15.0.0 |
187-
| 0.14.1 | 8 | 2.0.20 | 0.12.0-139 | 17.0.0 |
188-
| 0.15.0 | 8 | 2.0.20 | 0.12.0-139 | 18.1.0 |
189-
| 1.0.0-Beta2 | 8 / 11 | 2.0.20 | 0.12.0-383 | 18.1.0 |
186+
This table shows the mapping between main library component versions and minimum supported Java versions, along with
187+
other recommended versions.
188+
189+
| Kotlin DataFrame Version | Minimum Java Version | Kotlin Version | Kotlin Jupyter Version | Apache Arrow Version | Compiler Plugin Version | Compatible Kandy version |
190+
|--------------------------|----------------------|----------------|------------------------|----------------------|-------------------------|--------------------------|
191+
| 0.10.0 | 8 | 1.8.20 | 0.11.0-358 | 11.0.0 | | |
192+
| 0.10.1 | 8 | 1.8.20 | 0.11.0-358 | 11.0.0 | | |
193+
| 0.11.0 | 8 | 1.8.20 | 0.11.0-358 | 11.0.0 | | |
194+
| 0.11.1 | 8 | 1.8.20 | 0.11.0-358 | 11.0.0 | | |
195+
| 0.12.0 | 8 | 1.9.0 | 0.11.0-358 | 11.0.0 | | |
196+
| 0.12.1 | 8 | 1.9.0 | 0.11.0-358 | 11.0.0 | | |
197+
| 0.13.1 | 8 | 1.9.22 | 0.12.0-139 | 15.0.0 | | |
198+
| 0.14.1 | 8 | 2.0.20 | 0.12.0-139 | 17.0.0 | | |
199+
| 0.15.0 | 8 | 2.0.20 | 0.12.0-139 | 18.1.0 | | 0.8.0 |
200+
| 1.0.0-Beta2 | 8 / 11 | 2.0.20 | 0.12.0-383 | 18.1.0 | 2.2.20-dev-3524 | 0.8.1-dev-66 |
201+
| 1.0.0-Beta3n (notebooks) | 8 / 11 | 2.2.20 | 0.15.0-587 (K1 only) | 18.3.0 | - | 0.8.1n |
202+
| 1.0.0-Beta3 | 8 / 11 | 2.2.20 | 0.15.0-587 | 18.3.0 | 2.2.20 / IDEA 2025.2+ | 0.8.1 |
190203

191204
## Code of Conduct
192205

193-
This project and the corresponding community are governed by the [JetBrains Open Source and Community Code of Conduct](https://confluence.jetbrains.com/display/ALL/JetBrains+Open+Source+and+Community+Code+of+Conduct). Please make sure you read it.
206+
This project and the corresponding community are governed by
207+
the [JetBrains Open Source and Community Code of Conduct](https://confluence.jetbrains.com/display/ALL/JetBrains+Open+Source+and+Community+Code+of+Conduct).
208+
Please make sure you read it.
194209

195210
## License
196211

build.gradle.kts

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -158,7 +158,7 @@ val modulesUsingJava11 = with(projects) {
158158
dataframeGeoJupyter,
159159
examples.ideaExamples.titanic,
160160
examples.ideaExamples.unsupportedDataSources,
161-
tests,
161+
samples,
162162
plugins.dataframeGradlePlugin,
163163
)
164164
}.map { it.path }

core/README.md

Lines changed: 21 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -10,15 +10,29 @@ At the moment, these integrations are still part of the `:core` module:
1010
- (deprecated) csv/tsv
1111
- html
1212

13-
### Korro code samples
13+
### KoDEx
14+
15+
The code you're working on needs to be edited in [src](src), but the KDocs are processed by
16+
[KoDEx](https://github.com/Jolanrensen/kodex) when the project is published (or the task
17+
is run manually). The generated sources with adjusted KDocs will be overwritten
18+
in [generated-sources](generated-sources).
19+
See the [KDoc Preprocessing Guide](../KDOC_PREPROCESSING.md) for more information.
20+
21+
KDocs can also be exported to HTML, for them to be reused on the website.
22+
Elements annotated with `@ExportAsHtml` will have their generated content be copied over to
23+
[docs/StardustDocs/resources/snippets/kdocs](../docs/StardustDocs/resources/snippets/kdocs).
24+
25+
### ~~Korro~~ (NOTE: This is being moved to [:samples](../samples))
26+
27+
> [Should be removed and moved to `:samples` module](https://github.com/Kotlin/dataframe/issues/898).
28+
29+
Tests in this module contain code samples used for import into documentation
30+
using [Korro](https://github.com/devcrocod/korro).
1431

15-
Code samples for the documentation website reside
16-
in [core/.../test/.../samples/api](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples that depend solely on `dataframe-core`),
17-
and [tests/.../samples/api](../tests/src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api) (for samples can depend on other I/O modules)
18-
and they are copied over to Markdown files in [docs/StardustDocs/topics](../docs/StardustDocs/topics)
19-
by [Korro](https://github.com/devcrocod/korro).
2032

21-
### ~~Explainer dataframes~~ NOTE: This is being moved to [tests](../tests)
33+
### ~~Explainer dataframes~~ (NOTE: This is being moved to [:samples](../samples))
34+
35+
> [Should be removed and migrated to SampleHelper](https://github.com/Kotlin/dataframe/issues/898).
2236
2337
Aside from code samples, `@TransformDataFrameExpressions` annotated test functions also generate sample
2438
dataframe HTML files that can be used as iFrames on the documentation website.
@@ -31,15 +45,3 @@ retrieves the intermediate DataFrame expressions thanks to our "explainer" compi
3145
We can also generate "normal" DataFrame samples for the website. This can be done using the
3246
[OtherSamples class](./src/test/kotlin/org/jetbrains/kotlinx/dataframe/samples/api/OtherSamples.kt). Generated
3347
HTML files will be stored in [docs/StardustDocs/resources/snippets/manual](../docs/StardustDocs/resources/snippets/manual).
34-
35-
### KoDEx
36-
37-
The code you're working on needs to be edited in [src](src), but the KDocs are processed by
38-
[KoDEx](https://github.com/Jolanrensen/kodex) when the project is published (or the task
39-
is run manually). The generated sources with adjusted KDocs will be overwritten
40-
in [generated-sources](generated-sources).
41-
See the [KDoc Preprocessing Guide](../KDOC_PREPROCESSING.md) for more information.
42-
43-
KDocs can also be exported to HTML, for them to be reused on the website.
44-
Elements annotated with `@ExportAsHtml` will have their generated content be copied over to
45-
[docs/StardustDocs/resources/snippets/kdocs](../docs/StardustDocs/resources/snippets/kdocs).

0 commit comments

Comments
 (0)