Compiler plugin crash when updating to kotlin `2.3.x`

[CHervaudBetclic]

Hello,

When trying to update my project which uses dataframe to kotlin 2.3.0 and 2.3.10, I encountered a FileAnalysisException from the dataframe compiler plugin.

I found those related issues talking about K2 incompatibility: #1650 & #1445.

I tried setting the kotlin.reflect.jvm.useK1Implementation system property using the following but it didn't work:

tasks {
  compileKotlin {
    dependsOn(generateOpenAPICode, generateKongBack)
    kotlinDaemonJvmArguments.add("-Dkotlin.reflect.jvm.useK1Implementation=true")
  }
}

Could you help me on how to disable K2 in my project to dataframe compiler project doesn't crash?

[koperagen]

Hi! DataFrame compiler plugin doesn't exist in K1, it's K2 specific, so let's try to resolve the issue you have. Can you please share the exception message?

[CHervaudBetclic]

Yeah sure, here is the stacktrace:

org.jetbrains.kotlin.util.FileAnalysisException: Somewhere in file /Users/myuser/dev/myproject/build/generated/src/main/kotlin/com/myapp/models/MyDto.kt: org.jetbrains.kotlin.utils.exceptions.KotlinIllegalArgumentExceptionWithAttachments: Unexpected status. Expected is FirResolvedDeclarationStatus, but was FirDeclarationStatusImpl
	at org.jetbrains.kotlin.util.AnalysisExceptionsKt.wrapIntoFileAnalysisExceptionIfNeeded(AnalysisExceptions.kt:62)
	at org.jetbrains.kotlin.fir.FirCliExceptionHandler.handleExceptionOnFileAnalysis(Utils.kt:269)
	at org.jetbrains.kotlin.fir.resolve.transformers.plugin.FirCompanionGenerationTransformer.transformFile(FirCompanionGenerationProcessor.kt:113)
	at org.jetbrains.kotlin.fir.resolve.transformers.plugin.FirCompanionGenerationTransformer.transformFile(FirCompanionGenerationProcessor.kt:39)
	at org.jetbrains.kotlin.fir.declarations.FirFile.transform(FirFile.kt:47)
	at org.jetbrains.kotlin.fir.resolve.transformers.FirTransformerBasedResolveProcessor.processFile(FirResolveProcessor.kt:53)
	at org.jetbrains.kotlin.fir.resolve.transformers.FirTotalResolveProcessor.process(FirTotalResolveProcessor.kt:36)
	at org.jetbrains.kotlin.fir.pipeline.AnalyseKt.runResolution(analyse.kt:24)
	at org.jetbrains.kotlin.fir.pipeline.FirUtilsKt.resolveAndCheckFir(firUtils.kt:76)
	at org.jetbrains.kotlin.cli.pipeline.jvm.JvmFrontendPipelinePhase.executePhase(JvmFrontendPipelinePhase.kt:269)
	at org.jetbrains.kotlin.cli.pipeline.jvm.JvmFrontendPipelinePhase.executePhase(JvmFrontendPipelinePhase.kt:55)
	at org.jetbrains.kotlin.cli.pipeline.PipelinePhase.phaseBody(PipelinePhase.kt:68)
	at org.jetbrains.kotlin.cli.pipeline.PipelinePhase.phaseBody(PipelinePhase.kt:58)
	at org.jetbrains.kotlin.config.phaser.NamedCompilerPhase.invoke(CompilerPhase.kt:102)
	at org.jetbrains.kotlin.backend.common.phaser.CompositePhase.invoke(PhaseBuilders.kt:22)
	at org.jetbrains.kotlin.config.phaser.CompilerPhaseKt.invokeToplevel(CompilerPhase.kt:53)
	at org.jetbrains.kotlin.cli.pipeline.AbstractCliPipeline.runPhasedPipeline(AbstractCliPipeline.kt:109)
	at org.jetbrains.kotlin.cli.pipeline.AbstractCliPipeline.execute(AbstractCliPipeline.kt:68)
	at org.jetbrains.kotlin.cli.jvm.K2JVMCompiler.doExecutePhased(K2JVMCompiler.kt:51)
	at org.jetbrains.kotlin.cli.jvm.K2JVMCompiler.doExecutePhased(K2JVMCompiler.kt:42)
	at org.jetbrains.kotlin.cli.common.CLICompiler.execImpl(CLICompiler.kt:89)
	at org.jetbrains.kotlin.cli.common.CLICompiler.exec(CLICompiler.kt:359)
	at org.jetbrains.kotlin.daemon.CompileServiceImpl.compile(CompileServiceImpl.kt:1706)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:360)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200)
	at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:714)
	at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:598)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:844)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:721)
	at java.base/java.security.AccessController.doPrivileged(AccessController.java:400)
	at java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:720)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: org.jetbrains.kotlin.utils.exceptions.KotlinIllegalArgumentExceptionWithAttachments: Unexpected status. Expected is FirResolvedDeclarationStatus, but was FirDeclarationStatusImpl
	at org.jetbrains.kotlin.fir.symbols.impl.UtilsKt.errorInLazyResolve(Utils.kt:42)
	at org.jetbrains.kotlin.fir.symbols.impl.UtilsKt.resolvedStatus(Utils.kt:30)
	at org.jetbrains.kotlin.fir.symbols.impl.FirClassLikeSymbol.getResolvedStatus(FirClassLikeSymbol.kt:56)
	at org.jetbrains.kotlinx.dataframe.plugin.extensions.TopLevelExtensionsGenerator.matchedClasses_delegate$lambda$0(TopLevelExtensionsGenerator.kt:216)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:86)
	at org.jetbrains.kotlinx.dataframe.plugin.extensions.TopLevelExtensionsGenerator.getMatchedClasses(TopLevelExtensionsGenerator.kt:41)
	at org.jetbrains.kotlinx.dataframe.plugin.extensions.TopLevelExtensionsGenerator.fields_delegate$lambda$0(TopLevelExtensionsGenerator.kt:54)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:86)
	at org.jetbrains.kotlinx.dataframe.plugin.extensions.TopLevelExtensionsGenerator.getFields(TopLevelExtensionsGenerator.kt:53)
	at org.jetbrains.kotlinx.dataframe.plugin.extensions.TopLevelExtensionsGenerator.getTopLevelCallableIds(TopLevelExtensionsGenerator.kt:74)
	at org.jetbrains.kotlin.fir.extensions.FirDeclarationGenerationExtension.topLevelCallableIdsCache$lambda$0(FirDeclarationGenerationExtension.kt:93)
	at kotlin.UnsafeLazyImpl.getValue(Lazy.kt:100)
	at org.jetbrains.kotlin.fir.caches.FirThreadUnsafeValue.getLazyValue(FirThreadUnsafeCachesFactory.kt:94)
	at org.jetbrains.kotlin.fir.caches.FirThreadUnsafeValue.getValue(FirThreadUnsafeCachesFactory.kt:95)
	at org.jetbrains.kotlin.fir.extensions.FirExtensionDeclarationsSymbolProvider$symbolNamesProvider$1.getPackageNamesWithTopLevelCallables(FirExtensionDeclarationsSymbolProvider.kt:181)
	at org.jetbrains.kotlin.fir.extensions.FirSwitchableExtensionDeclarationsSymbolProvider$symbolNamesProvider$1.getPackageNamesWithTopLevelCallables(FirSwitchableExtensionDeclarationsSymbolProvider.kt:51)
	at org.jetbrains.kotlin.fir.resolve.providers.FirCompositeCachedSymbolNamesProvider.computePackageNamesWithTopLevelCallables(FirCachedSymbolNamesProvider.kt:158)
	at org.jetbrains.kotlin.fir.resolve.providers.impl.FirCachingCompositeSymbolProvider$symbolNamesProvider$1.computePackageNamesWithTopLevelCallables(FirCachingCompositeSymbolProvider.kt:54)
	at org.jetbrains.kotlin.fir.resolve.providers.FirCachedSymbolNamesProvider.topLevelCallablePackageNames_delegate$lambda$0(FirCachedSymbolNamesProvider.kt:75)
	at kotlin.SafePublicationLazyImpl.getValue(LazyJVM.kt:125)
	at org.jetbrains.kotlin.fir.resolve.providers.FirCachedSymbolNamesProvider.getTopLevelCallablePackageNames(FirCachedSymbolNamesProvider.kt:72)
	at org.jetbrains.kotlin.fir.resolve.providers.FirCachedSymbolNamesProvider.getPackageNamesWithTopLevelCallables(FirCachedSymbolNamesProvider.kt:103)
	at org.jetbrains.kotlin.fir.resolve.providers.FirCachedSymbolNamesProvider.getTopLevelCallableNamesInPackage(FirCachedSymbolNamesProvider.kt:106)
	at org.jetbrains.kotlin.fir.resolve.providers.FirSymbolNamesProvider.mayHaveTopLevelCallable(FirSymbolNamesProvider.kt:129)
	at org.jetbrains.kotlin.fir.resolve.providers.impl.FirCachingCompositeSymbolProvider.getTopLevelFunctionSymbolsTo(FirCachingCompositeSymbolProvider.kt:109)
	at org.jetbrains.kotlin.fir.resolve.providers.FirSymbolProvider.getTopLevelFunctionSymbols(FirSymbolProvider.kt:52)
	at org.jetbrains.kotlinx.serialization.compiler.fir.services.DependencySerializationInfoProvider.useGeneratedEnumSerializer_delegate$lambda$0(DependencySerializationInfoProvider.kt:25)
	at kotlin.UnsafeLazyImpl.getValue(Lazy.kt:100)
	at org.jetbrains.kotlin.fir.caches.FirThreadUnsafeValue.getLazyValue(FirThreadUnsafeCachesFactory.kt:94)
	at org.jetbrains.kotlin.fir.caches.FirThreadUnsafeValue.getValue(FirThreadUnsafeCachesFactory.kt:95)
	at org.jetbrains.kotlin.fir.caches.FirCacheWithPostComputeKt.getValue(FirCacheWithPostCompute.kt:50)
	at org.jetbrains.kotlinx.serialization.compiler.fir.services.DependencySerializationInfoProvider.getUseGeneratedEnumSerializer(DependencySerializationInfoProvider.kt:23)
	at org.jetbrains.kotlinx.serialization.compiler.fir.SerializationFirUtilsKt.isEnumWithLegacyGeneratedSerializer(SerializationFirUtils.kt:232)
	at org.jetbrains.kotlinx.serialization.compiler.fir.SerializationFirUtilsKt.shouldHaveGeneratedSerializer(SerializationFirUtils.kt:237)
	at org.jetbrains.kotlinx.serialization.compiler.fir.SerializationFirResolveExtension.getNestedClassifiersNames(SerializationFirResolveExtension.kt:64)
	at org.jetbrains.kotlin.fir.scopes.impl.FirGeneratedMemberDeclarationsStorage.getClassifierStorage$providers(FirGeneratedScopes.kt:169)
	at org.jetbrains.kotlin.fir.scopes.impl.FirGeneratedClassNestedClassifierScope$Companion.create(FirGeneratedScopes.kt:125)
	at org.jetbrains.kotlin.fir.scopes.impl.FirDeclaredMemberScopeProvider.createNestedClassifierScope(FirDeclaredMemberScopeProvider.kt:126)
	at org.jetbrains.kotlin.fir.scopes.impl.FirDeclaredMemberScopeProvider.access$createNestedClassifierScope(FirDeclaredMemberScopeProvider.kt:40)
	at org.jetbrains.kotlin.fir.scopes.impl.FirDeclaredMemberScopeProvider$nestedClassifierCache$1.invoke(FirDeclaredMemberScopeProvider.kt:55)
	at org.jetbrains.kotlin.fir.scopes.impl.FirDeclaredMemberScopeProvider$nestedClassifierCache$1.invoke(FirDeclaredMemberScopeProvider.kt:51)
	at org.jetbrains.kotlin.fir.caches.FirThreadUnsafeCache.getValue(FirThreadUnsafeCachesFactory.kt:57)
	at org.jetbrains.kotlin.fir.scopes.impl.FirDeclaredMemberScopeProvider.nestedClassifierScope(FirDeclaredMemberScopeProvider.kt:188)
	at org.jetbrains.kotlin.fir.scopes.impl.FirDeclaredMemberScopeProviderKt.nestedClassifierScope(FirDeclaredMemberScopeProvider.kt:173)
	at org.jetbrains.kotlin.fir.extensions.FirExtensionDeclarationsSymbolProvider.generateClassLikeDeclaration(FirExtensionDeclarationsSymbolProvider.kt:109)
	at org.jetbrains.kotlin.fir.extensions.FirExtensionDeclarationsSymbolProvider.access$generateClassLikeDeclaration(FirExtensionDeclarationsSymbolProvider.kt:29)
	at org.jetbrains.kotlin.fir.extensions.FirExtensionDeclarationsSymbolProvider$classCache$1.invoke(FirExtensionDeclarationsSymbolProvider.kt:46)
	at org.jetbrains.kotlin.fir.extensions.FirExtensionDeclarationsSymbolProvider$classCache$1.invoke(FirExtensionDeclarationsSymbolProvider.kt:45)
	at org.jetbrains.kotlin.fir.caches.FirThreadUnsafeCache.getValue(FirThreadUnsafeCachesFactory.kt:57)
	at org.jetbrains.kotlin.fir.extensions.FirExtensionDeclarationsSymbolProvider.getClassLikeSymbolByClassId(FirExtensionDeclarationsSymbolProvider.kt:277)
	at org.jetbrains.kotlin.fir.extensions.FirSwitchableExtensionDeclarationsSymbolProvider.getClassLikeSymbolByClassId(FirSwitchableExtensionDeclarationsSymbolProvider.kt:62)
	at org.jetbrains.kotlin.fir.resolve.transformers.plugin.FirCompanionGenerationTransformer.generateCompanion(FirCompanionGenerationProcessor.kt:80)
	at org.jetbrains.kotlin.fir.resolve.transformers.plugin.FirCompanionGenerationTransformer.generateAndUpdateCompanion(FirCompanionGenerationProcessor.kt:61)
	at org.jetbrains.kotlin.fir.resolve.transformers.plugin.FirCompanionGenerationTransformer.transformRegularClass(FirCompanionGenerationProcessor.kt:56)
	at org.jetbrains.kotlin.fir.resolve.transformers.plugin.FirCompanionGenerationTransformer.transformRegularClass(FirCompanionGenerationProcessor.kt:39)
	at org.jetbrains.kotlin.fir.declarations.FirRegularClass.transform(FirRegularClass.kt:54)
	at org.jetbrains.kotlin.fir.visitors.FirTransformerUtilKt.transformInplace(FirTransformerUtil.kt:20)
	at org.jetbrains.kotlin.fir.declarations.impl.FirFileImpl.transformDeclarations(FirFileImpl.kt:83)
	at org.jetbrains.kotlin.fir.declarations.impl.FirFileImpl.transformDeclarations(FirFileImpl.kt:29)
	at org.jetbrains.kotlin.fir.resolve.transformers.plugin.FirCompanionGenerationTransformer.transformFile(FirCompanionGenerationProcessor.kt:51)
	... 35 more

I don't know if this could be the cause of the issue, but it fails on a generated file (by openapi generator)

[koperagen]

Yes, it's related to these declarations. One of the classes annotated with @DataSchema causes this error. Can you share the file?

[CHervaudBetclic]

Yeah here it is, but it's not using @DataSchema

/**
 *
 * Please note:
 * This class is auto generated by OpenAPI Generator (https://openapi-generator.tech).
 * Do not edit this file manually.
 *
 */

@file:Suppress(
    "ArrayInDataClass",
    "EnumEntryName",
    "RemoveRedundantQualifierName",
    "UnusedImport"
)

package com.myapp.models


import kotlinx.serialization.SerialName
import kotlinx.serialization.Serializable

/**
 * 
 *
 * Values: INVALID_BODY,MISSING_HEADER
 */
@Serializable
enum class MyDto(val value: kotlin.String) {

    @SerialName(value = "INVALID_BODY")
    INVALID_BODY("INVALID_BODY"),

    @SerialName(value = "MISSING_HEADER")
    MISSING_HEADER("MISSING_HEADER");

    /**
     * Override [toString()] to avoid using the enum variable name as the value, and instead use
     * the actual value defined in the API spec file.
     *
     * This solves a problem when the variable name and its value are different, and ensures that
     * the client sends the correct enum values to the server always.
     */
    override fun toString(): kotlin.String = value

    companion object {
        /**
         * Converts the provided [data] to a [String] on success, null otherwise.
         */
        fun encode(data: kotlin.Any?): kotlin.String? = if (data is MyDto) "$data" else null

        /**
         * Returns a valid [CreateBrand400ErrorCode] for [data], null otherwise.
         */
        fun decode(data: kotlin.Any?): MyDto? = data?.let {
          val normalizedData = "$it".lowercase()
          values().firstOrNull { value ->
            it == value || normalizedData == "$value".lowercase()
          }
        }
    }
}

[koperagen]

Oh :(( So looks like plugin fails whenever it encounters an enum class. I need to debug. I wonder, you actually use @DataSchema? There's a compiler flag to completely disable "top level extension generation". It removes the exception, but makes them somewhat useless.

kotlin {
    compilerOptions {
        freeCompilerArgs.add("-P")
        freeCompilerArgs.add("plugin:org.jetbrains.kotlin.dataframe:disableTopLevelExtensionProperties=true")
    }
}

But if you use them, there's an ugly workaround for extension properties to appear even for such DataSchema:

DataFrame.readCsv("...").cast<MySchema>()
  // dummy operation, typesafe API is generated  
  .addId("fix").remove { fix }
 // now use any properties from MySchema

Thank you for reporting such issue!

[CHervaudBetclic]

Oh yeah you're right, whether it's in the build/generated or not, any enum will make the plugin fail

And yeah I do use @DataSchema, I will just stick to kotlin 2.2.x for now until the issue is resolved on your side.

Thanks for your reactivity!

[koperagen]

Hi, the fix is in 2.4.0-dev-6762, Beta1 will follow in coming weeks
JetBrains/kotlin@f6a6531

I'd appreciate if you could check how's compilation of your project on this version to make sure we don't have other problems. Here's Kotlin dev repository:
build.gradle.kts

plugins {
    kotlin("jvm") version "2.4.0-dev-6762"
    kotlin("plugin.dataframe") version "2.4.0-dev-6762"
    kotlin("plugin.serialization") version "2.4.0-dev-6762"
}


repositories {
    maven("https://packages.jetbrains.team/maven/p/kt/dev/")
    mavenCentral()
}

settings.gradle.kts

pluginManagement {
    repositories {
        maven("https://packages.jetbrains.team/maven/p/kt/dev/")
        mavenCentral()
        gradlePluginPortal()
    }
}

[CHervaudBetclic]

Hello,

I've just tried but it seems it doesn't find some of my plugins

Execution failed for task ':build-logic:generateExternalPluginSpecBuilders'.
> Could not resolve all files for configuration ':build-logic:compileClasspath'.
   > Could not find org.jetbrains.kotlin:kotlin-gradle-plugin:2.4.0-dev-6762.
     Searched in the following locations:
       - https://plugins.gradle.org/m2/org/jetbrains/kotlin/kotlin-gradle-plugin/2.4.0-dev-6762/kotlin-gradle-plugin-2.4.0-dev-6762.pom
     Required by:
         project :build-logic
   > Could not find org.jetbrains.kotlin.plugin.serialization:org.jetbrains.kotlin.plugin.serialization.gradle.plugin:2.4.0-dev-6762.
     Searched in the following locations:
       - https://plugins.gradle.org/m2/org/jetbrains/kotlin/plugin/serialization/org.jetbrains.kotlin.plugin.serialization.gradle.plugin/2.4.0-dev-6762/org.jetbrains.kotlin.plugin.serialization.gradle.plugin-2.4.0-dev-6762.pom
     Required by:
         project :build-logic

[koperagen]

It doesn't seem to try to resolve from Kotlin dev repository. Add this to settings.gradle.kts

pluginManagement {
    repositories {
        maven("https://packages.jetbrains.team/maven/p/kt/dev/")
        mavenCentral()
        gradlePluginPortal()
    }
}

[zaleslaw]

We are waiting for a reply from the user