While upgrading the pyspark version of sparkpools to 3.4 in Synapse Studio. We faced this error
An error occurred while calling o6063.save.\n: java.lang.NoSuchMethodError: 'org.apache.spark.sql.types.StructType org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(java.sql.ResultSet, org.apache.spark.sql.jdbc.JdbcDialect, boolean)'\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305)\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266)\n\tat com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79)\n\tat com.microsoft.sqlserver.jdbc.spark.DefaultSource.createRelation(DefaultSource.scala:66)\n\tat
We have updgraded the pyspark(3.4.0) and delta-spark(2.4.0) version in requirement.txt file also but issue is same.
While upgrading the pyspark version of sparkpools to 3.4 in Synapse Studio. We faced this error
An error occurred while calling o6063.save.\n: java.lang.NoSuchMethodError: 'org.apache.spark.sql.types.StructType org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$.getSchema(java.sql.ResultSet, org.apache.spark.sql.jdbc.JdbcDialect, boolean)'\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.matchSchemas(BulkCopyUtils.scala:305)\n\tat com.microsoft.sqlserver.jdbc.spark.BulkCopyUtils$.getColMetaData(BulkCopyUtils.scala:266)\n\tat com.microsoft.sqlserver.jdbc.spark.Connector.write(Connector.scala:79)\n\tat com.microsoft.sqlserver.jdbc.spark.DefaultSource.createRelation(DefaultSource.scala:66)\n\tatWe have updgraded the pyspark(3.4.0) and delta-spark(2.4.0) version in requirement.txt file also but issue is same.