Databricks Runtime 18.2 (Beta)

The following release notes provide information about Databricks Runtime 18.2 (Beta).

This version incorporates all features, improvements, and bug fixes from all previous Databricks Runtime releases. Databricks released this version in April 2026.

Behavioral changes

• XPath no longer fetches external DTDs
• NULL struct preservation in INSERT, MERGE, and streaming writes with schema evolution
• NullType (VOID) support in Delta tables
• SHOW CREATE TABLE supports metric views
• Fix for LEFT OUTER JOIN LATERAL dropping rows
• NATURAL JOIN respects case-insensitive column matching
• SQL UDF dependency validation in Unity Catalog
• Optimized writes for partitioned Unity Catalog tables created with CRTAS
• AWS SDK v1 dependencies are shaded
• Fix incorrect EPSG authority for ESRI-defined SRID 102100

XPath no longer fetches external DTDs

When you evaluate XPath over XML, Azure Databricks no longer loads external Document Type Definitions (DTDs) declared in the document. Previously, XPath could fail when the XML contained an external DTD reference that pointed to a malformed URL or an unreachable endpoint. Because DTD validation is separate from XPath evaluation, queries that already succeeded return the same results as before. Queries that previously failed only during external DTD retrieval can now succeed.

NULL struct preservation in INSERT, MERGE, and streaming writes with schema evolution

For INSERT, MERGE, and streaming writes that use schema evolution, a NULL struct in the source is now stored as NULL in the target. Previously, that value was incorrectly materialized as a non-null struct with every field set to NULL, while the same operations without schema evolution preserved NULL structs correctly. If your code relied on receiving a non-null struct whose fields were all NULL, update your code to handle a NULL struct instead.

NullType (VOID) support in Delta tables

Delta tables now support VOID (Spark's NullType) columns. VOID columns are no longer dropped from the table schema upon reads. Writes are unaffected. See VOID type for restrictions on where VOID columns can appear in the schema.

SHOW CREATE TABLE supports metric views

SHOW CREATE TABLE now supports metric views. Previously, running this command on a metric view raised an error. The output for metric views includes the fully qualified three-part name with catalog (for example, CREATE VIEW catalog.db.my_metric_view ...), making it easier to recreate the metric view in the correct location.

Fix for LEFT OUTER JOIN LATERAL dropping rows

A bug that incorrectly dropped rows from LEFT OUTER JOIN LATERAL queries is now fixed. Queries using this construct now return the correct results. To temporarily revert to the previous behavior, set spark.databricks.sql.optimizer.lateralJoinPreserveOuterSemantic to true.

NATURAL JOIN respects case-insensitive column matching

NATURAL JOIN now correctly uses case-insensitive column matching when spark.sql.caseSensitive is set to false (the default). Previously, NATURAL JOIN used case-sensitive comparison to identify common columns, causing columns that differed only in case (for example, ID versus id) to not be recognized as matching. This caused NATURAL JOIN to silently produce cross-join results. This fix aligns NATURAL JOIN behavior with USING joins, which already handled case-insensitivity correctly. Queries affected by this bug now return correct results with properly joined columns.

SQL UDF dependency validation in Unity Catalog

Unity Catalog now enforces dependency validation for SQL user-defined functions (UDFs) to prevent access control bypass. Previously, SQL functions created through the REST API could reference dependencies the user did not have access to. SQL UDFs with invalid dependency configurations are now blocked from execution.

Optimized writes for partitioned Unity Catalog tables created with CRTAS

Optimized writes are now correctly applied to partitioned Unity Catalog tables created with CREATE OR REPLACE TABLE ... AS SELECT (CRTAS). Previously, CRTAS on new partitioned Unity Catalog tables did not apply optimized writes, resulting in a higher number of small files per partition. This fix can increase write latency. To revert to the previous behavior, set spark.databricks.delta.optimizeWrite.UCTableCRTAS.enabled to false.

AWS SDK v1 dependencies are shaded

AWS SDK v1 dependencies bundled with Databricks Runtime are now shaded and no longer directly available on the classpath. If your code depends on AWS SDK v1 libraries previously provided by Databricks Runtime, add them as explicit dependencies in your project. This change prepares for the migration to AWS SDK v2, following the end of AWS support for SDK v1.

Fix incorrect EPSG authority for ESRI-defined SRID 102100

The Coordinate Reference System (CRS) mapping for SRID 102100 now correctly uses ESRI:102100 instead of the incorrect EPSG:102100. This fix ensures geospatial data is stored with the correct authority for better interoperability with other systems.

New features and improvements

• CREATE OR REPLACE support for temporary tables
• agg() alias for measure() function
• Snowflake JDBC driver upgrade
• pyspark.pipelines.testing namespace alias
• Improved Auto Loader listing performance
• Delta table history includes write option flags
• Structured Streaming rewind and replay support

CREATE OR REPLACE support for temporary tables

CREATE OR REPLACE TEMP TABLE syntax is now supported, allowing you to create or replace temporary tables in a single statement. This eliminates the need to explicitly drop and recreate temporary tables.

agg() alias for measure() function

agg() is now available as an alias for the measure() function. This change is fully backward compatible. Existing queries that use measure() continue to work without modification, and agg() produces identical results when used with the same arguments.

Snowflake JDBC driver upgrade

The Snowflake JDBC driver is upgraded from 3.22.0 to 3.28.0.

pyspark.pipelines.testing namespace alias

pyspark.pipelines.testing is now available as a convenience alias for dlt.testing APIs. Import Lakeflow Spark Declarative Pipelines pipeline testing utilities through either namespace.

Improved Auto Loader listing performance

Auto Loader now uses a more efficient listing method that improves listing speed for cloud storage sources. If your stream triggers overlap due to long-running listing operations, this optimization can result in increased cloud listing API costs. Monitor your trigger intervals and adjust scheduling to prevent overlapping operations if cost increases are observed.

Delta table history includes write option flags

Delta table history (DESCRIBE HISTORY) now includes write option flags in the operationParameters column for WRITE and REPLACE TABLE operations. When the following options are explicitly enabled, they appear as boolean flags in the history (only included when true):

For WRITE and REPLACE TABLE operations:

• isDynamicPartitionOverwrite: present when dynamic partition overwrite mode was used
• canOverwriteSchema: present when schema overwrite (overwriteSchema) was enabled
• canMergeSchema: present when schema merge (mergeSchema) was enabled

For REPLACE TABLE operations:

• predicate: present when replaceWhere was used
• isV1WriterSaveAsTableOverwrite: present when the replace was triggered by a .saveAsTable overwrite

Structured Streaming rewind and replay support

Structured Streaming now supports rewind and replay for streaming pipelines. This feature enables reprocessing from an earlier point in the stream to recover from failures such as schema changes, malformed input data, or logic errors, without requiring a full state reset. This doesn't modify the default behavior of existing streaming workloads.

Library upgrades

• Upgraded Python libraries:

  No libraries were upgraded in this version.

• Upgraded R libraries:

  No libraries were upgraded in this version.

• Upgraded Java libraries:

  • io.delta.delta-sharing-client_2.13 from 1.3.9 to 1.3.10

Apache Spark

Databricks Runtime 18.2 includes Apache Spark 4.1.0. This release includes all Spark fixes and improvements included in Databricks Runtime 18.1, as well as the following additional bug fixes and improvements made to Spark:

• SPARK-56219 Revert "[SC-225028][PS] Align groupby idxmax and idxmin skipna=False behavior with pandas 2/3"
• SPARK-56204 Strip Alias wrappers from inline table row expressions in parser
• SPARK-56186 Retire pypy
• SPARK-56202 Refactor streaming join tests: split Base/Suite hierarchy and simplify mode dispatch
• SPARK-56221 Feature parity between spark.catalog.* vs DDL commands
• SPARK-56301 Fix typos in error-conditions.json
• SPARK-55729 Support state data source reader for new state format v4 on stream-stream join
• SPARK-56256 Add emptyDataFrame API to SparkSession
• SPARK-56205 Validate base state store checkpoint ID before committing microbatch
• SPARK-55827 Fix type hint for datasource workers
• SPARK-55579 Rename PySpark error classes to be eval-type-agnostic
• SPARK-56247 Fix the fall back behavior and type hint of inheritable_thread_target
• SPARK-56244 Refine benchmark class layout in bench_eval_type.py
• SPARK-56262 Remove the unnecessary mypy check disable for types
• SPARK-55969 regr_r2 should treat first param as dependent variable
• SPARK-56179 Consolidate error classes for type mismatch - part 3
• SPARK-55630 Skip updating matched flag for non-outer side in stream-stream join v4
• SPARK-56217 Fix bucketBy exceptions in connect
• SPARK-56225 Improve View WITH SCHEMA EVOLUTION error message
• SPARK-55865 Rename _LEGACY_ERROR_TEMP_1266 to CANNOT_TRUNCATE_EXTERNAL_TABLE
• SPARK-55861 Rename _LEGACY_ERROR_TEMP_2045 to UNSUPPORTED_TABLE_CHANGE
• SPARK-56166 Use ArrowBatchTransformer.enforce_schema to replace column-wise type coercion logic
• SPARK-56245 Fix DataFrame.eval inplace assignment on pandas 3
• SPARK-56062 Isolate memory_profiler to improve import time
• SPARK-55964 system catalog wins over user catalog for BUILTIN, and SESSION schemas.
• SPARK-56226 Catch analysis errors before InternalFrame.__init__ in .loc
• SPARK-55723 Generalize enforce_schema error to PySparkTypeError
• SPARK-54878 Add sortKeys option to to_json function
• SPARK-56219 Align groupby idxmax and idxmin skipna=False behavior with pandas 2/3
• SPARK-44065 Optimize BroadcastHashJoin skew in OptimizeSkewedJoin
• SPARK-56179 Revert "[SC-225014][PYTHON] Consolidate error classes for type mismatch - part 3"
• SPARK-53399 Merge Python UDFs
• SPARK-56224 Polish type annotations for accumulators.py
• SPARK-55448 Fix query events loss when session closes during query execution
• SPARK-55862 Rename _LEGACY_ERROR_TEMP_2027 to UNEXPECTED_OPERATOR_IN_CORRELATED_SUBQUERY
• SPARK-56201 Run SPARK-49829 tests with VCF joins now that StateDataSource supports it
• SPARK-56179 Consolidate error classes for type mismatch - part 3
• SPARK-56184 Replace assert with proper SparkRuntimeException in partition column parsing
• SPARK-56206 Fix case-insensitive duplicate CTE name detection
• SPARK-55866 Rename _LEGACY_ERROR_TEMP_2145 to OPTION_VALUE_EXCEEDS_ONE_CHARACTER
• SPARK-56067 Lazy import psutil to improve import speed
• SPARK-56066 Lazy import numpy to improve import speed
• SPARK-55719 Remove deprecation warning for spark.sql.hive.convertCTAS
• SPARK-56179 Consolidate error classes for type mismatch - part 2
• SPARK-55510 Update structured-streaming-state-data-source.md doc to reflect deleteRange
• SPARK-56050 Eagerly resolve IDENTIFIER() with string literals at parse time
• SPARK-56151 Improve CreateVariable display string
• SPARK-55751 Add metrics on state store loads from DFS
• SPARK-56188 Align Series.map({}) with pandas 3 empty-dict behavior
• SPARK-55964 Revert "[SC-223957] system catalog wins over user catalog for BUILTIN, and SESSION schemas."
• SPARK-55577 Refactor SQL_SCALAR_ARROW_ITER_UDF wrapper, mapper, and serializer logic
• SPARK-55596 DSV2 Enhanced Partition Stats Filtering
• SPARK-56179 Revert "[SC-224777][PYTHON] Consolidate error classes for type mismatch - part 2"
• SPARK-55964 system catalog wins over user catalog for BUILTIN, and SESSION schemas.
• SPARK-56050 Revert "[SC-224153][SQL] Eagerly resolve IDENTIFIER() with string literals at parse time"
• SPARK-56179 Consolidate error classes for type mismatch - part 2
• SPARK-56102 UnionEstimation code cleanup
• SPARK-51712 Swallow non-fatal Throwables when resolving tables/views in spark.catalog.listTables()
• SPARK-55881 Add queryId, errorMessage, and rootExecutionId to SQL execution REST API
• SPARK-56050 Eagerly resolve IDENTIFIER() with string literals at parse time
• SPARK-55628 Integrate stream-stream join state format V4
• SPARK-56187 Fix Series.argsort null ordering for pandas 3
• SPARK-56167 Align astype with pandas 3 default string behavior
• SPARK-56018 Use ruff as formatter
• SPARK-56042 Fix swapped external/internal col family count metrics in RocksDBStateStoreProvider
• SPARK-56179 Consolidate error classes for type mismatch - part 1
• SPARK-56089 Align asinh/acosh with fdlibm algorithm for cross-engine compatibility
• SPARK-55453 Fix LIKE pattern matching for supplementary Unicode characters
• SPARK-52785 Simplifying super() syntax in PySpark
• SPARK-56169 Fix ClassCastException in error reporting when GetStructField child type is changed by plan transformation
• SPARK-55557 Hyperbolic functions should not overflow with large inputs
• SPARK-47997 Add errors parameter to DataFrame.drop and Series.drop
• SPARK-55008 Display Query ID in SparkUI
• SPARK-54660 Add RTM trigger to python
• SPARK-56047 Propagate distinctCount through Union in CBO statistics estimation
• SPARK-56111 Add SparkContext.isDriver() and use it across the codebase
• SPARK-55999 Enable forceSnapshotUploadOnLag by default
• SPARK-55610 Add getExecutorInfos to StatusTracker in Python
• SPARK-55728 Introduce conf for file checksum threadpool size and support disabling the threadpool
• SPARK-55686 SizeEstimator takes care of Compact Object Headers
• SPARK-56044 HistoryServerDiskManager does not delete app store on release when app is not in active map
• SPARK-55809 HeapHistogram uses DiagnosticCommandMBean instead of jmap subprocess
• SPARK-56122 Use pandas-aware numeric dtype check in Series.cov
• SPARK-56113 Improve pandas 3 string restoration in pandas-on-Spark
• SPARK-56118 Match pandas 3.0 bool handling in GroupBy.quantile
• SPARK-53823 Implement allow list for real time mode
• SPARK-55977 Fix isin() to use strict type matching like pandas
• SPARK-54027 Kafka Source RTM support
• SPARK-50284 Change docs for parseJson function
• SPARK-56035 [SQL] Introduce AggregationValidator for single-pass resolver Aggregate validation
• SPARK-55557 Revert "[SC-223720][SQL] Hyperbolic functions should not overflow with large inputs"
• SPARK-56075 Remove a batch of dead python error classes
• SPARK-55967 Unify column conversion for connect dataframe
• SPARK-53915 Add RealTimeScanExec and ability to execute long running batches
• SPARK-55557 Hyperbolic functions should not overflow with large inputs
• SPARK-55147 Scope timestamp range for time-interval join retrieval in V4 state format
• SPARK-56056 Support simpler worker profiling with viztracer
• SPARK-55948 Add DSv2 CDC connector API, analyzer resolution, and SQL CHANGES clause
• SPARK-54599 Reapply "[SC-219008][PYTHON] Refactor PythonExcept…
• SPARK-55390 Consolidate SQL_SCALAR_ARROW_UDF wrapper, mapper, and serializer logic
• SPARK-56023 Better load balance in LowLatencyMemoryStream
• SPARK-55986 Upgrade black to 26.3.1
• SPARK-55667 Move check_dependencies to init
• SPARK-55145 Support Avro for timestamp based RocksDB state key encoders
• SPARK-53970 Remove incorrect 'optional' tag for messageName…
• SPARK-55059 Revert "[SC-224058][PYTHON] Remove empty table workaround in toPandas"
• SPARK-50111 Add subplots support for pie charts in Plotly backend
• SPARK-56081 Align idxmax and idxmin NA handling with pandas 3
• SPARK-56080 Align Series.argmax/argmin with pandas 3.0 NA handling
• SPARK-56060 Handle pandas 3 null string conversion in describe() for empty timestamp frames
• SPARK-55059 Remove empty table workaround in toPandas
• SPARK-55995 Support TIMESTAMP WITH LOCAL TIME ZONE in SQL syntax
• SPARK-55976 Use Set instead of Seq for write privileges
• SPARK-56073 Simplify the build of PythonRunnerConfMap
• SPARK-55887 Special handling for CollectLimitExec/CollectTailExec to avoid full table scans
• SPARK-55980 Always apply _cast_back_float in numeric arithmetic
• SPARK-55357 Fix docstring for timestamp_add
• SPARK-55667 Revert "[SC-223289][PYTHON][CONNECT] Move check_dependencies to init"
• SPARK-54285 Revert "[PYTHON] Cache timezone info to avoid expensive timestamp conversion"
• SPARK-56021 Increase AutoSnapshotRepair default maxChangeFileReplay threshold from 50 to 500
• SPARK-55870 Add docs for Geo types
• SPARK-55962 Use getShort instead of getInt casting in putShortsFromIntsLittleEndian on Little Endian platforms
• SPARK-55903 Simplify MERGE Schema Evolution and Check Write Privileges
• SPARK-55326 Release remote session when SPARK_CONNECT_RELEASE_SESSION_ON_EXIT is set
• SPARK-55667 Move check_dependencies to init
• SPARK-55884 Add v1StatsToV2Stats to DataSourceV2Relation
• SPARK-55929 Add missing toString() to TableChange.UpdateColumnDefaultValue
• SPARK-55851 Clarify types of datasource partition and read
• SPARK-55828 Add DSV2 TableChange toString and fix missing error class for Merge Into Schema Evolution
• SPARK-55790 Build a complete SRS registry using PROJ 9.7.1 data
• SPARK-55645 Add serdeName to CatalogStorageFormat
• SPARK-54796 Fix NPE caused by race condition between Executor initialization and shuffle migration
• SPARK-55983 New single-pass analyzer functionality and bugfixes
• SPARK-55964 Cache coherence: clear function registry on DROP DATABASE
• SPARK-55868 Fix Predicate Pushdown for InMemoryTable for V2Filters
• SPARK-55973 LeftSemi optimization for stream-stream join
• SPARK-54665 Fix boolean vs string comparison to match pandas behavior
• SPARK-55539 Allow casting from GeographyType to GeometryType
• SPARK-55695 Avoid double planning in row-level operations
• SPARK-55904 Utilize _check_same_session to narrow down types
• SPARK-55965 Add warning when pandas >= 3.0.0 is used with PySpark
• SPARK-55493 [SS] Do not mkdirs in streaming checkpoint offset/commit log directory in StateDataSource
• SPARK-55851 Revert "[SC-223270][PYTHON] Clarify types of datasource partition and read"
• SPARK-55645 Revert "[SC-221839][SQL] Add serdeName to CatalogStorageFormat"
• SPARK-55640 Propagate WKB parsing errors for Geometry and Geography
• SPARK-55693 Avoid deadlock by making SparkSession.observationManager a non-lazy val
• SPARK-55528 Add default collation support for SQL UDFs
• SPARK-55860 Use UNABLE_TO_INFER_SCHEMA instead of UNABLE_TO_INFER_SCHEMA_FOR_DATA_SOURCE
• SPARK-55275 Add InvalidPlanInput sql states for sql/connect
• SPARK-55645 Add serdeName to CatalogStorageFormat
• SPARK-55716 Support NOT NULL constraint enforcement for V1 file source table inserts
• SPARK-53226 Make ClosureCleaner work with Java22+
• SPARK-55997 Set upper bound to prefixScan in RocksDB state store provider
• SPARK-55851 Clarify types of datasource partition and read
• SPARK-55954 Remove the incorrect overload type hint for fillna
• SPARK-56016 Preserve named Series columns in concat with ignore_index on pandas 3
• SPARK-55502 Unify UDF and UDTF Arrow conversion error handling
• SPARK-55989 Preserve non-int64 index dtypes in restore_index
• SPARK-55955 Remove overload type hint for drop
• SPARK-55945 [SDP] Support structured identifiers for flows in SDP eager analysis protos
• SPARK-55714 JDK might throw ArithmeticException without message
• SPARK-55991 Fix unicode related SQL text corruption with parameters
• SPARK-55696 Add explicit error to Encoders.bean for interface class
• SPARK-55533 Support IGNORE NULLS / RESPECT NULLS for collect_set
• SPARK-55987 Fix V4 windowed join timestamp extraction using findJoinKeyOrdinalForWatermark
• SPARK-55946 Set up pandas_priority so mixed binary ops dispatch correctly to pandas-on-Spark
• SPARK-55264 Add ExecuteOutput command to Spark Connect pipelines proto
• SPARK-47672 Avoid double eval from filter pushDown w/ projection pushdown
• SPARK-55780 Replace PNG logo with SVG in Spark Web UI
• SPARK-55821 Enforce keyword-only arguments in serializer init methods
• SPARK-55621 Fix ambiguous and unnecessary unicode usage
• SPARK-55662 Implementation of idxmin Axis argument
• SPARK-55631 ALTER TABLE must invalidate cache for DSv2 tables
• SPARK-55692 Fix SupportsRuntimeFiltering and SupportsRuntimeV2Filtering documentation
• SPARK-55928 New linter for config effectiveness in views and UDFs
• SPARK-55440 Types Framework - Phase 1a - Core Type System Foundation
• SPARK-55631 Revert "[SC-221596][SQL] ALTER TABLE must invalidate cache for DSv2 tables"
• SPARK-55631 ALTER TABLE must invalidate cache for DSv2 tables
• SPARK-55683 Optimize VectorizedPlainValuesReader.readUnsignedLongs
• SPARK-55892 Fix unable to load state store because reused SST file was deleted by maintenance
• SPARK-55946 Revert "[SC-223027][PS] Set up pandas_priority so mixed binary ops dispatch correctly to pandas-on-Spark"
• SPARK-55891 Preserve the SQL scripting context inside EXECUTE IMMEDIATE
• SPARK-55907 Fix incorrect error positions for invalid data types in CREATE FUNCTION
• SPARK-55946 Set up pandas_priority so mixed binary ops dispatch correctly to pandas-on-Spark
• SPARK-55694 Block constraints in CTAS/RTAS at parser level
• SPARK-55682 ServiceLoader returned iterator may throw NoClassDefFoundError on hasNext()
• SPARK-55155 Fix SET CATALOG to use special chars and backticks in the identifier name
• SPARK-55932 Fix XML to variant parser hang on negative scale
• SPARK-55673 Add more tests for nested type encoder
• SPARK-55679 Fix dectecting sun.io.serialization.extendedDebugInfo on Java 25
• SPARK-55957 Add 'DATA_SOURCE_NOT_FOUND' in Catalog.ERROR_HANDLING_RULES
• SPARK-55052 Add AQEShuffleRead properties to Physical Plan Tree
• SPARK-55652 Optimize VectorizedPlainValuesReader.readShorts() with direct array access for heap buffers
• SPARK-55659 Improve EventLogFileWriter to log stop operation
• SPARK-54666 Leave numeric types unchanged on to_numeric
• SPARK-55654 Enable TreePattern pruning for EliminateSubqueryAliases and ResolveInlineTables
• SPARK-55533 Revert "[SC-220538][SQL] Support IGNORE NULLS / RESPECT NULLS for collect_set"
• SPARK-55901 Raise an error from Series.replace() with no arguments
• SPARK-55896 Use numpy functions instead of builtins
• SPARK-55655 Make CountVectorizer vocabulary deterministic when counts are equal
• SPARK-55811 [SQL] Catch NonFatal instead of UnresolvedException when calling nodeWithOutputColumnsString
• SPARK-55533 Support IGNORE NULLS / RESPECT NULLS for collect_set
• SPARK-55435 Use StringBuilder instead of StringBuffer
• SPARK-54807 Allow qualified names for built-in and session functions (#198171)
• SPARK-55854 Tag pass-through duplicate attributes in Expand output to prevent AMBIGUOUS_REFERENCE
• SPARK-55261 Implement Parquet read support for Geo types
• SPARK-55416 Streaming Python Data Source memory leak when end-offset is not updated
• SPARK-55465 Support GeometryType in convert_numpy
• SPARK-55801 Fix type hint of _SimpleStreamReaderWrapper.getCache
• SPARK-55800 Remove the unused type check for datetime.date
• SPARK-55663 Unify module for data source functions
• SPARK-55665 Unify how workers establish connection with the executor
• SPARK-53446 Optimize BlockManager remove operations with cached block mappings
• SPARK-55867 Fix StringMethods with pandas 3
• SPARK-55501 Fix listagg distinct + within group order by bug
• SPARK-55558 Add support for Tuple/Theta set operations
• SPARK-55636 Add detailed errors in case of deduplication of invalid columns
• SPARK-55788 Support ExtensionDType for integers in Pandas UDF
• SPARK-55464 Support GeographyType in convert_numpy
• SPARK-55530 Support Geo result sets in Hive and Thrift server
• SPARK-55525 Fix UDTF_ARROW_TYPE_CONVERSION_ERROR with undefined error message parameter
• SPARK-55626 Don't load metadata columns on Table unless needed in V2TableUtil
• SPARK-55533 Revert "[SC-220538][SQL] Support IGNORE NULLS / RESPECT NULLS for collect_set"
• SPARK-55435 Revert "[SC-219656][CORE][SQL] Use StringBuilder instead of StringBuffer"
• SPARK-55533 Support IGNORE NULLS / RESPECT NULLS for collect_set
• SPARK-54452 Fix empty response from SparkConnect server for spark.sql(...) inside FlowFunction
• SPARK-55638 Refactor WKT serialization in GeometryModel
• SPARK-55551 Improve BroadcastHashJoinExec output partitioning
• SPARK-54314 Improve Server-Side debuggability in Spark Connect by capturing client application's file name and line numbers
• SPARK-55517 Optimize VectorizedPlainValuesReader.readBytes() with direct array access for heap buffers
• SPARK-55495 Fix EventLogFileWriters.closeWriter to handle checkError
• SPARK-55279 Add sketch_funcs group for DataSketches SQL functions
• SPARK-55435 Use StringBuilder instead of StringBuffer
• SPARK-55064 Support query level indeterminate shuffle retry
• SPARK-55411 SPJ may throw ArrayIndexOutOfBoundsException when join keys are less than cluster keys
• SPARK-55451 Cursors must start collecting results on OPEN, not first FETCH
• SPARK-54687 Add more edge cases with generators
• SPARK-55691 GetStatus client
• SPARK-55277 Add protobuf_funcs group for Protobuf SQL functions
• SPARK-55822 Rename _LEGACY_ERROR_TEMP_0052 to CREATE_VIEW_WITH_IF_NOT_EXISTS_AND_REPLACE
• SPARK-55236 Address unexpected exception in some CoarseGrainedExecutorBackendSuite test cases
• SPARK-55275 SQL State Coverage: IllegalStateException
• SPARK-55462 Reapply "[SC-221123][PYTHON] Support VariantType in convert_numpy"
• SPARK-55062 Support proto2 extensions in protobuf functions
• SPARK-55248 Clean up Jackson deprecated API usage in streaming.checkpointing.Checksum
• SPARK-55250 Reduce Hive client calls on CREATE NAMESPACE
• SPARK-55247 Clean up deprecated API usage related to o.a.c.io.input.BoundedInputStream
• SPARK-55198 spark-sql should skip comment line with leading whitespaces
• SPARK-55826 Rename _LEGACY_ERROR_TEMP_0006 to MERGE_INSERT_VALUE_COUNT_MISMATCH
• SPARK-55127 Add avro_funcs group for Avro SQL functions
• SPARK-54914 [SQL] Fixing DROP operator in pipe syntax to support qualified column names
• SPARK-55113 EnsureRequirements should copy tags
• SPARK-55074 Add test for Merge Into ANSI type coercion
• SPARK-54217 Synchronize PythonRunner's MonitorThread kill decision
• SPARK-54374 Enlarge the SVG viewBox attribute of SQL plan visualization initialization
• SPARK-54971 Add WITH SCHEMA EVOLUTION syntax for SQL INSERT
• SPARK-55065 Avoid making two JDBC API calls
• SPARK-55033 Fix stringArgs of DSv2 writing commands
• SPARK-55041 Cleanup some unused private funcion/val from core module
• SPARK-55338 Centralize Spark Connect request decompression logic in gRPC interceptor
• SPARK-55825 Rename _LEGACY_ERROR_TEMP_1309 to PARTITION_BY_NOT_ALLOWED_WITH_INSERT_INTO
• SPARK-55492 Validate that eventTime in withWatermark is top-level column
• SPARK-55802 Fix integer overflow when computing Arrow batch bytes
• SPARK-55694 Block constraints in CTAS/RTAS at parser level
• SPARK-55843 Handle the unit of datetime64 and timedelta64 dtypes
• SPARK-55824 Rename _LEGACY_ERROR_TEMP_1034 to WINDOW_FUNCTION_NOT_ALLOWED_IN_CLAUSE
• SPARK-55819 Refactor ExpandExec to be more succinct
• SPARK-55341 Add storage level flag for cached local relations
• SPARK-54599 Revert "[SC-219008][PYTHON] Refactor PythonException so it can take errorClass with sqlstate"
• SPARK-46167 Add axis implementation to DataFrame.rank
• SPARK-54599 Refactor PythonException so it can take errorClass with sqlstate
• SPARK-55529 Reapply [ES-1721989][SC-220716][PYTHON] Restore Arrow-level batch merge for non-iterator applyInPandas
• SPARK-55794 Always alias OuterReferences
• SPARK-55583 Validate Arrow schema types in Python data source
• SPARK-37711 Reduce pandas describe job count from O(N) to O(1)
• SPARK-46168 Add axis argument for idxmax
• SPARK-46162 Implement nunique with axis=1
• SPARK-55552 Add VariantType support to ColumnarBatchRow.copy() and MutableColumnarRow
• SPARK-55647 Fix ConstantPropagation incorrectly replacing attributes with non-binary-stable collations
• SPARK-55747 Fix NPE when accessing elements from an array that is null
• SPARK-55757 Improve spark.task.cpus validation
• SPARK-55699 Inconsistent reading of LowLatencyClock when used together with ManualClock
• SPARK-55702 Support filter predicate in window aggregate functions
• SPARK-55510 Fix deleteRange of Rocksdb state store to call changelogWriter
• SPARK-55739 Optimize OnHeapColumnVector.putIntsLittleEndian/putLongsLittleEndian using Platform.copyMemory on little-endian platforms
• SPARK-55730 Not make timezone lower case
• SPARK-55701 ES-1694761[SS] Fix race condition in CompactibleFileStreamLog.allFiles
• SPARK-55462 Revert "[SC-221123][PYTHON] Support VariantType in convert_numpy"
• SPARK-55144 Introduce new state format version for performant stream-stream join
• SPARK-55606 Server-side implementation of GetStatus API
• SPARK-55462 Support VariantType in convert_numpy
• SPARK-55600 Fix pandas to arrow loses row count when schema has 0 columns on classic
• SPARK-55700 Fix handling integer keys on Series with non-integer index
• SPARK-55349 Consolidate pandas-to-Arrow conversion utilities in serializers
• SPARK-55681 Fix singleton DataType equality after deserialization (reattempt)
• SPARK-55681 Revert "[SC-221427][SC-214079][SQL] Fix singleton DataType equality after deserialization"
• SPARK-55681 Fix singleton DataType equality after deserialization
• SPARK-55674 Optimize 0-column table conversion in Spark Connect
• SPARK-55323 Reapply "[SC-218885][PYTHON] Move UDF metadata to EvalConf to simplify worker protocol"
• SPARK-55322 Reapply [SC-221062][SQL] MaxBy and MinBy Overload with K Elements
• SPARK-55323 Revert "[SC-218885][PYTHON] Move UDF metadata to EvalConf to simplify worker protocol"
• SPARK-55615 Move SparkContext import into class branch
• SPARK-55323 Move UDF metadata to EvalConf to simplify worker protocol
• SPARK-55648 Handle an unexpected keyword argument error groupby(axis) with pandas 3
• SPARK-55647 Revert "[SC-221274][SQL] Fix ConstantPropagation incorrectly replacing attributes with non-binary-stable collations"
• SPARK-55646 Refactored SQLExecution.withThreadLocalCaptured to separate thread-local capture from execution
• SPARK-54854 Add a UUIDv7 queryId to SQLExecution Events
• SPARK-55619 Fix custom metrics in case of coalesced partitions
• SPARK-55647 Fix ConstantPropagation incorrectly replacing attributes with non-binary-stable collations
• SPARK-55322 Revert "[SC-221062][SQL] MaxBy and MinBy Overload with K Elements"
• SPARK-54740 Start faulthandler early in daemon mode
• SPARK-55493 [SS] Do not mkdirs in streaming checkpoint state directory in StateDataSource
• SPARK-55322 MaxBy and MinBy Overload with K Elements
• SPARK-55625 Fix StringOps to make str dtype work properly
• SPARK-55161 Reapply "[SC-218867][PYTHON] Support profilers on python data source"
• SPARK-55505 Fix NPE on reading EXECUTION_ROOT_ID_KEY in concurrent scenarios
• SPARK-55111 Recheckin Unfinished repartitioning detection on query restart
• SPARK-55593 Unify aggregation state for vector_avg/vector_sum
• SPARK-55500 Fix analyzer cycle between ApplyDefaultCollation, ExtractWindowExpressions and CollationTypeCasts
• SPARK-55494 Introduce iterator/prefixScan with multi-values in StateStore API
• SPARK-55561 Add retries for all Kafka admin client methods
• SPARK-55296 Support CoW mode with pandas 3
• SPARK-55479 Fix style issues in SparkShreddingUtils
• SPARK-55372 Fix SHOW CREATE TABLE for tables / views with default collation
• SPARK-55333 Enable DateType and TimeType in convert_numpy
• SPARK-55129 Introduce new key encoders for timestamp as a first class (UnsafeRow)
• SPARK-46163 DataFrame.update parameters filter_func and errors
• SPARK-55372 Revert "[SC-220571][SQL] Fix SHOW CREATE TABLE for tables / views with default collation"
• SPARK-55480 Remove all unused noqa for ruff
• SPARK-55471 Add optimizer support for SequentialStreamingUnion
• SPARK-55584 Produce better error on scalar subquery to EXEC IMMEDIATE
• SPARK-55161 Revert "[SC-218867][PYTHON] Support profilers on python data source"
• SPARK-55506 Pass explicit input schema to to_pandas in CogroupPandasUDFSerializer
• SPARK-55586 Add jdbc.py Example
• SPARK-55161 Support profilers on python data source
• SPARK-55529 Revert "[SC-220716][PYTHON] Restore Arrow-level batch merge for non-iterator applyInPandas"
• SPARK-55385 Mitigate the recomputation in zipWithIndex
• SPARK-55529 Restore Arrow-level batch merge for non-iterator applyInPandas
• SPARK-55389 Consolidate SQL_MAP_ARROW_ITER_UDF wrapper, mapper, and serializer logic
• SPARK-55406 Reimplement the thread pool for ExecutePlanResponseReattachableIterator
• SPARK-55372 Fix SHOW CREATE TABLE for tables / views with default collation
• SPARK-55367 Use venv for run-pip-tests
• SPARK-55355 Upgrade mypy version to the latest
• SPARK-55460 Remove E203 from ruff's ignore list
• SPARK-55541 Support Geometry and Geography in catalyst type converters
• SPARK-55449 Enable WKB parsing and writing for Geography
• SPARK-55339 Implement WKT writer support for Geo objects
• SPARK-54122 Implement TwsTester in Scala
• SPARK-54805 Implement TwsTester in PySpark
• SPARK-55256 Reapply "[SC-218596][SQL] Support IGNORE NULLS / RESPECT NULLS for array_agg and collect_list"
• SPARK-55156 Deal with include_groups for groupby.apply
• SPARK-55401 Add retry logic and timeout handling to pyspark install download
• SPARK-55229 Implement DataFrame.zipWithIndex in PySpark
• SPARK-55462 Support UserDefinedType in convert_numpy
• SPARK-55483 Fix NPE in PivotFirst when pivot column is a non-atomic type with null values
• SPARK-55490 Make groupby(as_index=False) include a grouping that is not in the DataFrame with pandas 3
• SPARK-55473 Replace itertools.tee with chain in applyInPandasWithState
• SPARK-55404 Always raise KeyboardInterrupt from SIGINT handler
• SPARK-55407 Replace logger.warn with logger.warning

Databricks ODBC/JDBC driver support

Databricks supports ODBC/JDBC drivers released in the past 2 years. Please download the recently released drivers and upgrade (download ODBC, download JDBC).

System environment

• Operating System: Ubuntu 24.04.4 LTS
• Java: Zulu21.48+15-CA
• Scala: 2.13.16
• Python: 3.12.3
• R: 4.5.1
• Delta Lake: 4.1.0

Installed Python libraries

Installed R libraries

R libraries are installed from the Posit Package Manager CRAN snapshot on 2025-11-20.

Installed Java and Scala libraries (Scala 2.13 cluster version)