gen18 release notes

Changelog

Generation 18 is a port of the legacy dm18 generation. There are not changes compared to this generation.

Available tags

platform-3.3.0-hadoop-3.3.2-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.3-gen18
• platform-3.3.0-gen18

Dependency versions

name	version
Spark	3.3.0
Hadoop	3.3.2
Java	8
Scala	2.12
Python	3.8
Hive	2.3.9
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.1026
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.3.9
ADLS gen2 (abfss://)	Azure Storage 7.0.1
Azure Blob Storage (wasbs://)	Azure Storage 7.0.1
AWS Glue	Hive 2.3.9

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.2.1-hadoop-3.3.1-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.2-gen18
• platform-3.2.1-gen18

Dependency versions

name	version
Spark	3.2.1
Hadoop	3.3.1
Java	8
Scala	2.12
Python	3.8
Hive	2.3.9
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.901
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.3.9
ADLS gen2 (abfss://)	Azure Storage 7.0.1
Azure Blob Storage (wasbs://)	Azure Storage 7.0.1
AWS Glue	Hive 2.3.9

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.2.0-hadoop-3.3.1-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.2.0-gen18

Dependency versions

name	version
Spark	3.2.0
Hadoop	3.3.1
Java	8
Scala	2.12
Python	3.8
Hive	2.3.9
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.901
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.3.9
ADLS gen2 (abfss://)	Azure Storage 7.0.1
Azure Blob Storage (wasbs://)	Azure Storage 7.0.1
AWS Glue	Hive 2.3.9

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.1.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.1-gen18
• platform-3.1.3-gen18

Dependency versions

name	version
Spark	3.1.3
Hadoop	3.2.0
Java	8
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.1.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.1.2-gen18

Dependency versions

name	version
Spark	3.1.2
Hadoop	3.2.0
Java	8
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.1.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.1.1-gen18

Dependency versions

name	version
Spark	3.1.1
Hadoop	3.2.0
Java	8
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.0.3-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.0-gen18
• platform-3.0.3-gen18

Dependency versions

name	version
Spark	3.0.3
Hadoop	3.2.0
Java	8
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.0.2-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.0.2-gen18

Dependency versions

name	version
Spark	3.0.2
Hadoop	3.2.0
Java	8
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.0.1-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.0.1-gen18

Dependency versions

name	version
Spark	3.0.1
Hadoop	3.2.0
Java	8
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.0.0-hadoop-3.2.0-java-8-scala-2.12-python-3.8-gen18

Additional tags

• platform-3.0.0-gen18

Dependency versions

name	version
Spark	3.0.0
Hadoop	3.2.0
Java	8
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-2.4.7-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen18

Additional tags

• platform-2.4.7-gen18
• platform-2.4-gen18

Dependency versions

name	version
Spark	2.4.7
Hadoop	3.1.0
Java	8
Scala	2.12
Python	3.7
Hive	Not supported
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.271
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.5
ADLS gen2 (abfss://)	Azure Storage 5.4.0
Azure Blob Storage (wasbs://)	Azure Storage 5.4.0
AWS Glue	Not supported

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.9.3

platform-2.4.6-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen18

Additional tags

• platform-2.4.6-gen18

Dependency versions

name	version
Spark	2.4.6
Hadoop	3.1.0
Java	8
Scala	2.12
Python	3.7
Hive	Not supported
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.271
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.5
ADLS gen2 (abfss://)	Azure Storage 5.4.0
Azure Blob Storage (wasbs://)	Azure Storage 5.4.0
AWS Glue	Not supported

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.9.3

platform-2.4.5-hadoop-3.1.0-java-8-scala-2.12-python-3.7-gen18

Additional tags

• platform-2.4.5-gen18

Dependency versions

name	version
Spark	2.4.5
Hadoop	3.1.0
Java	8
Scala	2.12
Python	3.7
Hive	Not supported
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.271
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.5
ADLS gen2 (abfss://)	Azure Storage 5.4.0
Azure Blob Storage (wasbs://)	Azure Storage 5.4.0
AWS Glue	Not supported

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.9.3

platform-3.3.0-hadoop-3.3.2-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.3.0
Hadoop	3.3.2
Java	11
Scala	2.12
Python	3.8
Hive	2.3.9
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.1026
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.3.9
ADLS gen2 (abfss://)	Azure Storage 7.0.1
Azure Blob Storage (wasbs://)	Azure Storage 7.0.1
AWS Glue	Hive 2.3.9

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.2.1-hadoop-3.3.1-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.2.1
Hadoop	3.3.1
Java	11
Scala	2.12
Python	3.8
Hive	2.3.9
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.901
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.3.9
ADLS gen2 (abfss://)	Azure Storage 7.0.1
Azure Blob Storage (wasbs://)	Azure Storage 7.0.1
AWS Glue	Hive 2.3.9

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.2.0-hadoop-3.3.1-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.2.0
Hadoop	3.3.1
Java	11
Scala	2.12
Python	3.8
Hive	2.3.9
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.901
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.3.9
ADLS gen2 (abfss://)	Azure Storage 7.0.1
Azure Blob Storage (wasbs://)	Azure Storage 7.0.1
AWS Glue	Hive 2.3.9

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.1.3-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.1.3
Hadoop	3.2.0
Java	11
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.1.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.1.2
Hadoop	3.2.0
Java	11
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.1.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.1.1
Hadoop	3.2.0
Java	11
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.0.3-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.0.3
Hadoop	3.2.0
Java	11
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.0.2-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.0.2
Hadoop	3.2.0
Java	11
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.0.1-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.0.1
Hadoop	3.2.0
Java	11
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-3.0.0-hadoop-3.2.0-java-11-scala-2.12-python-3.8-gen18

Dependency versions

name	version
Spark	3.0.0
Hadoop	3.2.0
Java	11
Scala	2.12
Python	3.8
Hive	2.3.7
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.375
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.9
ADLS gen2 (abfss://)	Azure Storage 7.0.0
Azure Blob Storage (wasbs://)	Azure Storage 7.0.0
AWS Glue	Hive 2.3.7

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.10.0

platform-2.4.7-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen18

Dependency versions

name	version
Spark	2.4.7
Hadoop	3.1.0
Java	8
Scala	2.11
Python	3.7
Hive	Not supported
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.271
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.5
ADLS gen2 (abfss://)	Azure Storage 5.4.0
Azure Blob Storage (wasbs://)	Azure Storage 5.4.0
AWS Glue	Not supported

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.9.3

platform-2.4.6-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen18

Dependency versions

name	version
Spark	2.4.6
Hadoop	3.1.0
Java	8
Scala	2.11
Python	3.7
Hive	Not supported
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.271
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.5
ADLS gen2 (abfss://)	Azure Storage 5.4.0
Azure Blob Storage (wasbs://)	Azure Storage 5.4.0
AWS Glue	Not supported

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.9.3

platform-2.4.5-hadoop-3.1.0-java-8-scala-2.11-python-3.7-gen18

Dependency versions

name	version
Spark	2.4.5
Hadoop	3.1.0
Java	8
Scala	2.11
Python	3.7
Hive	Not supported
Pyarrow	3.0.0

Supported connectors

name	version
S3 (s3a:// or s3://)	AWS 1.11.271
GCS (gs://)	2.1.5
ADLS gen1 (adl://)	ADLS SDK 2.2.5
ADLS gen2 (abfss://)	Azure Storage 5.4.0
Azure Blob Storage (wasbs://)	Azure Storage 5.4.0
AWS Glue	Not supported

Additonnal formats

name	version
Delta	Not supported
Snowflake	Spark Snowflake 2.9.3