19 Pull requests merged by 12 people

• [Kernel] Track nested fields in ArrayType / MapType

  #3426 merged Jul 26, 2024

• [DO_NOT_MERGE][4.0 Preview] Docs changes

  #3268 merged Jul 26, 2024

• [Spark] Extend coordinated commit changes to other suites

  #3305 merged Jul 25, 2024

• [Spark] Fix the semantic of shouldRewriteToBeIcebergCompatible in REORG UPGRADE UNIFORM

  #3412 merged Jul 25, 2024

• [Spark] Flip isDeltaTable.throwOnError to true

  #3422 merged Jul 25, 2024

• [Spark] Add Delta log throttling class

  #3418 merged Jul 25, 2024

• [Hudi] Refactor tests

  #3415 merged Jul 24, 2024

• [Storage][Kernel] Add InMemoryCommitCoordinator

  #3377 merged Jul 24, 2024

• Match table exhaustively for ConvertToDelta to avoid scala.MatchError

  #3411 merged Jul 24, 2024

• [Kernel] Wrap calls into the engine implementation with KernelEngineException

  #3407 merged Jul 24, 2024

• [Kernel] Add parsing support for coordinated commits Map type table properties

  #3400 merged Jul 23, 2024

• Revert "[Spark] Add custom not matched for insert clauses expr (#3405)"

  #3410 merged Jul 23, 2024

• [UNIFORM] Remove unnecessary iceberg patch and disable cleanup of files in expire snapshots via cleanExpiredFiles API

  #3399 merged Jul 23, 2024

• [Kernel] Add default coordinated commits handler implementation

  #3396 merged Jul 23, 2024

• [Spark] Add custom not matched for insert clauses expr

  #3405 merged Jul 22, 2024

• [Spark] Minor refactor in Delta Protocol Transition Suite

  #3402 merged Jul 22, 2024

• [Spark] Identity Columns Value Generation (without MERGE support)

  #3023 merged Jul 19, 2024

• [Kernel] Add coordinated commits interfaces and table properties

  #3370 merged Jul 19, 2024

• [Spark] Batch resolve DeltaMergeAction in ResolveMergeInto

  #3366 merged Jul 19, 2024

9 Pull requests opened by 8 people

• [doc] Clarify nullability for ALTER TABLE ADD COLUMNS

  #3401 opened Jul 22, 2024

• [Spark] Python DeltaTableBuilder API for Identity Columns

  #3404 opened Jul 22, 2024

• [Kernel] Add exception principles for Kernel to solidify the rules for our exception framework

  #3408 opened Jul 23, 2024

• [Spark][UniForm] Add test suite/utils for enable UniForm without rewrite

  #3414 opened Jul 23, 2024

• [Spark][Kernel][Protocol] Merge InCommitTimestamp RFC, and remove the -preview suffix from feature name and configs.

  #3416 opened Jul 24, 2024

• [Test Spark RC] Upgrade Spark version to use 3.5.2 RC2

  #3417 opened Jul 24, 2024

• [3423] Fix unnecessary DynamoDB GET calls during LogStore::listFrom

  #3425 opened Jul 25, 2024

• [DELTA] Catch fatal error in error matching in commitLarge

  #3428 opened Jul 26, 2024

• [Spark][UniForm] Add test suite/utils for enable UniForm without rewrite

  #3429 opened Jul 26, 2024

5 Issues closed by 2 people

• [BUG] java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.TableIdentifier.copy(Ljava/lang/String;Lscala/Option;)Lorg/apache/spark/sql/catalyst/TableIdentifier

  #3409 closed Jul 23, 2024

• General question: removing optimized/deleted data files without VACUUM

  #3389 closed Jul 22, 2024

• [feature request] Delta table API method to rollback/revert latest committed changes

  #497 closed Jul 21, 2024

• bug on injecting into roblox

  #2560 closed Jul 21, 2024

• [Feature Request] Improve UniForm Hudi support for Lists/Maps and Schema Evolution

  #2738 closed Jul 21, 2024

9 Issues opened by 7 people

• [Feature Request][Kernel] Handle nested schemas when assigning id / physical name for column mapping mode

  #3427 opened Jul 26, 2024

• [Feature Request] Add a salt column to DeltaOptimizeWriterExec

  #3424 opened Jul 25, 2024

• [BUG] ProvisionedThroughputExceededException when performing VACUUM using S3DynamoDBLogStore

  #3423 opened Jul 25, 2024

• [Feature Request][DeltaTable]Read spark table from hive metastore

  #3421 opened Jul 25, 2024

• [Feature Request][Storage]Remove CoordinatedCommitsUtils from Storage test package

  #3420 opened Jul 24, 2024

• [Feature Request][Kernel] Add methods to FieldMetadata to retrieve metadata items with the correct type

  #3419 opened Jul 24, 2024

• [BUG] Merge multiple match check is too strict

  #3413 opened Jul 23, 2024

• [Feature Request] Support Coordinated Commits in Delta Kernel

  #3406 opened Jul 22, 2024

• [QUESTION] support of "parquet io cache"

  #3403 opened Jul 22, 2024

26 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [Kernel] Update the Kernel read with coordinated commit support

  #3381 commented on Jul 26, 2024 • 25 new comments

• [Kernel] Add column mapping metadata update functionality

  #3393 commented on Jul 26, 2024 • 21 new comments

• [Spark, Protocol] UTC normalize timestamp partition values

  #3378 commented on Jul 22, 2024 • 18 new comments

• [Protocol]: Allow storing UTC adjusted timestamp partition values

  #3398 commented on Jul 22, 2024 • 2 new comments

• [Spark][WIP] Fix moving between compatibility versions in REORG TABLE (UPGRADE UNIFORM)

  #3382 commented on Jul 23, 2024 • 2 new comments

• [Spark] Use location as the table path

  #3330 commented on Jul 21, 2024 • 1 new comment

• [SPARK] Optimize : SELECT COUNT(*) FROM Table WHERE partitition=1

  #3345 commented on Jul 22, 2024 • 0 new comments

• [Spark] Refactor to remove duplicated code in Java unit test

  #3324 commented on Jul 24, 2024 • 0 new comments

• [WIP][Spark] Python DeltaTableBuilder API for Identity Columns

  #3044 commented on Jul 22, 2024 • 0 new comments

• [Spark] ALTER TABLE ALTER COLUMN SYNC IDENTITY SQL support

  #3005 commented on Jul 22, 2024 • 0 new comments

• [Feature Request] [MERGE] Avoid copy rows when match clause is false. Potential for huge perf impact.

  #1812 commented on Jul 25, 2024 • 0 new comments

• [BUG][Spark] Reliability issues when upserting data to ADLS using Delta in Synapse Analytics

  #3286 commented on Jul 25, 2024 • 0 new comments

• [BUG] MergeIntoCommand not visible in QueryExecutionListener when using Python/Scala API

  #1521 commented on Jul 25, 2024 • 0 new comments

• [PROTOCOL RFC] Column Mapping Usage Tracking

  #2682 commented on Jul 25, 2024 • 0 new comments

• [Kernel] Add unittests for reading cloned (shallow/deep) Delta tables in Kernel

  #2705 commented on Jul 25, 2024 • 0 new comments

• [Feature Request] Identity Column

  #1959 commented on Jul 24, 2024 • 0 new comments

• [Feature Request][Kernel] Improvement of ICT Read

  #3321 commented on Jul 24, 2024 • 0 new comments

• [Feature Request] data deduplication on existing delta table

  #1767 commented on Jul 23, 2024 • 0 new comments

• [Feature Request][Kernel] Ensure writers use the column-mapping transformed physical schema

  #3384 commented on Jul 23, 2024 • 0 new comments

• [Feature Request][Spark] Add Scala API for cloning

  #3391 commented on Jul 22, 2024 • 0 new comments

• [BUG]

  #3372 commented on Jul 21, 2024 • 0 new comments

• [Question] Are fast parallel writes in Delta Tables on S3 possible?

  #2531 commented on Jul 21, 2024 • 0 new comments

• [doc] NOT NULL in ALTER TABLE ADD COLUMNS is not mentioned in the doc

  #3360 commented on Jul 21, 2024 • 0 new comments

• [BUG][Spark] Merge generate ONE HUGE file

  #3394 commented on Jul 21, 2024 • 0 new comments

• [BUG][Spark] Unable to run Spark Connect 3.5.1 with Delta 3.2.0

  #3332 commented on Jul 21, 2024 • 0 new comments

• [BUG] Delta Lake MERGE INTO will hang Spark session without result (10000 columns).

  #3118 commented on Jul 21, 2024 • 0 new comments