Skip to content

Spark Procedures🔗

To use Iceberg in Spark, first configure Spark catalogs. Stored procedures are only available when using Iceberg SQL extensions in Spark 3.

Usage🔗

Procedures can be used from any configured Iceberg catalog with CALL. All procedures are in the namespace system.

CALL supports passing arguments by name (recommended) or by position. Mixing position and named arguments is not supported.

Named arguments🔗

All procedure arguments are named. When passing arguments by name, arguments can be in any order and any optional argument can be omitted.

CALL catalog_name.system.procedure_name(arg_name_2 => arg_2, arg_name_1 => arg_1);

Positional arguments🔗

When passing arguments by position, only the ending arguments may be omitted if they are optional.

CALL catalog_name.system.procedure_name(arg_1, arg_2, ... arg_n);

Snapshot management🔗

rollback_to_snapshot🔗

Roll back a table to a specific snapshot ID.

To roll back to a specific time, use rollback_to_timestamp.

Info

This procedure invalidates all cached Spark plans that reference the affected table.

Usage🔗

Argument Name Required? Type Description
table ✔️ string Name of the table to update
snapshot_id ✔️ long Snapshot ID to rollback to

Output🔗

Output Name Type Description
previous_snapshot_id long The current snapshot ID before the rollback
current_snapshot_id long The new current snapshot ID

Example🔗

Roll back table db.sample to snapshot ID 1:

CALL catalog_name.system.rollback_to_snapshot('db.sample', 1);

rollback_to_timestamp🔗

Roll back a table to the snapshot that was current at some time.

Info

This procedure invalidates all cached Spark plans that reference the affected table.

Usage🔗

Argument Name Required? Type Description
table ✔️ string Name of the table to update
timestamp ✔️ timestamp A timestamp to rollback to

Output🔗

Output Name Type Description
previous_snapshot_id long The current snapshot ID before the rollback
current_snapshot_id long The new current snapshot ID

Example🔗

Roll back db.sample to a specific day and time. 

CALL catalog_name.system.rollback_to_timestamp('db.sample', TIMESTAMP '2021-06-30 00:00:00.000');

set_current_snapshot🔗

Sets the current snapshot ID for a table.

Unlike rollback, the snapshot is not required to be an ancestor of the current table state.

Info

This procedure invalidates all cached Spark plans that reference the affected table.

Usage🔗

Argument Name Required? Type Description
table ✔️ string Name of the table to update
snapshot_id long Snapshot ID to set as current
ref string Snapshot Reference (branch or tag) to set as current

Either snapshot_id or ref must be provided but not both.

Output🔗

Output Name Type Description
previous_snapshot_id long The current snapshot ID before the rollback
current_snapshot_id long The new current snapshot ID

Example🔗

Set the current snapshot for db.sample to 1: 

CALL catalog_name.system.set_current_snapshot('db.sample', 1);

Set the current snapshot for db.sample to tag s1: 

CALL catalog_name.system.set_current_snapshot(table => 'db.sample', ref => 's1');

cherrypick_snapshot🔗

Cherry-picks changes from a snapshot into the current table state.

Cherry-picking creates a new snapshot from an existing snapshot without altering or removing the original.

Only append and dynamic overwrite snapshots can be cherry-picked.

Info

This procedure invalidates all cached Spark plans that reference the affected table.

Usage🔗

Argument Name Required? Type Description
table ✔️ string Name of the table to update
snapshot_id ✔️ long The snapshot ID to cherry-pick

Output🔗

Output Name Type Description
sou