Sling v1.5.24 (2026-08-09T21:32:00Z)

New Features

• ADBC auto-setup: Automatically downloads the ADBC driver manager (conda-forge) when missing, and auto-installs drivers via the dbc CLI when a driver path is not found locally (disable with SLING_DISABLE_DBC_AUTO_INSTALL). Windows discovery for adbc_driver_manager.dll is included.

Bug Fixes

• Redshift AWS credential chain: Falls back to the default AWS credential chain (env vars, shared profiles, IAM roles) when Redshift/S3 credentials are not explicitly configured, with clearer errors and redacted credential logging.

• Soft-merge delete guard NULL-safe: CDC soft-merge _sling_synced_op != 'D' guards now use COALESCE across dialects so pre-CDC target rows with NULL _sling_synced_op are no longer skipped (including Redshift).

• ADBC connection keys and auth: Corrected driver property keys (e.g. Snowflake/Trino uri), added Snowflake PAT and key-pair (JWT) auth, fixed BigQuery auth_type handling, and improved adbc_uri overrides per database type.

• ADBC library load diagnostics: Load failures (outdated libstdc++/glibc, wrong arch, ADBC_DRIVER_MANAGER_LIB) now return actionable remediation steps, including an LD_PRELOAD hint on Linux when needed.

• Column select vs SQL expressions: * inside SQL/JSONPath expressions (e.g. concat('a*', id), '$[*].amount') is no longer treated as a column glob; unmatched globs warn instead of silently no-oping.

• DuckDB Arrow / concurrency: Upgraded DuckDB, fixed copy_method → arrow_http when DUCKDB_USE_ARROW is set, improved Arrow stream error handling, and fall back to CSV when Arrow is unavailable due to a locked instance.

• Bool vs int cast to string: Boolean→string casting no longer uses integer comparisons/CAST that break on PostgreSQL/Redshift; integers targeting string columns use a plain varchar cast.

• Incremental state time format: time.Time state values are stored as RFC3339 so watermarks round-trip correctly with cast.ToTime (avoids full re-pulls).

• GCS auth collision: Resolved multiple-credential-option errors with newer Google API clients; added GC_KEY_BODY, GC_KEY_FILE, KEYFILE, and GC_CRED_API_KEY aliases.

Sling v1.5.23 (2026-08-03T16:51:29Z)

New Features

• Three-part object names (database.schema.table): Table names can now be qualified with a database/catalog for dialects that support it (Snowflake, Databricks, Trino, BigQuery, DuckDB/DuckLake/MotherDuck, Fabric, SQL Server, Azure, Azure DWH). The database qualifier now flows into metadata queries (columns, primary keys, indexes, DDL), Snowflake internal-stage creation, and temp tables, which inherit the target's database. Dialects that don't support it now reject 3-part names with a clear error instead of silently misbehaving.

• Oracle read performance (prefetch_rows): New connection property to control the number of rows fetched per network round-trip, defaulting to 100 instead of go-ora's 25. Large table reads are significantly faster; lower it (e.g. 25) for tables with large BLOB/CLOB columns to reduce memory usage.

• ScyllaDB improvements: Column metadata is now read from system_schema.columns (instead of select * limit 1), with proper native CQL → Sling type mapping, keyspace/table/view introspection, truncate/add column/drop column support, CQL-safe DROP TABLE generation, NetworkTopologyStrategy for keyspace creation, and json mapped to text.

• Upsert-by-insert for key-value stores: incremental and backfill modes with a primary key now work with direct writes for ScyllaDB, MongoDB, and Azure Table, where INSERT is already an upsert-by-PK — no more falling back to a temp table or erroring out.

• Go 1.26: Toolchain and dependencies bumped to Go 1.26.0.

Bug Fixes

• ClickHouse pending mutations block table drops: DropTable now kills in-flight mutations referencing the table before dropping it, avoiding failures when a previous delete/update mutation is still running.

• ClickHouse long loads fail with "bad connection" at commit: Large ClickHouse loads that held a single transaction open for the whole run would fail when the connection was reaped underneath mid-commit. Loads now commit on a wall-clock cadence (batch_max_duration, default 5 minutes for ClickHouse), and ClickHouse connections use a bounded max lifetime/idle time so the pool retires stale connections cleanly.

• S3 transfers truncated on large/slow objects: Removed the client-wide 300s HTTP timeout, which capped the entire request including body reads. Connect/handshake/header phases remain bounded by transport timeouts; a whole-request deadline can still be opted into via HTTP_TIMEOUT.

• SQL Server merge double-casting: The MERGE template now uses casted field variants (set_fields_casted, src_insert_fields_casted), since src is a derived table where the cast has already been applied — previously the cast was applied twice.

• SQL Server integer-to-string cast: Integers targeting a string column are now cast with a plain cast(... as varchar); only actual boolean columns get the 1/0 → 'true'/'false' conversion. Previously integer columns were incorrectly rendered as true/false.

• Nested format variables missing during pre-render: Nested maps (source.type, target.type, stream.table, …) are now built before target-object pre-rendering and refreshed afterwards, so JMESPath lookups resolve in all cases.

• Wildcard streams not matching explicit stream config: Wildcard expansion now matches explicit stream entries on schema.table first, so a 2-part stream definition still matches when the table renders with a database qualifier.

• Command/SSH hook output bypassing log sinks: command hook output (both local and SSH) is now routed through env.Println instead of writing straight to os.Stdout/os.Stderr, so it is properly captured by log sinks. Partial trailing lines are flushed at the end.

Sling v1.5.22 (2026-07-26T23:16:01Z)

New Features

• ScyllaDB connector: New scylladb database connection type for basic import/export, including connection template, default port (9042), and NoSQL type registration.

• MariaDB CDC reader: Dedicated binlog reader for MariaDB (separate from MySQL) handling the 4-column SHOW BINLOG STATUS DDL, the MariaDB GTID scheme, and transaction-boundary reads when resuming by file/offset.

• CDC change_feed option: Reference a pre-provisioned, DBA-managed server-side CDC object instead of relying on the engine default — a PostgreSQL publication, a SQL Server capture instance, or an Oracle GoldenGate stream. Takes precedence over engine-specific properties (e.g. gg_stream).

• Composite unique keys: Nested unique key groups (e.g. unique: [[a, b]]) are now preserved as composite unique indexes in generated DDL instead of being flattened into separate single-column constraints.

• Redshift IAM role & profile auth: Added AWS_ROLE_ARN and AWS_PROFILE support for S3 authentication during Redshift COPY/UNLOAD, alongside the existing access-key/session-token paths. AWS_ROLE_ARN=default uses the cluster's attached default IAM role.

• Pipeline execution state & CLI args: Pipeline runtime state now exposes execution-level fields (ID, start time, status, errors), and CLI arguments are surfaced via SLING_CLI_ARGS_MAP with dashed keys normalized to underscores and list values (primary-key, select, streams) preserved as structured data.

Bug Fixes

• on_failure: defer in hook group loops: Deferred step failures no longer abort the group mid-loop. They are collected across all loop iterations (concurrent and sequential) and surfaced only after every iteration completes; hard failures still abort immediately.

• CDC replay_from in shared-reader groups: replay_from was silently ignored when multiple streams shared a CDC reader — the group computed its start position from each stream's already-advanced position. Stream positions are now rewound before the shared start position is computed, matching single-stream semantics.

• MariaDB CDC table filtering (#775): Wildcard-expanded stream names retained their identifier quoting (e.g. `sf`.`fakturs`), so no binlog rows event ever matched the filter set and runs reported "read 0 CDC events". Table names are now normalized (quotes stripped, lower-cased) before filtering.

• Replication runtime state race: ReplicationConfig.RuntimeState now locks state access to prevent race conditions from concurrent callers.

Sling v1.5.21 (2026-07-15T13:52:59Z)

New Features

• CSV skip_lines source option: New skip_lines source option to skip a fixed number of leading lines (e.g. preamble rows) before the header is parsed.
• Runtime variables in stream SQL: Stream sql: queries are now re-rendered after start-hooks run, so {store.*}, {state.*}, and {execution.*} variables resolve at query time (also exposed in the format map).
• SQL Server pre-fetched access-token auth: Added ActiveDirectoryServicePrincipalAccessToken support — connect and BCP-import using a pre-fetched Azure AD access token (delivered via the security-token protocol, not the DSN).
• Redshift Serverless auto-detection: Connections to redshift-serverless.amazonaws.com now automatically use the Redshift connector.
• Composite table-key form: Primary/unique/partition keys now accept the wrapped/composite list form (e.g. primary: [[user_id, brand_id]]), flattened to behave identically to the flat form.
• API composite primary-key dedup: A composite primary_key declared on an API spec now deduplicates records at the extraction layer (first-seen wins), preventing duplicate-key violations on merge.

Bug Fixes

• StarRocks multi-chunk stream load: Probe the FE with an empty body to discover the CN/BE URL up front and reuse the redirect host across chunks — avoids re-uploading each chunk's full body and the resulting timeouts on large loads.
• Redshift merge/CDC SQL: Removed unsupported table aliases from DELETE/UPDATE statements in merge, delete-insert, and change-capture templates.
• Unsized decimal cast overflow: An explicit decimal cast (type only, no precision) no longer locks the sample-inferred precision, which could silently saturate/overflow later rows on strict engines like StarRocks.
• Explicitly-typed columns preserved during casting: Columns pinned via columns: are no longer reverted to string/decimal mid-stream when a value fails to cast.
• Oracle sqlldr UTF-8: Added CHARACTERSET AL32UTF8 to the sqlldr control template to correctly load multi-byte UTF-8 data.
• DuckDB http_timeout: The raised http_timeout is now applied on every session (not only when the httpfs extension is registered), with a http_timeout connection property to override it.
• DuckDB describe errors: A failing Describe (e.g. a missing table) now surfaces the real underlying error instead of returning an empty result.
• Snowflake private-key validation: Key-pair auth now returns actionable errors for wrong-format keys (PKCS#1, OpenSSH) and mismatched passphrase cases, with conversion instructions.
• API authenticator errors: Authenticate no longer dereferences a nil authenticator, and unsupported authentication types now return a clear error.
• Spurious API state warning: The "use SLING_STATE for incremental mode" warning is no longer shown for non-incremental API syncs.
• Deprecated queues: warning: Downgraded the deprecated top-level queues: spec-field warning to debug (queues are auto-detected).
• API rule messages: Retry/rule messages now render request-scoped variables so their interpolated suffixes resolve correctly.

Sling v1.5.20 (2026-06-07T10:37:59Z)

See https://github.com/slingdata-io/sling-cli/ for more details.

Sling v1.5.19 (2026-05-25T11:50:17Z)

New Features

• Column reordering and renames in select: The select clause now supports reordering and renaming columns inline. Combined with stricter handling of select exclusions and excluded PKs, this gives finer control over output schemas without needing downstream transforms.

• Oracle LOB / binary fidelity to Snowflake: End-to-end row-level fidelity for Oracle LONG, LONG RAW, and large BLOB columns when replicating to Snowflake. Snowflake COPY VIA STAGE now casts hex-encoded binary to BINARY with explicit sizing up to 64MB, Snowflake parquet file formats set binary_as_text = false, DuckDB staging decodes hex-encoded binary to BLOB for parquet, DuckDB read_csv max_line_size is raised for large hex blobs, and BinaryToHex conversion is optimized from O(n²) to O(n). Type inference is refined to preserve native binary types and apply explicit lengths.

• null_if function: New function for normalizing values to null when they match a target — useful for cleaning sentinel values out of API responses and CSV inputs.

• JSON-RPC error detection in endpoints: API endpoints now recognize JSON-RPC error envelopes and surface them through the standard rule/retry pipeline instead of treating them as successful payloads.

• Idle DB-connection reaping in sling lsp: The language server now reaps idle database connections instead of holding them open for the lifetime of the editor session.

• FileType.Exts method: New helper returns all valid extensions for a FileType, enabling correct detection of multi-extension formats (e.g. .xlsm now recognized as Excel file).

Bug Fixes

• Inline iterate.over preserved on dynamic API endpoints: Inner iterate.over is now preserved across dynamic endpoint expansion, and the over field's JSON tag is corrected so YAML/JSON specs round-trip correctly.

• BigQuery errors masked by context.Canceled: BigQuery job-wait logic is consolidated into waitForJob, which re-fetches job status with a detached context if job.Wait returns context.Canceled. Real BigQuery API errors now surface from LoadFromReader, CopyFromGCS, and CopyToGCS instead of being hidden behind a generic cancellation error.

• GCS path format in importViaGoogleStorage: Corrected the GCS URI format used when importing via Google Storage, fixing failures on BigQuery loads that route through GCS.

• Auth-sequence responses skipped default rules: Default retry/fail rules are now applied to authentication-sequence responses in dbio/api. Previously these responses bypassed the rule pipeline, producing unclear errors on auth failures. Rule application is also guarded for idempotency to prevent stacking on re-compilation.

• Endpoint rule stacking on re-compilation: Added an idempotency guard to checkResponse so endpoint rules aren't duplicated when specs are re-compiled, and ensured mutations to setup/teardown call requests/responses persist by indexing directly into slices.

• CDC metadata column validation: Added support for retrieving and validating CDC metadata columns after the initial snapshot, with additional debug logging on CDC operations to make troubleshooting easier.

• Temp-table double-drop: Skip dropping a temp table that has already been marked as dropped, and make the temp-table cleanup key unique to avoid collisions between concurrent tasks.

Sling v1.5.18 (2026-05-12T20:15:08Z)

New Features

• Queue-only API endpoints: New queue_only flag on API spec endpoints. Such endpoints drain synchronously to populate downstream queues and emit no records of their own, and are hidden from broad enumeration (sling conns discover, wildcard replication expansion) while remaining addressable by name. queue_only is also honored as a defaults value in the spec.

• Inline iterate lists/objects for dynamic API endpoints: Dynamic endpoints now accept native YAML arrays and objects in iterate (in addition to JSON literals and JMESPath strings). Nested YAML maps are normalized to map[string]any for consistent template/JMESPath use.

• sling conns exec output formats and --limit: sling conns exec now supports --output csv|json|arrow with streaming (Arrow IPC / CSV go straight to stdout, logs to stderr) and a --limit flag (default 100, 0 = unlimited). The limit is pushed down via the dialect's limit_sql template so the database truncates server-side. sling conns discover also gains CSV output.

Bug Fixes

• CDC nil target options on shared-reader path: CDC's shared-reader write path bypassed TaskExecution.Execute() and left target options (ColumnCasing, AddNewColumns, AdjustColumnType) nil. Now calls Config.SetDefault() on the cloned config so system-wide defaults apply.

• Dataflow.Pause timeout on slow stream starts: Pause now waits up to a configurable timeout (SLING_PAUSE_TIMEOUT, default 30s) and returns false if any stream can't pause in time or the dataflow context is canceled, instead of using a short random timer.

• MySQL LoadDataLocal missing batches: Reader handler is now registered per batch in MySQL's LoadDataLocal, ensuring every batch is loaded instead of only the first.

• DuckDB Describe errors no longer masked: SELECT/WITH queries fail fast on Describe errors in DuckDB instead of swallowing the error and returning misleading results.

• Nil reader panic in ConsumeCsvReaderChl: Guards against a nil reader to prevent panics.

• Spurious HTTP-URL check in GenerateInsertStatement: Removed an unnecessary check that could misclassify insert targets.

• chunk() function return type: The pipeline chunk() function now returns *ChunkResult instead of a bare channel for consistent downstream consumption.

• Cleaner error/messages: Clearer row-count-mismatch error in WriteToDb/writeToDbDirectly, simplified pause error handling on those same paths, corrected DuckDB pipeline path and sling command output message, clarified the sling run CLI description.

Sling v1.5.17 (2026-05-05T02:25:34Z)

New Features

• CDC shared-reader replication: New CDCGroupRunner orchestrates multi-stream CDC replications through a shared reader, with phased prepare/read-into-cache/write-and-commit execution and bounded-parallel writes.
• Connection retry in HookQuery: Automatically retries query hooks against dead database connections instead of failing the run.
• NewArrowWriter options: Arrow writer accepts additional options for finer control over output, with quieter logging in InferColumnTypes.
• sling agent service commands: Manage the Sling agent as a system service on Windows, macOS, and Linux (install/uninstall/start/stop).
• sling project CLI subcommands: New commands for managing project executions, files, and connections — including filters, output formatting, create/update/delete on jobs and files, and confirmation prompts on destructive actions.
• MCP file:// SQL support: MCP server can now execute queries from saved SQL files.
• LSP working-directory awareness: Language server resolves env and project files relative to the current working directory.

Bug Fixes

• Concurrent map iteration in API requests: Locked context in NewSingleRequest to prevent concurrent map iteration and write panics during API calls.
• Source options on zipped files: Source options (delimiter, header, etc.) are now propagated when reading from zip archives. (thanks @hermogenes)
• Standardized casing for file paths across scripts and documentation to avoid case-sensitivity issues on Linux.

Sling v1.5.15 (2026-04-15T22:44:09Z)

New Features

• CDC support for PostgreSQL, SQL Server, Oracle, and MongoDB: Change Data Capture now works with PostgreSQL (logical replication), SQL Server, Oracle, and MongoDB sources — previously only MySQL was supported.
• CDC initial snapshot cursor-based resume: Initial CDC snapshots now use keyset-based chunking for string/UUID primary keys, enabling reliable cursor-based resume after interruption.
• CDC catch-up mode with end position: CDC ReadChanges now captures the current position before streaming so catch-up runs terminate without waiting for new events.
• Default adjust_column_type for CDC: CDC replications now default adjust_column_type to true, automatically widening target columns (varchar length, decimal precision) when the source schema changes.
• adjust_column_type precision expansion: The adjust_column_type option now detects and expands varchar length and decimal precision/scale when the source columns grow wider, avoiding truncation or precision loss.
• description parameter for SQL query actions: MCP database.query and worksheet logging now accept an optional description field for improved context and observability.
• MCP worksheet query logging: MCP SQL queries are now appended to a worksheet log file with metadata (connection, description, timestamp, duration, columns, row count), with automatic size trimming.

Bug Fixes

• Oracle SID/service_name conflict: Fixed Oracle connection strings where the SID option could conflict with service_name, causing connection failures.
• Progress update crash when replication is nil: Fixed a panic when updating task progress on a task with no associated replication.
• API spec record limit not stopping iteration: Fixed an issue where the API spec record limit was not properly stopping inner iteration loops during rule evaluation.

Sling v1.5.14 (2026-04-06T15:41:03Z)

New Features

• SQL Server BCP with Entra ID authentication: BCP bulk loading now supports Entra ID token-based auth and Azure CLI integration for passwordless authentication.
• DuckDB Arrow IPC streaming: Implement Arrow IPC streaming for DuckDB connections, improving read/write performance.
• CDC full-refresh mode: Added support for full-refresh mode with CDC options, clearing state before running an initial snapshot.
• CDC UUID primary key support: CDC initial snapshots now handle string/UUID primary keys with offset-based chunking, in addition to integer PKs.
• length() function supports scalar types: The length() function now safely handles integers, floats, booleans, and dates by converting to string representation, allowing use in wildcard transforms with ternary guards.
• sling project jobs run --wait flag: New --wait / -w flag polls execution status every 5 seconds until the job reaches a terminal state, then reports rows, bytes, and duration.
• OAuth2 custom credentials and spec URL: Enhanced OAuth2 initialization to support custom client credentials and spec URL for flexible token management.

Bug Fixes

• Oracle sqlldr column sizing: Fixed sqlldr control file generation to use target column precision instead of hardcoding char(400000), preventing excessive memory allocation during Oracle bulk imports.
• TINYINT type handling: Corrected data type mapping for tinyint in metadata queries across MariaDB, MySQL, and StarRocks templates.
• Cross-platform path handling: Replaced path package with filepath across the codebase for correct behavior on Windows.
• Token save in agent mode: Allow local SaveToken to proceed even when running in agent mode.