Problem

Production query nodes intermittently fail S3/DynamoDB operations with:

Failed to get AWS credentials: ... DispatchFailure { source: ConnectorError { kind: Timeout, source: HttpTimeoutError { kind: "HTTP connect", duration: 3.1s } } }

which surfaces to callers as a 500.

AwsCredentialAdapter (lance-io/src/object_store/providers/aws.rs) proactively refreshes credentials credentials_refresh_offset (default 60s) before they expire. During that window the cached credentials are still valid, but the adapter treated the cache as a miss and performed a blocking refresh. When that refresh hit a transient error from the underlying provider — e.g. an IMDS/STS HTTP connect timeout — get_credential discarded the still-valid cached credentials and returned a hard error.

The same adapter backs both the S3 object store and the DynamoDB external manifest store (via OSObjectStoreToAwsCredAdaptor), so a single transient credential-provider blip during the refresh window turns into a failed request.

This is the gap left by the earlier credential-caching/refresh-offset work: hard expiry is handled, but a failed proactive refresh was not.

Fix

When a refresh fails but the cached credentials have not actually expired yet, fall back to the cached credentials and log a warning; the next call retries the refresh. Truly-expired credentials still surface the error rather than being served.

Added a unit test (test_aws_credential_adapter_falls_back_to_cached_on_refresh_failure) using a provider that succeeds once and then fails, asserting that the still-valid cached credentials are served while valid, and that an error is returned once they expire.

Notes

• This addresses transient failures during the refresh window. A cold-start credential fetch (empty cache) against an unreachable IMDS/STS endpoint will still fail, as there is nothing valid to fall back to.
• cargo fmt/cargo clippy could not be run in this environment (the pinned toolchain's rustfmt/clippy components are unavailable offline). The change was kept formatting-clean by hand; please confirm in CI.