SNOW-2499292 Add serializable Arrow batches for distributed fetch by sfc-gh-pfus · Pull Request #1833 · snowflakedb/gosnowflake

sfc-gh-pfus · 2026-08-06T07:12:39Z

TL;DR

Adds SerializableArrowBatch with ArrowBatch.ToSerializable and SerializableArrowBatch.ToArrowBatch to enable distributed fetch of Arrow result batches across machines with no Snowflake connection.

What changed?

Introduced arrowbatches.SerializableArrowBatch, a portable JSON-serializable descriptor that captures everything needed to fetch a result chunk: the presigned URL, SSE-C headers, inline IPC bytes (for the first batch), row count, column metadata, timezone, and conversion options.
Added ArrowBatch.ToSerializable() to convert a live batch handle into a SerializableArrowBatch before fetching.
Added SerializableArrowBatch.ToArrowBatch(opts ...Option) to reconstruct a fully functional ArrowBatch on a worker node using only an injected *http.Client — no Snowflake session required.
Extracted chunkDownloadHeaders in chunk_downloader.go as a shared helper so the same SSE-C/GCS/Azure header logic is used by both the live download path and the serializable descriptor construction.
Populated BatchRaw.Descriptor (a new ChunkDescriptor field on ia.BatchRaw) in rows.go for both remote chunks (URL + headers) and the inline first batch (base64-decoded RowSetBase64).
Added internal/arrow/download.go with OpenChunkStream, DecodeIPCRecords, and DownloadBatchRecords — low-level primitives that perform HTTP download and IPC decoding without any driver state, used by the reconstructed batch's Download closure.
Added JSON struct tags to ConversionOptions fields so they survive serialization as part of SerializableArrowBatch.
Added a cmd/arrow/serializablebatches example demonstrating the full producer → serialize → worker → fetch flow.
Updated doc.go with documentation covering the distributed fetch pattern, security considerations (presigned URLs and SSE-C keys are sensitive), and expiry caveats.

How to test?

Unit tests in arrowbatches/serializable_test.go cover:

Inline batch round-trip (no HTTP client needed).
Remote batch download with plain and gzip-compressed payloads, including SSE-C header forwarding.
Error cases: missing HTTP client for remote batches, expired/403 URLs, unsupported formats.
Full JSON marshal/unmarshal round-trip including nested RowTypes and all ConversionOptions flags.
loadLocation behavior for fixed-offset, Local, named IANA, and empty timezone names.
Verification that ConversionOptions (e.g. HigherPrecision) carried in the serialized form are actually applied during fetch, not just round-tripped as data.

Integration tests in arrowbatches/serialization_integration_test.go (require a live account) cover:

A 100,000-row result serialized, JSON round-tripped, and re-fetched with a connection-less HTTP client, with value and row-count integrity checks.
Structured/nested column types surviving the serialize → reconstruct → fetch path.

Why make this change?

Arrow batches returned by GetArrowBatches are live handles tied to the Snowflake connection that produced them. Distributed compute frameworks (Spark, Hadoop, etc.) need to partition a large result on a coordinator node and download chunks in parallel on worker nodes that have no Snowflake session. SerializableArrowBatch provides the seam for this: the coordinator serializes batch descriptors (which embed presigned URLs and decryption keys), ships them to workers, and each worker fetches its chunk independently using only a standard HTTP client.

sfc-gh-boler

Just a first pass and some high level commentary. This is a big enough change that it deserves a technical design doc and a PRD.

sfc-gh-boler · 2026-08-06T19:16:34Z

+//
+// Call ToSerializable before Fetch: after a successful Fetch the in-memory records are
+// released, though the descriptor itself remains valid for re-serialization.
+func (rb *ArrowBatch) ToSerializable() (SerializableArrowBatch, error) {


Just "Serialize" or "Marshal" to be consistent with the usual naming scheme.

But this is a bit different - this is not already serialised, this function just takes not serialisable struct and turns it to the serialisable struct.

sfc-gh-boler · 2026-08-06T19:16:49Z

+// The returned batch's Fetch downloads (if remote) and transforms the records exactly
+// as a batch obtained directly from GetArrowBatches would.
+//
+// A remote batch (URL set) requires an *http.Client via WithHTTPClient. Inline batches


It it requires a param it should be passed directly, not through options.

It's not required, for inline chunk it is not used at all. But apart from global timeout, http.DefaultClient is good enough. I'm marking it as optional.

sfc-gh-boler · 2026-08-06T19:19:07Z

+// Expiry: presigned URLs are valid only for a server-controlled window (typically a few
+// hours). A remote SerializableArrowBatch must be shipped and fetched within it. Inline
+// batches (InlineData set) embed their data and never expire.
+type SerializableArrowBatch struct {


It'd just wall it "Serialized". ArrowBatch suffix is pointless, because it is present in the package name already.

But it's not serialised, it's only able to be serialised. I can shorten the name, it may be more go-like, but is arrowbatches.Serializable fine?

sfc-gh-boler · 2026-08-06T19:20:25Z

+	RowTypes []query.ExecResponseRowType `json:"rowTypes"`
+	// TimezoneName is the batch's timezone name (loc.String()).
+	TimezoneName string `json:"timezoneName,omitempty"`
+	// TimezoneOffsetSeconds is the batch's timezone offset, used as a fallback when the


What's the point of TZName if we give the offset already?

Because Go is not Snowflake :) Snowflake uses offset only for timestamps, but to create a correct timezone in Go you need both. It is needed for LTZ.

sfc-gh-boler · 2026-08-06T19:20:56Z

+	TimezoneName string `json:"timezoneName,omitempty"`
+	// TimezoneOffsetSeconds is the batch's timezone offset, used as a fallback when the
+	// worker cannot resolve TimezoneName (e.g. fixed-offset zones, or missing tzdata).
+	TimezoneOffsetSeconds int `json:"timezoneOffsetSeconds,omitempty"`


TZs don't have offsets in seconds. That's confusing. Minutes would be fine.

But Go API requires seconds to create a location:

// FixedZone returns a [Location] that always uses // the given zone name and offset (seconds east of UTC). func FixedZone(name string, offset int) *Location {

sfc-gh-boler · 2026-08-06T19:35:09Z

+	}
+	resp, err := client.Do(req)
+	if err != nil {
+		return nil, fmt.Errorf("arrow: fetching chunk: %w", err)


is this going to print the URL containing keys (presigned)?

It shouldn't, it goes through our secret detector.

sfc-gh-boler · 2026-08-06T19:35:45Z

+	for k, v := range headers {
+		req.Header.Set(k, v)
+	}
+	resp, err := client.Do(req)


what's the retry and cancelation story?

Re: cancellation - this request takes context, so it should honour cancellations as usual.
Re: retry - we have an ugly retry mechanism, but it is in our main package, not available here. I can move it to internal package (http?), but I'd prefer doing this in another PR.

sfc-gh-boler · 2026-08-06T19:37:04Z

+			if inlineBytes, err := base64.StdEncoding.DecodeString(scd.RowSet.RowSetBase64); err == nil {
+				batch.Descriptor = ia.ChunkDescriptor{InlineData: inlineBytes}
+			} else {
+				logger.WithContext(scd.ctx).Warnf("failed to decode RowSetBase64 for arrow batch serialization: %v", err)


So we log an error, but there is no signalling to the caller it seems.

Good catch. I also think that there's no point in decoding it here, since this field is used only for serialising it later. So I kept base64 instead of byte array.

sfc-gh-boler · 2026-08-06T19:38:37Z

+// field is kept for forward compatibility with the streaming/JSON result path.
+const arrowFormatName = "arrow"
+
+// SerializableArrowBatch is a portable, self-contained description of an ArrowBatch.


Do we even want to expose those fields? They seem internal to Marsha/Unmarshal.

Yes, this is needed for JSON serialisation.

sfc-gh-boler · 2026-08-06T19:39:05Z

+// hours). A remote SerializableArrowBatch must be shipped and fetched within it. Inline
+// batches (InlineData set) embed their data and never expire.
+type SerializableArrowBatch struct {
+	// Format is the chunk payload format; currently always "arrow".


What does this mean specifically? In other words, when is this going to change?

If we want to provide a contract between orchestrators and executors (distributed system), we need a version number here.

What about orchestration between different drivers? Is the plan to support it?

This means, that this should be always Arrow, not JSON.
This is never an external executors, it should be always handled by a drivers. So all these should be an implementation detail.
About other drivers - only Go driver supports arrow batches in this form. We can think about improvements in UD, if needed.

sfc-gh-pfus · 2026-08-07T06:24:53Z

SNOW-2499292 Add serializable Arrow batches for distributed fetch #1833 👈 (View in Graphite)
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

arcticowl-ai-review · 2026-08-25T10:04:39Z

+	desc := ia.ChunkDescriptor{
+		URL:              s.URL,
+		Headers:          s.Headers,
+		InlineDataBase64: s.InlineData,
+		UncompressedSize: s.UncompressedSize,
+	}


🟠 MEDIUM · OpenChunkStream follows arbitrary attacker-influenced URL from a deserialized descriptor with no scheme/host allowli · CWE-918

github-advanced-security Bot found potential problems Aug 6, 2026

View reviewed changes

Comment thread cmd/arrow/serializablebatches/serializable_batches.go Dismissed

Comment thread cmd/arrow/serializablebatches/serializable_batches.go Dismissed

sfc-gh-pfus force-pushed the pfus-08-06-snow-2499292_add_serializable_arrow_batches_for_distributed_fetch branch from eb4b175 to 8cdcbee Compare August 6, 2026 07:27

sfc-gh-pfus marked this pull request as ready for review August 6, 2026 07:30

sfc-gh-pfus requested a review from a team as a code owner August 6, 2026 07:30

sfc-gh-pfus force-pushed the pfus-08-06-snow-2499292_add_serializable_arrow_batches_for_distributed_fetch branch 2 times, most recently from e006472 to 78d6c6d Compare August 6, 2026 08:31

sfc-gh-boler requested changes Aug 6, 2026

View reviewed changes

sfc-gh-pfus force-pushed the pfus-08-06-snow-2499292_add_serializable_arrow_batches_for_distributed_fetch branch from 78d6c6d to da861b8 Compare August 7, 2026 06:24

arcticowl-ai-review Bot reviewed Aug 7, 2026

View reviewed changes

Comment thread arrowbatches/serializable.go Resolved

sfc-gh-pfus requested a review from sfc-gh-boler August 7, 2026 07:11

sfc-gh-pfus added the critical_blocker_force_override label Aug 7, 2026

sfc-gh-pfus added 2 commits August 18, 2026 09:31

SNOW-2499292 Add serializable Arrow batches for distributed fetch

83a4e55

review fixes

a9ad80d

sfc-gh-pfus force-pushed the pfus-08-06-snow-2499292_add_serializable_arrow_batches_for_distributed_fetch branch from da861b8 to a9ad80d Compare August 25, 2026 09:56

arcticowl-ai-review Bot removed the critical_blocker_force_override label Aug 25, 2026

arcticowl-ai-review Bot reviewed Aug 25, 2026

View reviewed changes

sfc-gh-pfus marked this pull request as draft September 18, 2026 07:31

Conversation

sfc-gh-pfus commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TL;DR

What changed?

How to test?

Why make this change?

Uh oh!

Uh oh!

Uh oh!

sfc-gh-boler left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sfc-gh-pfus commented Aug 7, 2026

Uh oh!

Uh oh!

arcticowl-ai-review Bot Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sfc-gh-pfus commented Aug 6, 2026 •

edited

Loading