Feat/enable native decimal support#282
Open
nikhilmehta16 wants to merge 4 commits intodatabricks:mainfrom
Open
Feat/enable native decimal support#282nikhilmehta16 wants to merge 4 commits intodatabricks:mainfrom
nikhilmehta16 wants to merge 4 commits intodatabricks:mainfrom
Conversation
Remove blocking condition for decimal types in ScanRow method. The decimal128Container infrastructure is complete and functional, so native decimal scanning now works properly. Only interval types remain unsupported and continue to be blocked. Signed-off-by: Nikhil <mehta.nikhil@prophecy.io>
Update 'Error on retrieving not implemented native arrow types' test to expect success for decimal columns (index < 4 instead of < 3). Decimal types now work, only interval types should still error. Signed-off-by: Nikhil <mehta.nikhil@prophecy.io>
Signed-off-by: Nikhil <mehta.nikhil@prophecy.io>
Signed-off-by: Nikhil <mehta.nikhil@prophecy.io>
shivam2680
reviewed
Aug 6, 2025
| err := ars.ScanRow(dest, 0) | ||
|
|
||
| if i < 3 { | ||
| if i < 4 { |
Contributor
There was a problem hiding this comment.
Can you add a separate test to verify changes in ScanRow? That tests ScanRow no longer blocks decimal types
shivam2680
reviewed
Aug 6, 2025
Contributor
shivam2680
left a comment
There was a problem hiding this comment.
Thanks for making these changes! LGTM for code changes
can you update doc.go to indicate data type change.
DECIMAL(p,s) --> float64
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enables native Arrow decimal type support by removing unnecessary blocking condition in
ScanRowand enabling native decimal support by default. Thedecimal128Containerinfrastructure was already complete - these changes allow decimal columns to return native Arrowdecimal128types instead of raw bytes.Changes
UseArrowNativeDecimal = trueinArrowConfig.WithDefaults()ScanRowerror conditionWhy Native Arrow Decimal128?
This provides significant advantages over the previous raw bytes approach:
decimal128types with precision/scale metadataarrow.DECIMAL128instead ofarrow.STRINGImpact
Before:
UseArrowNativeDecimal = falseby default[]byte) in Arrow records requiring manual parsingarrow.STRINGtype for decimal columnsAfter:
UseArrowNativeDecimal = trueby defaultarrow.Decimal128types in Arrow recordsarrow.DECIMAL128with precision/scale metadatafloat64available viadecimal128Value.ToFloat64(scale)For Direct Arrow Usage
Users working with Arrow records directly now get:
Configuration Note
The
UseArrowNativeDecimaloption is currently internal and not exposed through the public API. Users cannot currently override this setting, but the new default behavior provides proper Arrow type semantics.Testing
✅ Updated test expectations to reflect new default behavior
✅ All tests pass locally
✅ Arrow records now contain native decimal128 types instead of strings
Breaking Change
This is a minor breaking change for users directly accessing Arrow records, as decimal columns now return
arrow.Decimal128Arrayinstead ofarrow.StringArray. However, this provides the correct Arrow type semantics and eliminates the need for manual parsing.