Repository navigation
Support for pyspark.sql.Column in dataclasses-json #295
Description
Hello,
In my dataclasses i am using Column field of pyspark.sql as well. Is it possible to extend dataclasses-json to define json encoding and decoding for Column field types as well?
My encoding and decoding functions will look like below
def encoder(self:Column) -> str:
return f'{self._jc}'
def decoder(self, colString) -> Column:
return expr(colString)I am extending Column class in python to add encoder and decoder method to enable to_json and from_json call on Column fields. I also tried specifying these encoder and decoder methods in my dataclass via below
@dataclass_json
@dataclass(frozen=True)
class ColumnExpression:
target: str
expression: Column = field(
metadata=config(
encoder=Column.encoder,
decoder=Column.decoder
)
)I am getting below error when i try to call to_json() on the instance of my own ColumnExpression dataclass
Traceback (most recent call last):
File "/Users/bhupeshgoel/Documents/codebase/ComponentBuilderPython/cb/server/ComponentBuilderBase.py", line 167, in <module>
print(cc.to_json())
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dataclasses_json/api.py", line 50, in to_json
return json.dumps(self.to_dict(encode_json=False),
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dataclasses_json/api.py", line 86, in to_dict
return _asdict(self, encode_json=encode_json)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dataclasses_json/core.py", line 323, in _asdict
value = _asdict(getattr(obj, field.name), encode_json=encode_json)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/dataclasses_json/core.py", line 338, in _asdict
return copy.deepcopy(obj)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
state = deepcopy(state, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
state = deepcopy(state, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
state = deepcopy(state, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
state = deepcopy(state, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 270, in _reconstruct
state = deepcopy(state, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 146, in deepcopy
y = copier(x, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 230, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/copy.py", line 161, in deepcopy
rv = reductor(4)
TypeError: cannot pickle '_thread.RLock' object
May i know if i am doing anything wrong here?
Activity
george-zubrienko commentedon Jun 14, 2023
Hi @bhupesh-simpledatalabs, this question is not really related to library functionality, so I'm going to move this to discussions instead. As to "am I doing something wrong", I'd say yes, you can't have a type that is linked to a distributed collection in a file format that is designed for single-node storage.