Cannot parse real-time microstructure data

[bastienqueste]

At line ~131, "pl.read_csv(f, separator=';')" in seaexplorer.py, read_csv cannot parse extreme microstructure values as i64:

ComputeError: Could not parse 2.22e-101as dtypei64` at column 'MR1000G-RDL_EPS2' (column number 22).
The current offset in the file is 490928 bytes.

You might want to try:

• increasing infer_schema_length (e.g. infer_schema_length=10000),
• specifying correct dtype with the dtypes argument
• setting ignore_errors to True,
• adding 2.22e-101 to the null_values list.`

[bastienqueste]

Probably should state that this is not surprising. 2.22e-101 is a stupid value for Rockland and Alseamar to be returning.

[jklymak]

Hmm I'm surprised it won't parse that exponent. If you can link an example or suggest a PR I'm sure this is easy to fix

[callumrollo]

It probably shouldn't be parsing microstructure data as 64 bit integer. I assume it would read fine into a float? Maybe we should start catching and specifying dtypes

[bastienqueste]

I've just made it look a little further before inferring a data type - small slowdown but doesn't fail: out = pl.read_csv(f, separator=';', infer_schema_length=10000)

…

______________________________________ Although I may work and communicate at strange hours, I do not expect replies outside of your own working hours. CLICK HERE TO BOOK A MEETING OR CHECK ***@***.***?anonymous&ep=plink> Dr Bastien Y. Queste Associate professor / Biträdande lektor Department of Marine Sciences, University of Gothenburg (UGOT) Natrium, Medicinaregatan 7B, SE-405 30 Gothenburg, Sweden Tel: +46-73-504-28-39 Home page: http://www.byqueste.com ORCID: https://orcid.org/0000-0002-3786-2275

________________________________ From: Callum Rollo ***@***.***> Sent: 22 January 2024 14:13 To: c-proof/pyglider ***@***.***> Cc: Bastien Queste ***@***.***>; Author ***@***.***> Subject: Re: [c-proof/pyglider] Cannot parse real-time microstructure data (Issue #160) It probably shouldn't be parsing microstructure data as 64 bit integer. I assume it would read fine into a float? Maybe we should start catching and specifying dtypes — Reply to this email directly, view it on GitHub<#160 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AABHOCUWKU42PDBEJE5IH5LYPZQXDAVCNFSM6AAAAABCCFA64WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBTHE4DCMRWGA>. You are receiving this because you authored the thread.Message ID: ***@***.***>