[Bug]: ParquetInput

[paolodelano]

Apache Hop version?

2.3.0

Java version?

19.0.1

Operating system

Windows

What happened?

Used the ParquetFile Input Transform to read from S3. When the file size is > 2GB I get the following exception:

2023/02/07 09:26:55 - Parquet File Input.0 - ERROR: Unexpected error
2023/02/07 09:26:55 - Parquet File Input.0 - ERROR: org.apache.hop.core.exception.HopException:
2023/02/07 09:26:55 - Parquet File Input.0 - Error read file s3://bucket/file.parquet
2023/02/07 09:26:55 - Parquet File Input.0 - Negative initial size: -1383794380
2023/02/07 09:26:55 - Parquet File Input.0 -
2023/02/07 09:26:55 - Parquet File Input.0 - at org.apache.hop.parquet.transforms.input.ParquetInput.processRow(ParquetInput.java:101)
2023/02/07 09:26:55 - Parquet File Input.0 - at org.apache.hop.pipeline.transform.RunThread.run(RunThread.java:55)
2023/02/07 09:26:55 - Parquet File Input.0 - at java.base/java.lang.Thread.run(Thread.java:1589)
2023/02/07 09:26:55 - Parquet File Input.0 - Caused by: java.lang.IllegalArgumentException: Negative initial size: -1383794380
2023/02/07 09:26:55 - Parquet File Input.0 - at java.base/java.io.ByteArrayOutputStream.(ByteArrayOutputStream.java:78)
2023/02/07 09:26:55 - Parquet File Input.0 - at org.apache.hop.parquet.transforms.input.ParquetInput.processRow(ParquetInput.java:84)
2023/02/07 09:26:55 - Parquet File Input.0 - ... 2 more
2023/02/07 09:26:55 - Parquet File Input.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)

We suspect there is a bug in the ParquetInput.java code below where a Long is cast to an Int

  long size = fileObject.getContent().getSize();
  InputStream inputStream = HopVfs.getInputStream(fileObject);

  // Reads the whole file into memory...
  //
  ByteArrayOutputStream outputStream = new ByteArrayOutputStream((int) size);

Issue Priority

Priority: 2

Issue Component

Component: Transforms

[bmoAtolcd]

All yours, @helorymarceau! You can assign this issue to yourself.

[helorymarceau]

.take-issue