"Error: Cannot create a string longer than 0x1fffffe8 characters"

[thbar]

I think I stumbled upon a technical limitation while comparing large (but similar to a very large extend) CSV files.

❯ daff 80921.20240125.120915.280075/stop_times.txt 80921.20240125.180818.104400/stop_times.txt
/Users/thbar/.asdf/installs/nodejs/16.17.1/lib/node_modules/daff/bin/daff.js:11375
                throw(e);
                ^

Error: Cannot create a string longer than 0x1fffffe8 characters
    at Object.slice (node:buffer:599:37)
    at Buffer.toString (node:buffer:819:14)
    at Object.readFileSync (node:fs:506:41)
    at Object.tio.getContent (/Users/thbar/.asdf/installs/nodejs/16.17.1/lib/node_modules/daff/bin/daff.js:11195:15)
    at $hx_exports.coopy.Coopy.loadTable (/Users/thbar/.asdf/installs/nodejs/16.17.1/lib/node_modules/daff/bin/daff.js:2359:21)
    at $hx_exports.coopy.Coopy.run (/Users/thbar/.asdf/installs/nodejs/16.17.1/lib/node_modules/daff/bin/daff.js:2933:17)
    at $hx_exports.coopy.Coopy.coopyhx (/Users/thbar/.asdf/installs/nodejs/16.17.1/lib/node_modules/daff/bin/daff.js:3014:15)
    at run_daff_base (/Users/thbar/.asdf/installs/nodejs/16.17.1/lib/node_modules/daff/bin/daff.js:11326:18)
    at Object.daff.run_daff_main (/Users/thbar/.asdf/installs/nodejs/16.17.1/lib/node_modules/daff/bin/daff.js:11343:13)
    at Object.<anonymous> (/Users/thbar/.asdf/installs/nodejs/16.17.1/lib/node_modules/daff/bin/daff.js:11361:11) {
  code: 'ERR_STRING_TOO_LONG'
}

The files are closer from 750MB, and the limit seems to be around 0x1fffffe8 = 512MB.

I wonder if this is can be fixed easily or not !

[thbar]

Ha! Looks like I hit buffer.constants.MAX_STRING_LENGTH 🤣