How to deal with very large datasets?

[davidgasquez]

Talking about 1TB datasets. They'll need to be partitioned.

[TomNicholas]

Go down this rabbit hole and you end up with chunked cloud-native big data repositories. The latest and greatest technology for this is Icechunk and Iceberg.

[davidgasquez]

I've been there! 😅 Didn't document much but ended up building some cool stuff on top of Delta/Iceberg, IPFS, and DuckDB.

This discussion is something I haven't thought a lot since there are already great solutions as you mention.

I need to do another deep dive into catalogs since lots of things have happened in the last few months!