Adding New Carla Maps

[Aditya-Gupta26]

This PR adds new Carla towns (11,12,13) to our datasets. There is a slight window size modification in visualize.c to support rendering of large sized maps.

[greptile-apps]

Too many files changed for review. (1537 files found, 100 file limit)

[daphne-cornelisse]

Thank you! I don't think it is a good idea to submit more than 1000 binaries to github. Maybe we should put these on huggingface or just add the json files with the script to process them? @eugenevinitsky, what do you think?

Context: these towns are too big, so Aditya cut them up in smaller parts

[eugenevinitsky]

Hmm, yeah it depends on how big the files are in total. We don't want to store many GB on github. But also, are they so big that 1000 is the right size? Also...what happens when you just load the whole map?

[eugenevinitsky]

There are a lot of open questions here for me.

[daphne-cornelisse]

@Aditya-Gupta26 could you please comment on this? Maybe add a description to the PR with your reasoning process? thank you :)

[Aditya-Gupta26]

Towns 11, 12 and 13 are pretty large in dimensions. For context -

• Town 13 is a 10 * 10 km^2 map, while Town 4 (our current largest map) is about 1 * 1 km^2
• Town 11 is even bigger - roughly about 20 * 20 km^2

These led to the creation of several small jsons and eventually several small map binaries

The JSON files for each of these maps (Town11,12,13) are pretty large (even without any agents). Here are the sizes (MB)-

Maybe uploading them to HuggingFace makes more sense given their size ?

Regarding the binaries -

Each of the 3 maps has 8 scenarios. Further -
Town 11 (Biggest) = split 3 times over which led to 64 small maps for each scenario. Total = 648 =512 maps
Town12 = split 2 times over which led to 16 small maps for each scenario. Total = 168 =128 maps
Town13 = split 2 times over which led to 16 small maps for each scenario. Total = 16*8 =128 maps

Hence, the total maps are 512+128+128 = 768

The reason we see 1536 (768*2) bins is because I created separate folder for each of the Towns, and one folder containing all towns.

The .bin file sizes are still smaller, which is why I thought of uploading them here.

About loading the whole map, it just takes a lot of time - and while training it causes big breaks in training while running the init_grid_map() function (which happens at the start and reset).

[eugenevinitsky]

Question, why is the grid map reset happen / expensive?

[Aditya-Gupta26]

The init_grid_map() runs at the start, which is pretty expensive. For every reset - we still have to loop through the entities - which takes time for these maps

[eugenevinitsky]

Yeah, my question is, what part of the entity cost has to be redone? Presumably just the cost of the dynamic entites which is O(100)

[Aditya-Gupta26]

Right, if we can loop through only the dynamic entities - it'll be much better. as of now we loop through all entities (num_entities), which, for the entire map is not useful.
However, the small map slices work just fine with the current setup btw