Repository navigation
Explicit chunk size to fit memory usage under 64 GB - #31
Merged
Merged
Conversation
i-am-sijia
marked this pull request as ready for review
October 5, 2026 17:20
Collaborator
Author
|
Got the machine memory peak down from 160 GB to 108 GB when running the summarize step. Deducting the 23 GB idle RAM, the summarize step now requires 85 GB RAM. Require ActivitySim source code change: https://github.com/wsp-sag/activitysim/tree/reduce-summarize-mem Before: After: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR does the following:
explicit_chunksize from Bound full-population memory use and repair benchmark qualification #29, which are saved in themodel\configs_explicit_chunkfoldernum_processesfor multiprocessing from4to16, based on CTPS machine spec (24 processors in total)summarizestep under multiprocessing, previously it was commented out@mmilkovits @jpn-- @mnbina The time series chart below shows the memory usage of a full-size model run without the
summarizestep, using MA/RI/NH population (3.6M households), running with 16 processors, sharrow turned on and pre-compiled. The machine uses 23 GB RAM when idle. During the run, the machine memory peaked at 75 GB, meaning the max memory used by the lighthouse was about 52 GB. Total run time was 250 minutes. This run was conducted on a WSP machine.Update:
summarizestep currently takes over 140 GB RAM. There might be some low hanging fruits in the activitysim source code that can reduce the memory needed forsummarize, I have some ideas and am exploring that.