Skip to content

ci: Add initial nightly/weekly CI test scripts for wan model#100

Open
chtruong814 wants to merge 27 commits intomainfrom
chtruong/nightly-ci
Open

ci: Add initial nightly/weekly CI test scripts for wan model#100
chtruong814 wants to merge 27 commits intomainfrom
chtruong/nightly-ci

Conversation

@chtruong814
Copy link
Collaborator

@chtruong814 chtruong814 commented Feb 4, 2026

Add initial nightly/weekly CI test scripts for wan model

  • Move existing automodel scripts to tests/scheduled directory
  • Add a run_sbatch_test.sh script to launch the scheduled tests on a slurm cluster
  • Update the test scripts to just contain the torch run launch command and other parameters needed for the job
  • Update the Dockerfile to copy over the source code and install DFM in the final step. Since it's the final step, the impact to the build time with cache should be minimal. Can consider multi-stage if the final stage is more involved

Huy Vu2 and others added 27 commits January 14, 2026 14:06
fix small errors
c0a7312
cicd tests config
5d1b27d
update script
038304c
@chtruong814
Update script for pretrain megatron wan testing
5691b0c 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
update nightly/weekly scripts
6c68a7e
@chtruong814
Fix test
3da7a88 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix test env var eval
ff60785 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix test script
f4df286 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Test
73bf04e 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix dockerfile
c2d34ab 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Add release stage
7baf76a 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix torchrun in docker
d5d7f43 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Create run_example.sh
33e66e2 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Ensure RDZV_PORT is set
5932ad0 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Update weekly tests
2ea9e6a
@chtruong814
Fix weekly tests
9ea3827 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix tests
d4a1172 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Add comment to script
e6c5837 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Merge remote-tracking branch 'origin/main' into chtruong/nightly-ci
d261dfe 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Refactor megatron test scripts to allow pretrain dir and iters
7f3c2d9 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Refactor automodel nightly tests
af8c73b 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Ensure megatron wan video test loads checkpoint from image model
4b513b8 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix automodel overrides
63c607b 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Fix checkpoint load for automodel
9d01e1b 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Refactor test location
1db21eb 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Update Dockerfile to include DFM source
f7bf200 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814
Move nightly weekly scripts to tests/scheduled
0ed6982 
Signed-off-by: Charlie Truong <chtruong@nvidia.com>
@chtruong814 chtruong814 requested a review from a team as a code owner February 4, 2026 14:55
@copy-pr-bot
Copy link

copy-pr-bot bot commented Feb 4, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@chtruong814
Copy link
Collaborator Author

chtruong814 commented Feb 4, 2026

/ok to test 0ed6982

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@huvunvidia huvunvidia Awaiting requested review from huvunvidia

@linnanwang linnanwang Awaiting requested review from linnanwang

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@chtruong814