Harmonize rank model submissions to Scout

[fellen31]

As a bioinformatics pipeline developer,
I want rank models to be accessed by Scout in the same way for all non-legacy and future pipelines that uses rank models,
So that development and updates become easier for pipeline devs.

Work impact

Answer the following questions:

• Is there currently a workaround for this issue? If so, what is it?
  • Bioinformaticians diving into cg and scout codebase
• How much time would be saved by implementing this feature on a weekly basis?
  • .5 headaches
• How many users are affected by this issue?
  • Pipeline developers for pipelines using rank models
• Are customers affected by this issue?
  • Yes, Nallo cases are currently shown in Scout with a different rank model leading to missing minimum and maximum values and rule explanations. This is because there is was no clear documentation (at least in my opinion) on how rank model submissions were handled before Fix #5186 - add rank_model_url to case load model scout#6090 in neither Scout, Atlas or Servers.

Acceptance Criteria

• Paths to rank models for SNVs and SVs is set by the cg config.
• SNV and SV rank models are copied to the case analysis directory on config-case so we know which models were used for a specific case.
• SNV and SV rank models are stored in the HK bundle on store.
• SNV and SV rank models are uploaded to Scout from the files stored in the HK bundle
• Rank model versions are parsed from versions within the HK bundle file, not the filename. (Could later consider if this should be the responsibility of cg or Scout)
• There exists documentation on how the upload of Rank models to Scout works.

Notes

• Additional information:

  • The current implementation for Raredisease:
    • Differs from the implementation for Nallo, creating ambiguity which may lead to errors if a developer assumes they are handled the same.
    • Parses the rank models from the params file which contains paths to the reference directory. A limitation of this is that if you update the rank model file in the reference directory while there are analyses running, the case will be uploaded with a different rank model than what it was run with. The customers may then benchmark the results of the rank model with the wrong assumptions.
  • Hardcodes the keys used for parsing the SNV and SV rank models from the params file within cg, creating more dependencies for the pipeline.
  • There is no documentation for the limitations above that I can find. Who is responsible for maintaining that? Is the bioinformatics developer tasked with updating the rank model expected to know in detail how the cg code works? Happy to discuss how this should be handled.

• Related user stories:

  • Original issue by @dnil: [raredisease hg38] add rank_model_urls to Scout load config #4940
  • Current raredisease implementation: Add rank model files to scout #5047
  • A related issue also exists for the variant catalog file: Move cg/cg/resources/*_bundle_filenames.yaml to another repo #4777 (comment) that is currently hardcoded to the reference directory in e.g. https://github.com/Clinical-Genomics/servers/blob/4a7de21589aa2cfc783e9b2b7d0efab1c59833ab/config/hasta.scilifelab.se/pipeline_deliverables/raredisease/raredisease_deliverables_stage.yaml#L168.

[RasmusBurge-CG]

Hi @fellen31,

We assume this only affects RD and Nallo at the moment. Is this assumption correct?

What is the aim of the document? Should it explain the system or be more about what you need to do when there is a new rank model version?

Could you provide us with the paths to the rank models? (just making sure we take the right ones)

[fellen31]

Hi @fellen31,

We assume this only affects RD and Nallo at the moment. Is this assumption correct?

I think only Nallo and RD uses rank models today, oncoflow will use rank models when it goes into production.

What is the aim of the document? Should it explain the system or be more about what you need to do when there is a new rank model version?

I think the aim of the documentation should be what a pipeline developer needs to know when updating rank models, and make any limitations of the current approach clear, while including enough explanation about the system for the developer to understand why they need to do it a certain way.

The de facto (unwritten) documentation today would be something like:

Updating rank models

"When you as a pipeline developer needs to update the rank model, you need to have detailed knowledge of parts of the cg and scout codebase that handles rank model upload and fetching, together with how the customers use said rank models in their own validations."

---

But I don't think pipeline developers should have to dig into to scout and cg codebases, and be expected to know how each customer uses the rank models. So with todays approach it could be something like:

Updating rank models

"Rank models submissions to Scout are handled by different approaches for Nallo and Raredisease.

Nallo

For the Nallo pipeline, the rank model version is set by NALLO_RANK_MODEL_VERSION_SNV and NALLO_RANK_MODEL_VERSION_SV constants in cg/constants/scout.py, that gets added to the Scout load config. Scout then concatenates a RANK_MODEL_LINK_PREFIX and SV_RANK_MODEL_LINK_PREFIX set in config/CGVS/config/app/cg-services-prod.scilifelab.se/scout38.conf.py, with the version number from cg in the Scout load config to fetch the rank models from the reference-files repository. For example "https://raw.githubusercontent.com/Clinical-Genomics/reference-files/master/nallo/rank_model/grch38_rank_model_snvs_-v" + "0.1" + "-.ini".

Limitations with the current approach includes:

• The Nallo SNV rank model has to be stored in https://raw.githubusercontent.com/Clinical-Genomics/reference-files/master/nallo/rank_model/ with the expected pattern: grch38_rank_model_snvs_-v" + VERSION + "-.ini".
• The Nallo SV rank model has to be stored in https://raw.githubusercontent.com/Clinical-Genomics/reference-files/master/nallo/rank_model/ with the expected pattern: grch38_rank_model_svs_-v" + VERSION + "-.ini".
• You cannot change the content of the rank model file without changing the version in the filename, otherwise files will get out of sync (with for example missing rule explanations in Scout) and customers relying on the correct rank model will do validations based on wrong assumptions.
• There is a versions line within the file, that has no effect.
• If you update NALLO_RANK_MODEL_VERSION_SNV and/or NALLO_RANK_MODEL_VERSION_SV while there are analyses running, these analysis will not be uploaded and displayed with the same rank model version as was used in the analysis.

When you want update the rank model you need to:

• Create a new rank model with a new version in the reference files directory.
• Release a new version of cg with updated NALLO_RANK_MODEL_VERSION_SNV and/or NALLO_RANK_MODEL_VERSION_SV.
• Check if there are any analyses that are running/have not been uploaded. If there are, you will need to wait for all to finish uploaded, or stop and restart from scratch after the rank model has been updated.

Raredisease

For the raredisease pipeline, the rank model version is supplied to the Scout config by parsing the version from the file path of score_config_snv and score_config_sv with a regex ((r"v(\d+\.\d+)") in the nextflow params file that is created on by cg workflow raredisease config-case and later stored in housekeeper with cg workflow raredisease store. The rank model itself is uploaded via the Scout config also by parsing the score_config_snv and score_config_sv keys in the params file. score_config_snv and score_config_sv in the params file points to files in the raredisease reference directory on Hasta.

Limitations with the current approach include:

• Rank models filenames needs to include a version matching (r"v(\d+\.\d+)".
• There is a versions line within the file, that has no effect.
• You cannot change the content of the rank model file without changing the version in the filename, otherwise files may get out of sync (with for example missing rule explanations in Scout) if you update the reference files while there are analyses running/uploading. Customers relying on the correct rank model may then do validations based on wrong assumptions.
• You cannot change score_config_snv and score_config_sv parameters in the pipeline without a cg update.

When you want update the rank model you need to:

• Create a rank model file in the reference directory, with a different version in the file name that matches (r"v(\d+\.\d+)".
• Check if there are any analyses that are running/have not been uploaded. If there are, you will need to wait for all to finish uploaded, or stop and restart from scratch after the rank model has been updated."

---

The aim of this issue is to harmonize the approaches and limit the limitations and considerations, so the documentation can be kept simple and minimal.

Could you provide us with the paths to the rank models? (just making sure we take the right ones)

Currently in use for RD: https://github.com/Clinical-Genomics/servers/blob/803596e0e3d0abebb993b88e36d771840f717357/config/hasta.scilifelab.se/nextflow/raredisease/gms_rd_v3.0_stage_params_hg38.yaml#L37-L38
Currently in use for Nallo: https://github.com/Clinical-Genomics/servers/blob/803596e0e3d0abebb993b88e36d771840f717357/config/hasta.scilifelab.se/nextflow/nallo/nallo_v1.0_production_params.yaml#L21-L22