Custom DevSkim rules (on yaml files) not showing in VSCode custom tab

[JaneX8]

Describe the bug
I don't manage to get DevSkim custom rules to show up in the Problems Tab in VSCode. I wrote a json file containing:

To Reproduce

[{
    "id": "TestRule",
    "name": "Basic Test Rule",
    "description": "A simple test rule.",
    "applies_to": ["yaml"],
    "tags": ["test"],
    "patterns": [{
        "pattern": "redirects",
        "type": "regex",
        "message": "Detected redirects."
    }]
}]

When I run:

devskim analyze -I "C:\pathToTestFolder" -r "C:\pathToCustomDevSkimRules"

The output shows a match on my custom rule. However when I simply open the yaml file in VSCode nothing is shown in the Problems tab.

I also added this:

    "MS-CST-E.vscode-devskim.rules.customRulesPaths": [
        "C:\\pathToCustomDevSkimRules"
    ],
    "MS-CST-E.vscode-devskim.logging.level": "debug",

Inside C:\Users\username\AppData\Roaming\VSCodium\User\settings.json. Other DevSkim rules do show up in the Problems tab.

Expected behavior
My custom rules to show up in the Problems tab in VSCode.

Versions(please complete the following information):

• OS: Windows 11
• Devskim Version: devskim 1.0.44+10b85ce690

Other

I saw #300 and my file extensions are actually yaml.

[JaneX8]

Ok, I just wasted an hour debugging what was wrong. And despite restarting VSCodium and disabling and reenabling the DevSkim extensions it wasn't enough. What did the trick was

What did fix it in VSCodium/VSCode: I did Ctrl+Shift+P (or Cmd+Shift+P on macOS), then type Reload Window and hit Enter.

[JaneX8]

The question remains: How do I properly and quickly debug my rules? Who lints the linter? Is there a JSONSchema definition for DevSkim rules? Can I have autocomplete/syntax checking on my DevSkim rules in VSCode? I now have to constantly reload the window after every change or run the command line option and find a tracelog of a failed json parse:

Unhandled exception. System.Text.Json.JsonException: The JSON value could not be converted to Microsoft.DevSkim.FixType. Path: ...

[JaneX8]

Ok, only reloading the whole VSCode window after every change makes it work. This is a bug, it should work automatically.

[gfs]

The DevSkim CLI has a verify command that does some checks for consistency/correctness, but there isn't an entrypoint to that in the VsCode extension at this point.

The VsCode extension uses a Language Server model, so the VsCode front end launches and then talks to a C# language server that is running the C# DevSkim engine. I think what you may be encountering with it working after a reload is that when the language server relaunches it will load the specified rules from disc with your options - but while VsCode is already running any changes to those files will be reflected on disc but are not hot reloaded into the language server.

[JaneX8]

I think that VSCode is the main reason I'm going crazy with DevSkim now because I'm trying to write custom rules and also using verify and analyze CLI and they produce different outputs (as expected) while not showing in VSCode problems tab, even after reloading the window. That's a pity because VSCode is the main reason I want to use DevSkim. I think it makes a very powerful combination if custom rules are easily writable and testable.

Also, it would be nice if we can figure out a way to reload that Language Server once changes are detected in the Custom Rules folders.

Here is an unexplained example that works in CLI but not showing in VSCode.

test.yaml

dfds:
    xyz:
      - kqe: afa
        asf:
          - 1801
    abc:
      - asd: something
        test:
          - "anything1"
          - "XXanything2"
          - "XXanything3"

example_yaml_rule.json

[{
    "name": "NestedRegexExtractorRule",
    "id": "AMLPath.NestedRegexExtractor",
    "description": "Extract regex patterns dynamically from the nested requests section of the YAML file.",
    "severity": "critical",
    "tags": ["test"],
    "patterns": [{
        "pattern": ".*",
        "ymlpaths": [
            "/dfds/abc/*/test/*"
        ],
        "type": "regex",
        "scopes": [
            "code"
        ],
        "modifiers": [
            "i"
        ],
        "confidence": "high"
    }]
}]

devskim analyze -I ".\test.yaml" -r ".\example_yaml_rule.json"

{"$schema":"https://schemastore.azurewebsites.net/schemas/json/sarif-2.1.0-rtm.6.json","version":"2.1.0","runs":[{"tool":{"driver":{"name":"devskim","fullName":"Microsoft DevSkim Command Line Interface","version":"1.0.44+10b85ce690","informationUri":"https://github.com/microsoft/DevSkim/","rules":[{"id":"AMLPath.NestedRegexExtractor","name":"Nestedregexextractorrule","fullDescription":{"text":"NestedRegexExtractorRule: Extract regex patterns dynamically from the nested requests section of the YAML file."},"help":{"text":"Extract regex patterns dynamically from the nested requests section of the YAML file.","markdown":"Visit [https://github.com/Microsoft/DevSkim/blob/main/guidance/](https://github.com/Microsoft/DevSkim/blob/main/guidance/) for guidance on this issue."},"shortDescription":{"text":"Extract regex patterns dynamically from the nested requests section of the YAML file."},"defaultConfiguration":{"level":"error"},"helpUri":"https://github.com/Microsoft/DevSkim/blob/main/guidance/","properties":{"precision":"","problem.severity":"error","DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}}]}},"versionControlProvenance":[{"repositoryUri":"https://github.com/Rebalance3603/scanning-tool.git","revisionId":"2deea1a5840e3e0303253ec1680e9dd980948a3d","branch":"main"}],"results":[{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"charOffset":193,"snippet":{"text":"","rendered":{"markdown":"``"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":11,"startColumn":12,"charOffset":180,"charLength":13,"snippet":{"text":"\"XXanything3\"","rendered":{"text":"\"XXanything3\"","markdown":"`\"XXanything3\"`"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":10,"startColumn":25,"endLine":10,"endColumn":25,"charOffset":166,"snippet":{"text":"","rendered":{"markdown":"``"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":10,"startColumn":12,"endLine":10,"endColumn":25,"charOffset":153,"charLength":13,"snippet":{"text":"\"XXanything2\"","rendered":{"text":"\"XXanything2\"","markdown":"`\"XXanything2\"`"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":9,"startColumn":23,"endLine":9,"endColumn":23,"charOffset":139,"snippet":{"text":"","rendered":{"markdown":"``"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}},{"ruleId":"AMLPath.NestedRegexExtractor","level":"error","message":{"text":"NestedRegexExtractorRule"},"locations":[{"physicalLocation":{"artifactLocation":{"uri":"test.yaml"},"region":{"startLine":9,"startColumn":12,"endLine":9,"endColumn":23,"charOffset":128,"charLength":11,"snippet":{"text":"\"anything1\"","rendered":{"text":"\"anything1\"","markdown":"`\"anything1\"`"}},"sourceLanguage":"yaml"}}}],"properties":{"tags":["test"],"DevSkimSeverity":"Critical","DevSkimConfidence":"Unspecified"}}],"columnKind":"utf16CodeUnits"}]}

Nothing is shown in the Problems tab. Other DevSkim rules work and do show there.

I didn't see anything worth mentioning under the output tab of DevSkim VS Code Client.

[gfs]

Sorry! I understand the frustration. I can definitely note this as a feature request to add hot-reload when custom rules are changed - as a workaround you can cause the language server to reload the rules from disk by changing a setting, you however, then also need to trigger a new analysis on the file - the trigger is scan is event driven from VS Code itself - so that should be if the file is opened or changed, so even if we could hot reload rules, we'd need something to trigger a rescan.

[gfs]

FWIW, the example rule I shared in the other thread does seem to be working in VS Code. Resharing here:

[{
    "name": "YamlPathValidate",
    "id": "YmlPath",
    "tags": ["test"],
    "description": "values under a key named test that start with XX",
    "severity": "critical",
    "patterns": [
      {
        "pattern":"XX\\w*",
        "ymlpaths" : ["/**/test/*"],
        "type": "regex",
        "scopes": [
          "code"
        ],
        "modifiers": [
          "i"
        ],
        "confidence": "high"
      }
    ]
  }]

[JaneX8]

I can confirm that one works indeed.

[JaneX8]

I'm having the suspicion that regex patterns are not behaving the same on CLI and VSCode, or maybe it has to do with the way it is escaped and passed through the language server. The combination of ymlpath with negative lookaheads in regexes for example are working in CLI but not showing in VSC. As do some other patterns like the simple .* that I showed above.

[gfs]

I tested with the rule you were testing above and it seems to be working for me in VS Code as well.

[JaneX8]

The VsCode extension uses a Language Server model, so the VsCode front end launches and then talks to a C# language server that is running the C# DevSkim engine. I think what you may be encountering with it working after a reload is that when the language server relaunches it will load the specified rules from disc with your options - but while VsCode is already running any changes to those files will be reflected on disc but are not hot reloaded into the language server.

Probably this. I just experienced that despite reloading the window, a previous rule kept working and the new one did not yet. I need to find a way to either manually reload that language server, and then preferably get the extension to do hot reloading at some point.

Possibly related:

• Feature Request: Command to reset / reload the language server vscode-cpptools#1152
• Restart language server – a generic solution vscode#76405

[gfs]

Thanks for the references. I believe we are using the language client library mentioned in 76405, I'll try to take a look and see how this could fit in. The simplest solution might be to add an action to the command palate in vscode to trigger a reload.