Allowing escape characters in comments and as identifiers

[hs-apotell]

Hello -

Looking for some assistance with getting the rules correct for my use case. My language supports naming identifiers with all printable characters as long as it starts with a backslash and ends with a space (ESCAPED_IDENTIFIER in the lexer rules below). Comments are much like most high level languages like C++ and Java (starting with two forward slashes and ending with a new line).

The problem is when comments include escape characters like in the following input -

// print("The final results is: \"{result}\"")
var \MyVariable = 10

In the above example, comment line has embedded escaped quotes. This comments terminates prematurely right before the first escape backslash. On the second line, however, the escaped identifier is parsed correctly as long as the first line succeeds.

How do I update these rules to cover both these use cases? When in comment, continue parsing till the end of the line (accepting backslashes) but treat backslashes as start of escaped identifier when not parsing a comment?

Appreciate your time and valued input.

lexer grammar TestLexer;

CR: '\n' -> channel(HIDDEN);
ESCAPED_IDENTIFIER: '\\' ~[WS\r\t\n]*? WS;
LINE_COMMENT: '//' ~[\\\r\n]*;
VAR: 'var';
ASSIGN_OP: '=';
CONSTANT: [1-9] [0-9]+;
IDENTIFIER: [a-zA-Z_] [a-zA-Z0-9_$]*;
WS: [ ]+;
ANY: .;

parser grammar TestParser;
options { tokenVocab=TestLexer; }

program: (var_decl | LINE_COMMENT)* EOF;
var_decl: VAR WS+ ESCAPED_IDENTIFIER WS* ASSIGN_OP WS* CONSTANT;

[The-Futurist]

Hello -

Looking for some assistance with getting the rules correct for my use case. My language supports naming identifiers with all printable characters as long as it starts with a backslash and ends with a space (ESCAPED_IDENTIFIER in the lexer rules below). Comments are much like most high level languages like C++ and Java (starting with two forward slashes and ending with a new line).

The problem is when comments include escape characters like in the following input -

// print("The final results is: \"{result}\"")
var \MyVariable = 10

In the above example, comment line has embedded escaped quotes. This comments terminates prematurely right before the first escape backslash. On the second line, however, the escaped identifier is parsed correctly as long as the first line succeeds.

How do I update these rules to cover both these use cases? When in comment, continue parsing till the end of the line (accepting backslashes) but treat backslashes as start of escaped identifier when not parsing a comment?

Appreciate your time and valued input.

lexer grammar TestLexer;

CR: '\n' -> channel(HIDDEN);
ESCAPED_IDENTIFIER: '\\' ~[WS\r\t\n]*? WS;
LINE_COMMENT: '//' ~[\\\r\n]*;
VAR: 'var';
ASSIGN_OP: '=';
CONSTANT: [1-9] [0-9]+;
IDENTIFIER: [a-zA-Z_] [a-zA-Z0-9_$]*;
WS: [ ]+;
ANY: .;

parser grammar TestParser;
options { tokenVocab=TestLexer; }

program: (var_decl | LINE_COMMENT)* EOF;
var_decl: VAR WS+ ESCAPED_IDENTIFIER WS* ASSIGN_OP WS* CONSTANT;

@hs-apotell Did you solve this? let me know, because I might have an idea.

This rule LINE_COMMENT: '//' ~[\\\r\n]*; tells the lexer to terminate the comment when it sees a \ (or a CR or a LF). So the character you want to include as being part of the comment, is not being included.

All you need to do is: LINE_COMMENT: '//' ~[\r\n]*;

[hs-apotell]

Yes, my last PR on chipsalliance/surelog solves the issue.

[hs-apotell]

The trouble with your recommended solution is that comments within macro definitions fail to parse because the comment will be terminated with a '\', as a continuation token like such -

#define ADD(x, y) \
  // This is a comment \
  x + y

The solution was to have two definitions for line comment - one that includes the backslash and other one doesn't. Use one variant in macros and other for non-macro use. Order of rules also is also important in this solution. The escaped line comment takes precedence over escaped identifiers.

Here's the PR that implements the solution - Apotell/Surelog@24c1053