Skip to content

examples/protobuf: add explicit lexer with comment support - #486

Closed
AdamMagued wants to merge 1 commit into
alecthomas:masterfrom
AdamMagued:fix/446-protobuf-comments-lexer
Closed

AdamMagued wants to merge 1 commit into
alecthomas:masterfrom
AdamMagued:fix/446-protobuf-comments-lexer

Conversation

@AdamMagued

Copy link
Copy Markdown

Fixes #446.

Problem and Root Cause

The protobuf example in _examples/protobuf did not define a custom lexer and instead relied on Participle default lexer (TextScannerLexer). Go underlying text/scanner.Scanner defaults to GoTokens, which sets scanner.SkipComments. This discards all // and /* ... */ comments at the scanner level before tokens reach Participle. Consequently, attempts to preserve, capture, or match comments fail because no comment tokens exist in the token stream.

Solution

  1. Define protoLexer using lexer.MustSimple to tokenize protobuf source including line comments (//[^\r\n]), block comments (/ ... */), whitespace, identifiers, strings, floats, ints, and punctuation.
  2. Configure the protobuf parser with participle.Lexer(protoLexer) and participle.Elide("Comment", "Whitespace"). This enables standard Participle comment handling: comments throughout the file are cleanly elided by default, while allowing explicit capture on AST nodes (such as Entry).
  3. Add Comment string @Comment to Entry so entry-level comments can be preserved in the AST when desired.
  4. Add regression test TestComments in _examples/protobuf/main_test.go verifying that line comments, block comments, and inline comments parse properly without syntax errors and entry comments are captured into the AST.

Verification

  • Run go test -v ./... in _examples/protobuf (both TestExe and TestComments pass).
  • Run go test ./... and cd _examples && go test ./... (all 20 example packages pass).
  • Verified gofmt -l is clean on modified files.

@alecthomas

Copy link
Copy Markdown
Owner

Thanks for the PR, but that issue should have been closed - the current example is working as intended.

@alecthomas alecthomas closed this Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Failing to parse "//" while trying to parse comments

2 participants