Skip to content

Comments

feat: add built-in help system with fuzzy search for INPUT parameters#6935

Open
zhubonan wants to merge 12 commits intodeepmodeling:developfrom
bonan-group:abacus-develop-built-in-help
Open

feat: add built-in help system with fuzzy search for INPUT parameters#6935
zhubonan wants to merge 12 commits intodeepmodeling:developfrom
bonan-group:abacus-develop-built-in-help

Conversation

@zhubonan
Copy link

@zhubonan zhubonan commented Jan 30, 2026

I have added a built-in help system for INPUT parameters.

Usage

# Show general help
abacus -h

# Get detailed help for a parameter
abacus -h ecutwfc

# Search for parameters matching keyword
abacus -s ecut
# Finds: ecutwfc, ecutrho, lcao_ecut, etc.

# Typo suggestions
abacus -h ecutwf
# Error: Unknown parameter 'ecutwf'
# Did you mean: ecutwfc

Implementation

  • source/source_io/input_help.{h,cpp} - Help system class with search and fuzzy matching
  • source/source_io/input_help_data.h - Auto-generated parameter data (4295 lines, 458 parameters)
  • source/source_io/parse_args.cpp - Command-line argument handling
  • tools/generate_help_data.py - Build-time code generator from markdown
  • Documentation integrated into docs/quick_start/input.md

@zhubonan
feat: add built-in help system with fuzzy search for INPUT parameters
de854d2 
- Add -h/--help flag for parameter documentation
- Add -s/--search flag for keyword search
- Implement fuzzy matching for typo suggestions (Levenshtein distance)
- Case-insensitive parameter lookup
- Auto-generate help data from markdown at build time
- 458 parameters documented with comprehensive metadata

Usage: abacus -h <param>, abacus -s <keyword>

refactor(help): improve help system with critical fixes and performance optimizations

docs(help): integrate help system documentation into quick start guide

Add concise help system section to docs/quick_start/input.md covering:
- Basic commands (-h, -s flags)
- Example output for parameter lookup
- Fuzzy matching for typo suggestions
- Case-insensitive lookup behavior

Remove standalone documentation files that are now redundant:
- docs/HELP_SYSTEM.md
- docs/FUZZY_SEARCH.md
@zhubonan
fix(test): fix help system unit tests
462862a 
- Add input_help.cpp to MODULE_IO_parse_args test sources to fix linker error
- Fix input_help_test.cpp include path and use value syntax instead of pointer
- Fix parse_args_test.cpp death tests to use EXPECT_EXIT regex matching
  instead of CaptureStderr which doesn't work properly with fork()
@mohanchen mohanchen requested a review from ZhouXY-PKU January 31, 2026 01:28
zhubonan and others added 2 commits January 31, 2026 17:22
@zhubonan
refactor(help): address PR review feedback and improve fuzzy matching
498d7b2 
This commit addresses all code review comments and fixes a critical UX issue
in the help system's fuzzy matching algorithm.

## Documentation Fixes
- Fix incorrect "case-sensitive" documentation (should be "case-insensitive")
- Update API documentation for find_similar_parameters() with new algorithm details
- Add inline comments explaining 3-tier matching strategy

## Architecture Improvements
- Add std::ostream& parameter to show_general_help() and show_parameter_help()
- Enable flexible stream redirection (stdout for success, stderr for errors)
- Maintain backward compatibility with default parameters
- Eliminate code duplication by reusing show_general_help() in error handler

## UX Fix: Fuzzy Matching Algorithm

### Problem
Pure Levenshtein distance suggested semantically irrelevant parameters:
- "abacus -h relax" incorrectly suggested "dmax" and "nelec"

### Solution
Implement 3-tier semantic matching strategy:
1. Prefix matches (e.g., "relax" → "relax_new") - Priority 0
2. Substring matches (e.g., "cut" → "ecutwfc") - Priority 1
3. Levenshtein distance for typos - Priority 10+
@zhubonan
Merge branch 'develop' into abacus-develop-built-in-help
afd8386
ZhouXY-PKU
ZhouXY-PKU approved these changes Feb 5, 2026
View reviewed changes
Copy link
Collaborator

@ZhouXY-PKU ZhouXY-PKU left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job! Thanks for your contribution!

@zhubonan
Copy link
Author

zhubonan commented Feb 5, 2026

I think there are still some room for improvement in terms of the formatting for the help information printed. Please don't merge this yet

mohanchen
mohanchen reviewed Feb 7, 2026
View reviewed changes
docs/quick_start/input.md
Copy link
Collaborator

@mohanchen mohanchen Feb 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will be refactored

@mohanchen mohanchen added Feature Discussed The features will be discussed first but will not be implemented soon Refactor Refactor ABACUS codes labels Feb 7, 2026
@zhubonan
feat(docs): auto-generate INPUT docs from C++ source with built-in he…
d198db9 
…lp system

Migrate INPUT parameter documentation from a manually-maintained markdown
file to auto-generation from C++ source. Documentation fields (category,
type, description, default_value, unit, availability) are now embedded
directly in Input_Item registrations in read_input_item_*.cpp files,
serving as the single source of truth for both the built-in --help system
and the Sphinx documentation.

Key changes:
- Add docs/generate_docs_from_source.py to parse C++ source and generate
  input-main.md (492 parameters across 34 categories)
- Hook generator into Sphinx build via builder-inited event in conf.py
- Add [NOTE] documentation to 36 parameters preserving notes from the
  previous manually-maintained docs (version compat, usage caveats)
- Fix generator regex to handle comments before Input_Item declarations
- Rename tmp_item to item in smearing_sigma_temp block for generator
  compatibility
- Remove old tools/generate_help_data.py and input_help_data.h pipeline
@zhubonan
Merge branch 'abacus-develop-built-in-help' of github.com:bonan-group…
9374230 
…/abacus-develop into abacus-develop-built-in-help

# Conflicts:
#	docs/advanced/input_files/input-main.md
@zhubonan
Copy link
Author

zhubonan commented Feb 8, 2026

I am refactoring the code in this PR with different approach. The input-main.md is now generated from the .cpp files which contains the parameters definitions and descriptions. This way the list of parameters and descript only needs to be maintained at the single phase.

@zhubonan
feat(docs): replace regex-based doc generation with YAML pipeline
34e3ef3 
The old generate_docs_from_source.py parsed C++ source files with regex
to extract Input_Item metadata, which was fragile and broke on code
refactoring. Replace it with a pipeline that uses the binary's own
parameter registry:

  abacus --generate-parameters-yaml > docs/parameters.yaml
  python docs/generate_input_main.py docs/parameters.yaml

- Add ParameterHelp::generate_yaml() with YAML serialization helpers
- Add --generate-parameters-yaml CLI flag to parse_args.cpp
- Add POST_BUILD command in CMakeLists.txt to auto-regenerate
  docs/parameters.yaml on every build
- Create docs/generate_input_main.py (YAML-to-markdown converter)
- Update docs/conf.py Sphinx hook to use YAML pipeline
- Track docs/parameters.yaml in the repository
- Delete docs/generate_docs_from_source.py
- Add 5 unit tests for YAML generation
@zhubonan
fix(docs): quote numeric YAML values, remove auto-generation, update …
e090140 
…contributing guide

- Quote numeric-looking default values in yaml_quote_if_needed() so
  PyYAML parses them as strings (fixes 92 parameters losing defaults)
- Also quote .inf, -.inf, .nan YAML special values
- Fix Python generate_input_main.py to use != '' instead of truthiness
  checks, preventing falsy-but-valid values (e.g. 0) from being dropped
- Replace sys.exit(1) with FileNotFoundError in generate() for safe use
  as a library from conf.py
- Remove POST_BUILD command from CMakeLists.txt (not portable across
  CMake generators, breaks cross-compilation)
- Update CONTRIBUTING.md "Documenting INPUT Parameters" section to
  describe the YAML-based workflow and remind developers to regenerate
  docs/parameters.yaml when adding or modifying parameters
- Regenerate docs/parameters.yaml with all values properly quoted
@mohanchen mohanchen added Useful Information Useful information for others to learn/study Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS labels Feb 10, 2026
zhubonan and others added 4 commits February 21, 2026 20:09
@zhubonan
fix(docs): reorder add_item() calls in read_input_item_*.cpp to match…
f2f0540 
… legacy doc order

The built-in help system generates input-main.md from C++ source, with
parameter order determined by add_item() call order. After introducing
the YAML-based doc pipeline, 19 of 32 documentation sections had
different parameter ordering compared to the old hand-maintained
input-main.md.

Reorder item blocks within each of the 13 read_input_item_*.cpp files
to restore the original documentation order. This is a pure mechanical
reordering with no logic changes. 28 of 32 sections now match; the
4 remaining differences are due to cross-file constraints (constructor
call order) that cannot be resolved without moving items between files.

Add a comment to each item_*() function noting that add_item() call
order determines generated documentation order.
@zhubonan
Add pyyaml as requirements for documentation generation
f4f5a96
@zhubonan
Commit updated input-main.md and parameters.yaml
9ba07ba
@zhubonan
Merge branch 'develop' into abacus-develop-built-in-help
0280d93
@zhubonan
Copy link
Author

zhubonan commented Feb 22, 2026

Changes since the initial approach:

Documentation pipeline now is : C++ source → abacus --generate-parameters-yaml → parameters.yaml → generate_input_main.py → input-main.md

This means the documentation of each input parameter is only written once (in the .cpp files)

  • Extended Input_Item with 6 documentation fields (category, type, description, default_value, unit,
    availability)
  • Annotated all ~490 parameters across the 12 read_input_item_*.cpp files with structured metadata based on the previous input-main.md
  • Added input_help.{h,cpp} — help system with case-insensitive lookup, substring search, and fuzzy matching
    (Levenshtein). Builds its registry via ReadInput in rank=-1 mode, no code generation needed
  • Added docs/generate_input_main.py — generates input-main.md from the YAML dump, replacing the
    regex-based scraper in the initial implementation
  • Added unit tests for the help system (input_help_test.cpp)
  • Updated docs/CONTRIBUTING.md with developer workflow for adding parameters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mohanchen mohanchen mohanchen left review comments

@ZhouXY-PKU ZhouXY-PKU ZhouXY-PKU approved these changes

Assignees

No one assigned

Labels

Compile & CICD & Docs & Dependencies Issues related to compiling ABACUS Feature Discussed The features will be discussed first but will not be implemented soon Refactor Refactor ABACUS codes Useful Information Useful information for others to learn/study

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@zhubonan @mohanchen @ZhouXY-PKU