Review and update how-to guides: build-test-validate, release-management, development-workflow, data-projects

[miohansen]

Overview

Conduct a comprehensive review and update of key how-to guide sections in docs/how-to-guides/eac/ to ensure accuracy, completeness, and alignment with current system functionality. This review focuses on four critical areas that form the core user workflows.

Scope: 36 guides across 4 sections

Sections to Review

1. Build, Test and Validate (13 guides)

Location: docs/how-to-guides/eac/commands/build-test-validate/

Files:

• index.md - Overview and navigation
• build-single-module.md
• build-changed-modules.md
• check-dependencies.md
• debug-test-failures.md
• execute-manual-tests.md
• fix-windows-defender-false-positives.md
• manage-risk-compliance.md
• run-test-suites.md
• run-tests-for-module.md
• scan-for-security-issues.md
• validate-before-commit.md
• validate-specifications.md

Review Focus:

• Command syntax matches current implementation (r2r eac vs go run ./go/eac/commands)
• Test suite names correct (unit, integration, acceptance, production-verification)
• Manual test workflow accurate (export-import-merge process)
• Build commands reflect Docker-based workflow
• Security scan commands up to date (Trivy, Semgrep, ZAP, etc.)
• Examples work with current CLI
• Output examples match actual command output
• Flag documentation complete and accurate
• Links to reference documentation work
• Cross-references to other guides accurate

Specific Items:

1. execute-manual-tests.md: Verify aligns with new manual test documentation (Document manual test workflow in docs/reference/eac/testing #100)
2. run-test-suites.md: Check test suite names (@L0-L4 tags, suite naming)
3. validate-before-commit.md: Ensure complete pre-commit checklist
4. scan-for-security-issues.md: Verify all scanners documented

2. Release Management (11 guides)

Location: docs/how-to-guides/eac/commands/release-management/

Files:

• index.md - Overview and workflow diagram
• check-ci-before-release.md
• cleanup-container-images.md
• create-release-tag.md
• generate-changelog.md
• prepare-module-release.md
• release-workflow-variants.md
• understanding-release-folder.md
• view-approval-comments.md
• view-changelog-release-notes.md
• view-specifications.md

Review Focus:

• Changelog-driven release process accurately described
• CDe vs RA workflow patterns clearly explained
• Git tag format correct ({module}/{version})
• Conventional commit format up to date
• SemVer vs CalVer rules accurate
• CI workflow triggers documented correctly
• Release folder structure matches implementation
• Quality gates and approval processes current
• Container cleanup commands accurate
• GitHub Actions workflow references correct

Specific Items:

1. prepare-module-release.md: Verify complete end-to-end workflow
2. release-workflow-variants.md: Check CDe and RA patterns match CD Model
3. understanding-release-folder.md: Confirm folder structure is current
4. generate-changelog.md: Verify release-this command behavior
5. check-ci-before-release.md: Ensure CI check commands work

3. Development Workflow (4 guides)

Location: docs/how-to-guides/eac/commands/development-workflow/

Files:

• index.md - Overview and common workflows
• create-feature-workspace.md
• create-pull-request.md
• make-commits-with-ai.md
• merge-workspace-changes.md

Review Focus:

• Git worktree commands accurate
• work-create, work-commit, work-merge commands documented
• AI commit message generation workflow current
• PR creation with create-pr command accurate
• Workspace merge strategies explained
• Branch naming conventions documented
• Parallel development workflow clear
• Integration with Git operations correct
• Examples reflect current CLI behavior

Specific Items:

1. create-feature-workspace.md: Verify work-create command and worktree setup
2. make-commits-with-ai.md: Check AI provider configuration and work-commit
3. create-pull-request.md: Verify create-pr command and AI PR description
4. merge-workspace-changes.md: Confirm work-merge squash behavior

4. Data Projects (8 guides)

Location: docs/how-to-guides/eac/data-projects/

Files:

• index.md - Overview and CD model mapping
• cd-model-for-data-projects.md
• cicd-pipeline-databricks.md
• databricks-asset-bundles.md
• develop-data-pipeline-with-specs.md
• environment-management.md
• monitoring-data-quality.md
• testing-data-pipelines.md

Review Focus:

• 12-stage CD Model application to data projects accurate
• Databricks Asset Bundles structure current (v2 format)
• Unity Catalog integration documented correctly
• Delta Lake versioning and cloning explained
• CI/CD pipeline examples work with GitHub Actions
• Testing strategies (unit, integration, data quality) complete
• Environment isolation (PLTE setup) clear
• Monitoring and observability patterns up to date
• Gherkin specification examples applicable to data pipelines

Specific Items:

1. cd-model-for-data-projects.md: Verify all 12 stages mapped correctly
2. databricks-asset-bundles.md: Check Asset Bundle v2 YAML structure
3. cicd-pipeline-databricks.md: Ensure GitHub Actions examples work
4. testing-data-pipelines.md: Verify pytest and chispa examples
5. environment-management.md: Confirm Unity Catalog and PLTE setup

Review Criteria

For each guide, check:

Accuracy

• Commands execute successfully with current CLI
• Flag names and values are correct
• Output examples match actual output
• File paths and directory structures are accurate
• Workflow steps work in sequence

Completeness

• All command flags documented
• Error scenarios covered
• Prerequisites clearly stated
• Expected outcomes described
• Troubleshooting section present (if applicable)

Clarity

• Step-by-step instructions clear and numbered
• Examples are realistic and practical
• Terminology consistent across guides
• Diagrams/visuals aid understanding (where appropriate)
• Code blocks properly formatted and syntax-highlighted

Currency

• Reflects current CLI version and features
• No deprecated commands or flags
• References to external tools (Docker, Git, GitHub CLI) are current
• Links to other documentation work
• Version numbers match current release

Alignment

• Consistent with reference documentation
• Matches CD Model principles
• Follows repository conventions
• Cross-references accurate
• Terminology matches glossary

Testing Approach

Command Validation

For each guide with command examples:

1. Execute commands in clean environment
2. Capture output and compare to documented examples
3. Verify flags and options work as described
4. Test error scenarios if documented
5. Check file outputs match descriptions

Workflow Validation

For workflow guides (multi-step processes):

1. Follow complete workflow start to finish
2. Verify each step produces expected result
3. Check integration between steps
4. Test variations (different options, flags)
5. Validate outcomes match descriptions

Cross-Reference Validation

For guides with links:

1. Test all internal links to reference documentation
2. Verify external links (GitHub, Databricks docs)
3. Check command references link to correct pages
4. Validate navigation (.nav.yml entries)

Update Process

For Each Guide

1. Read thoroughly: Understand the guide's purpose and workflow
2. Execute commands: Run all examples in test environment
3. Note discrepancies: Document any inaccuracies or errors
4. Update content: Fix errors, update examples, improve clarity
5. Add missing content: Fill gaps in coverage
6. Test updates: Verify corrected examples work
7. Review cross-references: Ensure links still valid
8. Commit changes: Clear commit message describing updates

Documentation Standards

Command Examples:

# Good: Full command with explanation
r2r eac test eac-commands --suite unit

# Bad: Incomplete or ambiguous
test something

Output Examples:

# Good: Actual output with formatting
✓ Build complete
Module: eac-commands
Duration: 45.2s
Artifacts: out/build/eac-commands/

# Bad: Generic placeholder
[output here]

Step-by-Step Instructions:

1. Check for pending changes
   ```bash
   r2r eac release pending my-module

Expected output: Shows version bump needed

2. Generate changelog
   [next step...]

## Priority Areas

**High Priority** (Must be accurate for release):
1. Release management guides (critical for release process)
2. Manual test execution guide (aligns with #100)
3. Build and test guides (core development workflow)

**Medium Priority** (Important for developer experience):
1. Development workflow guides (daily development tasks)
2. Validation guides (quality assurance)
3. Security scanning guides (compliance)

**Lower Priority** (Specialized use cases):
1. Data projects guides (specific to data team)
2. Windows Defender guide (platform-specific)

## Success Criteria

### For Each Guide
- [ ] All commands execute successfully
- [ ] Output examples match actual output
- [ ] No broken links or references
- [ ] Workflow steps complete end-to-end
- [ ] Troubleshooting section helpful
- [ ] Prerequisites clearly stated

### Overall
- [ ] All 36 guides reviewed and updated
- [ ] Command reference documentation aligned
- [ ] Cross-references validated
- [ ] Navigation (.nav.yml) accurate
- [ ] No deprecated content
- [ ] User testing completed (sample users follow guides)

## Testing with Real Users

After updates, conduct user testing:

**Test Scenarios**:
1. New developer follows "Build, Test and Validate" workflow
2. Release manager prepares release using release management guides
3. Feature developer uses development workflow guides
4. Data engineer follows data projects guides

**Validation**:
- User completes workflow without getting stuck
- User understands each step
- Commands work as documented
- Outcomes match expectations
- User can troubleshoot issues using guide

## Related Documentation

**Reference Documentation** (must align with):
- `docs/reference/eac/commands/` - Command reference
- `docs/reference/eac/testing/` - Test suite reference
- `docs/reference/repository/continuous-delivery/` - CD Model reference

**Other How-To Guides** (cross-reference checks):
- `docs/how-to-guides/eac/commands/getting-started/` - Beginner guides
- `docs/how-to-guides/eac/commands/documentation/` - Documentation guides
- `docs/how-to-guides/local-setup/` - Environment setup

## Deliverables

1. **Updated Guides**: All 36 guides reviewed and corrected
2. **Change Log**: Document of changes made per guide
3. **Test Results**: Validation that all commands work
4. **Issue List**: Any command bugs or gaps found
5. **User Feedback**: Results from user testing

## Timeline Estimate

**Per Guide Review**:
- Simple guide (4 steps): ~30 minutes
- Medium guide (6-10 steps): ~1 hour
- Complex guide (multi-workflow): ~2 hours

**Total Estimated Effort**:
- Build-test-validate (13 guides): ~15 hours
- Release management (11 guides): ~18 hours
- Development workflow (4 guides): ~6 hours
- Data projects (8 guides): ~12 hours
- **Total**: ~51 hours

**Phased Approach**:
- Phase 1: Release management (Week 1) - Critical for releases
- Phase 2: Build-test-validate (Week 2) - Core workflows
- Phase 3: Development workflow (Week 3) - Daily tasks
- Phase 4: Data projects (Week 4) - Specialized

## Related Issues

- #100 - Manual test documentation (aligns with execute-manual-tests.md)
- Template system documentation (#96) - May reference in development workflow
- Extension development (#97, #98) - May need how-to guides

## Priority

**High** - Critical for documentation quality and user experience
- Essential for new users learning the system
- Required for release process validation
- Impacts developer productivity
- Foundation for onboarding documentation