Feat/ascend 910b1 support

README.md

@@ -10,7 +10,7 @@ In shared development environments with multiple GPUs, researchers and developer
 
 You peacefully share a host but want a helper to avoid accidental conflicts.
 
-- You have a single host with GPUs (NVIDIA or AMD) shared by multiple users
+- You have a single host with NVIDIA GPUs, AMD GPUs, or Huawei Ascend NPUs shared by multiple users
 - You all log in and run commands manually for development and/or testing
 - You can still talk to each other about playing nice and sharing your GPUs
 
@@ -35,7 +35,8 @@ canhazgpu admin --gpus 8
 canhazgpu status
 
 # Run vLLM with an automatic 2 GPU reservation.
-# - CUDA_VISIBLE_DEVICES is set in the environment before running the command.
+# - The provider visibility variable is set before running the command
+#   (CUDA_VISIBLE_DEVICES for NVIDIA/AMD; ASCEND_RT_VISIBLE_DEVICES for Ascend).
 # - If GPUs are unavailable, waits in queue until they become available.
 canhazgpu run --gpus 2 -- vllm serve my/model --tensor-parallel-size 2
 
@@ -61,9 +62,12 @@ canhazgpu reserve --gpus 1 --duration 4h
 # Reserve specific GPU IDs manually
 canhazgpu reserve --gpu-ids 0,2 --duration 2h
 
-# Reserve GPUs and set CUDA_VISIBLE_DEVICES in one step (for scripting)
+# Reserve NVIDIA/AMD GPUs and set CUDA_VISIBLE_DEVICES in one step (for scripting)
 export CUDA_VISIBLE_DEVICES=$(canhazgpu reserve --gpus 2 --short)
 
+# For Ascend, use the CANN visibility variable instead.
+export ASCEND_RT_VISIBLE_DEVICES=$(canhazgpu reserve --gpus 2 --short)
+
 # Release manual reservations when done
 canhazgpu release
 
@@ -83,8 +87,8 @@ canhazgpu web --port 8080
 - **MRU-per-user allocation**: Smart GPU affinity using most recently used per-user strategy with LRU fallback
 - **Specific GPU reservation**: Reserve exact GPU IDs when needed (e.g., --gpu-ids 1,3)
 - **Unreserved usage detection**: Identifies GPUs in use without proper reservations
-- **Real-time validation**: Uses nvidia-smi or amd-smi to verify actual GPU usage
-- **Multi-provider support**: Supports both NVIDIA and AMD GPUs with automatic detection
+- **Real-time validation**: Uses nvidia-smi, amd-smi, or npu-smi to verify actual device usage
+- **Multi-provider support**: Supports NVIDIA, AMD, and Huawei Ascend devices with automatic detection
 - **Flexible reservations**: Support for both command execution and manual reservations
 - **Reservation reporting**: Track and analyze GPU reservation patterns over time by user
 - **Web dashboard**: Real-time monitoring interface with status and reservation reports
@@ -121,6 +125,7 @@ For detailed usage, configuration, and administration:
 - **GPUs** with appropriate management tools:
   - **NVIDIA GPUs**: nvidia-smi available
   - **AMD GPUs**: amd-smi available (ROCm 5.7+)
+  - **Huawei Ascend NPUs** (including 910B1): npu-smi available and the CANN runtime configured for the user
 - **System access** to `/proc` filesystem or `ps` command
 
 ## Installation
@@ -146,11 +151,13 @@ Then initialize the GPU pool:
 canhazgpu admin --gpus $(nvidia-smi -L | wc -l)  # For NVIDIA
 # OR
 canhazgpu admin --gpus $(amd-smi list --json | jq 'length')  # For AMD
+# OR (example: eight Ascend NPUs)
+canhazgpu admin --gpus 8 --provider ascend  # For Huawei Ascend
 ```
 
 ## How It Works
 
-1. **Validation**: Uses nvidia-smi or amd-smi to detect actual GPU usage and identify conflicts
+1. **Validation**: Uses nvidia-smi, amd-smi, or npu-smi to detect actual device usage and identify conflicts
 2. **Coordination**: Uses Redis for distributed state management and race condition prevention
 3. **Queueing**: FCFS queue; the first entry whose full request can be satisfied is allocated, so no GPUs are held by a job that cannot start yet
 4. **Allocation**: MRU-per-user (Most Recently Used per user) strategy provides GPU affinity with LRU fallback for fair distribution

deploy/canhazgpu-guard.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=canhazgpu accelerator reservation guard
+Wants=network-online.target redis.service
+After=network-online.target redis.service
+
+[Service]
+Type=simple
+User=root
+ExecStart=/usr/local/bin/canhazgpu guard --enforce --exclude-users "" --interval 1s --grace 5s --confirmations 1 --max-warnings 1 --warn-interval 5s --kill-grace 5s --max-kills-per-hour 0 --channels log --log-file /home/ajhou/.cache/gpu.log
+Restart=on-failure
+RestartSec=5s
+
+[Install]
+WantedBy=multi-user.target

docs/admin-troubleshooting.md

@@ -93,6 +93,8 @@ redis-cli get "canhazgpu:provider"
 canhazgpu admin --gpus 8 --provider nvidia --force
 # OR
 canhazgpu admin --gpus 8 --provider amd --force
+# OR
+canhazgpu admin --gpus 8 --provider ascend --force
 
 # Let system auto-detect
 canhazgpu admin --gpus 8 --force
@@ -110,6 +112,43 @@ canhazgpu admin --gpus 4 --provider nvidia
 
 # Use AMD provider for AMD GPUs  
 canhazgpu admin --gpus 2 --provider amd
+
+# Use Huawei Ascend provider for Ascend NPUs
+canhazgpu admin --gpus 8 --provider ascend
+```
+
+## Huawei Ascend NPU Issues
+
+### npu-smi Permission Denied
+
+**Symptoms:**
+```bash
+❯ npu-smi info
+DrvMngGetConsoleLogLevel failed. (ret=4)
+dcmi module initialize failed. ret is -8005
+```
+
+**Cause:** The account cannot read the Ascend device nodes. On typical CANN
+installations the nodes are owned by the configured runtime group, commonly
+`HwHiAiUser`.
+
+**Solution:** An administrator must add the account to that group, then the
+user must start a completely new login session:
+
+```bash
+sudo usermod -aG HwHiAiUser <username>
+
+# After logging out and back in
+id -nG
+npu-smi info
+```
+
+Read `/etc/ascend_install.info` to confirm the site's `UserGroup`; do not
+assume `HwHiAiUser` if the installation uses a different group. Once
+`npu-smi info` succeeds for the user, initialize the pool with:
+
+```bash
+canhazgpu admin --gpus 8 --provider ascend
 ```
 
 ## NVIDIA GPU Issues
@@ -623,4 +662,4 @@ ps aux | grep -E "(redis|nvidia|python)"
 - `dmesg` output for hardware issues
 - Any custom monitoring logs
 
-This troubleshooting guide covers the most common issues encountered in production deployments of canhazgpu. Most problems can be resolved by following these systematic approaches.
+This troubleshooting guide covers the most common issues encountered in production deployments of canhazgpu. Most problems can be resolved by following these systematic approaches.

docs/commands.md

@@ -13,7 +13,7 @@ Commands:
   release     Release manually reserved GPUs held by the current user
   report      Generate GPU usage reports
   reserve     Reserve GPUs manually for a specified duration
-  run         Reserve GPUs and run a command with CUDA_VISIBLE_DEVICES set
+  run         Reserve GPUs and run a command with device visibility set
   schedule    Show the GPU booking schedule for a day
   status      Show current GPU allocation status
   violations  Show GPU usage that bypassed the reservation system
@@ -103,7 +103,7 @@ canhazgpu admin --gpus <count> [--force] [--provider <type>]
 **Options:**
 - `--gpus`: Number of GPUs available on this machine (required)
 - `--force`: Force reinitialization even if already initialized
-- `--provider`: GPU provider type (`nvidia`, `amd`, or `fake`). Auto-detected if not specified.
+- `--provider`: Device provider type (`nvidia`, `amd`, `ascend`, or `fake`). Auto-detected if not specified.
 
 **Examples:**
 ```bash
@@ -116,6 +116,9 @@ canhazgpu admin --gpus 8 --provider nvidia
 # Use AMD GPUs
 canhazgpu admin --gpus 4 --provider amd
 
+# Use Huawei Ascend NPUs
+canhazgpu admin --gpus 8 --provider ascend
+
 # Use fake provider for development/testing (no real GPUs required)
 canhazgpu admin --gpus 4 --provider fake
 
@@ -125,7 +128,7 @@ canhazgpu admin --gpus 4 --force
 
 !!! tip "Fake Provider for Development"
     Use `--provider fake` to develop and test canhazgpu on systems without actual GPUs.
-    The fake provider simulates GPU behavior without requiring nvidia-smi or amd-smi.
+    The fake provider simulates GPU behavior without requiring nvidia-smi, amd-smi, or npu-smi.
 
 !!! warning "Destructive Operation"
     Using `--force` will clear all existing reservations. Use with caution in production.
@@ -305,11 +308,11 @@ canhazgpu run --wait 30m --gpus 4 -- python train.py
 ```
 
 **Behavior:**
-1. Validates actual GPU availability using nvidia-smi
+1. Validates actual device availability using the configured provider
 2. Excludes GPUs that are in use without reservation
 3. If GPUs unavailable, waits in queue (unless `--nonblock` is set)
 4. Reserves the requested number of GPUs using MRU-per-user allocation (with LRU fallback)
-5. Sets `CUDA_VISIBLE_DEVICES` to the allocated GPU IDs
+5. Sets `CUDA_VISIBLE_DEVICES` for NVIDIA/AMD or `ASCEND_RT_VISIBLE_DEVICES` for Ascend to the allocated device IDs
 6. Runs your command
 7. Automatically releases GPUs when the command finishes
 8. Maintains a heartbeat while running to keep the reservation active
@@ -398,9 +401,13 @@ canhazgpu reserve --start 14:00 --end 16:00 --gpus 2
 ```
 
 **Important Note:**
-Unlike the `run` command, `reserve` does NOT automatically set `CUDA_VISIBLE_DEVICES`. You can use `--short` for easy shell integration:
+Unlike the `run` command, `reserve` does NOT automatically set the provider visibility variable. You can use `--short` for easy shell integration:
 ```bash
+# NVIDIA or AMD
 export CUDA_VISIBLE_DEVICES=$(canhazgpu reserve --gpus 2 --short)
+
+# Huawei Ascend
+export ASCEND_RT_VISIBLE_DEVICES=$(canhazgpu reserve --gpus 2 --short)
 ```
 
 **Use Cases:**
@@ -822,7 +829,7 @@ When no remote hosts are configured, the dashboard shows the traditional single-
 
 All allocation commands (`run` and `reserve`) automatically:
 
-1. **Scan for unreserved usage** using nvidia-smi
+1. **Scan for unreserved usage** using the configured provider
 2. **Exclude unreserved GPUs** from the available pool
 3. **Hold back GPUs needed by scheduled bookings** during the window the reservation would cover
 4. **Provide detailed error messages** if insufficient GPUs remain
@@ -844,4 +851,4 @@ When multiple GPUs are available, the system uses **Most Recently Used per User*
 
 ### Status Integration
 
-The `status` command shows comprehensive information about all reservation types and validates actual usage against reservations, making it easy to identify and resolve conflicts.
+The `status` command shows comprehensive information about all reservation types and validates actual usage against reservations, making it easy to identify and resolve conflicts.

docs/configuration.md

@@ -51,7 +51,7 @@ guard:
   channels: ["process", "tty", "log"]
   log-file: ""
   exclude-users: ["root"]
-  exclude-commands: ["Xorg", "nvidia-smi", "amd-smi", "dcgm-exporter", "nvidia-persistenced"]
+  exclude-commands: ["Xorg", "nvidia-smi", "amd-smi", "npu-smi", "dcgm-exporter", "nvidia-persistenced"]
   notify-holder: true
 
 # Default settings for 'run' command
@@ -241,4 +241,4 @@ web:
   host: "0.0.0.0"
 ```
 
-This configuration provides sensible defaults while allowing easy customization for different environments and use cases.
+This configuration provides sensible defaults while allowing easy customization for different environments and use cases.

docs/dev-architecture.md

@@ -407,11 +407,12 @@ Current MRU-per-user allocation could be enhanced with:
 
 ### 2. GPU Provider System
 
-The system supports multiple GPU providers through a unified interface:
+The system supports multiple accelerator providers through a unified interface:
 
 **Available Providers:**
 - **NVIDIA**: Uses nvidia-smi for NVIDIA GPU management
 - **AMD**: Uses amd-smi (ROCm 5.7+) for AMD GPU management
+- **Huawei Ascend**: Uses npu-smi and CANN logical device IDs
 - **Fake**: Simulated provider for development and testing without real GPUs
 
 **Provider Architecture:**
@@ -575,4 +576,4 @@ type UsageRecord struct {
 - Large GPU count scenarios
 - Memory leak detection
 
-This architecture provides a robust, scalable foundation for GPU resource management while maintaining simplicity and ease of deployment.
+This architecture provides a robust, scalable foundation for GPU resource management while maintaining simplicity and ease of deployment.

docs/dev-contributing.md

@@ -17,7 +17,7 @@ cd canhazgpu
 # System requirements
 # - Go 1.25+
 # - Redis server
-# - NVIDIA drivers with nvidia-smi
+# - A supported accelerator tool (nvidia-smi, amd-smi, or npu-smi)
 
 # Go dependencies (automatic)
 go mod download
@@ -451,7 +451,7 @@ func (e *AllocationEngine) AllocateGPUs(ctx context.Context, req *AllocationRequ
 
 ### 3. Adding New GPU Providers
 
-The system already supports NVIDIA, AMD, and Fake providers. To add support for new GPU hardware:
+The system already supports NVIDIA, AMD, Huawei Ascend, and Fake providers. To add support for new accelerator hardware:
 
 1. **Implement the GPUProvider interface:**
 ```go
@@ -602,7 +602,7 @@ Relates to #456
 - Operating system and version
 - Go version (for development issues)
 - Redis version
-- GPU driver version (NVIDIA or AMD)
+- Accelerator driver version (NVIDIA, AMD, or Ascend)
 - Complete error messages
 - Steps to reproduce
 - Expected vs actual behavior
@@ -616,7 +616,7 @@ Clear description of the bug
 - OS: Ubuntu 22.04
 - Go: 1.23.0 (if building from source)
 - Redis: 7.0.0
-- GPU Provider: nvidia / amd
+- GPU Provider: nvidia / amd / ascend
 - GPU Driver: 535.129.03
 - canhazgpu version: 1.0.0
 
@@ -695,4 +695,4 @@ Contributors are recognized in:
 - Release notes
 - Documentation acknowledgments
 
-Thank you for contributing to canhazgpu! Your efforts help make GPU resource management better for everyone.
+Thank you for contributing to canhazgpu! Your efforts help make GPU resource management better for everyone.

docs/dev-testing.md

@@ -23,7 +23,7 @@ This guide explains how to run tests for canhazgpu and understand the testing in
 #### Integration Tests (Slower)
 - Run with: `make test` or `make test-integration`
 - Duration: 5-30 seconds per test
-- Dependencies: Redis server, nvidia-smi (optional)
+- Dependencies: Redis server and a supported provider tool (`nvidia-smi`, `amd-smi`, or `npu-smi`; optional)
 - Tests real system interactions
 
 ## Running Tests
@@ -72,8 +72,8 @@ When running full tests (`make test`), these tests may take time:
 
 2. **GPU Validation Tests** (5-10 seconds)
    - `TestDetectGPUUsage_Integration` 
-   - Calls nvidia-smi command
-   - Logs: Indicates nvidia-smi availability
+   - Calls the available provider tool
+   - Logs: Indicates provider availability
 
 3. **Heartbeat Manager Tests** (1-3 seconds)
    - `TestHeartbeatManager_Wait`
@@ -83,7 +83,7 @@ When running full tests (`make test`), these tests may take time:
 
 4. **GPU Allocation Tests** (2-10 seconds)
    - `TestAllocationEngine_AllocateGPUs_Structure`
-   - Combines Redis + nvidia-smi validation
+   - Combines Redis + provider validation
    - Logs: Indicates each phase
 
 ### Test Logging
@@ -107,8 +107,8 @@ Integration tests include verbose logging to explain timing:
    - Tests automatically skip if unavailable
    - Uses database 15 (test database)
 
-2. **nvidia-smi** (optional)
-   - Used for GPU detection tests
+2. **Provider tool** (optional)
+   - `nvidia-smi`, `amd-smi`, or `npu-smi` is used for device detection tests
    - Tests gracefully handle missing command
    - Expected to fail on non-GPU systems
 
@@ -126,7 +126,7 @@ SKIP: Redis not available for testing: dial tcp :6379: connect: connection refus
 ```
 **Solution**: Start Redis server or run `make test-short`
 
-### nvidia-smi Not Found
+### Provider Tool Not Found
 ```
 nvidia-smi not available or failed: exec: "nvidia-smi": executable file not found
 ```
@@ -208,4 +208,4 @@ make test
 make test-coverage
 ```
 
-This testing infrastructure ensures reliable GPU allocation while providing fast feedback during development.
+This testing infrastructure ensures reliable GPU allocation while providing fast feedback during development.

docs/features-guard.md

@@ -88,7 +88,7 @@ Safety rails:
 - **Escalation ladder**: SIGINT → SIGTERM → SIGKILL, `--kill-grace` apart, so a job gets the chance to shut down cleanly
 - **Circuit breaker**: at most `--max-kills-per-hour` terminations (default 3); beyond that the guard only warns, since a storm of kills is more likely a bug than a room full of offenders
 - **Unknown owners are never terminated**: if the process owner cannot be determined it might be a system process
-- **Allow lists**: `--exclude-users` (default `root`) and `--exclude-commands` (default `Xorg,nvidia-smi,amd-smi,dcgm-exporter,nvidia-persistenced`)
+- **Allow lists**: `--exclude-users` (default `root`) and `--exclude-commands` (default `Xorg,nvidia-smi,amd-smi,npu-smi,dcgm-exporter,nvidia-persistenced`)
 - **Dry run**: `--dry-run` records what would have happened, including in `canhazgpu violations`
 
 ## Avoiding false positives

docs/features-idle-timeout.md

@@ -76,7 +76,7 @@ The same rules apply as for manual reservations:
 
 **Reservations without an idle timeout are exempt.** That includes reservations created before this feature existed, `canhazgpu run --idle-timeout 0`, and any reservation whose stored timeout is zero. They stay tied to the lifetime of their process.
 
-**Reservations are only released when usage can actually be checked.** If `nvidia-smi`/`amd-smi` cannot be queried, idle detection is skipped for that pass rather than guessed at. The same applies to usage that cannot be attributed to an owner.
+**Reservations are only released when usage can actually be checked.** If `nvidia-smi`, `amd-smi`, or `npu-smi` cannot be queried, idle detection is skipped for that pass rather than guessed at. The same applies to usage that cannot be attributed to an owner.
 
 **Memory counts as usage.** A GPU holding a loaded model with no active kernel is "in use" as far as canhazgpu is concerned — this matches how the rest of the tool defines usage. Lower `--memory-threshold` if you want stricter accounting.