refactor: reorganize tools/ into tests/ and scripts/

.github/labeler.yml

@@ -21,7 +21,7 @@ smartem-devtools:claude:
 
 smartem-devtools:e2e-test:
   - changed-files:
-      - any-glob-to-any-file: tools/e2e/**
+      - any-glob-to-any-file: tests/e2e/**
 
 # Type-of-work labels
 documentation:
@@ -34,6 +34,6 @@ devops:
   - changed-files:
       - any-glob-to-any-file:
           - .github/**
-          - tools/k8s/**
-          - tools/github/**
+          - scripts/k8s/**
+          - scripts/github/**
           - k8s/**

.github/workflows/gitflow.yml

@@ -15,7 +15,7 @@ on:
     branches: [main]
     paths:
       - 'core/github-tags-config.ts'
-      - 'tools/github/**'
+      - 'scripts/github/**'
 
 jobs:
   check-labels:
@@ -38,7 +38,7 @@ jobs:
       - name: Check labels
         env:
           GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: npx tsx tools/github/sync-labels.ts --check --verbose
+        run: npx tsx scripts/github/sync-labels.ts --check --verbose
 
   sync-labels:
     name: Sync Labels
@@ -60,4 +60,4 @@ jobs:
       - name: Sync labels
         env:
           GH_TOKEN: ${{ secrets.LABEL_SYNC_TOKEN }}
-        run: npx tsx tools/github/sync-labels.ts --sync --verbose
+        run: npx tsx scripts/github/sync-labels.ts --sync --verbose

claude-code/CLAUDE.md

@@ -127,16 +127,16 @@ Developer tooling, documentation, and workspace configuration for the SmartEM ec
 - Core TypeScript config for multi-repo management
 - API specs (SmartEM, Athena)
 - Kubernetes manifests for dev/stage/prod (k8s/)
-- K8s development scripts (tools/k8s/)
-- E2E test scripts (tools/e2e/)
+- K8s development scripts (scripts/k8s/)
+- E2E test scripts (tests/e2e/)
 - EPUPlayer for EPU filesystem recording/playback (packages/smartem-epuplayer/)
 
 **Key directories:**
 - `docs/` - Markdown documentation, ADRs, how-to guides (synced to webui as MDX)
 - `claude-code/` - Skills, repo guidelines, architecture docs
 - `webui/` - React developer dashboard
 - `core/` - Repository and workspace config definitions
-- `tools/e2e/` - E2E test runners for SmartEM (single and multi-microscope)
+- `tests/e2e/` - E2E test runners for SmartEM (single and multi-microscope)
 - `packages/smartem-epuplayer/` - EPU filesystem recording/playback for dev/test
 
 **Tech**: Python 3.11+, Node.js, React 19, Vite, TypeScript

claude-code/shared/skills/devops/SKILL.md

@@ -25,10 +25,10 @@ Kubernetes deployment, container management, and CI/CD operations for the SmartE
 cd repos/DiamondLightSource/smartem-decisions
 
 # Start local k3s cluster with all services
-./tools/k8s/dev-k8s.sh up
+./scripts/k8s/dev-k8s.sh up
 
 # Stop and cleanup
-./tools/k8s/dev-k8s.sh down
+./scripts/k8s/dev-k8s.sh down
 
 # Check cluster status
 kubectl get pods -n smartem-decisions

claude-code/smartem-decisions/settings.json

@@ -34,8 +34,9 @@
       "Bash(psql:*)",
       "Bash(pg_dump:*)",
       "Bash(pg_restore:*)",
-      "Bash(./tools/k8s/dev-k8s.sh:*)",
-      "Bash(./tools/*)",
+      "Bash(./scripts/k8s/dev-k8s.sh:*)",
+      "Bash(./scripts/*)",
+      "Bash(./tests/*)",
       "WebFetch(domain:github.com)",
       "WebFetch(domain:diamondlightsource.github.io)",
       "WebFetch(domain:docs.anthropic.com)",

docs/backend/api-server.md

@@ -6,7 +6,7 @@ The core backend service providing HTTP API, database operations, and message qu
 
 ```bash
 # create env and launch service stack locally:
-./tools/k8s/dev-k8s.sh up
+./scripts/k8s/dev-k8s.sh up
 
 # launch RabbitMQ worker (consumer)
 python -m smartem_backend.consumer              # ERROR level (default)

docs/decision-records/decisions/0006-sealed-secrets-kubernetes-secrets-management.md

@@ -46,7 +46,7 @@ be safely committed to version control whilst maintaining security.
 
 Implementation will include:
 - Installation of sealed-secrets controller in each Kubernetes environment
-- Integration with existing deployment workflow via tools/k8s/generate-sealed-secrets.sh script
+- Integration with existing deployment workflow via scripts/k8s/generate-sealed-secrets.sh script
 - Documentation in docs/how-to/manage-kubernetes-secrets.md for team procedures
 - Environment-specific encryption keys for development, staging, and production clusters
 

docs/decision-records/decisions/0013-json-config-with-typescript-types.md

@@ -96,7 +96,7 @@ React components
 ### Modified
 - `webui/src/config/index.ts` - now imports from `core/index.ts` and transforms
 - `webui/scripts/prebuild.ts` - simplified to docs sync only
-- `tools/github/sync-labels.ts` - updated import path
+- `scripts/github/sync-labels.ts` - updated import path
 
 ### Deleted
 - `core/github-labels-config.ts`

docs/development/e2e-simulation.md

@@ -8,21 +8,21 @@ For the simplest test execution, use the automated test runner script:
 
 ```bash
 # Check k3s status first (DO NOT restart if already running)
-./tools/k8s/dev-k8s.sh status
+./scripts/k8s/dev-k8s.sh status
 
 # Run automated test (Test Type 2: Pre-Acquisition Agent Setup)
-./tools/run-e2e-test.sh
+./tests/e2e/run-e2e-test.sh
 
 # With custom parameters (paths relative to workspace root)
-./tools/run-e2e-test.sh \
+./tests/e2e/run-e2e-test.sh \
   ../../testdata/recordings/bi37708-42_epurecording.tar.gz \
   ../../tmp/epu-test-dir \
   0.1
 
 # Or override via environment variables
 SMARTEM_TEST_RECORDING=/path/to/recording.tar.gz \
 SMARTEM_EPU_DIR=/path/to/epu-dir \
-./tools/run-e2e-test.sh
+./tests/e2e/run-e2e-test.sh
 ```
 
 **Script parameters** (all optional):
@@ -64,7 +64,7 @@ If you've already set up once and just need to run another test:
 
 ```bash
 # 1. Check k3s status (DO NOT restart if already running)
-./tools/k8s/dev-k8s.sh status
+./scripts/k8s/dev-k8s.sh status
 
 # 2. Prepare clean environment
 unset POSTGRES_HOST POSTGRES_PORT POSTGRES_DB POSTGRES_USER POSTGRES_PASSWORD
@@ -115,13 +115,13 @@ For testing multiple concurrent microscopes and acquisition sessions simultaneou
 
 ```bash
 # Check k3s status first (DO NOT restart if already running)
-./tools/k8s/dev-k8s.sh status
+./scripts/k8s/dev-k8s.sh status
 
 # Run multi-microscope test with 3 microscopes (default)
-./tools/run-e2e-test-multi-microscope.sh
+./tests/e2e/run-e2e-test-multi-microscope.sh
 
 # With custom parameters (paths relative to workspace root)
-./tools/run-e2e-test-multi-microscope.sh \
+./tests/e2e/run-e2e-test-multi-microscope.sh \
   3 \
   ../../testdata/recordings/bi37708-42_epurecording.tar.gz \
   ../../tmp/epu-test-dir \
@@ -130,7 +130,7 @@ For testing multiple concurrent microscopes and acquisition sessions simultaneou
 # Or override via environment variables
 SMARTEM_TEST_RECORDING=/path/to/recording.tar.gz \
 SMARTEM_EPU_DIR=/path/to/epu-dir \
-./tools/run-e2e-test-multi-microscope.sh 3
+./tests/e2e/run-e2e-test-multi-microscope.sh 3
 ```
 
 **Script parameters** (all optional):
@@ -239,7 +239,7 @@ The test setup simulates a complete SmartEM workflow:
 ### Environment Setup
 - Python 3.12+ with venv activated: `source .venv/bin/activate`
 - Full development install: `pip install -e .[all]`
-- Local k3s cluster running: `./tools/k8s/dev-k8s.sh up`
+- Local k3s cluster running: `./scripts/k8s/dev-k8s.sh up`
 - Environment file: `.env` (created from `.env.example` - see Environment File Setup below)
 
 ### Test Data
@@ -298,7 +298,7 @@ ADMINER_PORT=8080
 CORS_ALLOWED_ORIGINS=*
 ```
 
-**Usage**: Automatically loaded by `./tools/run-e2e-test.sh` script
+**Usage**: Automatically loaded by `./tests/e2e/run-e2e-test.sh` script
 
 ## Database Operations
 
@@ -506,7 +506,7 @@ pkill -f uvicorn
 pkill -f fastapi
 
 # 5. Verify k3s services are running (DO NOT restart if already running)
-./tools/k8s/dev-k8s.sh status
+./scripts/k8s/dev-k8s.sh status
 
 # 6. Clean RabbitMQ (restart to ensure empty queue)
 kubectl rollout restart deployment/rabbitmq -n smartem-decisions
@@ -773,7 +773,7 @@ pgrep -f "smartem_backend|smartem_agent|epuplayer"
 rm -rf ../epu-test-dir
 
 # Stop k3s cluster (if needed)
-./tools/k8s/dev-k8s.sh down
+./scripts/k8s/dev-k8s.sh down
 ```
 
 ## Troubleshooting

docs/development/github-labels.md

@@ -11,7 +11,7 @@ The SmartEM ecosystem uses a standardised set of GitHub labels across four repos
 - `DiamondLightSource/smartem-devtools`
 - `DiamondLightSource/fandanGO-cryoem-dls`
 
-Labels are defined in `core/github-labels-config.ts` and synced using `tools/github/sync-labels.ts`.
+Labels are defined in `core/github-labels-config.ts` and synced using `scripts/github/sync-labels.ts`.
 
 ## Prerequisites
 
@@ -90,7 +90,7 @@ npm run labels:check
 Or directly:
 
 ```bash
-npx tsx tools/github/sync-labels.ts --check
+npx tsx scripts/github/sync-labels.ts --check
 ```
 
 This will:
@@ -110,7 +110,7 @@ npm run labels:sync
 Or directly:
 
 ```bash
-npx tsx tools/github/sync-labels.ts --sync
+npx tsx scripts/github/sync-labels.ts --sync
 ```
 
 This will:
@@ -123,23 +123,23 @@ This will:
 To sync only specific repositories:
 
 ```bash
-npx tsx tools/github/sync-labels.ts --sync --repo smartem-decisions
-npx tsx tools/github/sync-labels.ts --sync --repo smartem-frontend --repo smartem-devtools
+npx tsx scripts/github/sync-labels.ts --sync --repo smartem-decisions
+npx tsx scripts/github/sync-labels.ts --sync --repo smartem-frontend --repo smartem-devtools
 ```
 
 ### Verbose Output
 
 For detailed output including conforming labels:
 
 ```bash
-npx tsx tools/github/sync-labels.ts --check --verbose
+npx tsx scripts/github/sync-labels.ts --check --verbose
 ```
 
 ## CI/CD Integration
 
 The `gitflow.yml` workflow automates label management:
 
-- **On push to main**: Runs `--check` mode when `core/github-labels-config.ts` or `tools/github/**` changes
+- **On push to main**: Runs `--check` mode when `core/github-labels-config.ts` or `scripts/github/**` changes
 - **Manual dispatch**: Can trigger `--sync` mode via GitHub Actions UI
 
 ### Running Sync via CI/CD

docs/operations/environment-variables.md

@@ -28,7 +28,7 @@ The project uses multiple environment configuration patterns for different deplo
 **When to Use**:
 - Day-to-day development with `python -m smartem_backend.consumer`
 - Running API server locally with `python -m smartem_backend.api_server`
-- E2E testing with `./tools/run-e2e-test.sh`
+- E2E testing with `./tests/e2e/run-e2e-test.sh`
 - Manual debugging and development workflows
 
 **Setup**:
@@ -52,14 +52,14 @@ HTTP_API_PORT=8000
 **Used By**:
 - Python code via `load_dotenv()` in `smartem_backend/consumer.py`, `smartem_backend/utils.py`
 - Docker `entrypoint.sh` when running containers outside Kubernetes
-- E2E test scripts (`./tools/run-e2e-test.sh`)
+- E2E test scripts (`./tests/e2e/run-e2e-test.sh`)
 
 ### 2. Kubernetes Development: `.env.k8s.development`
 
 **Use Case**: Deploying SmartEM to local K8s cluster (k3s) for development.
 
 **When to Use**:
-- Running `./tools/k8s/dev-k8s.sh up`
+- Running `./scripts/k8s/dev-k8s.sh up`
 - Setting up local development cluster with all services
 
 **Setup**:
@@ -89,7 +89,7 @@ RABBITMQ_PASSWORD=password
 ```
 
 **Used By**:
-- `./tools/k8s/dev-k8s.sh` script to create K8s Secrets and ConfigMaps
+- `./scripts/k8s/dev-k8s.sh` script to create K8s Secrets and ConfigMaps
 
 ### 3. Kubernetes Staging: `.env.k8s.staging`
 
@@ -109,7 +109,7 @@ cp .env.example.k8s.staging .env.k8s.staging
 
 **Used By**:
 - CI/CD pipelines for staging deployments
-- `DEPLOY_ENV=staging ./tools/k8s/dev-k8s.sh up` (if adapted for remote clusters)
+- `DEPLOY_ENV=staging ./scripts/k8s/dev-k8s.sh up` (if adapted for remote clusters)
 
 ### 4. Kubernetes Production: `.env.k8s.production`
 
@@ -167,7 +167,7 @@ cp .env.example.k8s.development .env.k8s.development
 nano .env.k8s.development
 
 # 3. Start K8s cluster (creates infrastructure)
-./tools/k8s/dev-k8s.sh up
+./scripts/k8s/dev-k8s.sh up
 
 # 4. .env is already configured to connect to K8s NodePorts
 # No changes needed unless you want custom ports
@@ -236,7 +236,7 @@ When running in Kubernetes:
 
 ```bash
 # 1. Start K8s cluster
-./tools/k8s/dev-k8s.sh up
+./scripts/k8s/dev-k8s.sh up
 
 # 2. Use .env (already configured for NodePorts)
 cat .env
@@ -261,7 +261,7 @@ cp .env.example.k8s.development .env.k8s.development
 # Edit with your credentials
 
 # 2. Deploy
-./tools/k8s/dev-k8s.sh up
+./scripts/k8s/dev-k8s.sh up
 
 # 3. All services run in pods
 kubectl get pods -n smartem-decisions
@@ -280,10 +280,10 @@ kubectl get pods -n smartem-decisions
 test -f .env || cp .env.example .env
 
 # 2. Start K8s cluster
-./tools/k8s/dev-k8s.sh up
+./scripts/k8s/dev-k8s.sh up
 
 # 3. Run E2E test (script loads .env automatically)
-./tools/run-e2e-test.sh
+./tests/e2e/run-e2e-test.sh
 
 # The script does:
 # - source .env (loads variables)