AgentCloud: Autonomous Hybrid AI Incident Response

Overview

AgentCloud is an advanced, lightweight, multi-agent system designed to simulate a cloud environment and autonomously detect, diagnose, and mitigate system incidents. It leverages a Hybrid AI Architecture, combining the lightning-fast classification speed of a fine-tuned DistilBERT model with the adaptive reasoning of a local, CPU-only Qwen2.5-0.5B-Instruct model (GGUF, via llama.cpp).

At its core, this project demonstrates a ReAct (Reason + Act) loop for autonomous server operations that runs fully offline, with no GPU and no API key required.

Key Features

• Hybrid AI Diagnosis Routing: Rather than passing every log to a slow LLM, AgentCloud intercepts logs using a fine-tuned local DistilBERT classifier. If the ML model is highly confident (>= 0.40 score), it instantly resolves the anomaly. If confidence is low, it gracefully falls back to the Generative LLM.
• Autonomous Learning & Memory: Powered by an embedded SQLite ledger and a FAISS-based semantic vector index using SentenceTransformers, AgentCloud remembers its past failures. If a mitigation strategy fails, the Planning Agent consults the Memory Agent and dynamically pivots to a new strategy in the future based on cosine similarity of the incident context.
• Fully Local & Air-gapped: The entire pipeline operates on CPU using Hugging Face Transformers for diagnosis and a local Qwen2.5-0.5B-Instruct GGUF model (llama.cpp) for planning — no GPU, no cloud API required by default. An OpenAI-compatible backend remains available as an opt-in alternative.
• Ablation & Evaluation Suite: Includes a rigorous evaluation framework to measure Mean Time To Recovery (MTTR), tool accuracy, and success rates.

System Architecture

The project is structured into a sequential agentic pipeline that operates continuously:

Log Generator -> Monitoring -> Diagnosis -> Planning -> Execution -> Memory

1. Simulator (Log Generator): Generates real-time mock cloud metric logs.
2. Monitoring Agent: Scans continuous streaming logs using time-windowed thresholds to instantly alert on anomalous behavior.
3. Diagnosis Agent: The core Hybrid AI router. Passes alerts to the DistilBERT ML model. If confidence is insufficient, prompts the configured LLM backend (local Qwen by default). If the LLM is unavailable, utilizes a deterministic symbolic fallback algorithm.
4. Planning Agent: Decides the mitigation route (restart_service, block_ip, isolate_server, reroute_traffic). Prefers a memory-recalled strategy from a similar past incident; otherwise consults the local Qwen2.5-0.5B-Instruct model for an adaptive plan; any missing/invalid/unsafe output falls through to a deterministic rule-based planner.
5. Execution Agent: Simulates the system commands internally against a mock SimState.
6. Memory Agent: Stores a long-term ledger of incidents and actions in SQLite, and indexes successful incident resolutions into a FAISS semantic vector memory. This allows the system to demonstrate "Failure -> Recovery" learning and retrieve similar past contexts using cosine similarity.

Supported Scenarios

You can run the system to test specific failure cases by changing the --scenario flag. The simulator supports:

• normal: Standard cloud operation with fluctuating metrics
• overload: Massive CPU and traffic spikes triggering node load anomalies
• intrusion: Coordinated IP brute-force and failed auth bursts
• crash: Unexpected service segmentation faults and offline nodes
• mixed: (Default) Rotates randomly through all of the above

Dataset & ML Classifier

The Diagnosis Agent utilizes a distilbert-base-uncased model fine-tuned for Sequence Classification to categorize anomalies into four labels: normal, overload, crash, and intrusion.

The Dataset

The model was trained using the Loghub HDFS (Hadoop Distributed File System) Dataset. Download: https://www.kaggle.com/datasets/ayenuryrr/loghub-hdfs-hadoop-distributed-file-system-data

To process the raw 1.5GB hdfs.log file into a lightweight format suitable for Hugging Face Transformers, we use the convert.py script. It extracts log levels, services, and uses keyword heuristics to tag the raw data into a train.jsonl file.

Installation & Setup

AgentCloud requires the PyTorch and Hugging Face ecosystem.

# Clone the repository
git clone https://github.com/yourusername/AgentCloud.git
cd AgentCloud

# Set up a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install the Machine Learning Stack
pip install -r requirements.txt

Training the Model

By default, the repository excludes the massive ML model weights. You must train the model locally using the provided scripts.

1. Download the hdfs.log from Kaggle and place it in agentcloud/data/raw/hdfs.log.
2. Convert the raw logs into the training dataset:

python3 agentcloud/convert.py

3. Fine-tune the DistilBERT model. This will output the trained weights to agentcloud/models/cloudsec-model/:

python3 train_model.py

Quickstart

Run a completely simulated autonomous cycle using the ML model and deterministic fallback:

python3 -m agentcloud.main --scenario mixed --max-events 5

Live simulator logs will be written to logs/cloud.jsonl in JSONL format. Long-term semantic memory and past resolutions are stored in database.db (SQLite).

If you want to view the raw JSON line logs in real-time as they are generated by the simulator, add the --show-logs flag:

python3 -m agentcloud.main --scenario mixed --max-events 5 --show-logs

Generative LLM Integration

Both the DiagnosisAgent (low-confidence fallback) and the PlanningAgent (adaptive remediation) can consult a reasoning LLM, selected via AGENTCLOUD_LLM_BACKEND. Every LLM-generated output is still validated against the same symbolic safety schema before use, so a missing/misbehaving model degrades gracefully to the deterministic fallback rather than failing the run.

Local Qwen (default, CPU-only, no API key)

Weights are downloaded once (Qwen2.5-0.5B-Instruct, GGUF, Q4_K_M quantization, ~491MB) from the official Qwen/Qwen2.5-0.5B-Instruct-GGUF repo and cached under ~/.cache/huggingface:

pip install llama-cpp-python huggingface_hub  # already in requirements.txt
python3 -m agentcloud.main --scenario mixed --max-events 5

Useful overrides:

• AGENTCLOUD_LOCAL_MODEL_PATH: load a .gguf file already on disk instead of downloading.
• AGENTCLOUD_QWEN_GGUF_FILE: pick a different quantization from the HF repo (default *q4_k_m.gguf).
• AGENTCLOUD_LLM_THREADS: CPU thread count for inference (default 4).
• AGENTCLOUD_FORCE_NO_LLM=1: disable LLM consultation everywhere (diagnosis and planning), forcing pure symbolic/ML behavior — used by the benchmark runner's --no-llm flag.

OpenAI-compatible (opt-in)

If you prefer a heavier, hosted reasoning engine instead:

export AGENTCLOUD_LLM_BACKEND="openai"
export OPENAI_API_KEY="sk-..."
export AGENTCLOUD_LLM_MODEL="gpt-4o-mini"
python3 -m agentcloud.main --scenario mixed --max-events 5

Evaluation & Ablations

AgentCloud includes a robust research suite to measure the effectiveness of its cognitive components.

Memory Learning Demo

Force an incident to fail its first mitigation, triggering the Memory Agent to log the failure. On the second occurrence, watch the system intelligently pivot its strategy to achieve recovery:

python3 -m agentcloud.main --demo-learning --tick 0.02 --trace-file logs/trace.jsonl

Ablation Studies

Turn off specific cognitive components to calculate success differentials:

# Disable the Memory Agent (system will repeat past mistakes)
python3 -m agentcloud.main --scenario mixed --max-events 5 --no-memory

# Disable the Planning Agent (defaults to an imperative fallback map)
python3 -m agentcloud.main --scenario mixed --max-events 5 --no-planning

Evaluation Runner

Run massive scenarios across multi-incident episodes to automatically calculate Success Rate, Tool Accuracy, and Mean Time To Recovery (MTTR):

python3 -m agentcloud.evaluation --scenario mixed --episodes 10 --max-incidents 5

# Compare against ablations
python3 -m agentcloud.evaluation --scenario mixed --episodes 10 --max-incidents 5 --no-memory

Out of scope (current research artifact)

The current codebase is a single-process, CPU-only research artifact. The following items are deliberately not built in this snapshot:

• Kubernetes Execution Layer: Replace the simulated SimState Execution Agent with actual Kubernetes API bindings to autonomously restart failing real-world pods.
• Dashboard Web UI: Develop a lightweight FastAPI + React dashboard to visualize the real-time agentic trace graph and confidence scores.

Other roadmap items (RAG over Markdown playbooks, multi-provider LLM abstraction beyond Qwen/OpenAI) have been intentionally omitted from this artifact. See docs/ROADMAP.md for the full forward-looking plan.

Project Outputs

• Logs: Continuous logs trace out to logs/cloud.jsonl.
• Memory Database: Stored autonomously generated incidents are recorded permanently in database.db.
• Trace Output: Run any main.py loop with --trace-file logs/trace.jsonl to append extremely verbose JSON trace records for the agentic thought-loop.