`
- inside the `qbsearch-input`. I will try clicking [653] first.
- page_context_url: https://github.com/
- page_context_title: "GitHub \xB7 Change is constant. GitHub keeps you ahead. \xB7\
- \ GitHub"
-- description: Press Enter key
- output: null
- cssSelector: null
- xpath: null
- elementTag: null
- elementHash: null
- selectorStrategies: null
- target_text: element
- container_hint: null
- position_hint: null
- interaction_type: null
- type: key_press
- key: Enter
-- description: Scroll down 1.0 pages
- output: null
- type: scroll
- scrollX: 0
- scrollY: 800
- agent_reasoning: The previous step of inputting "browser-use" and pressing Enter
- was successful, and I am now on the search results page. The immediate goal is
- to find and click the "browser-use" repository link. Since the search was executed,
- the current page should display the results. I need to scroll down to see the
- search results or use `find_text` to locate the repository link. Since the current
- view is still showing the search modal/overlay, I will scroll down to see the
- results page content.
- page_context_url: https://github.com/
-- description: Scroll down 1.0 pages
- output: null
- type: scroll
- scrollX: 0
- scrollY: 800
- agent_reasoning: The previous step of scrolling down was performed because the search
- results were expected to be below the current view, which was still showing the
- search modal/overlay after pressing Enter. However, the current browser state
- still shows the search overlay elements (like the query builder and suggestions)
- at the top, and the main content of the search results page is still below. I
- need to scroll further down to see the search results, or the search result page
- might not have loaded correctly after the Enter key, or the scroll didn't move
- the viewport enough relative to the search results. I will try scrolling down
- again to ensure the search results are visible. The page info shows 11.1 pages
- below, so scrolling is necessary.
- page_context_url: https://github.com/
-- description: Scroll down 5.0 pages
- output: null
- type: scroll
- scrollX: 0
- scrollY: 4000
- agent_reasoning: The previous two steps involved scrolling down, but the browser
- state still shows the search overlay prominently, and the search results are not
- yet visible. I need to scroll down significantly more to get past the initial
- landing page content and see the search results page content. The page info suggests
- 11.1 pages below, so I will scroll down by a larger amount (e.g., 5 pages) to
- ensure I reach the search results.
- page_context_url: https://github.com/
-- description: Scroll down 10.0 pages
- output: null
- type: scroll
- scrollX: 0
- scrollY: 8000
- agent_reasoning: The previous steps involved navigating to GitHub, searching for
- "browser-use", and repeatedly scrolling down. Despite three attempts at scrolling,
- the browser state still shows the search overlay and the initial landing page
- content, not the actual search results page content. This suggests that either
- the search was not executed correctly, or the search results are very far down,
- or the page structure is confusing and the search results are hidden. Since I
- pressed Enter after inputting the text, the search should have executed. I will
- try scrolling down one more time, but this time I will use `scroll(down=True,
- pages=10)` to ensure I reach the bottom of the page and see the search results.
- If that fails, I will suspect the search execution failed and try to find a "Search"
- button to click instead of relying on the Enter key, or re-evaluate the search
- input. Given the page info shows 11.1 pages below, scrolling to the bottom is
- the next logical step to confirm the search results are loaded.
- page_context_url: https://github.com/
-- description: Click on 'browser-use, Search all of GitHub' (News/Press Releases)
- output: null
- cssSelector: ''
- xpath: html/body/div[1]/div[3]/header/div/div[2]/div/div/qbsearch-input/div[1]/div/modal-dialog/div/div/div/form/query-builder/div[1]/div[2]/ul/li/ul/li
- elementTag: li
- elementHash: abaa6b85ca
- selectorStrategies: null
- target_text: browser-use, Search all of GitHub
- container_hint: null
- position_hint: null
- interaction_type: null
- type: click
- agent_reasoning: 'Despite multiple attempts to scroll down, the search results page
- content is still not visible, and the search overlay remains prominent. This strongly
- suggests that either the search was not successfully executed (i.e., pressing
- Enter in the shadow DOM input didn''t trigger the navigation), or the search results
- are not on the current page but require a click on a specific search result link
- within the overlay to navigate to the results page.
-
-
- Looking at the current interactive elements, I see a suggestion: `[10021]
`. This list item represents the search result/suggestion
- "browser-use, Search all of GitHub". I should click this element to execute the
- search and navigate to the results page.'
- page_context_url: https://github.com/
- page_context_title: "GitHub \xB7 Change is constant. GitHub keeps you ahead. \xB7\
- \ GitHub"
-- description: "Click on Browser Use (on Repository search results \xB7 GitHub)"
- output: null
- cssSelector: ''
- xpath: html/body/div[1]/div[4]/main/react-app/div/div/div[1]/div/div/div[2]/div/div/div[1]/div[4]/div/div/div[1]/div/div[1]/h3/div/div[2]/a
- elementTag: a
- elementHash: 7a2302537b
- selectorStrategies: null
- target_text: Browser Use
- container_hint: null
- position_hint: null
- interaction_type: null
- type: click
- agent_reasoning: The previous step of clicking the search suggestion successfully
- navigated to the repository search results page for "browser-use". I can confirm
- this by the current URL and the page title. The next immediate goal is to click
- on the correct repository link, which is "browser-use/browser-use" based on the
- user request and the search results list. I see element [14628] which is an `
`
- tag with the text "browser-use/ browser-use". This is the target repository. After
- clicking, I will be on the repository page and can extract the star count.
- page_context_url: https://github.com/
- page_context_title: "Repository search results \xB7 GitHub"
-input_schema: []
diff --git a/workflows/storage/workflows/bfaf87d9-f5d9-4245-b210-5bfdf302a9b2.workflow.yaml b/workflows/storage/workflows/bfaf87d9-f5d9-4245-b210-5bfdf302a9b2.workflow.yaml
deleted file mode 100644
index eee23dc3..00000000
--- a/workflows/storage/workflows/bfaf87d9-f5d9-4245-b210-5bfdf302a9b2.workflow.yaml
+++ /dev/null
@@ -1,135 +0,0 @@
-workflow_analysis: null
-name: '
-
- Go to GitHub, search for the browser-use repository, click on it,
-
- and extract the star count.
-
- '
-description: "Workflow for: \nGo to GitHub, search for the browser-use repository,\
- \ click on it,\nand extract the star count.\n"
-version: 1.0.0
-steps:
-- description: Navigate to https://github.com
- output: null
- type: navigation
- url: https://github.com
- agent_reasoning: The first step is to navigate to GitHub. Since the current page
- is empty, I will use the `navigate` action. The ultimate goal is to find the "browser-use"
- repository and extract its star count.
-- description: "Click on 'Search or jump to\u2026' (Filings section)"
- output: null
- cssSelector: ''
- xpath: html/body/div[1]/div[3]/header/div/div[2]/div/div/qbsearch-input/div[1]/button
- elementTag: button
- elementHash: e53e5f0519
- selectorStrategies: null
- target_text: "Search or jump to\u2026"
- container_hint: null
- position_hint: null
- interaction_type: null
- type: click
- agent_reasoning: "The previous step of waiting for 3 seconds was successful, and\
- \ the page content is now visible. The immediate goal is to find the search bar\
- \ and input \"browser-use\". I see an interactive element `[520]
+ if (!labelText) {
+ const parentCell = el.closest('td, th');
+ if (parentCell) {
+ const prevCell = parentCell.previousElementSibling;
+ if (prevCell && (prevCell.tagName === 'TD' || prevCell.tagName === 'TH')) {
+ const cellText = prevCell.textContent?.trim() || '';
+ // Only use if it looks like a label (short text, ends with colon, etc.)
+ if (cellText && cellText.length < 50) {
+ labelText = cellText.replace(/[:ļ¼]\s*$/, '').trim();
+ }
+ }
+ }
+ }
+
return labelText;
} catch (error) {
debugMessage('Error in safeGetLabelText', error.message);
@@ -838,6 +853,7 @@ async def extract_semantic_mapping(self, page: 'Page') -> Dict[str, Dict]:
'element_type': element_type,
'deterministic_id': element_id,
'original_text': text,
+ 'label_text': element_info.get('label_text', ''), # IMPORTANT: Include label text for input field matching
'dom_path': element_info.get('dom_path', ''),
'container_context': element_info.get('container_context', {}),
'sibling_context': element_info.get('sibling_context', {}),
diff --git a/workflows/workflow_use/workflow/service.py b/workflows/workflow_use/workflow/service.py
index 3b70c572..284e06c0 100644
--- a/workflows/workflow_use/workflow/service.py
+++ b/workflows/workflow_use/workflow/service.py
@@ -49,6 +49,9 @@ def __init__(
page_extraction_llm: BaseChatModel | None = None,
fallback_to_agent: bool = True,
use_cloud: bool = False,
+ debug: bool = False,
+ debug_log_folder: str | Path | None = None,
+ step_wait_time: float = 0.1,
) -> None:
"""Initialize a new Workflow instance from a schema object.
@@ -59,6 +62,9 @@ def __init__(
llm: Optional language model for fallback agent functionality
fallback_to_agent: Whether to fall back to agent-based execution on step failure
use_cloud: Whether to use browser-use cloud browser service instead of local browser
+ debug: Whether to enable debug mode (captures screenshots for each step)
+ debug_log_folder: Custom folder path for debug logs and screenshots (default: ./logs/workflow_debug)
+ step_wait_time: Time to wait between steps in seconds (default: 0.1)
Raises:
ValueError: If the workflow schema is invalid (though Pydantic handles most).
@@ -77,6 +83,13 @@ def __init__(
self.fallback_to_agent = fallback_to_agent
+ # Debug mode settings
+ self.debug = debug
+ self.debug_log_folder = Path(debug_log_folder) if debug_log_folder else Path('./logs/workflow_debug')
+
+ # Step execution settings
+ self.step_wait_time = step_wait_time
+
# Initialize multi-strategy element finder
self.element_finder = ElementFinder()
@@ -96,6 +109,9 @@ def load_from_file(
browser: Browser | None = None,
page_extraction_llm: BaseChatModel | None = None,
use_cloud: bool = False,
+ debug: bool = False,
+ debug_log_folder: str | Path | None = None,
+ step_wait_time: float = 0.1,
) -> Workflow:
"""Load a workflow from a file."""
with open(file_path, 'r', encoding='utf-8') as f:
@@ -108,6 +124,9 @@ def load_from_file(
llm=llm,
page_extraction_llm=page_extraction_llm,
use_cloud=use_cloud,
+ debug=debug,
+ debug_log_folder=debug_log_folder,
+ step_wait_time=step_wait_time,
)
# --- Runners ---
@@ -297,6 +316,73 @@ async def _run_agent_step(self, step: AgenticWorkflowStep, step_index: int) -> A
return await agent.run()
+ async def _run_extraction_step(self, step, step_index: int) -> ActionResult:
+ """
+ Lightweight extraction that uses LLM directly without spinning up an agent.
+ Much faster and cheaper than using a full agent for simple page content extraction.
+ """
+ from browser_use.agent.views import ActionResult
+
+ # Get extraction goal
+ extraction_goal = ''
+ if hasattr(step, 'goal'):
+ extraction_goal = step.goal
+ elif hasattr(step, 'extractionGoal'):
+ extraction_goal = step.extractionGoal
+ else:
+ extraction_goal = 'Extract information from the page'
+
+ # Get current page content using markdown extraction
+ page = await self.browser.get_current_page()
+ page_text, _ = await page._extract_clean_markdown()
+ page_url = await page.get_url()
+
+ # Use page_extraction_llm if available, otherwise fall back to main llm
+ extraction_llm = self.page_extraction_llm or self.llm
+
+ # Build extraction prompt
+ # Limit page text to avoid token limits
+ truncated_page_text = page_text[:10000] if page_text else ''
+
+ extraction_prompt = f"""You are extracting information from a web page.
+
+Page URL: {page_url}
+
+Extraction Goal:
+{extraction_goal}
+
+Page Content:
+{truncated_page_text}
+
+Instructions:
+- Extract the requested information accurately
+- Return ONLY the extracted data, no explanations
+- If the information is not found, return an empty string or appropriate null value
+- Format the output as requested in the extraction goal
+
+Extracted Information:"""
+
+ # Call LLM directly
+ messages = [UserMessage(content=extraction_prompt)]
+ response = await extraction_llm.ainvoke(messages)
+
+ # Extract the text content from response
+ # ainvoke returns ChatInvokeCompletion with a 'completion' attribute
+ extracted_content = ''
+ if hasattr(response, 'completion'):
+ extracted_content = response.completion
+ elif isinstance(response, str):
+ extracted_content = response
+
+ logger.info(f'Extracted content: {extracted_content[:200]}...')
+
+ # Return as ActionResult
+ return ActionResult(
+ is_done=False,
+ extracted_content=extracted_content,
+ include_in_memory=True,
+ )
+
# async def _fallback_to_agent(
# self,
# step_resolved: WorkflowStep,
@@ -516,27 +602,11 @@ async def _execute_step(self, step_index: int, step_resolved: WorkflowStep) -> A
action_name = step_resolved.type or '[No action specified]'
- # Extraction steps ALWAYS use agent/LLM, even in deterministic mode
+ # Extraction steps use lightweight LLM extraction (no agent needed)
is_extraction_step = action_name in ['extract', 'extract_page_content']
if is_extraction_step:
- logger.info(f'Step {step_index + 1}: Extraction step detected - using agent with LLM')
- # Convert to agent step
- from workflow_use.schema.views import AgentTaskWorkflowStep
-
- # Create agent task based on extraction step type
- if action_name == 'extract':
- task = getattr(step_resolved, 'extractionGoal', 'Extract information from the page')
- else: # extract_page_content
- task = getattr(step_resolved, 'goal', 'Extract text from the page')
-
- # Create an AgentTaskWorkflowStep on the fly
- agent_step = AgentTaskWorkflowStep(
- type='agent',
- task=task,
- description=step_resolved.description if hasattr(step_resolved, 'description') else None,
- output=step_resolved.output if hasattr(step_resolved, 'output') else None,
- )
- result = await self._run_agent_step(agent_step, step_index)
+ logger.info(f'Step {step_index + 1}: Extraction step detected - using lightweight LLM extraction')
+ result = await self._run_extraction_step(step_resolved, step_index)
# Check if this is a selector step without cssSelector - use semantic execution
elif action_name in ['click', 'input', 'key_press', 'select_change']:
@@ -702,6 +772,47 @@ async def run_step(self, step_index: int, inputs: dict[str, Any] | None = None):
# await self.browser.close() # <-- Commented out for testing
return result
+ async def _capture_debug_screenshot(self, step_index: int, step_description: str, prefix: str = '') -> None:
+ """Capture a screenshot for debugging purposes.
+
+ Args:
+ step_index: The index of the current step
+ step_description: Description of the step for the filename
+ prefix: Optional prefix for the filename (e.g., 'before', 'after', 'error')
+ """
+ if not self.debug:
+ return
+
+ try:
+ # Create debug log folder if it doesn't exist
+ self.debug_log_folder.mkdir(parents=True, exist_ok=True)
+
+ # Clean step description for filename (remove special characters)
+ import re
+
+ clean_description = re.sub(r'[^\w\s-]', '', step_description)
+ clean_description = re.sub(r'[-\s]+', '_', clean_description)
+ clean_description = clean_description[:50] # Limit length
+
+ # Create timestamp for uniqueness
+ from datetime import datetime
+
+ timestamp = datetime.now().strftime('%Y%m%d_%H%M%S')
+
+ # Build filename
+ prefix_str = f'{prefix}_' if prefix else ''
+ filename = f'step_{step_index + 1:02d}_{prefix_str}{clean_description}_{timestamp}.png'
+ screenshot_path = self.debug_log_folder / filename
+
+ # Capture screenshot
+ page = await self.browser.get_current_page()
+ await page.screenshot(path=str(screenshot_path), full_page=True)
+
+ logger.info(f'šø Debug screenshot saved: {screenshot_path}')
+
+ except Exception as e:
+ logger.warning(f'Failed to capture debug screenshot: {e}')
+
async def run(
self,
inputs: dict[str, Any] | None = None,
@@ -730,10 +841,24 @@ async def run(
results: List[ActionResult | AgentHistoryList] = []
+ # Log debug mode status
+ if self.debug:
+ logger.info(f'š Debug mode enabled - screenshots will be saved to: {self.debug_log_folder}')
+ # Create debug folder at the start
+ self.debug_log_folder.mkdir(parents=True, exist_ok=True)
+
+ # Log step wait time if configured
+ if self.step_wait_time > 0.1: # Only log if it's been changed from default
+ logger.info(f'ā±ļø Step wait time configured: {self.step_wait_time}s between steps')
+
await self.browser.start()
try:
for step_index, step_dict in enumerate(self.schema.steps): # self.steps now holds dictionaries
- await asyncio.sleep(0.1)
+ # Wait between steps (configurable)
+ if step_index > 0: # Don't wait before the first step
+ await asyncio.sleep(self.step_wait_time)
+ if self.step_wait_time > 0:
+ logger.debug(f'Waited {self.step_wait_time}s between steps')
# Check if cancellation was requested
if cancel_event and cancel_event.is_set():
@@ -743,11 +868,23 @@ async def run(
# Use description from the step dictionary
step_description = step_dict.description or 'No description provided'
logger.info(f'--- Running Step {step_index + 1}/{len(self.schema.steps)} -- {step_description} ---')
+
+ # Capture screenshot before step execution (if debug enabled)
+ await self._capture_debug_screenshot(step_index, step_description, prefix='before')
+
# Resolve placeholders using the current context (works on the dictionary)
step_resolved = self._resolve_placeholders(step_dict)
# Execute step using the unified _execute_step method
- result = await self._execute_step(step_index, step_resolved)
+ try:
+ result = await self._execute_step(step_index, step_resolved)
+
+ # Capture screenshot after successful step execution (if debug enabled)
+ await self._capture_debug_screenshot(step_index, step_description, prefix='after')
+ except Exception as e:
+ # Capture screenshot on error (if debug enabled)
+ await self._capture_debug_screenshot(step_index, step_description, prefix='error')
+ raise # Re-raise the exception after capturing screenshot
results.append(result)
# Persist outputs using the resolved step dictionary