WebSearch AI

The Philosophy: Engineered for Reuse

WebSearch AI stands as a testament to the principles of Clean Architecture applied to Python tooling. Born from the need to standardize web search capabilities across the DeepPlants ecosystem, it was designed from day one to be a modular, maintainable, and high-performance library.

We built this because existing tools were often tightly coupled scripts that were hard to test and harder to extend.

Architectural Design

The library is structured around a strict separation of concerns, ensuring that the "what" (business logic) is decoupled from the "how" (implementation details).

Core Components

• SearchManager: The high-level orchestrator that coordinates the search and retrieval process.
• SearchEngine: An abstract interface for search providers. We implemented multiple backends (DuckDuckGo, Google Custom Search, etc.) that can be swapped via configuration without changing client code.
• HTTPFetcher: A dedicated module for network operations, handling retries, user-agent rotation, and connection pooling.
• ContentParser: The component responsible for turning raw HTML into clean text.

High-Performance Async Pipeline

Web scraping is inherently I/O bound. Waiting for HTTP requests sequentially is a waste of resources. WebSearch AI leverages Python's asyncio to execute searches and content fetches in parallel. The pipeline allows for:

• Concurrent Fetching: Retrieving dozens of pages simultaneously.
• Non-blocking Operations: Ensuring that slow server responses don't freeze the entire application.

Intelligent Content Extraction

Getting the HTML is easy; getting the content is hard. We integrated Trafilatura, a state-of-the-art library for text extraction.

• Noise Reduction: The library automatically strips ads, navigation menus, and footers.
• Metadata Extraction: It preserves critical metadata like publication dates and authors.
• Custom Filters: We implemented a filter system in websearch_ai.filters to further clean the output, ensuring that the text passed to downstream LLMs is dense with information and free of token-wasting boilerplate.

LLM Summarization Layer

The library includes an optional summarization layer. By injecting a dependency on an LLM provider, WebSearch AI can transform a list of search results into a coherent, cited answer. This allows developers to drop a "Google Search" capability into their agents with a single function call, receiving back a synthesized answer rather than a list of blue links.