Let me paint you a picture. It's 11 PM, I'm staring at my screen, and I need to research "Docker vs Podman" for a blog post. I've got 15 tabs open, my brain is fried, and I'm just... tired. Sound familiar?
That night, instead of actually doing the research, I did what any reasonable developer would do — I spent the next few hours building a tool to do it for me.
The result? An AI research assistant that takes any topic, searches the web, reads the articles, and writes me a nice little report. All in about 30 seconds flat.
Today, I'm going to show you exactly how I built it. No fluff, no over-engineering — just a clean, minimal Python script that actually works.
What Does This Thing Actually Do?
Before we dive into code, let me show you what we're building. You type something like this:
"Docker vs Podman"And you get back a report.md file that looks something like:
A proper title and timestamp
A well-structured synthesis of what different sources say
A numbered list of all the sources it actually used
The whole pipeline runs in five steps: search, extract, summarize, combine, and save. That's it.
The Architecture (It's Simpler Than You Think)
Here's the workflow in plain English:
You give it a topic — anything from "React vs Vue 2024" to "best indoor plants for low light"
It searches the web — using DuckDuckGo (no API keys needed for search!)
It grabs the top articles — extracts the actual content, not the ads and sidebars
It summarizes each source — using Google's Gemini AI
It writes a final report — combining all the summaries into one coherent document
The entire codebase is under 300 lines of Python split across five files. Each file does exactly one thing. No crazy abstractions, no frameworks you need to learn.
Step 1: Searching Without Breaking a Sweat
The first thing we need is search. Now, you could use Google Custom Search API, but that requires API keys and billing setup. I wanted something that works immediately.
DuckDuckGo to the rescue. There's a Python library called ddgs that gives you programmatic access to DuckDuckGo's search results. No API key required, no rate limiting headaches.
"""
search.py
Handles web search via DuckDuckGo and returns the top N result URLs
for a given research topic.
"""
from __future__ import annotations
import logging
from typing import List
from ddgs import DDGS
logger = logging.getLogger(__name__)
def search_topic(topic: str, num_results: int = 3) -> List[str]:
"""
Search DuckDuckGo for a topic and return a list of result URLs.
Args:
topic: The research topic / query string.
num_results: How many URLs to return (default 3).
Returns:
A list of URL strings. May contain fewer than num_results
entries if the search doesn't return enough results.
"""
if not topic or not topic.strip():
raise ValueError("Search topic must not be empty.")
urls: List[str] = []
try:
with DDGS() as ddgs:
results = ddgs.text(topic, max_results=num_results)
for result in results:
url = result.get("href") or result.get("url")
if url:
urls.append(url)
if len(urls) >= num_results:
break
except Exception as exc: # noqa: BLE001 - surface a clean error upstream
logger.error("Search failed for topic '%s': %s", topic, exc)
raise RuntimeError(f"Web search failed: {exc}") from exc
if not urls:
raise RuntimeError(f"No search results found for topic: '{topic}'")
logger.info("Found %d URLs for topic '%s'", len(urls), topic)
return urls
if __name__ == "__main__":
# Quick manual test: python search.py "Docker vs Podman"
import sys
logging.basicConfig(level=logging.INFO)
query = " ".join(sys.argv[1:]) or "Docker vs Podman"
for i, u in enumerate(search_topic(query), start=1):
print(f"{i}. {u}")The function takes a topic and a number of results (default is 3), hits DuckDuckGo, and returns a list of URLs. That's it. If the search fails or returns nothing useful, it raises an error early rather than failing silently later.
Pro tip: I set the default to 3 results because that's usually enough to get a good picture of a topic without hitting API rate limits or making the report too long.
Step 2: Actually Reading Web Pages (The Tricky Part)
Getting a URL is easy. Getting clean, readable text from that URL? That's where things get interesting.
Web pages are messy. They have navigation bars, sidebars, cookie notices, ads, related posts, comment sections, and about seventeen different tracking scripts. What you actually want is the article text — just the content.
This is where trafilatura comes in clutch. It's a Python library specifically designed for extracting article text from HTML pages. It's not just stripping HTML tags — it's actually smart enough to identify where the main content is.
"""
scraper.py
Fetches web pages and extracts clean, readable article text using
Trafilatura, with `requests` used for the raw HTTP fetch.
"""
from __future__ import annotations
import logging
from typing import Optional
import requests
import trafilatura
logger = logging.getLogger(__name__)
DEFAULT_HEADERS = {
"User-Agent": (
"Mozilla/5.0 (compatible; AIResearchAssistant/1.0; "
"+https://example.com/bot)"
)
}
def fetch_html(url: str, timeout: int = 15) -> Optional[str]:
"""
Download raw HTML for a URL. Returns None on failure instead of
raising, since a single bad URL shouldn't kill the whole pipeline.
"""
try:
response = requests.get(
url, headers=DEFAULT_HEADERS, timeout=timeout, allow_redirects=True
)
response.raise_for_status()
return response.text
except requests.RequestException as exc:
logger.warning("Failed to fetch %s: %s", url, exc)
return None
def extract_text(url: str, timeout: int = 15) -> Optional[str]:
"""
Fetch a URL and extract its main article text.
Tries trafilatura's own downloader first (it handles encoding/
compression well), then falls back to `requests` + trafilatura's
extractor if that fails.
Returns:
The extracted text, or None if extraction wasn't possible.
"""
downloaded = trafilatura.fetch_url(url)
if not downloaded:
downloaded = fetch_html(url, timeout=timeout)
if not downloaded:
logger.warning("Could not retrieve content for %s", url)
return None
text = trafilatura.extract(
downloaded,
include_comments=False,
include_tables=False,
favor_precision=True,
)
if not text or not text.strip():
logger.warning("No extractable text found at %s", url)
return None
return text.strip()
def extract_texts(urls: list[str], timeout: int = 15) -> dict[str, str]:
"""
Extract text for multiple URLs.
Returns:
A dict mapping URL -> extracted text, skipping URLs that
failed to yield usable content.
"""
results: dict[str, str] = {}
for url in urls:
text = extract_text(url, timeout=timeout)
if text:
results[url] = text
else:
logger.info("Skipping %s (no usable text extracted)", url)
return results
if __name__ == "__main__":
import sys
logging.basicConfig(level=logging.INFO)
test_url = sys.argv[1] if len(sys.argv) > 1 else "https://example.com"
content = extract_text(test_url)
print(content[:1000] if content else "No text extracted.")Here's my approach: I try trafilatura's built-in URL fetcher first (it handles encodings and compression well), and if that fails, I fall back to requests with a custom user agent. This gives us the best of both worlds.
I also made a deliberate design choice: if a page fails, we skip it. We don't crash the whole pipeline. Your research shouldn't die just because one random blog decided to block bots.
Step 3: Making Gemini Do the Hard Work
Now comes the fun part — actually understanding what these articles say.
I'm using Google's Gemini API with the Flash Lite model. Why Flash Lite? Because it's fast, cheap, and more than capable enough for summarizing articles. We're not writing poetry here; we're extracting key points.
For each URL we successfully extracted, we send the article text to Gemini with a prompt that goes something like: "Here's an article about [topic], give me the key points in a concise summary."
"""
gemini.py
Thin wrapper around the Gemini API for two jobs:
1. Summarizing a single page's extracted text.
2. Combining several summaries into one final research report.
Uses the new `google-genai` SDK (genai.Client), not the older
`google-generativeai` package.
"""
from __future__ import annotations
import logging
import os
from typing import List
from google import genai
from dotenv import load_dotenv
logger = logging.getLogger(__name__)
load_dotenv()
_MODEL_NAME = os.getenv("GEMINI_MODEL", "gemini-2.5-flash")
_API_KEY = os.getenv("GEMINI_API_KEY")
_client: genai.Client | None = None
def _get_client() -> genai.Client:
"""Lazily create and cache the Gemini Developer API client."""
global _client
if _client is not None:
return _client
if not _API_KEY:
raise RuntimeError(
"GEMINI_API_KEY is not set. Copy .env.example to .env and add your key."
)
_client = genai.Client(api_key=_API_KEY)
return _client
def summarize_page(url: str, text: str, topic: str, max_chars: int = 12000) -> str:
"""
Summarize a single page's extracted text in the context of the
overall research topic.
Args:
url: Source URL (included for context/attribution in the prompt).
text: Raw extracted article text.
topic: The overall research topic, to keep the summary focused.
max_chars: Truncate very long articles before sending to the API.
Returns:
A concise summary string.
"""
trimmed = text[:max_chars]
prompt = f"""You are a research assistant. Summarize the following web page
content in the context of this research topic: "{topic}"
Source URL: {url}
Instructions:
- Write 3-6 concise bullet points capturing the key facts, claims, and data relevant to the topic.
- Ignore navigation text, ads, or unrelated content.
- Do not invent information that isn't in the text.
- Keep it factual and neutral in tone.
Page content:
\"\"\"
{trimmed}
\"\"\"
"""
try:
client = _get_client()
response = client.models.generate_content(
model=_MODEL_NAME,
contents=prompt,
)
summary = (response.text or "").strip()
except Exception as exc: # noqa: BLE001
logger.error("Gemini summarization failed for %s: %s", url, exc)
raise RuntimeError(f"Gemini summarization failed for {url}: {exc}") from exc
if not summary:
raise RuntimeError(f"Gemini returned an empty summary for {url}")
return summary
def generate_final_report(topic: str, summaries: List[dict]) -> str:
"""
Combine per-page summaries into one cohesive markdown research report.
Args:
topic: The overall research topic.
summaries: List of dicts like {"url": str, "summary": str}.
Returns:
A full markdown-formatted report as a string (without the
surrounding file boilerplate; report.py handles that).
"""
if not summaries:
raise ValueError("Cannot generate a report with zero summaries.")
combined_sources = "\n\n".join(
f"Source: {item['url']}\nSummary:\n{item['summary']}" for item in summaries
)
prompt = f"""You are a research assistant producing a final report on the topic:
"{topic}"
Below are summaries gathered from {len(summaries)} different web sources.
Synthesize them into a single, well-organized research report.
Requirements:
- Start with a short overview paragraph of the topic.
- Organize the body into clear sections with headings where it makes sense
(e.g. by subtopic, or by comparing/contrasting viewpoints).
- Note any agreements or contradictions between sources.
- End with a short "Key Takeaways" bulleted list.
- Write in clear, neutral, professional prose. Do not fabricate facts
beyond what the summaries support.
- Do not include a sources/references section yourself; that will be
appended separately.
Summaries:
{combined_sources}
"""
try:
client = _get_client()
response = client.models.generate_content(
model=_MODEL_NAME,
contents=prompt,
)
report_text = (response.text or "").strip()
except Exception as exc: # noqa: BLE001
logger.error("Gemini final report generation failed: %s", exc)
raise RuntimeError(f"Gemini final report generation failed: {exc}") from exc
if not report_text:
raise RuntimeError("Gemini returned an empty final report.")
return report_textNotice I'm passing the original research topic to the summarization prompt. This isn't just a generic "summarize this" — it's "summarize this in the context of what we're researching." That small detail makes the summaries way more useful.
Step 4: Saving a Beautiful Report
The final step is formatting everything into a nice markdown file. This is honestly the simplest part of the whole thing.
"""
report.py
Assembles the final report.md file: metadata header, the Gemini-generated
report body, and a sources list.
"""
from __future__ import annotations
import datetime
import logging
import os
from typing import List
logger = logging.getLogger(__name__)
def build_markdown(topic: str, report_body: str, sources: List[str]) -> str:
"""
Assemble the full markdown document as a string.
"""
timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M")
sources_section = "\n".join(f"{i}. {url}" for i, url in enumerate(sources, start=1))
return f"""# Research Report: {topic}
*Generated on {timestamp}*
---
{report_body}
---
## Sources
{sources_section}
"""
def save_report(
topic: str,
report_body: str,
sources: List[str],
output_path: str = "report.md",
) -> str:
"""
Build and write the markdown report to disk.
Returns:
The absolute path of the written file.
"""
markdown = build_markdown(topic, report_body, sources)
abs_path = os.path.abspath(output_path)
with open(abs_path, "w", encoding="utf-8") as f:
f.write(markdown)
logger.info("Report written to %s", abs_path)
return abs_pathWe add a title, a timestamp, the synthesized report body, and a numbered list of all the sources we actually used. The "actually used" part is important — if a source failed to load or summarize, it doesn't appear here. The reader only sees what contributed to the report.
Putting It All Together
The main.py file orchestrates everything in sequence:
"""
main.py
Entry point for the AI Research Assistant.
Workflow:
User -> Topic -> Search -> Top URLs -> Extract Text
-> Gemini Summary (per URL) -> List of Summaries
-> Gemini Final Report -> report.md
"""
from __future__ import annotations
import argparse
import logging
import sys
from search import search_topic
from scraper import extract_texts
from gemini import summarize_page, generate_final_report
from report import save_report
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
)
logger = logging.getLogger("main")
def run(topic: str, num_results: int = 3, output_path: str = "report.md") -> str:
"""
Run the full research pipeline for a given topic.
Returns:
Path to the generated report.md file.
"""
logger.info("Step 1/5: Searching the web for: %s", topic)
urls = search_topic(topic, num_results=num_results)
for i, url in enumerate(urls, start=1):
logger.info(" URL %d: %s", i, url)
logger.info("Step 2/5: Extracting page text from %d URLs", len(urls))
page_texts = extract_texts(urls)
if not page_texts:
raise RuntimeError(
"Failed to extract usable text from any of the search results. "
"Try a different topic or check your network connection."
)
logger.info("Step 3/5: Summarizing %d pages with Gemini", len(page_texts))
summaries = []
for url, text in page_texts.items():
try:
summary = summarize_page(url, text, topic)
summaries.append({"url": url, "summary": summary})
logger.info(" Summarized: %s", url)
except RuntimeError as exc:
logger.warning(" Skipping %s due to summarization error: %s", url, exc)
if not summaries:
raise RuntimeError("Gemini failed to summarize any of the extracted pages.")
logger.info("Step 4/5: Generating final report with Gemini")
report_body = generate_final_report(topic, summaries)
logger.info("Step 5/5: Saving report to %s", output_path)
final_path = save_report(
topic=topic,
report_body=report_body,
sources=[item["url"] for item in summaries],
output_path=output_path,
)
return final_path
def main() -> None:
parser = argparse.ArgumentParser(
description="AI Research Assistant - search, summarize, and report on a topic."
)
parser.add_argument(
"topic",
nargs="*",
help="Research topic, e.g. 'Docker vs Podman'. If omitted, you'll be prompted.",
)
parser.add_argument(
"-n",
"--num-results",
type=int,
default=3,
help="Number of search results to use (default: 3).",
)
parser.add_argument(
"-o",
"--output",
default="report.md",
help="Output markdown file path (default: report.md).",
)
args = parser.parse_args()
topic = " ".join(args.topic).strip()
if not topic:
topic = input("Enter a research topic: ").strip()
if not topic:
print("No topic provided. Exiting.")
sys.exit(1)
try:
path = run(topic, num_results=args.num_results, output_path=args.output)
except Exception as exc: # noqa: BLE001
logger.error("Pipeline failed: %s", exc)
sys.exit(1)
print(f"\n[DONE] Report generated: {path}")
if __name__ == "__main__":
main()One detail I'm proud of: I wrapped each step in clear logging. When you run the tool, you can see exactly what's happening:
"Step 1/5: Searching the web..."
"Step 2/5: Extracting page text..."
And so on
This makes debugging way easier and gives you confidence that things are actually working.
What You Need to Get Started
The setup is deliberately minimal:
Python 3.10+ (nothing exotic)
A Gemini API key from Google AI Studio (they have a generous free tier)
Four Python packages:
google-genai,ddgs,trafilatura, andrequests
pip install -r requirements.txtCreate a .env file with your API key:
GEMINI_API_KEY=your_key_here
GEMINI_MODEL=gemini-2.0-flash-liteAnd you're ready to go.
Things I Learned Building This
Web scraping is still hard. Even with trafilatura, some sites just don't want to cooperate. That's why I built in graceful fallbacks and skipping logic.
Small AI models are underrated. You don't always need GPT-4 or Gemini Pro. For summarization tasks, the lighter models work great and cost a fraction of a cent per request.
Three sources is the sweet spot. For most topics, three good sources give you enough perspective without becoming redundant. But I made it configurable because sometimes you want more.
The "fail gracefully" pattern is underrated. Every function in this project either returns a meaningful value or a clear error. Nothing sneaky fails silently. This makes the whole pipeline way more reliable.
What's Next? (Stuff You Could Add)
This is intentionally minimal, but here are some ideas if you want to take it further:
Add date filtering — only use articles from the last year
Support multiple languages — search in French, summarize in English
Add a web interface — slap a simple Gradio or Streamlit UI on top
Save search history — so you can revisit past research topics
Add citation footnotes — show which source contributed which claim
Watch Me Build This Live
I made a complete walkthrough video where I build this from scratch and explain every decision along the way.
[Watch the video tutorial here — COMING SOON]
Wrapping Up
This little tool has genuinely changed how I research things. Instead of drowning in tabs, I just type a topic and get a clean report in seconds. It's not going to write a PhD thesis, but for getting up to speed on a new topic quickly? It's incredible.
The best part? The entire thing is about 250 lines of Python. There's no complex infrastructure, no vector databases, no LangChain abstractions. Just good old-fashioned piping data through a few well-defined steps.
The code is on GitHub. Clone it, break it, improve it, make it your own. That's what open source is about.
Got questions or ideas for features? Drop them in the comments below. I read every single one and I'm always looking for ways to make this better.
Happy researching! And remember — the best time to automate something is when you find yourself doing it at 11 PM on a Tuesday.