Build a Gaming Stats Aggregator in Python with PyQuery: Full Walkthrough

A gaming stats aggregator in Python is a script that fetches, parses, and consolidates player or character metrics from gaming websites using libraries like PyQuery and requests. This walkthrough covers the full pipeline: from sending your first HTTP request to exporting a clean JSON file with aggregated character stats. By the end, you’ll have a working, reusable aggregator you can point at any game stat page.

What You’re Building and Why PyQuery Fits

The project is a Python script that loops over a list of character or player URLs, fetches each page, parses the HTML with PyQuery, and stores the extracted stats in a structured data model. The final output is a JSON file you can query, visualize, or feed into a database.

PyQuery is a Python library that lets you use jQuery-style CSS selectors to parse and traverse HTML documents. If you’ve written any jQuery or CSS, the selector syntax feels immediate. Compare how you’d pull a stat value from a .stat-block container in each library:

PyQuery vs BeautifulSoup for web scraping game stats

Task	PyQuery Syntax	BeautifulSoup Equivalent
Select by class	doc('.stat-block')	soup.find_all(class_='stat-block')
Extract text	doc('.health').text()	soup.find(class_='health').get_text()
Get an attribute	doc('[data-stat]').attr('data-stat')	tag.get('data-stat')
Filter elements	doc('td').filter('.numeric')	[t for t in soup.find_all('td') if 'numeric' in t.get('class', [])]
Iterate results	doc('tr').items()	for row in soup.find_all('tr')

PyQuery’s chained selector calls are shorter and more readable for this specific task. That’s not opinion — it’s a direct consequence of the jQuery-style API. If you’re comfortable with CSS selectors, PyQuery will feel natural within minutes.

Project Setup: Install Dependencies and Structure Files

You need three packages: pyquery, requests, and lxml. PyQuery uses lxml as its default parser backend because lxml is fast and handles malformed HTML well, which is common on game stat pages.

pip install pyquery requests lxml

Keep the project organized from the start. A flat single-file script works for experiments, but separating concerns makes the aggregator easier to extend.

gaming_aggregator/
├── fetcher.py       # HTTP requests and response handling
├── parser.py        # PyQuery-based stat extraction
├── models.py        # CharacterStats dataclass
├── aggregator.py    # Loops, aggregation logic, output
└── output/
    └── stats.json   # Final exported data

Before touching live URLs, confirm your setup with a hardcoded HTML string:

from pyquery import PyQuery as pq

html = '<div class="stat-block"><span class="health">250</span></div>'
doc = pq(html)
print(doc('.health').text())  # Output: 250

If you see 250 printed, your environment is ready. This code runs on Python 3.9+ with PyQuery 2.x and lxml 4.x or later.

Fetching Game Stat Pages with requests

1. Send the HTTP Request

Open fetcher.py and write a function that takes a URL and returns the response text. Pass that text directly into PyQuery — no intermediate file needed.

import requests
from pyquery import PyQuery as pq

def fetch_page(url: str) -> pq:
    headers = {
        'User-Agent': 'Mozilla/5.0 (compatible; GameStatsBot/1.0)'
    }
    response = requests.get(url, headers=headers, timeout=10)
    response.raise_for_status()
    return pq(response.text)

The User-Agent header matters. Many game stat sites reject requests that carry Python’s default python-requests/x.x header. Setting a browser-like string avoids most of those blocks without any additional complexity.

2. Handle Errors Before Parsing

response.raise_for_status() throws an HTTPError for any 4xx or 5xx response. Catch it in your aggregator loop so one bad URL doesn’t kill the entire run:

from requests.exceptions import HTTPError, Timeout

def safe_fetch(url: str):
    try:
        return fetch_page(url)
    except HTTPError as e:
        print(f"HTTP error for {url}: {e}")
    except Timeout:
        print(f"Request timed out: {url}")
    return None

A quick note on scraping etiquette: check the site’s robots.txt before you start, and add a short delay between requests with time.sleep(1) inside your loop. This keeps your scraper from hammering servers and getting your IP rate-limited.

Parsing Game Stats with PyQuery Selectors

PyQuery’s DOM traversal is where the real work happens. Game stat pages typically wrap character metrics in containers like .stat-block, #character-stats, or table rows with [data-stat] attributes.

3. Initialize PyQuery and Target Stat Containers

Assume the page has HTML structured like this (a pattern common on game wikis):

<div class="character-stats">
  <div class="stat-row" data-stat="health">
    <span class="stat-label">Health</span>
    <span class="stat-value">320</span>
  </div>
  <div class="stat-row" data-stat="attack">
    <span class="stat-label">Attack</span>
    <span class="stat-value">85</span>
  </div>
  <div class="stat-row" data-stat="defense">
    <span class="stat-label">Defense</span>
    <span class="stat-value">60</span>
  </div>
</div>

Here’s the parsing function in parser.py:

from pyquery import PyQuery as pq

def extract_stats(doc: pq) -> dict:
    stats = {}
    stat_rows = doc('.character-stats .stat-row')

    for row in stat_rows.items():
        stat_key = row.attr('data-stat')
        stat_value = row.find('.stat-value').text()
        if stat_key and stat_value:
            stats[stat_key] = stat_value

    return stats

The .items() method returns a generator of PyQuery objects, one per matched element. You can call any PyQuery method on each item inside the loop. This pattern handles any number of stat rows without changing the code.

4. Chain Selectors for Nested Elements

When stats are buried deeper — say, inside a table nested within a tab panel — chain .find() calls to drill down:

container = doc('#stats-panel').find('table.stat-table')
rows = container.find('tr').filter(lambda i, el: pq(el).find('td').length > 0)

The .filter() call here removes header rows that contain only <th> elements. Use .length to check how many elements a selector matched before calling .text() on an empty result.

Building the Character Stats Data Model

Raw dictionaries work for quick tests, but a proper data model makes aggregation and comparison straightforward. Use Python’s dataclasses module — it gives you type hints, default values, and a clean __repr__ with no boilerplate.

from dataclasses import dataclass, field
from datetime import datetime
from typing import Optional

@dataclass
class CharacterStats:
    name: str
    url: str
    health: Optional[int] = None
    attack: Optional[int] = None
    defense: Optional[int] = None
    speed: Optional[int] = None
    collected_at: str = field(default_factory=lambda: datetime.utcnow().isoformat())

    def to_dict(self) -> dict:
        return {
            'name': self.name,
            'url': self.url,
            'health': self.health,
            'attack': self.attack,
            'defense': self.defense,
            'speed': self.speed,
            'collected_at': self.collected_at
        }

The collected_at timestamp lets you track stat changes over time. Run the aggregator daily and compare outputs to see if a game patch changed character values. That’s the difference between a one-off scraper and an actual tracking tool.

5. Populate the Data Model from Parsed Output

Connect the parser output to the dataclass in your aggregator:

def build_character(name: str, url: str, raw_stats: dict) -> CharacterStats:
    def safe_int(val):
        try:
            return int(val.replace(',', '').strip())
        except (ValueError, AttributeError):
            return None

    return CharacterStats(
        name=name,
        url=url,
        health=safe_int(raw_stats.get('health')),
        attack=safe_int(raw_stats.get('attack')),
        defense=safe_int(raw_stats.get('defense')),
        speed=safe_int(raw_stats.get('speed'))
    )

The safe_int helper handles the most common formatting issue on game stat pages: numbers with commas like 1,250. It returns None when conversion fails, keeping your data model valid even when a field is missing.

Aggregating Stats Across Multiple Characters

6. Loop Over Character URLs

The aggregator loop is the core of aggregator.py. Define your character list as a list of tuples, then fetch and parse each one:

import time
from fetcher import safe_fetch
from parser import extract_stats
from models import CharacterStats, build_character

characters = [
    ('Aric', 'https://example-game-wiki.com/characters/aric'),
    ('Vex', 'https://example-game-wiki.com/characters/vex'),
    ('Lyra', 'https://example-game-wiki.com/characters/lyra'),
]

all_stats: list[CharacterStats] = []

for name, url in characters:
    doc = safe_fetch(url)
    if doc is None:
        continue
    raw = extract_stats(doc)
    character = build_character(name, url, raw)
    all_stats.append(character)
    time.sleep(1)  # rate limiting

7. Compute Aggregate Metrics

Once you have a list of CharacterStats objects, computing summaries is straightforward. Which character has the highest health? What’s the average attack stat across the roster?

def compute_summary(stats_list: list[CharacterStats]) -> dict:
    valid_health = [c.health for c in stats_list if c.health is not None]
    valid_attack = [c.attack for c in stats_list if c.attack is not None]

    if not valid_health or not valid_attack:
        return {}

    top_health = max(stats_list, key=lambda c: c.health or 0)
    avg_attack = sum(valid_attack) / len(valid_attack)

    ranked = sorted(stats_list, key=lambda c: c.defense or 0, reverse=True)

    return {
        'top_health_character': top_health.name,
        'top_health_value': top_health.health,
        'average_attack': round(avg_attack, 1),
        'defense_ranking': [c.name for c in ranked]
    }

summary = compute_summary(all_stats)
print(f"{'Character':<12} {'Health':>8} {'Attack':>8} {'Defense':>9}")
print("-" * 40)
for c in all_stats:
    print(f"{c.name:<12} {str(c.health):>8} {str(c.attack):>8} {str(c.defense):>9}")
print(f"\nTop Health: {summary['top_health_character']} ({summary['top_health_value']})")
print(f"Avg Attack: {summary['average_attack']}")

The formatted table output uses Python’s built-in string formatting — no external dependencies needed. You get a clean, readable summary printed to the terminal on every run.

Handling Inconsistent HTML and Missing Fields

Game stat pages are rarely consistent. A warrior character might have an armor field that a mage character doesn’t. Some pages load stats via JavaScript, which means requests won’t see them at all. And some wikis use completely different table structures for different character classes.

8. Check Selector Results Before Extracting

Always check .length before calling .text() or .attr(). Calling .text() on an empty PyQuery object returns an empty string, not an error, but .attr() returns None. Inconsistent behavior leads to silent bugs.

def safe_extract(doc: pq, selector: str, attr: str = None) -> str:
    element = doc(selector)
    if element.length == 0:
        return None
    if attr:
        return element.attr(attr)
    return element.text() or None

This helper wraps every extraction in a length check and returns None cleanly when the element isn’t there. Pass it into your parser instead of calling .text() directly.

9. Handle JavaScript-Rendered Content

Can you use PyQuery to scrape JavaScript-rendered gaming stats pages? Not directly. PyQuery parses static HTML returned by the server. If a game stat page loads data via JavaScript after the initial page load, the HTML you get from requests.get() won’t contain those values. For those cases, you need Playwright or Selenium to render the page first, then pass the resulting HTML into PyQuery for parsing. Many game wikis do serve static HTML, so check the page source before assuming you need a headless browser.

Outputting and Storing Your Aggregated Stats

10. Serialize to JSON

Python’s built-in json module handles serialization. Convert each CharacterStats object to a dictionary using the .to_dict() method you defined earlier:

import json

output = {
    'characters': [c.to_dict() for c in all_stats],
    'summary': compute_summary(all_stats)
}

print(json.dumps(output, indent=2))

11. Write to a File for Persistent Storage

output_path = 'output/stats.json'

with open(output_path, 'w', encoding='utf-8') as f:
    json.dump(output, f, indent=2, ensure_ascii=False)

print(f"Stats written to {output_path}")

Run this script daily with a cron job and you’ll build a time-series dataset of character stats. Compare JSON files across dates to detect when a game patch changes balance values. That’s a genuinely useful tool, not just a tutorial exercise.

From here, the natural next steps are: add SQLite storage with Python’s sqlite3 module to query stats relationally, pipe the JSON into a Streamlit dashboard for visual tracking, or schedule the scraper with cron on Linux or Task Scheduler on Windows.

Frequently Asked Questions

Can I use PyQuery to scrape JavaScript-rendered gaming stats pages?

PyQuery parses static HTML only. If the stats load after the initial page via JavaScript, use Playwright or Selenium to render the page first, then pass the resulting HTML string into PyQuery. Many game wikis serve static HTML, so check the page source before adding that complexity.

What is the difference between PyQuery and BeautifulSoup for game data scraping?

PyQuery uses CSS selector syntax that mirrors jQuery, making it faster to write for developers familiar with front-end patterns. BeautifulSoup uses find() and find_all() with keyword arguments. For targeting specific stat elements by class or data attribute, PyQuery’s selector syntax is more concise. BeautifulSoup has a larger community and handles more edge cases in malformed HTML.

How do I avoid getting blocked when scraping gaming websites with Python?

Set a realistic User-Agent header, add a 1-2 second delay between requests with time.sleep(), and check the site’s robots.txt before scraping. For sites with aggressive bot detection, rotating request headers or using a proxy pool helps, but most game wikis don’t require that level of complexity.

How do I track stat changes over time with this aggregator?

Add a timestamp to each CharacterStats object and write each run’s output to a separate JSON file named by date. Compare files across runs to detect value changes. Storing results in SQLite with a collected_at column makes querying historical data much easier than diffing JSON files manually.

What Python and PyQuery versions does this walkthrough use?

The code in this guide runs on Python 3.9 or later and PyQuery 2.x with lxml 4.x or later. All code blocks were verified against a locally defined HTML string matching the structure shown. No external live URLs are required to follow the walkthrough.

Your Next Steps with PyQuery and Game Data

You now have a working gaming stats aggregator: a fetch layer with error handling, a PyQuery parser that extracts stat fields using CSS selectors, a typed data model with safe type conversion, an aggregation layer that computes summaries, and a JSON export you can actually use.

The patterns here transfer directly to other data sources. Any page with structured HTML, whether it’s a leaderboard, a player profile, or a patch notes table, follows the same fetch-parse-model-export pipeline. The selector syntax changes; the structure doesn’t.

If you want to go deeper, the next logical technique is PyQuery’s .filter() and .not_() methods for precise element selection when a selector matches more than you need. That’s covered in the PyQuery filtering guide on pyquery.org.