Setting Up a Movie Dataset You Can Actually Trust

Every article in this series builds on the same dataset: a CSV of Indian and Hollywood movies pulled from OMDb, currently sitting at 1,163 rows across 10 languages. Before any of that data reaches an LLM agent, a search tool, or a recommendation engine, it has to clear a much lower bar than “does the agent give good answers”: does the dataset actually contain what everything downstream assumes it contains. That’s a data-validation problem, not a machine-learning problem, and it’s the first thing worth getting right, because every later article in this series inherits whatever’s wrong here.

The loader

The loading code is deliberately thin. Anything smarter belongs in the layers that use the data, not in the layer that just reads it off disk.

# phase-1/src/data_loader.py
import pandas as pd
from pathlib import Path

def load_movies_data(csv_path: str) -> pd.DataFrame:
    """Load movies dataset from CSV file."""
    df = pd.read_csv(csv_path)
    return df

That’s the whole function. No schema validation, no null checks, no type coercion. pd.read_csv will happily load a file with a missing column, a rating stored as text, or a blank plot field, and hand back a DataFrame that looks fine until something three files away calls .str.lower() on a NaN and crashes.

The exploration pass

The project’s explore_data() function is a diagnostic pass over the loaded frame: row count, column list, language breakdown, rating range, year range, and the most common genres extracted from a comma-separated genre column.

def explore_data(df: pd.DataFrame):
    """Display basic statistics about the dataset."""
    print(f"\n📈 Total Movies: {len(df)}")
    print(f"📋 Columns: {list(df.columns)}")

    print(f"\n🌍 Languages in Dataset:")
    print(df['language'].value_counts())

    print(f"\n⭐ IMDB Rating Statistics:")
    print(f"  - Min: {df['imdb_rating'].min()}")
    print(f"  - Max: {df['imdb_rating'].max()}")
    print(f"  - Mean: {df['imdb_rating'].mean():.2f}")

    print(f"\n📅 Year Range: {df['year'].min()} to {df['year'].max()}")

    # Extract individual genres from comma-separated string
    all_genres = []
    for genres_str in df['genre']:
        genres = [g.strip() for g in genres_str.split(',')]
        all_genres.extend(genres)
    from collections import Counter
    genre_counts = Counter(all_genres)
    for genre, count in genre_counts.most_common(10):
        print(f"  - {genre}: {count}")

This is useful, and it’s also a trap if it’s the only check that ever runs: it’s a human reading printed output and eyeballing whether it looks right. That works the first time, when someone’s actually watching the terminal. It stops working the moment this loader runs unattended, which is exactly what happens a few articles from now when a scheduled script fetches new movies from OMDb and appends them to this same CSV.

What actually needs to be true

explore_data() answers “what does this look like.” A validation pass answers a narrower, more useful question: “is this still safe for everything downstream to assume.” Those are different jobs, and the second one needs to be assertions, not print statements:

# tests/test_dataset.py
import pandas as pd
import pytest

REQUIRED_COLUMNS = [
    'title', 'year', 'genre', 'language', 'imdb_rating',
    'plot', 'director', 'actors', 'runtime', 'imdb_id',
]

@pytest.fixture
def df():
    return pd.read_csv('data/movies.csv')


def test_required_columns_present(df):
    missing = set(REQUIRED_COLUMNS) - set(df.columns)
    assert not missing, f"Missing columns: {missing}"


def test_no_nulls_in_critical_fields(df):
    for col in ['title', 'plot', 'genre', 'language', 'imdb_id']:
        assert df[col].notna().all(), f"Null values found in '{col}'"


def test_rating_is_numeric_and_in_range(df):
    ratings = pd.to_numeric(df['imdb_rating'], errors='coerce')
    assert ratings.notna().all(), "Non-numeric imdb_rating values present"
    assert ratings.between(0, 10).all(), "Ratings outside the 0-10 range"


def test_year_is_a_plausible_four_digit_number(df):
    years = pd.to_numeric(df['year'], errors='coerce')
    assert years.notna().all(), "Non-numeric year values present"
    assert years.between(1900, 2030).all(), "Year outside a plausible range"


def test_imdb_id_is_unique(df):
    duplicates = df['imdb_id'][df['imdb_id'].duplicated()]
    assert duplicates.empty, f"Duplicate imdb_id values: {duplicates.tolist()}"

None of these are exotic. They’re the same five checks you’d run on any tabular dataset before trusting it: the columns you need exist, the fields you can’t recover from being empty aren’t empty, the numeric fields are actually numeric and in range, and the field you’re using as a unique key is actually unique. What makes them worth writing down as tests instead of running once by hand is that this CSV isn’t static. A later article in this series adds a script that fetches new movies from OMDb and appends rows to it on a schedule, and another repairs bad rows in place. Both of those touch this exact file, and both can reintroduce exactly the problems these five tests catch.

Where this breaks down

The imdb_rating and year checks matter more than they look. pd.read_csv infers column dtypes from the data it sees, and a single malformed row (a rating field that got quoted, a year that came through as "2024.0" instead of 2024) can silently flip a column from int64/float64 to object, which means every later .min(), .max(), or numeric comparison either raises or does something quietly wrong depending on what’s actually in the column. pd.to_numeric(..., errors='coerce') turns that silent failure into a loud one: anything that isn’t a clean number becomes NaN, and the .notna().all() assertion catches it immediately instead of three functions later.

The genre-parsing loop in explore_data() has a smaller version of the same issue: genres_str.split(',') assumes every row’s genre field is a non-null string. If a fetch script ever appends a row with a missing genre, this loop throws on .split() before it prints a single genre count. That’s a real, cheap thing to add to the test list: df['genre'].notna().all(), checked before anything tries to split it.

Takeaways

• A loader that just reads a CSV and returns it is fine, as long as something else in the pipeline actually validates what came back. Don’t let “it loaded without an exception” stand in for “it’s correct.”
• Print-and-eyeball exploration is a good first pass and a bad permanent test suite. The moment a dataset gets touched by anything unattended (a cron job, a scheduled fetch), the checks need to be assertions.
• Validate the fields everything downstream assumes are safe: required columns present, critical fields non-null, numeric fields actually numeric and in range, and any field used as a unique key actually unique.