A Naive Chatbot Is Not an Agent

The second phase of this project is a movie chatbot with memory: you ask it a question, it answers, and it remembers what you said earlier in the conversation. It’s called an agent in the code and the guide that goes with it. It isn’t one, and the gap between what this actually is and what the name implies is the most useful thing about looking at it closely, because it’s the same gap a lot of “AI agent” projects never close.

What it actually is

# phase-2/src/agent.py
from langchain.memory import ConversationBufferMemory
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langchain_openai import ChatOpenAI

class MovieRecommendationAgent:
    def __init__(self, csv_path: str):
        self.df = pd.read_csv(csv_path)
        self.memory = ConversationBufferMemory()

        self.llm = ChatOpenAI(
            model="gpt-3.5-turbo",
            temperature=0.7,
            api_key=os.getenv("OPENAI_API_KEY"),
        )

        self.prompt = PromptTemplate(
            input_variables=["history", "input"],
            template="""You are a friendly movie recommendation assistant.
You have knowledge about 520 Indian movies spanning multiple languages.

Conversation history:
{history}

User question: {input}

Please answer helpfully about Indian movies. Keep your answer concise (2-3 sentences)."""
        )

        self.chain = LLMChain(
            llm=self.llm,
            prompt=self.prompt,
            memory=self.memory,
        )

    def chat(self, user_input: str) -> str:
        try:
            response = self.chain.run(input=user_input)
            return response.strip()
        except Exception as e:
            return f"❌ Error: {str(e)}"

Every call to chat() does the same three things: format a prompt with the conversation history and the new input, send it to the model, append the exchange to memory. There is no branch in this code that depends on what the model decides. The self.df DataFrame is loaded in __init__ and never touched again outside get_dataset_info(): the model never sees it, never queries it, and has no way to look anything up. Every “fact” it states about a specific movie is either something it happened to learn during pretraining or something it’s making up that sounds plausible.

That’s the whole distinction. A chatbot takes text in, runs it through a model and a prompt template, and returns text out. An agent takes text in and the model itself decides what to do next: answer directly, or call a tool, look at what the tool returned, and decide again. Phase 2 has memory and a nice system prompt, and neither of those is the thing that makes something agentic. A real agentic rebuild, later in this series, swaps the model in for that decision entirely: given the same request, it chooses for itself whether to look anything up at all. That’s the contrast worth holding onto while reading this one.

There’s a second, more mundane issue: LLMChain and ConversationBufferMemory are both deprecated in current LangChain, superseded by LCEL (prompt | llm) and checkpointer-backed memory. That’s not the interesting problem with this code (a deprecated API still runs), but it’s worth knowing if you’re following along and wondering why current LangChain docs don’t mention either of them.

What you can even test here

This is the part worth sitting with. A pure chat wrapper like this one gives you almost nothing deterministic to assert on. The model’s exact wording is different every run, there’s no structured output, and there’s no tool call to inspect. Testing it looks like this:

def test_agent_responds_to_a_question():
    agent = MovieRecommendationAgent("data/movies.csv")
    response = agent.chat("What's Lagaan?")
    assert len(response) > 0
    assert "error" not in response.lower()


def test_memory_persists_across_turns():
    agent = MovieRecommendationAgent("data/movies.csv")
    agent.chat("I like action movies")
    response = agent.chat("What did I just say I liked?")
    assert "action" in response.lower()


def test_empty_input_does_not_crash():
    agent = MovieRecommendationAgent("data/movies.csv")
    response = agent.chat("")
    assert isinstance(response, str)

Every one of these is a real LLM call, every one is non-deterministic, and none of them can verify the thing that actually matters: whether a claim the agent makes about a specific movie is true. “Responds without crashing” and “remembers the previous turn” are legitimate things to check, but they’re the ceiling of what’s testable here, not the floor. There’s no way to write assert response.movie == "Lagaan" against free-form prose, and there’s no tool call to check because none is ever made.

Where this breaks down

Ask this agent “what’s a good Tamil thriller from the dataset” and it will answer confidently, using the phrase “the dataset” in its system prompt as license to sound authoritative, while having queried nothing. If the model happens to know the movie from pretraining, the answer might be right. If it doesn’t, it will still produce a fluent, specific-sounding recommendation, because nothing in this architecture distinguishes “I looked this up” from “this sounds right.” That’s not a prompt-engineering problem you can fix by asking it more firmly to only discuss real movies. It’s a structural problem: the model has no mechanism to ground its answer in the actual 1,163-row dataset, so there’s nothing to ground it with.

This is also why “does it respond, does memory work” is a legitimate but shallow test suite. It validates that the chain runs. It says nothing about whether the agent is telling the truth, because at this stage nothing in the system knows what the truth is either.

Takeaways

• Memory and a good prompt make something feel conversational. Neither makes it an agent. The thing that makes something agentic is the model deciding what to do next, including whether to call a tool, not us deciding it in advance.
• A pure LLM-chain wrapper has no deterministic surface to test against: no tool calls to inspect, no structured output, nothing but free-form text that changes every run. Testing it means checking that it runs and behaves reasonably, not that it’s correct.
• A chatbot that can only generate text has no way to distinguish a real fact from a plausible-sounding one. That’s not a bug you patch with a better system prompt: it’s what happens when there’s no tool between the model and the truth.