When I first started working with Large Language Models, I was surprised to learn that ChatGPT does not remember conversations like people do. Still, if you ask a follow-up question, it often knows what you mean. The technical reason behind how ChatGPT remembers your conversation is the context window.
When building LLM-powered apps, I often notice that beginners think the model saves every message and pulls it up when needed. Actually, the process is simpler. The app gathers previous messages, turns them into tokens, and sends them back to the model as part of the input.
Learning how this works helped me build better chatbots, RAG systems, and context-aware AI apps. In this article, I’ll show you what goes on behind the scenes.
How ChatGPT Remembers Your Conversation
ChatGPT Does Not Remember Conversations Like Humans
Suppose I start a conversation with ChatGPT:
Me: My name is Aman.
Then, a few messages later, I ask:
Me: What is my name?
ChatGPT may answer:
ChatGPT: Your name is Aman.
At first, this seems like memory. But in a basic LLM chat, the model does not actually keep the fact that my name is Aman stored anywhere inside itself.
Instead, the application can send the conversation history back to the model:
User: My name is Aman.
Assistant: Nice to meet you, Aman.
User: What is my name?
The model looks at the whole conversation so far and predicts what to say next.
One of the biggest lessons I learned while building LLM apps is that the language model and the conversation management system are actually two separate parts.
The model generates the response. The application determines which conversation context to send to the model.
So, What Is a Context Window?
A context window is simply how much information an LLM can handle at once.
The context is more than just the latest user prompt. Depending on the app, it might include:
→ System instructions
→ Developer instructions
→ Previous user messages
→ Previous assistant responses
→ Retrieved documents
→ Tool outputs
→ The current user prompt
All of this information uses up tokens.
For example, imagine the context window as a token budget of 10,000 tokens. Your application may send:
System Instructions: 500 tokens
Conversation History: 6,000 tokens
Retrieved Documents: 2,000 tokens
Current Prompt: 500 tokens
Altogether, the context now uses 9,000 tokens. The model has to process all of this before it can answer.
When I first built RAG and conversational AI systems, I focused a lot on prompts. Later, I realized that managing context is just as important. Even a great prompt can’t fix a poorly managed context.
What Are Tokens?
LLMs do not directly process sentences or paragraphs. Text is converted into smaller units called tokens.
A token can represent a complete word, part of a word, punctuation, or another text unit depending on the tokenizer.
For example: “Artificial Intelligence is changing the world.”
Conceptually, a tokenizer may break it into units similar to:
Artificial
Intelligence
is
changing
the
world
.
The actual tokenization depends on the model and its tokenizer. This matters because context window limits are measured in tokens, not messages.
For example, 100 short messages might use less context than just 10 messages if those contain big documents or long code blocks.
I often see learners make this mistake: they count chat messages when trying to fix context issues. I find it better to think about the actual token budget instead.
How Previous Messages Are Passed to the Model
Let us look at a simplified conversation history:
messages = [
{
"role": "system",
"content": "You are a helpful AI assistant."
},
{
"role": "user",
"content": "My name is Aman."
},
{
"role": "assistant",
"content": "Nice to meet you, Aman."
},
{
"role": "user",
"content": "What is my name?"
}
]
The role shows who sent the message. System messages set behavior or instructions. User messages are what the user says. Assistant messages are the model’s previous replies.
When the latest question is submitted, the application sends the available message history to the model.
Conceptually, the workflow looks like this:
Conversation History
↓
Tokenization
↓
Context Window Check
↓
LLM
↓
Generate Next Response
↓
Store Response in History
This loop continues as the conversation grows.
Want to go deeper into LLMs, RAG, and AI Agents? Check out my book Hands-On GenAI, LLMs & AI Agents for practical, hands-on learning.
ChatGPT’s Conversation Memory Algorithmic Implementation with Python
Let me build a simple Python implementation to demonstrate the underlying algorithmic idea.
We will create a conversation manager with a fixed token limit.
Step 1: Create the Conversation History
conversation = []
MAX_TOKENS = 50The conversation list stores our messages.
MAX_TOKENS represents the maximum context size supported by our simulated model.
I am deliberately using a very small number so that we can easily observe messages being removed.
Step 2: Estimate the Number of Tokens
def count_tokens(text):
return len(text.split())Real LLM applications use model-specific tokenizers. For simplicity, I am treating every word as one token. For example:
count_tokens("My name is Aman")The function returns:
4
This isn’t how tokenization works in real apps, but it’s enough to show how context management works.
Step 3: Add Messages to the Conversation
def add_message(role, content):
conversation.append({
"role": role,
"content": content
})Every time the user or assistant sends a message, we add it to the conversation history. For example:
add_message("user", "My name is Aman")
add_message("assistant", "Nice to meet you Aman")Our conversation now looks like this:
[
{
"role": "user",
"content": "My name is Aman"
},
{
"role": "assistant",
"content": "Nice to meet you Aman"
}
]
Step 4: Calculate the Total Context Size
def total_tokens():
return sum(
count_tokens(message["content"])
for message in conversation
)This function loops through every message and calculates the total number of tokens currently stored in the conversation. Conceptually:
Message 1 Tokens
+
Message 2 Tokens
+
Message 3 Tokens
↓
Total Context Tokens
Before sending the conversation to the LLM, we need to check whether this value exceeds the model’s context limit.
Step 5: Remove Old Messages When the Limit Is Reached
def manage_context():
while total_tokens() > MAX_TOKENS:
removed_message = conversation.pop(0)
print(
"Removed:",
removed_message["content"]
)This is the most important part of our implementation. The condition checks:
total_tokens() > MAX_TOKENSIf the conversation is too large, we remove the oldest message:
conversation.pop(0)Index 0 is the first message in the list.
The algorithm continues to remove old messages until the conversation fits within the context window. This is commonly known as a sliding window strategy.
Step 6: Simulate a Conversation
Now let us use our context manager:
add_message(
"user",
"My name is Aman and I work on artificial intelligence projects"
)
add_message(
"assistant",
"Nice to meet you Aman. Artificial intelligence is an exciting field"
)
add_message(
"user",
"I am currently learning how large language models process conversations"
)
add_message(
"assistant",
"Large language models process previous messages as part of their context"
)
add_message(
"user",
"Can you explain how context windows work in language models"
)
manage_context()
print(conversation)When the total token count goes over our MAX_TOKENS limit, the oldest messages get removed. In the end, only the latest messages might remain. For example:
Before Context Management
Message 1
Message 2
Message 3
Message 4
Message 5
↓
Token Limit Exceeded
↓
After Context Management
Message 3
Message 4
Message 5
Now imagine the user asks:
What is my name?
If the message “My name is Aman” was removed from the available context, the model may no longer have that information from the active conversation context.
This simple experiment shows why long LLM conversations can lose earlier details.
Final Thoughts
Once I understood context windows, I stopped thinking of ChatGPT as a system that simply “remembers everything.”
Now, I see it as a model that works with whatever information is currently in its context, plus any extra memory or retrieval tools the app provides.
I hope you like this article on how ChatGPT remembers your conversation is the context window.
For more AI and machine learning tips, follow me on Instagram. My book, Hands-On GenAI, LLMs & AI Agents, can also help you advance your AI career.
