Unexpected Behavior with Agent Memory in FastAPI using OpenWebUI Pipelines

[NTTLuke]

I’m experiencing an issue with the memory behavior of an agent set up in a FastAPI OpenWebUI pipeline. While I suspect I may be missing something, the behavior is inconsistent, and I’m unsure how to resolve it.

Here’s the relevant portion of my agent setup and chat handling code (sensitive information removed):

# Agent setup in the OpenWebUI pipeline `on_startup` method
self.agent = Agent(
    model=self.model,
    introduction="""
    You are a helpful and seasoned assistant bla bla bla
    """,
    instructions=[
        "list of my instructions"
    ],
    expected_output="The answer",
    storage=PgAgentStorage(table_name="my_table_name", db_url="my_db_url"), 
    add_history_to_messages=True,
    read_chat_history=True,
    num_history_responses=3,           
    tools=[LukeToolkit()],
    prevent_hallucinations=True,
    show_tool_calls=True,
    # debug_mode=True,
    markdown=True
)

# Chat handling method in the OpenWebUI pipeline `pipe` method
# for setting values dynamically

session_id = self.chat_id
user_id = body["user"]["id"]
self.agent.session_id = session_id
self.agent.user_id = user_id

try:
    for chunk in self.agent.run(
        message=user_message,
        # messages=messages,
        stream=True,
        # stream_intermediate_steps=True,
    ):
        yield chunk.content
except Exception as e:
    yield "Error during chat, please try again later."

Observed Issues

1. Memory Retrieval Without History Parameters
   When I remove add_history_to_messages=True and num_history_responses=3 as documented here](https://docs.phidata.com/agents/storage), the agent’s memory does not function correctly. It seems unable to retrieve old messages from the history.

2. Adding Chat History
   Adding add_history_to_messages=True and num_history_responses=3 resolves the memory issue. However, I expect the agent to retrieve history only from the current chat session, not from the entire user history (all the user chats).

   • My current approach effectively retrieves user-level history instead of session-specific history.
   • The desired behavior is for the agent to only reference the messages within the single chat session.

Additional Notes

• This behavior does not occur when testing the agent using the print_response method; in that case, the history is restricted to the current chat session as expected.
• I’m wondering if there’s a different or more appropriate approach for setting up the agent API side to achieve session-specific memory retrieval, or if I might be missing a key configuration.

Could someone clarify if I’m configuring the agent correctly in this scenario? Are there additional steps or tricks to ensure that the history retrieval is limited to the session context that in my case corresponds to the single chat?

Thank u :)

[ysolanky]

Hello @NTTLuke !

Memory Retrieval Without History Parameters: Yes, this is expected behaviour. By default we do not add any prior chat history to an Agent. add_history_to_messages param adds the chat history.

Adding Chat History: Can you please confirm if session_id is dynamically generated? An Agent would return chat history only as long as it is in the same session. So if self.chat_id is different for every new chat, an Agent should not add / have access to any other session.

The print_response function calls the run function behind the scenes. So behaviour should be consistent across both. Your approach is good. I am just curious to see if self.chat_id is assigning a new session id for each new chat.

[NTTLuke]

Hello @ysolanky
Thank you so much for your response.

Regarding your question: Yes, a new sessionId is assigned for each new chat. I’ve conducted further investigations and would like to share my findings along with a sample use case to illustrate the behavior.

The use case works using only add_history_to_messages=True , so I removed read_chat_history=True and num_history_responses=3 from my agent configuration

Use Case:

Scenario: Starting from scratch

1. User 1 creates a new chat (Chat 1) and starts writing messages.

   • In the Postgres database:
     • A new row is added with the correct userId and sessionId.
     • The memory column reflects only the messages specific to Chat 1.
   • When I retrieve all messages asking for references, it correctly returns the chat messages belonging to Chat 1.

2. User 1 creates a new chat (Chat 2) and writes a few messages.

   • In the Postgres database:
     • A new row is added with the correct userId and sessionId.
     • The memory column unexpectedly includes messages from Chat 1 as well.
       This might explain why, when requesting historical messages, references to Chat 1 appear alongside Chat 2’s messages.

3. User 1 creates a new chat (Chat 3) and writes a few messages.

   • In the Postgres database:
     • A new row is added with the correct userId and sessionId.
     • The memory column now includes messages from Chat 2.
       Similarly, this could be why references to Chat 1 and Chat 2 appear when retrieving historical messages.

To address this in different ways, I modified the approach by moving the agent to the pipe method and recreating the agent each time, rather than storing it in the self object. However, the behavior remained the same.

Based on this scenario, it seems that during the creation of a new chat, the agent retrieves all historical messages from the latest instance before saving them to PostgreSQL even if the agent has the empty memory (agent.memory.messages is [] when a new chat is created). So the issue is not related to "sharing" the session, as I initially supposed but I don't understand where is the trick.

Please let me know your thoughts—unless I’m starting to hallucinate worse than an LLM. 😂
Thanks in advance for your help!

[NTTLuke]

I just want to share even this to understand better the flow.

Specifically, when I remove read_chat_history=True and num_history_responses=3, the agent sometimes fails to incorporate the correct historical chat context. This behavior appears to occur at the following point in the code:

 # 2. Read existing session from storage
        self.read_from_storage()

 # 3. Prepare messages for this run
        system_message, user_messages, messages_for_model = self.get_messages_for_run(
            message=message, audio=audio, images=images, videos=videos, messages=messages, **kwargs
        )

At step 2, messages retrieved from Postgres are correctly stored in self.memory.messages

However, at step 3, these stored messages are not included in the messages_for_model variable or in the user_messages. As a result, models receives only the system prompt and the latest user message, without references to the storage data.

I tested a quick and admittedly "dirty" workaround by appending the stored messages manually at runtime:

messages_for_model.extend(self.memory.messages)

or

user_messages.extend(self.memory.messages)

With this change, the agent successfully incorporates the historical context.

I don't know if it's correct and I'm pointing to the wrong part but I just wanted to share with you because this was the reason of adding the read_chat_history and num_history_responses parameters.

thank you

[ysolanky]

@NTTLuke thank you for sharing the use cases. I see now that the issue is due to the use of the same instance of Agent. I was able to replicate scenarios 2 and 3 by using the same instance and just updating the session_id. For a fix, for each new self.chat_id, please create a new instance of the Agent class.

[ysolanky]

read_chat_history adds a tool to the Agent allowing it to search the storage. For example, when an Agent has access to this tool, you can ask a question like "What were my last 10 messages" and it would call the tool to search.

num_history_responses is the number of historical messages added to the Agent while using add_history_to_messages. We set a value of 3 as default to num_history_responses.

I am looking into the get_messages_for_run function now. Thank you for flagging this

[NTTLuke]

@NTTLuke thank you for sharing the use cases. I see now that the issue is due to the use of the same instance of Agent. I was able to replicate scenarios 2 and 3 by using the same instance and just updating the session_id. For a fix, for each new self.chat_id, please create a new instance of the Agent class.

Thank you @ysolanky ... I will try in this way !