How do you do the OpenAI LoCoMo evaluation?

[Salmon-f42]

I have read the code about LoCoMo evaluation in this repo, but I don't find how you generate the response with OpenAI memory, per my understanding, OpenAI doesn't support memory in their API, so how do you evaluate its performance on LoCoMo dataset? could you kindly explain your method to evaluate it?

[Van0SS]

+100
Very interested as well

[hush-cd]

I have read the code about LoCoMo evaluation in this repo, but I don't find how you generate the response with OpenAI memory, per my understanding, OpenAI doesn't support memory in their API, so how do you evaluate its performance on LoCoMo dataset? could you kindly explain your method to evaluate it?

Thank you for your thoughtful question. You're absolutely right that OpenAI's memory feature is not accessible via the API. Currently, memory is only supported through the ChatGPT web interface, and users can view and manage it under Settings > Personalization > Memory. Here’s how we handled memory extraction and evaluation for the LoCoMo dataset:

1. Memory Extraction via ChatGPT Web Interface

• For each conversation in the LoCoMo dataset, we manually input its sessions (multi-turn dialogues) one at a time into the ChatGPT web interface, with memory enabled.
• After all sessions for a given conversation had been submitted, we went to Settings > Personalization > Manage memories to view the Saved memories generated by ChatGPT.
• We manually copied and saved these memories to a local file. All sessions from the same conversation shared the same memory file.
• Before moving on to the next conversation, we deleted all previously saved memories from ChatGPT to avoid memory contamination across conversations.
  

2. Structuring Conversation Memory Files

• Each memory file stored the accumulated memories extracted across all sessions within a single conversation.
• This ensured that memory extraction was scoped cleanly at the conversation level, and no unintended context carried over between different conversations.

3. Prompting gpt-4o‑mini with Simulated Memory

• At test time, for each LoCoMo evaluation question, we loaded the corresponding conversation’s memory file.
• We constructed a single API prompt by embedding:
  1. A brief instruction to GPT‑4o‑mini.
  2. All of the pre‑extracted memory entries for that conversation.
  3. The test question itself.
• We then sent this prompt to the GPT‑4o‑mini API and recorded the model’s response.

4. Rationale

• Because the API does not support automatic memory retrieval, we supply GPT‑4o‑mini with exactly the same “knowledge” that ChatGPT’s memory feature would have provided.
• This manual memory‑injection ensures that our evaluation of GPT‑4o‑mini on LoCoMo is not handicapped by the lack of an official memory API and is at least as strong as, if not stronger than, the built‑in ChatGPT memory retrieval mechanism.

5. Example Memory‑Extraction Prompt

   Can you please extract relevant information from this conversation and create memory
   entries for each user mentioned? Please store these memories in your knowledge base in
   addition to the timestamp provided for future reference and personalized interactions.
   
   (1:56 pm on 8 May, 2023) Caroline: Hey Mel! Good to see you! How have you been?
   (1:56 pm on 8 May, 2023) Melanie: Hey Caroline! Good to see you! I’m swamped with the
   kids & work. What’s up with you? Anything new?
   (1:56 pm on 8 May, 2023) Caroline: I went to a LGBTQ support group yesterday and it was so
   powerful.
   ...
   

I hope this clarifies our methodology. Please let me know if you have any further questions!

[Salmon-f42]

Thanks for the detailed explanation! There are two types of memory that OpenAI is using, one is the saved memory, the other one is "Reference chat history". From the method you mentioned, it seems you only use the saved memory here and don't include the "chat history", if my understanding is right, the result with such method should be worse than their official one?

[hush-cd]

That's an excellent and very perceptive follow-up question. You are right in your observation.

To be precise, we did not use the "Reference chat history" feature. However, our methodology was designed to compensate for this. For each question in the LoCoMo evaluation, we attached the entire set of memories extracted from the corresponding conversation. In effect, this provides the model with all the contextual information from that conversation's history.

The reason for this approach stems from a limitation we identified in the LoCoMo evaluation process itself: the dataset's questions are not designated to be asked at a specific point within the conversation or session. This ambiguity makes it impossible to perfectly simulate ChatGPT's "Reference chat history" feature, as we wouldn't know which part of the history to reference. To mitigate this, we chose to provide the complete conversation memory, ensuring the model had all possible context to answer the question. It's also worth noting that in a live chat, it is unlikely that ChatGPT appends the entire conversation history to the prompt for every reply, so our method provides a very comprehensive context for the model.

This is a great point, and it's something our team has also discussed internally. We acknowledge that this aspect of the LoCoMo evaluation flow doesn't perfectly capture the sequential, turn-by-turn nature of memory extraction and retrieval in a real-world dialogue. For instance, in a live user conversation, MemOS doesn't wait for the entire dialogue to finish before performing memory extraction and then answering a batch of questions. Instead, both memory extraction and retrieval occur dynamically with each turn of the conversation.

We are actively working on refining our evaluation pipeline with this goal in mind. Our aim is to create a more realistic comparison that better simulates how both MemOS and OpenAI's memory functions would perform in an authentic, ongoing conversational setting.

Additional, our resulting evaluation scores for OpenAI's memory function are largely consistent with those reported by the mem0 project, which undertook a similar effort to replicate and evaluate this feature.

Thank you again for your insightful feedback. It is incredibly valuable for improving the rigor of our project. Please feel free to reach out if anything else comes to mind.

[Salmon-f42]

I see, if I remember correctly, when mem0 tested OpenAI's memory, OpenAI only enabled the saved memory and didn't support "chat history" at that time. BTW, I am also curious why you don't directly put the question to ChatGPT web interface, as you already used this method to extract the save memory, it seems to be nature to directly test on their web interface？

[hush-cd]

That's another excellent question, and your point about the timeline of mem0's testing is very astute. You are absolutely right to ask why we didn't simply continue using the ChatGPT web interface for the evaluation phase.

We considered this approach, but ultimately decided against it for two primary reasons: practical efficiency and maintaining evaluation integrity.

1. Efficiency and Scale:
   The most significant hurdle is the practical limitation of the web interface. The LoCoMo dataset contains 1,540 test questions. The ChatGPT web UI has daily usage caps—even for Plus subscribers, the GPT-4o message limit is currently around 80 requests every 3 hours. Attempting to run the full evaluation through the web interface would be extremely time-consuming, likely taking weeks to complete for a single run. This doesn't even account for the initial inputs required to extract the memory for each conversation. The API, by contrast, allows for much higher throughput.

2. Evaluation Integrity and the "Chat History" Flaw:
   This is the more critical, methodological reason. The LoCoMo evaluation flow dictates that we first input an entire conversation to build the memory, and then ask a series of questions about it.

   If we were to ask these questions sequentially in the ChatGPT web interface, the "Reference chat history" feature would become a source of contamination. For example, after we ask the first question and get an answer, the "recent history" for the second question would be our first question and its answer—not the original conversation content. This problem would compound with each question, meaning ChatGPT would be referencing a growing history of our Q&A session rather than the source material it's being tested on.

   Since LoCoMo's questions are not designed to be asked at a specific, logical point within the conversational flow, there is no "correct" recent history for ChatGPT to reference. Our solution—to bypass this by attaching the complete, static conversation memory to each question via the API—is a deliberate workaround to compensate for this limitation in the benchmark's design.

In summary, our current methodology is a pragmatic solution that balances the practical constraints of UI-based testing with the inherent structural limitations of the LoCoMo benchmark. We believe it provides a fairer and more repeatable assessment, even if it cannot perfectly replicate every feature of the live web product.

We are committed to further optimizing our evaluation process to address these issues in the future.

Thank you once again for pushing us to clarify these details. These discussions are incredibly helpful for us and the community.