-
|
I got an error during evaluating locomo benchmark through memos. However, I can't reproduce it even when I keep the same input and define the seed of ollama in
Currently I suggest that someone be assigned to:
For someone to reproduce the error, running the following commands in
|
Beta Was this translation helpful? Give feedback.
Replies: 0 comments 5 replies
-
|
This issue will be scheduled in the near future but will temporarily remain unaddressed. Below are some potential solutions for future reference:
该问题将在近期安排处理，但目前暂时搁置。以下是一些可行的备选方案：
|
Beta Was this translation helpful? Give feedback.
-
|
I'll reassign this issue to you because we now have the MemoryItem base model (https://github.com/MemTensor/MemOS/blob/main/src/memos/memories/textual/item.py), which enables constrained decoding. Here's what you need to do:
我把这个issue重新分配给你，因为我们现在已经有了MemoryItem模型（https://github.com/MemTensor/MemOS/blob/main/src/memos/memories/textual/item.py），可以实现约束解码了。 你需要：
|
Beta Was this translation helpful? Give feedback.
-
|
While evaluating from pydantic import BaseModel
from outlines import models, generate
class User(BaseModel):
name: str
last_name: str
id: int
model = models.transformers("microsoft/Phi-3-mini-4k-instruct")
generator = generate.json(model, User)
result = generator("Create a user profile with the fields name, last_name and id")
print(result) # ➜ User(name='John', last_name='Doe', id=11)Since |
Beta Was this translation helpful? Give feedback.
-
|
After careful consideration, I decided we will not adopt For instance, I attempted a vivid example: import uuid
from datetime import datetime
from typing import Literal
from outlines import generate, models
from pydantic import BaseModel, ConfigDict, Field, field_validator
class TextualMemoryMetadata(BaseModel):
"""Metadata for a memory item.
This includes information such as the type of memory, when it occurred,
its source, and other relevant details.
"""
user_id: str | None = Field(
default=None,
description="The ID of the user associated with the memory. Useful for multi-user systems.",
)
session_id: str | None = Field(
default=None,
description="The ID of the session during which the memory was created. Useful for tracking context in conversations.",
)
status: Literal["active", "archived", "deleted"] | None = Field(
default=None,
description="The status of the memory, e.g., 'active', 'archived', 'deleted'.",
)
type: Literal["procedure", "fact", "event", "opinion"] | None = Field(default=None)
memory_time: str | None = Field(
default=None,
description='The time the memory occurred or refers to. Must be in standard `YYYY-MM-DD` format. Relative expressions such as "yesterday" or "tomorrow" are not allowed.',
)
source: Literal["conversation", "retrieved", "web", "file"] | None = Field(
default=None, description="The origin of the memory"
)
confidence: float | None = Field(
default=None,
description="A numeric score (float between 0 and 100) indicating how certain you are about the accuracy or reliability of the memory.",
)
entities: list[str] | None = Field(
default=None,
description='A list of key entities mentioned in the memory, e.g., people, places, organizations, e.g., `["Alice", "Paris", "OpenAI"]`.',
)
tags: list[str] | None = Field(
default=None,
description='A list of keywords or thematic labels associated with the memory for categorization or retrieval, e.g., `["travel", "health", "project-x"]`.',
)
visibility: Literal["private", "public", "session"] | None = Field(
default=None, description="e.g., 'private', 'public', 'session'"
)
updated_at: str | None = Field(
default_factory=lambda: datetime.now().isoformat(),
description="The timestamp of the last modification to the memory. Useful for tracking memory freshness or change history. Format: ISO 8601.",
)
model_config = ConfigDict(extra="forbid")
@field_validator("memory_time")
@classmethod
def validate_memory_time(cls, v):
try:
if v:
datetime.strptime(v, "%Y-%m-%d")
except ValueError as e:
raise ValueError("Invalid date format. Use YYYY-MM-DD.") from e
return v
@field_validator("confidence")
@classmethod
def validate_confidence(cls, v):
if v is not None and (v < 0 or v > 100):
raise ValueError("Confidence must be between 0 and 100.")
return v
class TextualMemoryItem(BaseModel):
"""Represents a single memory item in the textual memory.
This serves as a standardized format for memory items across different
textual memory implementations.
"""
id: str = Field(default_factory=lambda: str(uuid.uuid4()))
memory: str
metadata: TextualMemoryMetadata = Field(default_factory=TextualMemoryMetadata)
model_config = ConfigDict(extra="forbid")
@field_validator("id")
@classmethod
def validate_id(cls, v):
try:
uuid.UUID(v)
except ValueError as e:
raise ValueError("Invalid UUID format") from e
return v
@classmethod
def from_dict(cls, data: dict) -> "TextualMemoryItem":
return cls(**data)
def to_dict(self) -> dict:
return self.model_dump(exclude_none=True)
def __str__(self) -> str:
"""Pretty string representation of the memory item."""
meta = self.metadata.model_dump(exclude_none=True)
meta_str = ", ".join(f"{k}={v}" for k, v in meta.items())
return f"<ID: {self.id} | Memory: {self.memory} | Metadata: {meta_str}>"
EXTRACTION_PROMPT_PART_1 = f"""You are a memory extractor. Your task is to extract memories from the given messages.
* You will receive a list of messages, each with a role (user or assistant) and content.
* Your job is to extract memories related to the user's long-term goals, interests, and emotional states.
* Each memory should be a dictionary with the following keys:
- "memory": The content of the memory (string). Rephrase the content if necessary.
- "metadata": A dictionary containing additional information about the memory.
* The metadata dictionary should include:
- "type": The type of memory (string), e.g., "procedure", "fact", "event", "opinion", etc.
- "memory_time": The time the memory occurred or refers to (string). Must be in standard `YYYY-MM-DD` format. Relative expressions such as "yesterday" or "tomorrow" are not allowed.
- "source": The origin of the memory (string), e.g., `"conversation"`, `"retrieved"`, `"web"`, `"file"`.
- "confidence": A numeric score (float between 0 and 100) indicating how certain you are about the accuracy or reliability of the memory.
- "entities": A list of key entities (array of strings) mentioned in the memory, e.g., people, places, organizations, e.g., `["Alice", "Paris", "OpenAI"]`.
- "tags": A list of keywords or thematic labels (array of strings) associated with the memory for categorization or retrieval, e.g., `["travel", "health", "project-x"]`.
- "visibility": The accessibility scope of the memory (string), e.g., `"private"`, `"public"`, `"session"`, determining who or what contexts can access it.
- "updated_at": The timestamp of the last modification to the memory (string). Useful for tracking memory freshness or change history. Format: ISO 8601 or natural language.
* Current date and time is {datetime.now().isoformat()}.
* Only return the list of memories in JSON format.
* Do not include any other text or explanation.
## Example
### Input
[
{{"role": "user", "content": "I plan to visit Paris next week."}},
{{"role": "assistant", "content": "Paris is a beautiful city with many attractions."}},
{{"role": "user", "content": "I love the Eiffel Tower."}},
{{"role": "assistant", "content": "The Eiffel Tower is a must-see landmark in Paris."}}
]
### Output
[
{{
"memory": "The user plans to visit Paris on 05-26-2025.",
"metadata": {{
"type": "event",
"memory_time": "2025-05-26",
"source": "conversation",
"confidence": 90.0,
"entities": ["Paris"],
"tags": ["travel", "plans"],
"visibility": "private",
"updated_at": "2025-05-19T00:00:00"
}}
}},
{{
"memory": "The user loves the Eiffel Tower.",
"metadata": {{
"type": "opinion",
"memory_time": "2025-05-19",
"source": "conversation",
"confidence": 100.0,
"entities": ["Eiffel Tower"],
"tags": ["opinions", "landmarks"],
"visibility": "session",
"updated_at": "2025-05-19T00:00:00"
}}
}}
]
"""
EXTRACTION_PROMPT_PART_2 = """
## Query
### Input
{messages}
### Output
"""
messages = [
{"role": "user", "content": "I love eating bananas."},
{"role": "assistant", "content": "Bananas are a great source of potassium."},
]
model = models.transformers("Qwen/Qwen2.5-0.5B-Instruct")
generator = generate.json(model, TextualMemoryItem)
result = generator(EXTRACTION_PROMPT_PART_1 + EXTRACTION_PROMPT_PART_2.format(messages=messages))
print(result)It took approximately thirty minutes (nearly all the time was related to |
Beta Was this translation helpful? Give feedback.
-
|
Here’s a best practice suggested by @J1awei-Yang to help others when facing issues with JSON data extraction. Best Practice: Reliable JSON Extraction from LLM Responses with RetryThe ProblemWhen calling a language model (LLM) to generate structured JSON output, it’s common for the model to occasionally return malformed JSON—such as extra text, missing commas, or unclosed braces. This causes response = llm.generate(prompt)
json.loads(response) # 💥 Raises JSONDecodeError if the response is not valid JSONThe Solution: Wrap Both the LLM Call and JSON Parsing in a RetryUse a retry mechanism (e.g., Tenacity) to automatically retry the entire process—regenerating the LLM output and re-parsing it—whenever a from tenacity import retry, retry_if_exception_type, stop_after_attempt
import json
import logging
logger = logging.getLogger(__name__)
@retry(
stop=stop_after_attempt(3),
retry=retry_if_exception_type(json.JSONDecodeError),
before_sleep=lambda state: logger.warning(
f"Failed to parse JSON from LLM output (attempt {state.attempt_number}): {state.outcome.exception()}"
),
)
def extract_structured_output(llm, prompt: str) -> list[dict]:
"""Call the LLM and parse its JSON output, retrying on failures."""
response = llm.generate([{"role": "user", "content": prompt}])
return json.loads(response)Usage Exampleuser_prompt = build_extraction_prompt(messages)
structured_data = extract_structured_output(llm, user_prompt)Bonus Tips
以下是@J1awei-Yang建议的一个最佳实践，用于帮助他人解决提取JSON数据时遇到的问题。 最佳实践：通过重试从LLM响应中可靠提取JSON问题描述当调用语言模型（LLM）生成结构化JSON输出时，模型偶尔会返回格式错误的JSON——例如多余文本、缺失逗号或未闭合的括号等。这会导致 response = llm.generate(prompt)
json.loads(response) # 💥 如果响应不是有效的JSON，将抛出JSONDecodeError解决方案：将LLM调用和JSON解析同时包装在重试机制中使用重试机制（如Tenacity），在发生 from tenacity import retry, retry_if_exception_type, stop_after_attempt
import json
import logging
logger = logging.getLogger(__name__)
@retry(
stop=stop_after_attempt(3),
retry=retry_if_exception_type(json.JSONDecodeError),
before_sleep=lambda state: logger.warning(
f"Failed to parse JSON from LLM output (attempt {state.attempt_number}): {state.outcome.exception()}"
),
)
def extract_structured_output(llm, prompt: str) -> list[dict]:
"""调用LLM并解析其JSON输出，失败时自动重试。"""
response = llm.generate([{"role": "user", "content": prompt}])
return json.loads(response)使用示例user_prompt = build_extraction_prompt(messages)
structured_data = extract_structured_output(llm, user_prompt)额外建议
|
Beta Was this translation helpful? Give feedback.
Here’s a best practice suggested by @J1awei-Yang to help others when facing issues with JSON data extraction.
Best Practice: Reliable JSON Extraction from LLM Responses with Retry
The Problem
When calling a language model (LLM) to generate structured JSON output, it’s common for the model to occasionally return malformed JSON—such as extra text, missing commas, or unclosed braces. This causes
json.loads()to raise aJSONDecodeError.The Solution: Wrap Both the LLM Call and JSON Parsing in a Retry
Use a retry mechanism (e.g., Tenacity) to automatically retry the entire process—…