chore: simplify schema creation from storage #1368
Merged
+27
−65
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
""" WalkthroughThe changes refactor schema creation logic by removing separate methods for creating schemas from ingestor and querier sources, consolidating them into a single method that fetches and stores schemas using a unified approach. Related function calls and helpers are updated or removed in accordance with this new schema retrieval strategy. Additionally, stream presence checks in the querier now consider schema emptiness to avoid redundant recreations. Changes
Sequence Diagram(s)sequenceDiagram
participant Caller
participant ObjectStorage
participant fetch_schema
Caller->>ObjectStorage: create_schema_from_storage(stream_name)
ObjectStorage->>fetch_schema: fetch_schema(stream_name)
fetch_schema-->>ObjectStorage: Schema object
ObjectStorage->>ObjectStorage: Serialize schema, store under schema path
ObjectStorage-->>Caller: Stored schema bytes
Possibly related PRs
Suggested labels
Suggested reviewers
Poem
📜 Recent review detailsConfiguration used: CodeRabbit UI 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
✨ Finishing Touches
🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
nikhilsinhaparseable
force-pushed
the
simplify-schema-merge
branch
from
8eb4934 to
29315f2
Compare
remove functions that creates schema from ingestors and queriers separately reused function `fetch_schema` that fetches all schema files and merges the schemas into one this ensures the schema is always the latest
nikhilsinhaparseable
force-pushed
the
simplify-schema-merge
branch
from
1a55a11 to
a4c8f78
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/handlers/http/query.rs (1)
442-446: Update the comment to match the corrected logic.The comment is confusing and doesn't accurately describe the intended behavior. After fixing the logical error, update the comment to be clearer.
- // fetch querier streams which have field list blank - // now missing streams should be list of streams which are in storage but not in querier - // and also have no fields in the schema - // this is to ensure that we do not create streams for querier which already exist in querier + // Find streams that need to be created/updated in the querier: + // 1. Streams that exist in storage but not in querier (need to be created) + // 2. Streams that exist in querier but have empty schema (need schema update) + // This avoids recreating streams that already exist with valid schemas
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
src/handlers/http/query.rs(1 hunks)src/migration/mod.rs(1 hunks)src/parseable/mod.rs(1 hunks)src/storage/object_storage.rs(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (3)
- src/parseable/mod.rs
- src/migration/mod.rs
- src/storage/object_storage.rs
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: de-sh
PR: parseablehq/parseable#1185
File: src/handlers/http/logstream.rs:255-261
Timestamp: 2025-02-14T09:49:25.818Z
Learning: In Parseable's logstream handlers, stream existence checks must be performed for both query and standalone modes. The pattern `!PARSEABLE.streams.contains(&stream_name) && (PARSEABLE.options.mode != Mode::Query || !PARSEABLE.create_stream_and_schema_from_storage(&stream_name).await?)` ensures proper error handling in both modes.
src/handlers/http/query.rs (2)
Learnt from: de-sh
PR: parseablehq/parseable#1185
File: src/handlers/http/logstream.rs:255-261
Timestamp: 2025-02-14T09:49:25.818Z
Learning: In Parseable's logstream handlers, stream existence checks must be performed for both query and standalone modes. The pattern `!PARSEABLE.streams.contains(&stream_name) && (PARSEABLE.options.mode != Mode::Query || !PARSEABLE.create_stream_and_schema_from_storage(&stream_name).await?)` ensures proper error handling in both modes.
Learnt from: nikhilsinhaparseable
PR: parseablehq/parseable#1340
File: src/query/mod.rs:64-66
Timestamp: 2025-06-18T06:39:04.775Z
Learning: In src/query/mod.rs, QUERY_SESSION_STATE and QUERY_SESSION serve different architectural purposes: QUERY_SESSION_STATE is used for stats calculation and allows dynamic registration of individual parquet files from the staging path (files created every minute), while QUERY_SESSION is used for object store queries with the global schema provider. Session contexts with schema providers don't support registering individual tables/parquets, so both session objects are necessary for their respective use cases.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (10)
- GitHub Check: Build Default x86_64-pc-windows-msvc
- GitHub Check: Build Default x86_64-apple-darwin
- GitHub Check: Build Default x86_64-unknown-linux-gnu
- GitHub Check: Build Default aarch64-apple-darwin
- GitHub Check: Build Default aarch64-unknown-linux-gnu
- GitHub Check: Build Kafka aarch64-apple-darwin
- GitHub Check: Build Kafka x86_64-unknown-linux-gnu
- GitHub Check: Quest Smoke and Load Tests for Distributed deployments
- GitHub Check: Quest Smoke and Load Tests for Standalone deployments
- GitHub Check: coverage
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
remove functions that creates schema from ingestors and queriers separately
reuse function
fetch_schemathat fetches all schema files and merges the schemas into onethis ensures the schema is always the latest
Summary by CodeRabbit