Skip to content

Kafka External Stream to support write the headers #904

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jovezhong opened this issue Feb 19, 2025 · 1 comment
Open

Kafka External Stream to support write the headers #904

jovezhong opened this issue Feb 19, 2025 · 1 comment
Assignees
Labels

Comments

@jovezhong
Copy link
Contributor

Use case

In #899 , we introduced the virtual column _tp_message_headers to read the headers as a map of string/string.

We also need to provide ways to write custom headers to Kafka. To comply with SQL INSERT syntax, the _tp_message_headers column probably need to be defined in the create external stream DDL.

This may not be implemented at high priority. If anyone has a strong use case, please leave comment below.

Describe the solution you'd like

Describe alternatives you've considered

Additional context

@zliang-min
Copy link
Collaborator

zliang-min commented Feb 21, 2025

This will be implemented exactly the same way as how _tp_message_key is implemented (in enterprise version) for writing: https://docs.timeplus.com/proton-kafka#messagekey .

I.e. firstly, create an external stream with specifying the column ( must use map(string, string) ):

CREATE EXTERNAL STREAM example (
  a int,
  b string,
  _tp_message_headers map(string, string)
) SETTINGS type='kafka', ...;

Then construct the headers when inserting data:

INSERT INTO example (a, b, _tp_message_headers) VALUES (1, 'one', {'key_1': 'value_1', 'key_2': 'value_2'});

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants