Clarification on partitioner usage: discrepancy between known issue comment in Producer.c and test case test_set_partitioner_murmur2_random

[arossowork]

Description

While reading through the codebase, I came across a comment in src/confluent_kafka/src/Producer.c that states:

/**
 * @brief KNOWN ISSUES
 *
 *  - Partitioners will cause a dead-lock with librdkafka, because:
 *     GIL + topic lock in topic_new  is different lock order than
 *     topic lock in msg_partitioner + GIL.
 *     This needs to be sorted out in librdkafka, preferably making the
 *     partitioner run without any locks taken.
 *     Until this is fixed the partitioner is ignored and librdkafka's
 *     default will be used.
 *
 */

However, in tests/test_Producer.py, I noticed a test function named test_set_partitioner_murmur2_random() which appears to validate the ability to set a partitioner without error.

This leads to some confusion on my end:

• Is the comment in Producer.c outdated and no longer valid?
• Or is test_set_partitioner_murmur2_random() testing only the API call, but the partitioner is actually not used at runtime (due to being ignored internally)?
• Could this represent a partial incompatibility, or is it just a legacy note that should be removed or clarified?

Environment

• Version: confluent-kafka==2.10.0
• Platform: N/A (just reading the source)

What I'm looking for

Clarification on whether custom partitioners are actually used at runtime in the Python client, and if the warning about deadlocks is still applicable in the current version.

[pranavrth]

The warning is still applicable. We still don't support custom partitioner. The test that you are referring is for the predefined partitioner present in librdkafka.

def test_set_partitioner_murmur2_random():
    """
    Test ability to set built-in partitioner type murmur2_random
    """
    Producer({'partitioner': 'murmur2_random'})