[Bug]: [woodpecker] After inserting 20 millio entities serially, both count(*) and binlog show that there are more data than expected

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20250605-e3826c29-amd64
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server

• mq: woodpecker
• config

    common:
      enabledJSONKeyStats: true
    dataCoord:
      enableActiveStandby: true
    indexCoord:
      enableActiveStandby: true
    log:
      level: debug
    queryCoord:
      enableActiveStandby: true
    rootCoord:
      enableActiveStandby: true
    streaming:
      walWriteAheadBuffer:
        capacity: 1m
        keepalive: 0.5s
    woodpecker:
      storage:
        type: local

client

1. create a collection with schema:

{'auto_id': False,
 'description': '',
 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}},
            {'name': 'json_1', 'description': '', 'type': <DataType.JSON: 23>}],
 'enable_dynamic_field': False} (base.py:329)

2. create vector index -> insert 20m data -> flush -> index again -> load
   
   
   

Expected Behavior

No response

Steps To Reproduce

zong-wp-stable-1-672269452

Milvus Log

pods:

wp-upsert-op-57-1037-etcd-0                                       1/1     Running     0                59m     10.104.32.29    4am-node39   <none>           <none>
wp-upsert-op-57-1037-milvus-standalone-764874f5f9-bcfcq           1/1     Running     0                58m     10.104.25.175   4am-node30   <none>           <none>
wp-upsert-op-57-1037-minio-7966f4878d-dkkpf                       1/1     Running     0                59m     10.104.32.28    4am-node39   <none>           <none>

Anything else?

No response

[xiaofan-luan]

@tinswzy

Do we have a idempotent id for each writes？

So even if we do retry it won't change the data

[yanliang567]

/assign @tinswzy
/unassign