Support scalar field indexing for milvus collection creation

[rgupta2508]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

def _create_index(self) -> None:
    """Create a index on the collection"""
    from pymilvus import Collection, MilvusException

    if isinstance(self.col, Collection) and self._get_index() is None:
        try:
            # If no index params, use a default HNSW based one
            if self.index_params is None:
                self.index_params = {
                    "metric_type": "L2",
                    "index_type": "HNSW",
                    "params": {"M": 8, "efConstruction": 64},
                }

            try:
                self.col.create_index(
                    self._vector_field,
                    index_params=self.index_params,
                    using=self.alias,
                )

            # If default did not work, most likely on Zilliz Cloud
            except MilvusException:
                # Use AUTOINDEX based index
                self.index_params = {
                    "metric_type": "L2",
                    "index_type": "AUTOINDEX",
                    "params": {},
                }
                self.col.create_index(
                    self._vector_field,
                    index_params=self.index_params,
                    using=self.alias,
                )
            logger.debug(
                "Successfully created an index on collection: %s",
                self.collection_name,
            )

        except MilvusException as e:
            logger.error(
                "Failed to create an index on collection: %s", self.collection_name
            )
            raise e

Error Message and Stack Trace (if applicable)

No response

Description

We are trying to use Langchain_milvus library to create milvus collection using metadata. Now latest version of milvus support Scalar Index for other column also. we have requirement to add Scalar Index for batter performance in filtering data.

Currently langchain milvus support to add index only for VECTOR field only.

We can use metadata_schema logic to support indexing on Scalar fields.

#23219

System Info

langchain-core==0.2.20
langchain-community==0.2.7

[RafaelXokito]

Hi, @rgupta2508 did you considered creating the scalar indexes using the collection instance?

collection.create_index(field_name=field_name, index_name=f"{field_name}_index")

[rgupta2508]

Hi, @rgupta2508 did you considered creating the scalar indexes using the collection instance?

collection.create_index(field_name=field_name, index_name=f"{field_name}_index")

@RafaelXokito Yes we can create index like this using collection instance but idea is creating collection before embedding started using langchain. currently collection will be created if it is not exists.

Initially we dont know if collection exists or not?, if collection exists, no indexing required in collection but if new collection is created while embedding, immediately we want to add index also because after completion of embedding it will take some time to add index on that column.

But yes using, pymilvus we can do all the thing.

[RafaelXokito]

@rgupta2508 I don't think I understood your concern, but here's what I suggest to create all the scalar indices you want:

pymilvus.connections.connect(uri=uri)
_is_new_collection = not pymilvus.has_collection(collection_name)
if _is_new_collection:
    collection = Collection(collection_name, _source_schema.schema)

    index_params = {
        "metric_type": "L2",
        "index_type": "HNSW",
        "params": {"M": 8, "efConstruction": 64},
    }
    collection.create_index(_source_schema.vector.name, index_params=index_params)

    existing_index_names = {index.field_name for index in collection.indexes}
    for field_name in ["scalar_1", "scalar_2"]:  #Specify which scalar field names you want
        if field_name not in existing_index_names:
            collection.create_index(field_name=field_name, index_name=f"{field_name}_index")
    collection.load()

_vector_store = Milvus(embedding_function=self._embedding_function,
                                               collection_name=collection_name)

Take into account that milvus keep a connections pool that will reuse by an "alias". Which makes really reliable this kind of approach.

[rgupta2508]

@rgupta2508 I don't think I understood your concern, but here's what I suggest to create all the scalar indices you want:

pymilvus.connections.connect(uri=uri)
_is_new_collection = not pymilvus.has_collection(collection_name)
if _is_new_collection:
    collection = Collection(collection_name, _source_schema.schema)

    index_params = {
        "metric_type": "L2",
        "index_type": "HNSW",
        "params": {"M": 8, "efConstruction": 64},
    }
    collection.create_index(_source_schema.vector.name, index_params=index_params)

    existing_index_names = {index.field_name for index in collection.indexes}
    for field_name in ["scalar_1", "scalar_2"]:  #Specify which scalar field names you want
        if field_name not in existing_index_names:
            collection.create_index(field_name=field_name, index_name=f"{field_name}_index")
    collection.load()

_vector_store = Milvus(embedding_function=self._embedding_function,
                                               collection_name=collection_name)

Take into account that milvus keep a connections pool that will reuse by an "alias". Which makes really reliable this kind of approach.

Ok now i got it, you are suggesting to create collection using pymilvus and using langchain milvus object for embedding.
But vector we can get after calling embedding api and vector dimension we are getting from vectors in langchain for creating collection.
Can you suggest , i am not sure how we can handle this.

But we can try creating scalar index after creating collection.

[RafaelXokito]

@rgupta2508 thank you for your feedback, tell me if in any case you need help with it.