Fix Semantic Query Rewrite Interception Drops Boosts

[Samiul-TheSoccerFan]

Closes ##128696

Match query: boost only on inference field

PUT my-index
{
  "mappings": {
    "properties": {
      "title": {
        "type": "text"
      },
      "author": {
        "type": "text"
      },
      "summary": {
        "type": "text",
        "copy_to": "semantic_summary"
      },
      "semantic_summary": {
        "type": "semantic_text"
      },
      "moral_lesson": {
        "type": "text",
        "copy_to": "semantic_moral"
      },
      "semantic_moral": {
        "type": "semantic_text"
      }
    }
  }
}

POST my-index/_doc/1
{
  "title": "Cinderella",
  "author": "Charles Perrault",
  "summary": "Cinderella is a young girl mistreated by her stepmother and stepsisters until she meets a fairy godmother.",
  "moral_lesson": "Goodness and kindness will always be rewarded."
}

POST my-index/_doc/2
{
  "title": "Little Red Riding Hood",
  "author": "Brothers Grimm",
  "summary": "A young girl encounters a cunning wolf on her way to visit her grandmother.",
  "moral_lesson": "Beware of strangers."
}

GET my-index/_search
{
  "query": {
    "match": {
      "semantic_moral": {
        "query": "danger",
        "boost": 2.0
      }
    }
  }
}

Match query: boost on both semantic_text and text fields

PUT normal-text-index/
{
  "mappings": {
    "properties": {
      "semantic1": {
        "type": "text"
      },
      "semantic2": {
        "type": "text"
      }
    }
  }
}

POST normal-text-index/_doc/1
{
  "semantic1": "Cinderella is a young girl mistreated by her stepmother and stepsisters until she meets a fairy godmother.",
  "semantic2": "Goodness and kindness will always be rewarded."
}

PUT semantic-text-index/
{
  "mappings": {
    "properties": {
      "semantic1": {
        "type": "semantic_text"
      },
      "semantic2": {
        "type": "semantic_text"
      }
    }
  }
}

POST semantic-text-index/_doc/1
{
  "semantic1": "A young girl encounters a cunning wolf on her way to visit her grandmother.",
  "semantic2": "Beware of strangers."
}

GET normal-text-index,semantic-text-index/_search
{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "semantic1": {
              "query": "wolf",
              "boost": 2.0
            }
          }
        },
        {
          "match": {
            "semantic2": {
              "query": "strangers",
              "boost": 1.0
            }
          }
        }
      ]
    }
  }
}

KNN

PUT _inference/text_embedding/my-e5-model
{
  "service": "elasticsearch",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1,
    "model_id": ".multilingual-e5-small"
  }
}

PUT test-knn-semantic-boost
{
  "mappings": {
    "properties": {
      "sem_field": {
        "type": "semantic_text",
        "inference_id": "my-e5-model"
      }
    }
  }
}

POST test-knn-semantic-boost/_doc/1?refresh=true
{
  "sem_field": "star wars droids"
}

POST test-knn-semantic-boost/_doc/2?refresh=true
{
  "sem_field": "fairy tale godmother"
}

// no boost given
GET test-knn-semantic-boost/_search
{
  "query": {
    "knn": {
      "field": "sem_field",
      "query_vector": [0.1, 0.2, 0.3],
      "k": 2,
      "num_candidates": 10
    }
  }
}

// boost preserve
GET test-knn-semantic-boost/_search
{
  "query": {
    "knn": {
      "field": "sem_field",
      "query_vector": [0.1, 0.2, 0.3], // the query throws error which is expected but the boost is propagated properly
      "k": 2,
      "num_candidates": 10,
      "boost": 5.0
    }
  }
}

Sparse

PUT test-sparse-semantic-boost
{
  "mappings": {
    "properties": {
      "sem_field": {
        "type": "semantic_text"
      }
    }
  }
}

POST test-sparse-semantic-boost/_doc/1?refresh=true
{
  "sem_field": "star wars droids"
}

POST test-sparse-semantic-boost/_doc/2?refresh=true
{
  "sem_field": "fairy tale godmother"
}

// sparse without boost
GET test-sparse-semantic-boost/_search
{
  "query": {
    "sparse_vector": {
      "field": "sem_field",
      "query": "driods"
    }
  }
}

// boost preserved
GET test-sparse-semantic-boost/_search
{
  "query": {
    "sparse_vector": {
      "field": "sem_field",
      "query": "driods",
      "boost": 5.0
    }
  }
}

[elasticsearchmachine]

Hi @Samiul-TheSoccerFan, I've created a changelog YAML for you.

[kderusso]

Looks great @Samiul-TheSoccerFan ! Can we please add some tests?

[Mikep86]

I'd prefer to see a solution based on copy constructors. Delegating the responsibility of generating a complete copy to the caller is error-prone and hard to maintain, which is how this class of bugs got introduced in the first place.

[Samiul-TheSoccerFan]

@elasticmachine update branch

[elasticmachine]

merge conflict between base and head

[Mikep86]

We still need to adjust the error ranges and address @kderusso's comments, everything else looks good.

[Samiul-TheSoccerFan]

@elasticmachine update branch

[Samiul-TheSoccerFan]

@elasticmachine update branch

[kderusso]

LGTM provided CI passes; thanks for all the iterations on this

[Mikep86]

LGTM!

[Samiul-TheSoccerFan]

@elasticmachine update branch

[elasticsearchmachine]

💔 Backport failed

Status	Branch	Result
❌	9.0	Commit could not be cherrypicked due to conflicts
❌	8.18	Commit could not be cherrypicked due to conflicts
❌	8.19	Commit could not be cherrypicked due to conflicts
❌	9.1	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 129282

[Samiul-TheSoccerFan]

💚 All backports created successfully

Status	Branch	Result
✅	9.1
✅	9.0
✅	8.19
✅	8.18

Questions ?

Please refer to the Backport tool documentation