Some questions about hybrid search, WeightedRanker and COSINE score

[LoveFishoO]

The website docs https://milvus.io/docs/reranking.md#Weighted-Scoring-WeightedRanker
The source code https://github.com/milvus-io/milvus/blob/648d5661ca8771fadc664f427ac330b083b8734e/internal/proxy/reScorer.go

Question 1

According the website docs, Milvus use $arctan$ to normalize the distance from different routes. However, according to the source code, for COSINE, Milvus use $(distance + 1) / 2$ to normalize this distance.

milvus/internal/proxy/reScorer.go

Line 90 in 648d566

return (1 + distance) * 0.5

Question 2

I searched for 4 columns using hybrid search and WeightedRanker.But the final score does not match my expectations.

Case:

# index params
index.add_index(field_name="vec1", index_type="FLAT", metric_type="COSINE")
index.add_index(field_name="vec2", index_type="FLAT", metric_type="COSINE")
index.add_index(field_name="vec3", index_type="FLAT", metric_type="COSINE")
index.add_index(field_name="vec4", index_type="FLAT", metric_type="COSINE")

# weight
rerank = WeightedRanker(0.3, 0.15, 0.25, 0.3)

# search param
for field in ['vec1','vec2','vec3','vec4']:
  search_param = {
                  "data": [query_vec],
                  "anns_field": field , 
                  "param": {
                      "metric_type": "COSINE", 
                      "radius": 0.99,
                      "range_filter": 1.0 
                  },
                  "limit": 50
              }

## Score calculated by myself
The cosine sim between query_vec and vec1 is 0.8377
The cosine sim between query_vec and vec2 is 0.8451
The cosine sim between query_vec and vec3 is 0.9690
The cosine sim between query_vec and vec4 is 0.9304

My expected scores is 0.3 * (0.8377+1) / 2 + 0.15 * (0.8451+1) / 2 + 0.25 * (0.9690+1) / 2 + 0.3 * (0.9304+1) / 2 = 0.9497

## Score from Milvus
0.2895

Question 3

In the case from Question2, I set radius and range_filter to filter. But why are data with scores below 0.99 not filtered?

---

Looking for your reply
kind regards

[yhmo]

Q1
The doc mentioned "For instance, the distance for IP ranges from [-∞,+∞], while the distance for L2 ranges from [0,+∞]. Milvus employs the arctan function, transforming values to the [0,1] range to provide a standardized basis for different metric types.".
Arctan is for IP and L2 whose range is unlimited(∞).
COSINE range is [-1, 1], is limited, the (1 + distance) * 0.5 makes sense.

        if isCosine {
		f := func(distance float32) float32 {
			return (1 + distance) * 0.5
		}
		return f
	}

	if isIP {
		f := func(distance float32) float32 {
			return 0.5 + float32(math.Atan(float64(distance)))/math.Pi
		}
		return f
	}

	f := func(distance float32) float32 {
		return 1.0 - 2*float32(math.Atan(float64(distance)))/math.Pi
	}

[LoveFishoO]

Thanks for your reply. How could i solve the Q2? This question prevents me from setting a threshold to filter the results.

[yhmo]

Q2
I write a script to test:
A collection with 4 vector fields. The collection only has 1 row.

import random

from pymilvus import (
    DataType, CollectionSchema, FieldSchema, Collection,
    utility, connections, AnnSearchRequest, WeightedRanker,
)

dim = 128
collection_name = "AAA"
metric_type = "COSINE"

connections.connect(host='localhost', port='19530')
print(utility.get_server_version())

if utility.has_collection(collection_name):
    utility.drop_collection(collection_name)

schema = CollectionSchema(fields=[
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=False),
    FieldSchema(name="vec1", dtype = DataType.FLOAT_VECTOR, dim=dim),
    FieldSchema(name="vec2", dtype = DataType.FLOAT_VECTOR, dim=dim),
    FieldSchema(name="vec3", dtype = DataType.FLOAT_VECTOR, dim=dim),
    FieldSchema(name="vec4", dtype = DataType.FLOAT_VECTOR, dim=dim),
])

collection = Collection(name=collection_name, schema=schema)
print(f"Collection '{collection_name}' created")

data = [
    [1], # id
    [[random.random() for _ in range(dim)]],  # vec1
    [[random.random() for _ in range(dim)]],  # vec2
    [[random.random() for _ in range(dim)]],  # vec3
    [[random.random() for _ in range(dim)]],  # vec4
]
collection.insert(data=data)

index_params = {
    'metric_type': metric_type,
    'index_type': "FLAT",
    'params': {},
}
collection.create_index(field_name="vec1", index_params=index_params)
collection.create_index(field_name="vec2", index_params=index_params)
collection.create_index(field_name="vec3", index_params=index_params)
collection.create_index(field_name="vec4", index_params=index_params)
collection.load()

req_list = []
weights = [0.3, 0.15, 0.25, 0.3]
query_vector = [random.random() for _ in range(dim)]
distances = []
for field_name in ["vec1", "vec2", "vec3", "vec4"]:
    search_param = {
        "data": [query_vector],
        "anns_field": field_name,
        "param": {"metric_type": metric_type},
        "limit": 10}
    res = collection.search(**search_param)
    print(f"Normal search result for field '{field_name}':\n{res}")
    distances.append(res[0][0].distance)

    req = AnnSearchRequest(**search_param)
    req_list.append(req)

hybrid_res = collection.hybrid_search(req_list, WeightedRanker(*weights), 10)

print("\nHybrid search by WightedRanker")
for hits in hybrid_res:
    for hit in hits:
        print(hit)

sum = 0.0
for i in range(len(distances)):
    active = (1 + distances[i])*0.5 # active func is (1 + distance) * 0.5
    sum = sum + weights[i]*active
print(f"\nSum of weight*active_func(distance): {sum}")

Run this script by several times, the "Sum of weight*active_func(distance)" is always equal to the result of hybrid search.

[yhmo]

Normal search result for field 'vec1':
data: ["['id: 1, distance: 0.7834256887435913, entity: {}']"], cost: 0
Normal search result for field 'vec2':
data: ["['id: 1, distance: 0.7359726428985596, entity: {}']"], cost: 0
Normal search result for field 'vec3':
data: ["['id: 1, distance: 0.7687178254127502, entity: {}']"], cost: 0
Normal search result for field 'vec4':
data: ["['id: 1, distance: 0.7612380385398865, entity: {}']"], cost: 0

Hybrid search by WightedRanker
id: 1, distance: 0.8829872608184814, entity: {}

Sum of weight*active_func(distance): 0.8829872354865074

[LoveFishoO]

Thanks a lot for your test script!
I found the reason because the error data appears less than 4 times in search results. Like the Q2 case
Check the id: 1095730

Normal search result for field 'vec1':
data: ["['id: 5321679, distance: 0.9257609248161316, entity: {}', 'id: 1135952, distance: 0.9257609248161316, entity: {}', 'id: 1135827, distance: 0.9257609248161316, entity: {}', 'id: 5410560, distance: 0.9221035242080688, entity: {}', 'id: 4320232, distance: 0.8955430388450623, entity: {}', 'id: 1031482, distance: 0.8906382322311401, entity: {}', 'id: 5300266, distance: 0.8859068155288696, entity: {}', 'id: 1092703, distance: 0.885138750076294, entity: {}', 'id: 5300186, distance: 0.8829507231712341, entity: {}', 'id: 1084254, distance: 0.8824491500854492, entity: {}']"], cost: 0
Normal search result for field 'vec2':
data: ["['id: 1061180, distance: 0.8648502230644226, entity: {}', 'id: 1061177, distance: 0.8648502230644226, entity: {}', 'id: 1061174, distance: 0.8648502230644226, entity: {}', 'id: 1061173, distance: 0.8648502230644226, entity: {}', 'id: 1060306, distance: 0.8648502230644226, entity: {}', 'id: 1060258, distance: 0.8648502230644226, entity: {}', 'id: 1058529, distance: 0.8648502230644226, entity: {}', 'id: 1046944, distance: 0.8648502230644226, entity: {}', 'id: 1041281, distance: 0.8648502230644226, entity: {}', 'id: 1026222, distance: 0.8648502230644226, entity: {}']"], cost: 0
Normal search result for field 'vec3':
data: ["['id: 1070025, distance: 0.9690285921096802, entity: {}', 'id: 1070007, distance: 0.9690285921096802, entity: {}', 'id: 1070001, distance: 0.9690285921096802, entity: {}', 'id: 1070000, distance: 0.9690285921096802, entity: {}', 'id: 1069659, distance: 0.9690285921096802, entity: {}', 'id: 1060453, distance: 0.9690285921096802, entity: {}', 'id: 1060452, distance: 0.9690285921096802, entity: {}', 'id: 1048630, distance: 0.9690285921096802, entity: {}', 'id: 1088464, distance: 0.9690285921096802, entity: {}', 'id: 1087875, distance: 0.9690285921096802, entity: {}']"], cost: 0
Normal search result for field 'vec4':
data: ["['id: 1087875, distance: 0.9690285921096802, entity: {}', 'id: 1138221, distance: 0.9690285921096802, entity: {}', 'id: 5300266, distance: 0.9690285921096802, entity: {}', 'id: 1132628, distance: 0.9690285921096802, entity: {}', 'id: 1136894, distance: 0.9690285921096802, entity: {}', 'id: 1086371, distance: 0.9690284132957458, entity: {}', 'id: 1073907, distance: 0.9690284132957458, entity: {}', 'id: 1136783, distance: 0.9690284132957458, entity: {}', 'id: 1138130, distance: 0.9690284132957458, entity: {}', 'id: 1095730, distance: 0.930426836013794, entity: {}']"], cost: 0

0.3 * (0+1) / 2 + 0.15 * (0+1) / 2 + 0.25 * (0+1) / 2 + 0.3  * (0.9304 + 1) / 2 = 0.2895

According to this logic, this score is correct.

Base on this logic, I have an anthor question. This data was obtained by searching for vec4 column. But why did not the score of vec1, vec2, vec3 column be calculated during the rerank phase?

According to my understanding, the first stage of retrieval is to use multiple ANN to recall data. Then, merge multiple data sources, calculate multi-dimensional scores for all data and sort them based on the scores.

Maybe based on Milvus logic, if a data can not be search by other route, this means this data is not similar in other route. However, due to the limitation of parameter limit, even data with scores greater than 0.98 may not be included in the results.

Anyway, Thanks a lot for your help!

[LoveFishoO]

Actually, I think the logic of Milvus may be problematic. What methods do I have to avoid score bias caused by a lack of scores.

[yhmo]

The script is just to show how the weight reranker works. All the 4 vector fields have the same dimension, the same vectors.
In practice, usually, different vector fields have different dimensions, different vectors come from different models. The query vectors also come from different models.
The reranker simply maps the results of different routes to range [0, 1] by (1 + distance) * 0.5, and sums up the weight values by distance1 * weight1 + distance2 * weight2 + ......
The reranker doesn't care what algorithm is applied to ANN search, it is just like a worker, "give me N routes topk result, let me sum up the weight values".

[LoveFishoO]

May I ask if there will be more powerful rerank or custom rerank functions updated in future versions?

[yhmo]

Q3
The search param is incorrect, should be:

search_param = {
        "data": [query_vector],
        "anns_field": field_name,
        "param": {
            "metric_type": metric_type,
            "params": {
                "radius": 0.9, # Radius of the search circle
                "range_filter": 1.0 # Range filter to filter out vectors that are not within the search circle
            }
        },
        "limit": 10}