Handling null values in Milvus

[Mahanchello]

Hey, just wanted to suggest the feature on adding handling for null, None, NaN values in Milvus.
There are workarounds like storing an empty string, -1 for ints, but could be really cool if Milvus would handle such values as well without erroring out on unknown DataType when inserting the data.
Thank you!

[xiaofan-luan]

Ahh you r right. Null support is definitely important on some cases.
I'm planning to support it on 2.4 but if anyone want to take it we can make it happen faster!

[jiaoew1991]

@smellthemoon and I are working on that, if you have some suggestion we can discuss here. 😊

[hakan458]

I just ran into this issue as well. I am inserting DataFrames in batches and I am hitting an exception when any row in a column has a null value. For some reason that causes the data type returned in the mapping to be UNKNOWN.
Right now I am going to handle this on my end using fillna() on my DataFrames but it would be great if Milvus could handle this!

@jiaoew1991 @smellthemoon have you made any progress on this? I could possibly help as well

[xiaofan-luan]

Good catch!

We are working on this feature for a while, should be able to release it on 2.4?

@smellthemoon @jiaoew1991

[Abhijnan-Bajpai]

Any updates on this feature?

[junjie-landing]

any update?

[xiaofan-luan]

there are already some pr for null support
#32183
#33602
The feature will be released with 2.5

[arrrrny]

same here. definitely need this

[smellthemoon]

same here. definitely need this

releted issue：#31728