about Changing the size of a VARCHAR field

[ian-lss]

Hello

I’m not sure if I missed it in the documentation,
but it seems that changing the size of a VARCHAR column is not supported in pymilvus.
If that’s the case, how can I change the field size while the service is online?

[yhmo]

max_length of a varchar field cannot be changed once the collection is created.
The max value of max_length is 65535. If you are unsure which value is proper, you can set max_length=65535 when creating the collection.

[ian-lss]

@yhmo

Thank you for your response.
If I always set the length of all VARCHAR fields to 65535,
what potential issues could arise?

[ian-lss]

I have one last question.
If there are no potential issues, why does Milvus require specifying the size for VARCHAR type instead of defaulting it to 65535?
Is it just to align with the interface of other databases?

[yhmo]

It is just like the VARCHAR of MySQL, variable string requires a max_length value to limit the length.
There is another type TEXT in MySQL, long length string without any limitation. Milvus doesn't support TEXT.

[ian-lss]

The reason for keeping the VARCHAR size tight in MySQL is that the buffer pool scans the storage space based on the VARCHAR size, not the actual data length.
Of course, the actual storage uses only the actual data length.
Additionally, since the VARCHAR length can be changed according to the service situation, it is possible to do so.

If Milvus scans the storage space for the query node based on the actual data length, then guiding developers to use 65535 for the max_length should not be an issue.
Since the length can be changed at any time from a service perspective, I plan to set it to 65535 as long as there are no performance issues. This is why I asked the question.

[yhmo]

Actually, the max_length value affects the internal logic of milvus (such as determining the size of insert buffer).

In ths source code of pkg/util/typeutil/schema.go

• EstimateSizePerRecord is used to determine the size of the insert buffer
• EstimateMaxSizePerRecord is used to determine the size of the read buffer of bulkinsert

max_length is calculated by these methods. So, different values will affect the internal behavior of milvus. From the source code, I didn't see it affects search/query performance.
An accurate max_length is welcome, 65535 is also ok if users don't like the limitation.

milvus/pkg/util/typeutil/schema.go

Line 105 in 9b37d3f

func estimateSizeBy(schema *schemapb.CollectionSchema, policy getVariableFieldLengthPolicy) (int, error) {

milvus/pkg/util/typeutil/schema.go

Line 50 in 9b37d3f

func getVarFieldLength(fieldSchema *schemapb.FieldSchema, policy getVariableFieldLengthPolicy) (int, error) {

[ian-lss]

@yhmo
Thank you for your response.
I fully understand your detailed explanation.

[xiaofan-luan]

/assign @PowderLi
please help on the feature of changing varchar length