how to set diskann disk size

[Uijeong97]

I want to use disk index using diskann.
But in milvus, the local storage path has a disk capacity of 100MB.

I have '/overlay' disk. and the local storage path set on /overlay
but I think the auto-detected capacity is too small for diskann to run properly.

This issue looks similar to this one. #30742

and I tried set LOCAL_STORAGE_SIZE environment variable of query, index node,
but it is no change.

What should I set up?

[Uijeong97]

I'm having issues starting with the index build.

[2024/07/09 15:33:18.915 +09:00] [DEBUG] [indexnode/indexnode_service.go:162] ["querying index build task"] [traceID=e89e51fe3525775d44e8c4ed2b797d12] [clusterID=by-dev-cr2-240704] [indexBuildID=451019045677138279] [state=Retry] [reason="failed to create index, C Runtime Exception: bad_function_call\n: segcore error[segcoreCode=2001]"]

[yhmo]

The issue #30742 mentioned the localStorage.path should be mounted to a correct disk.

localStorage:
  path: /var/lib/milvus/data/ # please adjust in embedded Milvus: /tmp/milvus/data/

[yhmo]

This doc mentioned NVMe SSD is required for disk index. NVME SSD is preferred. I am not sure what is NVME RDD and whether it is as fast as NVMe SSD.
https://milvus.io/docs/disk_index.md#DiskANN-related-Milvus-configurations

The Milvus data path should be mounted to an NVMe SSD for full performance
- For a Milvus Standalone instance, the data path should be /var/lib/milvus/data in the container where the instance runs.
- For a Milvus Cluster instance, the data path should be /var/lib/milvus/data in the containers where the QueryNodes and IndexNodes run.

[Uijeong97]

@yhmo

Sorry for the typo, I just want to say I have block storage hdd.

I'm wondering if it is possible to set the HDD disk as a disk index storage type,
if willing to sacrifice performance (like qps.).

[yhmo]

I didn't get a chance to test disk index running on a HDD disk. The performance must be poorer than NVMe SSD, I think it might be 5x~10x slower.

[Uijeong97]

@yhmo
Thank you for your answer. I'll have to try it, as the data volume is more than 1 billion.

[xiaofan-luan]

HDD might not be an option. Even cloud disk won't work.

DiskANN requires disk with at least 50000+ iops. Each topk requires 5-10 iops, that is saying, with 10 QPS and 100 topk, your disk will need roughly 10000 iops to handle. But HDD offers hundred iops mostly