Connection Refused on port 53100

[owenwould]

Hi,
Recently we have started using milvus, we have set up milvus on a ecs service (fargate) configured with s3. We can see the logs when inserting data into milvus so we are confident s3 is configured properly. We have noticed the below warning error in our logs and we are wondering if this was something we should just ignore or will cause milvus not to work.
`
resource "aws_ecs_task_definition" "milvus" {
family = "${var.instance_name}-milvus"
network_mode = "awsvpc"
pid_mode = "task"
requires_compatibilities = ["FARGATE"]

task_role_arn = aws_iam_role.milvus.arn
execution_role_arn = aws_iam_role.milvus_execution.arn

cpu = 2048
memory = 4096

container_definitions = jsonencode([
{
name = "milvus-etcd"

  command = ["etcd", "-advertise-client-urls", "http://127.0.0.1:2379", "-listen-client-urls", "http://0.0.0.0:2379", "--data-dir", "/etcd"]

  cpu              = 1024
  memory           = 2048
  essential        = true

  environment = [
    { name = "ETCD_AUTO_COMPACTION_MODE", value = "revision" },
    { name = "ETCD_AUTO_COMPACTION_RETENTION", value = "1000" },
    { name = "ETCD_QUOTA_BACKEND_BYTES", value = "4294967296" },
    { name = "ETCD_SNAPSHOT_COUNT", value = "50000" }
  ]

  healthCheck = {
    command  = ["CMD", "etcdctl", "endpoint", "health"],
    interval = 30,
    timeout  = 20,
    retries  = 3
  }

},
{
  name                  = "milvus-standalone"
  
  portMappings = [
    { hostPort = 19530, containerPort = 19530, protocol = "tcp" },
    { hostPort = 9091, containerPort = 9091, protocol = "tcp" },
  ]

  command = ["milvus", "run", "standalone"]

  cpu              = 1024
  memory           = 2048
  essential        = true

  environment = [
    { name = "ETCD_ENDPOINTS", value = "127.0.0.1:2379" },
    { name = "ETCD_LOG_LEVEL", value = "error" },
    { name = "LOG_LEVEL", value = "error" },
    { name = "PROXY_GIN_LOGGING", value = "false" },
    { name = "MINIO_LOG_LEVEL", value = "error" },
    { name = "MINIO_BUCKET_NAME", value = var.milvus_bucket.bucket },
    { name = "MINIO_ADDRESS", value = "s3.${var.aws_region}.amazonaws.com" },
    { name = "MINIO_REGION", value = var.aws_region },
    { name = "MINIO_PORT", value = "443" },
    { name = "MINIO_USE_SSL", value = "true" },
    { name = "MINIO_USE_IAM", value = "true" }, 
    { name = "MINIO_ACCESS_KEY_ID", value = "" },
    { name = "MINIO_SECRET_ACCESS_KEY", value = "" },
    { name = "ROOT_COORD_MIN_SEGMENT_SIZE_TO_ENABLE_INDEX", value = "100" }
  ]   
  }

  healthCheck = {
    command      = ["CMD", "curl", "-f", "http://localhost:9091/healthz"],
    interval     = 30,
    start-period = 90,
    timeout      = 20,
    retries      = 3
  }

  dependsOn = [
    { containerName = "milvus-etcd", condition = "HEALTHY" }
  ]
}

])

}
`

[yhmo]

Sorry missed this issue.
53100 is the port of rootcoord node. "Connection refused" indicated the rootcoord failed to initialize. A possible reason is the rootcoord could not connect to the Etcd service.
You can check the log of the rootcoord, there should be errors about etcd.

[owenwould]

Thanks I'll take a look

[owenwould]

I think the errors were just due to it taking some time for the rootcoord to initialise and erroring until it was ready, we increased the resources for milvus and so far have not seen it occur again.