'Duplicate Route' connection floods on implicit routes

[wjordan]

Observed behavior

When a large cluster establishes connections to an implicit route, all of the nodes can flood a host with a large number of redundant 'Duplicate Route' connections.

In production with a large cluster of 100 nodes, I've observed intermittent floods of Router connection closed: Duplicate Route logged, up to 10k connections/sec to a single server, when a server experiences a broken/unreliable network connection to the rest of the cluster.

Expected behavior

I would expect 'Duplicate Route' connections to be relatively rare and the rate of connection attempts for any given route to be limited to the retry rate (one per second), instead of an unbounded flood of duplicate connections being sent all at once.

Server and client version

nats-server: v2.10.16

Host environment

No response

Steps to reproduce

My proposed fix in f3d3565 includes a test demonstrating the issue, counting the total number of 'Duplicate Route' log entries when establishing a 10-server cluster:

nats-server/server/routes_test.go

Lines 4357 to 4369 in f3d3565

	func TestImplicitDuplicateRoutes(t *testing.T) {
	c := createClusterWithName(t, "duplicateRoute", 10)
	duplicates := 0
	for _, s := range c.servers {
	logger := s.Logger().(*checkDuplicateRouteLogger)
	logger.Mutex.Lock()
	duplicates += logger.count
	logger.Mutex.Unlock()
	}
	if duplicates > 0 {
	t.Fatalf("Got %d duplicate routes, expected 0", duplicates)
	}
	}

Without any fix, the number of duplicate routes:

n=10	n=30	n=50	n=70
150	7000	38000	80000

My best understanding is that there's a feedback loop on implicit routes:

1. When a route is registered (addRoute), the server sends an INFO broadcast to all servers in the cluster (forwardNewRouteInfoToKnownServers).
2. When each server handles this INFO message (processImplicitRoute), it creates an implicit-route connection (connectToRoute).
3. When the implicit-route connection is registered (addRoute), see 1.

Although processImplicitRoute skips the connection if it's explicitly-configured or if the route already exists, this only checks against registered routes (not unregistered routes or unestablished connections), so a flood of connection attempts can pile up before a successful route registration prevents future connections.

With a fix that checks the route-connection count before dialing a new connection, the number of duplicate routes isn't completely eliminated for larger clusters, but is significantly less:

n=10	n=30	n=50	n=70
0	0	500	5000