High leader memory usage with millions of messages (jetstream)

[michaelmohamed]

Observed behavior

I am using nats-server w/ jetstream via helm charts on k8s to process about 40 million messages.

I have 30 pull-based consumers.

The leader (nats-2) seems to be using a lot of memory compared to the nats-0 and nats-1. Is this normal? Is there a way to minimize the amount of memory used?

Other info:

Expected behavior

Lower memory use.

Server and client version

nats-server: v2.10.16
nats-py: v2.7.2

Host environment

I am using nats-server via helm charts on k8s (EKS)

Steps to reproduce

No response

[neilalexander]

Can you please provide memory profiles? If you'd rather not attach here then please feel free to email them to neil@nats.io.

Using your system account:

nats server request profile allocs ~/path/to/folder --level=0

Please also provide the output of free -m from within each pod.