Direct answer

-Xmx caps Java heap, not process RSS. Thread stacks, JIT code, metaspace, direct buffers and native libraries can all consume additional memory.

How to check it

  1. Record heap flags with jcmd <PID> VM.flags and compare OS process RSS with the configured -Xmx.
  2. If diagnosing a suspected JVM-native growth issue, restart a staging server with -XX:NativeMemoryTracking=summary and collect jcmd <PID> VM.native_memory summary.
  3. After warm-up establish jcmd <PID> VM.native_memory baseline and later use summary.diff to identify growing categories.

What to change

java -XX:NativeMemoryTracking=summary -Xms4G -Xmx4G -jar paper.jar --nogui
jcmd <PID> VM.native_memory summary
jcmd <PID> VM.native_memory baseline
jcmd <PID> VM.native_memory summary.diff
  1. Leave reasonable RAM headroom for native JVM use and the OS rather than consuming the whole panel quota with Xmx.
  2. Investigate which category grows: thread stacks, class metadata, code cache or another area; inspect plugin/native allocations separately.
  3. Disable NMT again after diagnosis if overhead is not acceptable; do not assume NMT sees every JNI/native library allocation.

Software and version notes

Compatibility and safety

Oracle documents that NMT must be enabled at JVM startup, does not track all third-party native allocations, and has measurable overhead. This is a diagnostic option, not a permanent speed tweak.

Verify the result

Compare JVM committed native memory, operating-system RSS and container OOM counters after matching periods of uptime. Ensure the growth is reproducible.

References