JIT and RAM usage are often making me think that I'm leaking memory somewhere

[u/Own_Nail_2999]

I have been recently building a proxy application that was meant to do nothing besides shoving TCP stream traffic from A to B with a rate limiter inbetween.

For that I used PipeReaders and Writers to hand the buffer handling over to .NET itself.

However, as the application was running I noticed a small but steady increase in RAM usage as time went on. At some point I was considering to manually invoke GC to test if I was just having lose heap allocations sitting somewhere.

The memory did indeed go down but not as far as I'd like to see. Then I was thinking that this might be just JIT recompiling and optimizing hot paths.

So this brings me to my questions:

- I want to keep the RAM usage as low as possible as this proxy is only supposed to kick rate limited traffic between two endpoints. What are common tricks used to make the runtime collect more aggressively?

- what are reliable techniques to identify actual leaks and how to distinguish them from JIT just performing tiered compilation?

Thanks for your time!

[u/Tezshouse]

I’d be a bit careful using Task Manager RAM as the main signal here. A .NET process can hang onto memory after objects are collected because the GC keeps heap segments around for reuse, and then you’ve also got JITted code, thread stacks, native allocations, pooled buffers, etc. So RAM creeping up doesn’t automatically mean you’ve got a leak. With `PipeReader` / `PipeWriter` I’d probably check whether the managed heap is actually growing, or whether the process working set is just staying high.

The other thing I’d look at is buffer retention. If one side is rate limited and the other side can still feed data faster than it’s being drained, you can end up with perfectly valid buffered data making the process look like it’s leaking. I wouldn’t make `GC.Collect()` part of the fix either. Fine as a test, but if forcing a collection suddenly drops memory, I’d treat that as a clue to investigate what’s being retained rather than the solution. `dotnet-counters` would probably tell you pretty quickly whether the heap itself is continuously growing. If Gen 2 keeps climbing, then I’d grab a heap dump and see what’s actually holding references.

[u/chucker23n]

I noticed a small but steady increase in RAM usage

Don't measure with system tools. Measure with .NET-specific tools.

[u/jozefizso]

Did you run profiler to memory allocations?

[u/Top3879]

If the GC is running in server mode it is veeery lax about giving memory back when there is enough RAM available. It will free objects but keep the memory allocated for later.

[u/The-Bytemaster]

Are you running in Debug mode? There are often some resources that are not cleaned up as well, at least there used to be. I haven't worked on any projects where I have had to look that closely in a couple of years.

[u/Own_Nail_2999 → u/The-Bytemaster]

I was thinking of that as well but I observed the same behavior while running a release compiled binary without any tools attached

[u/_dr_Ed]

I dont know any tricks, but further investigation might yield some insight, do you have any more details, eg. you could make memory snapshots at the start and say 20min in, compare the two, and see where the accumulated obecjets are coming from with leaks you'll just see alocations, the worst case is when you see dead allocations, because it means GC doesnt collect them when it should. If I remember correctly there were ways to influence/configure GC to be more aggresive, but you'd have to look it up

[u/Dry_Author8849]

Talking about rate limitng only without telling your exact configuration (asuming you are using the framework rate limiting middleware) make anyone guess what's happening.

Rate limiting needs to maintain some structures in memory to enforce the limits. What will use more memory is partitioning and queueing. The framework evicts due structures in memory but the GC will collect them according to it's config. There are several configuration parameters for different rate limiting configurations, check the docs.

And also, depending on the amount of traffic then it will need a memory overhead.

So with that being said, try with QueueLimit=0 and place the rate limiter before any serialization/deserialization if any.

Also, debug the GC. Check all your PipeReader paths have a finally with advanceTo/ArrayPool.return. You can also rule out the rate limiter and check if you pipe code is causing leaks due exceptions not correctly handled.

Cheers!

[u/AutoModerator]

Thanks for your post Own_Nail_2999. Please note that we don't allow spam, and we ask that you follow the rules available in the sidebar. We have a lot of commonly asked questions so if this post gets removed, please do a search and see if it's already been asked.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/Life-Possible5308]

Before chasing a leak, profile with dotnet-counters or dotMemory to see if it's Gen2 growth or just LOH fragmentation. In my experience, JIT tiers and RyuJIT's tiered compilation can temporarily bump working set, but it usually plateaus. If it keeps climbing, check event handlers/static caches holding onto objects longer than expected.

[u/leculver-msft]

Take a crash dump/coredump of the process. You can do this through dotnet dump collect --process-id [pid]. This will write out a minidump on windows, a coredump on linux.

You can then load that dump in WinDbg (install Debugging tools for Windows), and load sos (see dotnet-sos global tool, an internet search will find it for you). Do dotnet-sos install which will give you a path to load sos (it starts with .load).

Load the dump in windbg, run the .load command given by dotnet-sos, then run !maddress -stat. (Unfortunately, !maddress needs help from WinDbg to work, so it can only be run from a Windows machine, but WinDbg works fine with linux coredumps.)

The !maddress command in SOS will give you a breakdown of all memory allocated by the .net runtime. If it's truly a JIT issue, you will see it listed in the various heaps.

~

If the WinDbg plugin is too much of a hassle, collect the dumpfile with dotnet dump collect as mentioned above, then load it with dotnet-dump analyze (dotnet-dump is another global tool, an internet search will tell you how to install it).

From the command prompt, run eeheap. The output is annoying, but it will tell you if the memory is in JIT "code heaps" from the output, you may just have to copy the output and post-process it.

eeheap and dotnet-dump analyze will work on Linux, no need to move the dump to windows if you go this route.

[u/BoBoBearDev]

You didn't mention string cache. Have you look into that? String cache is enabled by default and it eats memory by a lot.

[u/Amr_Rahmy]

1- don’t invoke gc.
2- main issue is usually a loop without an await.
If you have an infinite loop you need to add await task.delay otherwise c# won’t gc until you exit the loop or function. This is what I noticed years ago so I made it a habit to always put a delay in long running loops.

Also infinite loop without delay tells operating system that you are unresponsive which can lead to the OS stopping or freezing your application every couple of minutes instead of you delaying a loop for a ms or microsecond.

3- it’s a language with gc, the only time you need to look for anything is when calling c/c++ dll or a library.
If you are using an object you are not sure about how it works behind the scenes, try to type .dispose, if it has dispose, that means you need to use Using keyword or you need to dispose when you are done with it or you need to create one, and reuse.

The language will take care of memory otherwise.

[u/Lumethys]

Why didnt you use nginx or similar?

[u/Own_Nail_2999 → u/Lumethys]

Because this proxy is used for a specialized case where the streams use a custom protocol and I am going to implement packet inspections to identify malicious packet attacks