6

As mentioned in some StackOverflow posts (like this one), I'm dealing with a difficult situation:

My company is developing some C# applications (being a client-server application). We have migrated the server application from one computer to another one, and on the starting computer, everything was working fine but on the destination computer, memory usage of the server application goes through the roof.

First I discovered that they had Trend Micro applications on that destination computer. Some of those binaries, like tmmon.dll get into applications and create a new thread. That situation is well-known for blowing up memory usage and one of my previous employers even has the rule that, when tmmon is found inside the dump of their application, the support for that particular case is finished. For my current employer, this seems to be their first encounter with Trend Micro.

I have informed the customer about this, the people I'm talking to seem to be unaware of the whole situation but after a week I see three differences:

  • No more Trend Micro threads are found in the dump of our server application.
  • No more Trend Micro DLLs are found in the "Modules" list of the dump of our server application.
  • The memory usage still increases (at a rate of ±100Mb/day), causing the server application needing to restart every two weeks, which is very bad.

I have discovered that, for a dump of more than 1Gb, only ±13Mb is actually reachable by .NET garbage collector, so I suspect a lot of the memory usage increased not to be caused by our .NET technology, so I would really try to get rid of this whole situation, but in fact this is the communication we currently have:

  • I : "The memory increase is caused by things, outside of our hands, so there's nothing we can do. It's up to you to make sure that those external things stop wasting memory."
  • Customer: "This is your application and you can't prove that that external thing is still present, so it's up to you to make sure the memory waste stops."

The situation is getting that far, that I'm using typical C++ dump analysis tools (heap_stat, PYKD.PYD) for analysing a C# dump, which is, next to the enormous work, not even giving the expected results.

Does anybody have an idea how I can get out of this mess?

Thanks in advance

Improve this question
9
  • "The memory increase is caused by things, outside of our hands, so there's nothing we can do. It's up to you to make sure that those external things stop wasting memory." Did you seriously say this? If a support vendor said that to me, I would be onto my commercial contact ASAP with a very large complaint. Commented Dec 1, 2023 at 8:18
  • @PhilipKendall: obviously I did not say it like that, it's just a means to clearly formulate the situation :-) Commented Dec 1, 2023 at 8:24
  • This question is not about technical aid, that part of the question is handled by the StackOverflow questions I refer to. My question is more about customer relations and communication: How can I explain my customer that the problem looks like my fault but is in fact not?. Commented Dec 1, 2023 at 8:27
  • 6
    GC isn't a silver bullet that magically makes all memory problems go away, it removes some classes of problems but it's still perfectly possible to keep an arbitrarily large set of objects in memory because you're referencing them all from somewhere. Commented Dec 1, 2023 at 8:51
  • 4
    Again, the downvoters and close voters around who misread this question as a request for debugging help. Sorry, but I have zero sympathy for this mindset. This is a question clearly answerable from a high-level view, from a software engineers perspective, so I encourage everyone who shares my opinion to counter-upvote the question. Commented Dec 1, 2023 at 11:12

2 Answers 2

Reset to default
6

causing the server application needing to restart every two weeks, which is very bad.

This is your problem. Not the fact your application needs to be restarted every two weeks - but the fact this causes trouble (whatever that means in your situation).

Even if the memory leak is not your fault today, you cannot guarantee that every now-and-then in future releases of your system a new memory leak will slip into production - simply because such errors are hard to find in reality. Hence, it is a really good idea to design server processes (or client/server systems) in a way they can be restarted from time to time (for example, automatically once a day) without causing too much hassle, maybe just an outtime of less than a minute.

The building bricks here are

  • short transactions

  • idempotent operations (so retrying an operation for which it is unclear if it was successful does not cause any harm)

  • a server process which does not hold too much unsaved state in memory over a longer period

  • a server which after a restart can pick up the pieces and continue where it was interrupted

  • automatic reconnects from the clients.

In the current world of web based systems this is pretty standard, any web server is usually restartable. Of course, when you have created a larger system over a longer period without restart capabilities in mind you may have a problem now, but if I would be in your shoes, I would seriously think about some backfitting.

Improve this answer
7
  • Thanks for your quick reply. The server application is not a webserver, but an application, handling a production environment, which is meant to be working 24/7, hence the need not to restart it too often. Commented Dec 1, 2023 at 9:28
  • 2
    @Dominique: and a downtime of, lets say, one minute per day isn't tolerable? I think that's what I would work on: to make this tolerable. One cannot prevent oneself from every memory leak in the future, so a robust system should mitigate this at the process level. Commented Dec 1, 2023 at 11:03
  • 1
    ... or maybe you can decrease the restart or failover time? Commented Dec 1, 2023 at 11:16
  • At startup, most of the time is spent at the loading of the database, and there already exists a script for removing old data from that database, so reducing the startup time won't solve the issue. Commented Dec 1, 2023 at 12:14
  • 2
    @Dominique: I gave you a general strategy on a broader scale, and a reason why it is a good idea to follow it regardless whether your current issues are your fault or someone elses. Adopting this strategy to your system and solving the issues is now up to you, that is something I cannot really help you with, Commented Dec 1, 2023 at 12:27
4

Does anybody have an idea how I can get out of this mess?

From a customer perspective, you have to prove it's not your application/process causing the memory pressure, or admit that there's a problem.

If it not your problem, taking a few snapshots of what is going on over time will show your customer it's not your issue. For example, on my windows laptop, using task manager its clear which app/processes are consuming the most memory:

Task manager snapshot

If your app or process is at the top of usage, it's your issue and it's your mess, which means you need to be transparent that it's your issue to your customer. It doesn't matter whether the memory is being managed, unmanaged, coming from a 3rd party dll, bad coding practices, and/or bad application design. If it's your app/process consuming loads of memory it is ultimately up to you and your team to rectify the issue. Take ownership. The customer will appreciate this even if the current situation is not ideal.

Some temporary relief from pain (triage/stabilize):

  • Add more memory, maybe the amount of memory is not enough to run your application, so add more (right size hardware).
  • If it's growing over time, then periodic restart. This is a more tactical fix, but you would be surprised how many servers are restarted on a periodic basis in production.

These usually take a minimal time to implement and do not cost a lot. Go over these options with your customer.

Long Term Health:

  • Fix/Redesign your application/process. This is easier said than done in most cases. Fixing code takes time and expense, more than upgrading memory and restarting a server periodically. Communicate with your customer what your long-term plan of attack is and what the ETA on the fix(es) will be. Then deliver software updates until problem is resolved. Most software has warts and sometimes serious problems, but there is always a way forward.
Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.