2

I would like to know how to determine the status of my server using the top command or if I have to change the server or add more resources. Next is the output the top command in my server.

enter image description here

Next are some other facts - The load average values vary from 28.XX to 77.XX. - CPUs %id most of the time is between 10.0 and 22.0 and sometimes drops to 30.0. - The server is running on a virtual machine. - The server the virtual machine is mounted on has a Intel(R) Xeon(R) CPU E5-2403 0 @ 1.80GHz, with 4 cores - The web applications, data base service, memcached, web server and other related server apps have been running for a week. - The presence.py service is the one that does the most amount of work and is currently checking the presence of 703 nodes.

I would say I do not need to add more RAM to the system but it certainly looks like the CPU is overwhelmed. We still have to add 100-200 more nodes so I think that the server is not going to be able to handle it. Am I right?

EDIT: presence.py background

The Presence service (presence.py) is a propietary application that runs in (twice the number of cores) processes, one main process and (twice the number of cores-1) worker processes. For each node registered a thread is created in one of the worker processes, so if we have 700 nodes each worker process will have ~100 threads running. Each thread checks the status of its node via telnet or HTTP once per second, so you can imagine the load of each process. This means most of the time each thread is sleeping or waiting for network i/o.

The Presence service started with a load of 350 nodes and had been working great but for some time, since we started increasing the load and surpassed the 600 nodes, it started to operate somewhat slow. For instance, if you run the following command curl www.google.com it takes many seconds to run it.

Improve this question
7
  • looks like presence.py needs optimizing or you are using it wrong. Link to the source? Commented Oct 13, 2016 at 16:21
  • @user1133275 Since it is our product I can't link you to any documentation nor source code but I added more data related to it. Commented Oct 13, 2016 at 17:26
  • @user1133275 just because it takes up a lot of resources doesn't mean it's not optimized or running improperly. Somethings just take a lot of resources because of the nature of the tasks they perform. Without knowing more about the process itself it can't be assumed to be problematic. Commented Oct 13, 2016 at 17:47
  • In theory adding 200 more nodes could increase workload for the process by about 28%, and your CPU usage could raise similarly, which would put you at 99% at most. Is this a production server? Do you have a test environment to try adding these extra nodes? It's really not possible to say exactly how much impact will be had on the process without knowing it intimately, but it seems that it would be pushing the hardware. Commented Oct 13, 2016 at 18:01
  • @Centimane No, we don't have a testing environment to add 200 more nodes. Nevertheless, connecting by SSH to the server and running commands as simple as curls take many seconds to complete. Isn't this enough to say that the hardware is more than overloaded? Commented Oct 13, 2016 at 18:16

2 Answers 2

Reset to default
1

replace presence.py with a lighter weight solution. something like

cat config.txt | while read C; do
    C=($C)
    nmap -p ${C[1]} ${C[0]} | grep open || mail -s "Warning; port ${C[1]} on server ${C[0]} is unreachable" ${C[2]} < /dev/null &
done;

referenced from crontab with a config like

echo -e "127.0.0.1\t22\[email protected]" > config.txt

There is no reason to use any significant CPU to check a port; it's strictly an IO limiting operation. (you should see lower user and higher wait CPU times.)

This Bash example is considered a heavy solution (no re-use), if you want it really light code it in c... your python is probably doing something like failing to recycle objects or some other needless work incurring oversight. Throwing more resources at inefficient code can work but it's often less expensive to just fix the inefficiency. You may also want to tune The network stack.

Improve this answer
3
  • Sorry but this question has nothing to do with replacing the Presence service. So your solution is off topic. Your code has nothing to do with what that service does. I quote: "Each thread checks the status of its node via telnet or HTTP once per second, so you can imagine the load of each process." Clearly it has nothing to do with a port. Commented Oct 13, 2016 at 20:38
  • @Storo even if it's purpose is not the port specifically it's not inappropriate for them to suggest that a lighter application would be the best course of action. Have you also considered reversing the relationship for presence.py? (Each node reports their status to the server with a timestamp, offloading much of the CPU work to the clients.) Commented Oct 13, 2016 at 20:43
  • @Storo if whatever service is providing the telnet or HTTP dies nmap will report the port not open, so unless you are checking for a responsive but otherwise broken service there is no need to use curl/telnet, but even if there were replacing nmap with curl/telnet would still be lighter than presence.py. Commented Oct 13, 2016 at 21:27
0

That's a reasonably high load average you got there and about 80% aggregate busy CPU. A busy userland, pretty busy kernel and even some software interrupts poking its head up a bit there.

I really hope you're doing asynchronous network calls there, that'll help the load but it's not about the program but the system. I'd say you need more CPU as you said. Also check how many threads you have as if there are a lot and they're all clamouring for CPU you may get context switch hitting things. Sometimes less threads is more!

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.