The actual Memory Usage on Linux

Hi,

It has been quite long since I have posted here. The main reason was the lots of work in the office and preparing for my next AWS Certification, which was AWS Solutions Architect Professional. I will write a short note on the exam details and what I did to clear it, later on.

What was happening ?

Coming back to this topic, I had setup Cloudwatch alerts for Memory Usage and Disk Usage and then alert the stakeholders when the usage breaches the threshold set. For me it was set to 90% for both the metrics. But upon observing closely, the alerts for memory usage were quite frequent, say 2 times per 5 minutes. This meant that something was definitely wrong and I needed to recheck the script deployed, fast.

Upon inspection, my previous script was calculating the memory from free -m command, using the first column:

             total       used       free     shared    buffers     cached
Mem:          7984       5423       2561          0        385       3130
-/+ buffers/cache:       1907       6077
Swap:            0          0          0

The calculation was as follows:

1	Memory Usage Percentage = (Used/Total) * 100

This comes out to be 67.92%.

Now I had used a custom script ps_mem.py LINK to calculate the approximate memory usage:

[ec2-user@ip-172-31-X-Y tmp]$ sudo ./ps_mem.py
 Private  +   Shared  =  RAM used	Program

144.0 KiB +  58.0 KiB = 202.0 KiB	agetty
168.0 KiB +  44.5 KiB = 212.5 KiB	rngd
196.0 KiB +  68.0 KiB = 264.0 KiB	atd
272.0 KiB +  76.5 KiB = 348.5 KiB	irqbalance
444.0 KiB + 164.5 KiB = 608.5 KiB	rpcbind
580.0 KiB + 140.5 KiB = 720.5 KiB	auditd
628.0 KiB +  98.5 KiB = 726.5 KiB	dbus-daemon
680.0 KiB + 115.0 KiB = 795.0 KiB	crond
532.0 KiB + 305.0 KiB = 837.0 KiB	mingetty (6)
780.0 KiB + 125.5 KiB = 905.5 KiB	init
356.0 KiB + 569.5 KiB = 925.5 KiB	mysqld_safe
856.0 KiB + 254.5 KiB =   1.1 MiB	rpc.statd
636.0 KiB + 570.5 KiB =   1.2 MiB	bash
492.0 KiB + 885.5 KiB =   1.3 MiB	udevd (3)
  1.1 MiB + 421.0 KiB =   1.5 MiB	ntpd
  1.0 MiB + 560.0 KiB =   1.5 MiB	sudo
  1.0 MiB + 547.5 KiB =   1.6 MiB	dhclient (2)
  2.4 MiB + 580.5 KiB =   3.0 MiB	cupsd
  1.4 MiB +   2.9 MiB =   4.3 MiB	sshd (3)
  3.4 MiB +   1.2 MiB =   4.6 MiB	sendmail.sendmail (2)
  4.7 MiB + 129.0 KiB =   4.8 MiB	rsyslogd
  9.9 MiB +  14.5 KiB =   9.9 MiB	docker-containerd
 34.7 MiB + 108.0 KiB =  34.8 MiB	dockerd
 43.5 MiB + 732.0 KiB =  44.2 MiB	mysqld
  1.4 GiB +   2.0 MiB =   1.4 GiB	java (2)
---------------------------------
                          1.5 GiB
=================================
Warning: Swap is not reported by this system.

As per the output above, the actual memory usage is about 1.5 GB or 1536 MB, which is 19.23%.

Calculating the memory usage, properly

I tried various flavours of linux, such as Ubuntu, RHEL, CentOS and Amazon Linux, and found variations of the commands such as:

free -m
cat /proc/meminfo

However, 1 specific command vmstat was giving the output almost properly among all the linux flavours. So I decided to go with it.

# vmstat -s
      8176604  total memory
      5555796  used memory
      4379188  active memory
       745036  inactive memory
      2620808  free memory
       394516  buffer memory
      3206024  swap cache
            0  total swap
            0  used swap
            0  free swap
       664123 non-nice user cpu ticks
           45 nice user cpu ticks
       138166 system cpu ticks
     75768535 idle cpu ticks
        39822 IO-wait cpu ticks
            0 IRQ cpu ticks
         1913 softirq cpu ticks
        12845 stolen cpu ticks
      5794205 pages paged in
     10155760 pages paged out
            0 pages swapped in
            0 pages swapped out
     57218586 interrupts
    101894616 CPU context switches
   1515624326 boot time
       589690 forks

In the above output, we are looking for:

total memory
free memory
buffer memory
swap cache, or simply cache.

Command 1:

1
2

# Gathering info
vmstat -s | sed -e 's/^[ \t]*//' | cut -d' ' -f1,3- | tr -s ' ' ',' | grep -e 'total,memory' -e 'free,memory' -e 'buffer,memory' -e 'swap,cache' | awk -F, '{print $1/1024","$2" "$3}'

Output:

7984.96,total memory
2553.59,free memory
386.012,buffer memory
3131.41,swap cache

With some mathematical formulas, I am able to calculate the near accurate (actual) memory usage on the system.

# Total Memory
MemTotal="$(cat /tmp/mem-usage.csv | grep -e 'total memory' | awk -F, '{print $1}')"
# Free Memory
MemFree="$(cat /tmp/mem-usage.csv | grep -e 'free memory' | awk -F, '{print $1}')"
# Buffers
Buffers="$(cat /tmp/mem-usage.csv | grep -e 'buffer memory' | awk -F, '{print $1}')"
# Cached
Cached="$(cat /tmp/mem-usage.csv | grep -e 'swap cache' | awk -F, '{print $1}')"
# Calculating total free memory (Free + Buffers + cached)
TotalMemFree="$(bc -l <<< $MemFree+$Buffers+$Cached)"
# Calculating total memory used (Total Memory - Total Free Memory)
TotalMemUsed="$(bc -l <<< $MemTotal-$TotalMemFree)"
# Percentage Used
mem="$(bc <<< "scale=2; $TotalMemUsed*100/$MemTotal")"
echo "$mem"

Conclusion

Till now, this is the best method I have come up with, to calculate the memory usage of the linux boxes. If you have any other method to calculate the memory usage, do let me know in the comments boxes.

The Memory and Disk Usage script is available on my Github account.

about stuff