Memory usage of an application is a key factor to monitor. Specially in production systems you need to set alarms to make sure that system is stable. Therefore memory usage can be considered as a probe to measure the health of the system. Usually production systems are installed on Linux servers. OS itself help in many ways to provide a clear view of application’s memory usage.
In this post, I am going to discuss different commands and tools which can be used to measure memory usage of applications, specially Java applications. This post will guide you from higher-level to lower-level under following topics.
- Monitoring Overall System Memory Usage
- Monitoring Application’s Memory Usage
- Analyzing Java Application’s Memory Usage
Further information on commands and tools can be gained by going through the external links provided.
Monitoring Overall System Memory Usage
For this purpose I am going to discuss using “free” command in Linux. Free command gives an overview of complete system memory usage. Therefore please note that this is not an efficient way of measuring an applications performance. Because the system can host many applications and each application has its own boundaries. However, let’s look in to some usages of free command. (hint: use free -h to get a human readable output)
[root@operation_node ~]# free -m total used free shared buffers cached Mem: 7873 7360 512 0 70 920 -/+ buffers/cache: 6369 1503 Swap: 11143 431 10712
In the above output, in the first line, you can see total physical memory is 7873 MB and 7360 MB is used. Because of that only 512 MB is remaining. However this does not imply that memory is completely being used. Linux OS is good at using memory effectively. Therefore it uses caching for make the memory access efficient. And that cached memory is showing as used in the first line.
What you should look at is the second line which removed the cache & buffer usage of the physical memory. In used column of second line shows actual use of memory without cache & buffer. In the free column of second row you can see 1503 MB of free memory, which is generated by accumulating free + cache & buffer. So actually you have 1503 MB of free physical memory for use. In addition, according to the third line, you have around 10 GB of swap memory ready for use. Please refer  for further information.
In modern versions of Linux kernels, the output of free command has changed. It would be look like something below.
me@my-pc ~ $ free -m
total used free shared buff/cache available
Mem: 7882 3483 299 122 4100 3922
Swap: 9535 0 9535
total = used + free + buffers + cache
available : is the amount of memory which is available for allocation to a new process or to existing processes.
free : is the amount of memory which is currently not used for anything. This number should be small, because memory which is not used is simply wasted.
Monitoring Application's Memory Usage
In many cases we want to target monitoring of a single application than overall system. Overall memory usage reflects the use of memory including OS-level operations. Therefore we can use top command in Linux for this purpose. Following is a sample output of top command.
What you should actually focus is the RES value and %MEM value of the application (for this, first you need to identify the process-id of an application using ps -aux | grep "application_name" command). You can use simple "e" to toggle the unit of displaying memory.
RES -- Resident Memory Size : The non-swapped physical memory a task has used.
%MEM -- Memory Usage (RES) : A task's currently used share of available physical memory (RAM).
Analyzing Java Application's Memory Usage
If your application is a Java application, then you might need to look at what objects consumes the high amount of memory. For that one option is taking a heap-dump of that Java application. You may use following command to take a heap dump of the application. Prior to that you should know the process-id of that running Java application.
jmap -dump:format=b,file=heap_dump.hprof <process_id>
Once you execute the command, you will get the file heap_dump.hprof containing heap usage of Java program. Since the file is in binary format, you need to use a special tool to analyze it. Commonly using tool to inspect heap-dump is Eclipse Memory Analyzer Tool (MAT) , which is built on top of Eclipse platform. You just need to download the pack and extract it. Executing MemoryAnalyzer will open up a GUI application to analyze the heap-dump. When you open the heap-dump using MAT, tool will prompt you to generate reports based on heap-dump. You may interest about Leak Suspects Report which shows the large object which takes large portion of the memory.
Another interesting view of this tool is the Dominator Tree, which shows large objects along with ones who keep them. According to the definition ;
An object x dominates an object y if every path in the object graph from the start (or the root) node to y must go through x.
In the dominator tree view, you will see list of objects and the amount of memory they took when you take the heap-dump.
In dominator tree view, you can go expanding each entry and see how they have composed. Two columns showing in this view are Shallow Heap and Retained Heap. By default the list is sorted by Retained Heap value, descending order. Following definition  clearly explain the meaning of those two values.
Shallow heap is the memory consumed by one object. An object needs 32 or 64 bits (depending on the OS architecture) per reference, 4 bytes per Integer, 8 bytes per Long, etc. Depending on the heap dump format the size may be adjusted (e.g. aligned to 8, etc...) to model better the real consumption of the VM.
Retained set of X is the set of objects which would be removed by GC when X is garbage collected.
Retained heap of X is the sum of shallow sizes of all objects in the retained set of X, i.e. memory kept alive by X.
Generally speaking, shallow heap of an object is its size in the heap and retained size of the same object is the amount of heap memory that will be freed when the object is garbage collected.
Therefore in case of out-of-memory or high memory indications, you should definitely focus on this Retained Heap values of the dominator tree view.
In this post I want to give an clear idea on using several tools to analyze the memory usage of an application running on Linux OS. I have used commands which comes with Linux itself. However you may find tools which can be installed to analyze memory. In the last section I spent on discussing how Eclipse Memory Analyzer can be used to examine heap usage of a Java program. Hope those will help you as well.
 Understanding Linux free memory : https://thecodecave.com/understanding-free-memory-in-linux/
 Usage of free memory : https://stackoverflow.com/questions/30772369/linux-free-m-total-used-and-free-memory-values-dont-add-up
 Ask Ubuntu clarification on free command : https://askubuntu.com/questions/867068/what-is-available-memory-while-using-free-command
 Super-user forum top command explanation : https://superuser.com/questions/575202/understanding-top-command-in-unix
 Linuxarea blog top command explanation : https://linuxaria.com/howto/understanding-the-top-command-on-linux
 Eclipse Memory Analyzer : https://www.eclipse.org/mat/
 MAT Shallow Heap and Retained Heap explanation : https://help.eclipse.org/mars/index.jsp?topic=%2Forg.eclipse.mat.ui.help%2Fconcepts%2Fdominatortree.html