stevepedwards.com/DebianAdmin linux mint IT admin tips info

Linux Performance Tools List – Check Utilisation, Saturation, Errors (USE)

This is a partial list of tools from Brendan's presentations, book and elsewhere – it's up to you to learn how to use them, when and why. Many can show rolling O/P by a 1 sec addition eg mpstat 1

sudo apt-get install sysstat 

apt-cache show sysstat

Description-en: system performance tools for Linux
The sysstat package contains the following system performance tools:
- sar: collects and reports system activity information;
- iostat: reports CPU utilization and disk I/O statistics;
- mpstat: reports global and per-processor statistics;
- pidstat: reports statistics for Linux tasks (processes);
- sadf: displays data collected by sar in various formats;
- nfsiostat: reports I/O statistics for network filesystems;
- cifsiostat: reports I/O statistics for CIFS filesystems.
.
The statistics reported by sar deal with I/O transfer rates,
paging activity, process-related activities, interrupts,
network activity, memory and swap space utilization, CPU
utilization, kernel activities and TTY statistics, among
others. Both UP and SMP machines are fully supported.

uptime 

11:07:33 up 1:33, 3 users, load average: 0.46, 0.59, 0.69 (1min, 5 min, 15 min)

perl -e 'while(1) {`uptime` =~ /average: ([\d.]+)/; printf("% 5s %s\n", $1, "#" x ($1 * 10)); sleep 3 }'

perlloadavg.png

nmon

nmon.png

bmon

bmon.png

free

free.png

sudo atop

atop.png

top/htop

htop.png

ps u | head

ps.png

netstat -s | head

netstat.png

sudo iotop

iotop.png

dstat

dstat.png

vmstat (-sm)

vmstat.png

slabtop

slabtop.png

iostat

iostat.png

man strace (USE WITH KNOWLEDGE AND CAUTION)

DESCRIPTION

In the simplest case strace runs the specified command until it exits. It intercepts and records the system calls which are called by a process and the signals which are received by a process. The name of each system call, its arguments and its return value are printed on standard error or to the file specified with the -o option.

strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them. Students, hackers and the overly-curious will find that a great deal can be learned about a system and its system calls by tracing even ordinary programs.

strace.png

sudo mpstat -P 0,1,2,3

mpstat0123.png

nicstat

nicstat.png

pidstat | head

pidstat.png

sudo lsof | head

lsof.png

ss -a | head

ss.png

ethtool -i eth0

ethtool.png

sudo sar (-BHRrSsw)

sar.png

If you run sar, then a nice addition is sadf as it formats sar data for other use - DB, csv etc. For example, it can show 6 cpu utilizations for each 4 cores which can be grepped nicely for an easy summary:

sadf -p -P 0,1,2,3 | head -24

sadf24.png

Each usage type can be grepped quickly showing 10 min separations for a clear timeline of usage e.g:

sadf -p -P 0,1,2,3 | grep user

sadfuser.pngsadfsystem.pngThese columns could be stripped and graphed quite easily via awk / gnuplot etc e.g:

sadf -p -P 0,1,2,3 | grep user | grep cpu0 | awk '{print $3,$4,$7}' > data1.dat

2016-08-17 11:55:01 10.28
2016-08-17 12:05:01 5.27
2016-08-17 12:15:01 17.73
2016-08-17 12:25:01 13.63
2016-08-17 12:35:01 4.38
2016-08-17 12:45:01 7.67
2016-08-17 12:55:01 8.40
2016-08-17 13:05:01 11.02
2016-08-17 13:15:01 13.57
2016-08-17 13:25:01 10.07
2016-08-17 13:35:01 5.68
2016-08-17 13:45:01 3.20
2016-08-17 13:55:01 4.32
2016-08-17 14:05:01 2.40

userCPU.png

cifsiostat

cifsiostat.png

Hard drive cached reads: sudo hdparm -T /dev/sd*

hdparmTreads.png

stevee@Dell490 ~ $ sudo hdparm -t /dev/sd*

/dev/sda:
Timing buffered disk reads: 316 MB in 3.02 seconds = 104.77 MB/sec

/dev/sda1:
Timing buffered disk reads: 100 MB in 0.96 seconds = 104.44 MB/sec

/dev/sda2:
Timing buffered disk reads: 312 MB in 3.00 seconds = 103.97 MB/sec

/dev/sda3:
Timing buffered disk reads: 290 MB in 3.01 seconds = 96.24 MB/sec

/dev/sdb:
Timing buffered disk reads: 222 MB in 3.01 seconds = 73.76 MB/sec

/dev/sdb1:
Timing buffered disk reads: 222 MB in 3.01 seconds = 73.64 MB/sec

/dev/sdb2:
Timing buffered disk reads: read(2097152) returned 1024 bytes

/dev/sdb5:
Timing buffered disk reads: 114 MB in 3.04 seconds = 37.45 MB/sec

blkiomon EXAMPLES
To get I/O statistics for /dev/sdw every 10 seconds for a period of one
hour, use the following command:

% blktrace /dev/sdw -a issue -a complete -w 3600 -o - | blkiomon -I
10 -h -

sudo blktrace /dev/sda -a issue -a complete -w 20 -o iofile.txt
=== sda ===
CPU 0: 0 events, 0 KiB data
CPU 1: 1 events, 1 KiB data
Total: 1 events (dropped 0), 1 KiB data

Webpage graph

sudo apt-get install bandwidthd

browser address bar: /var/lib/bandwidthd/htdocs/index.html

bandwidthd.png

or add it as an alias in Apache in the virtual host block e.g. remote view it on your apache server //dellmint/bd

vi /etc/apache2/sites-available/000-default.conf

Alias /bd /var/lib/bandwidthd/htdocs
<Directory /var/lib/bandwidthd/htdocs>
Options All
AllowOverride All
order allow,deny
allow from all
</Directory>

INFO RE SYSTEMTAP!! You may need a development kernel with debugging installed/enabled! Most tutorials COMPLETELY fail to mention this - and you wonder why the cmd examples don't work!

see here: http://posulliv.github.io/2010/02/26/installing-stap/

"SystemTap needs to have a debug version of the kernel available. Unfortunately, installing the debug information for a kernel on ubuntu is not a trivial operation to perform. In fact, there is a bug on Launchpad about this issue."

If you want to experiment - use a test PC with a dev kernel...

https://www.suse.com/documentation/sled11/singlehtml/book_sle_tuning/book_sle_tuning.html#sec.tuning.systemtap.setup

If you HAVE installed just systemtap and systemtap-doc, the list of available probes can be seen with:

man probe::[TAB]

Display all 289 possibilities? (y or n)

re the above stap kernel info - later, with my std kernel, I did get one of Brendan's stap cmds to work to create a flame graph in /Flamegraph as root (but much beyond this one cmd, it seems you need the full dev install a la suse link):

http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#perf

stap -s 32 -D MAXBACKTRACE=100 -D MAXSTRINGLEN=4096 -D MAXMAPENTRIES=10240 \
-D MAXACTION=10000 -D STP_OVERLOAD_THRESHOLD=5000000000 --all-modules \
-ve 'global s; probe timer.profile { s[backtrace()] <<< 1; }
probe end { foreach (i in s+) { print_stack(i);
printf("\t%d\n", @count(s[i])); } } probe timer.s(60) { exit(); }' \
> out.stap-stacks
 ./stackcollapse-stap.pl out.stap-stacks > out.stap-folded
 cat out.stap-folded | ./flamegraph.pl > stap-kernel.svg

stapflame.png

You can use perf for Brendan's script examples such as his "perf Hacktogram"

http://www.brendangregg.com/blog/2014-07-10/perf-hacktogram.html

in your Flamegraph directory (previous Post) as root - create this histogram in the terminal:

nethisto.png

 

Comments are closed.

Post Navigation