Search This Blog

2012-11-22

Gathering Performance Statistics

Some very common statistics you may want to poll for are the statistics that deal with IO.

There are many tools out there but by far the most standard of these is vmstat.

I will go over some commands and example usages of these commands.

CPU Statistics

vmstat -a -n 1
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
 r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 19995384 6760328 5634340    0    0     2   167    8   27  1  0 99  0  0
 0  0      0 19995384 6760328 5634340    0    0     0   220 1009  180  0  0 100  0  0
 0  0      0 19995384 6760328 5634340    0    0     0     0 1005  203  0  0 100  0  0
 0  0      0 19995384 6760328 5634340    0    0     0     0 1003  168  0  0 100  0  0
 0  0      0 19995384 6760328 5634340    0    0     0     0 1005  152  0  0 100  0  0

CPU usage
by User, System, Idle, Wait. This allows you to see if system processes or the lower priority user processes are using up the majority of cpu.
Context Switching 
allows you to see how many times the CPU is switching between tasks
Memory Usage:
Free memory (which is not being used at all), Amount of RAM that is being actively used or inactive (cached). Swap usage is also included

Disk Performance

vmstat -n -p sda1 1
sda1          reads   read sectors  writes    requested writes
                 303       2460         26         70
                 303       2460         26         70
                 303       2460         26         70
                 303       2460         26         70

IOPs (IOs per second).
Read IOPs, Write IO, requested IOs and the amount of sectors being read/written,

In Depth Disk Performance

vmstat -d -n 1
disk- ------------reads------------ ------------writes----------- -----IO------
       total merged sectors      ms  total merged sectors      ms    cur    sec
ram0       0      0       0       0      0      0       0       0      0      0
ram1       0      0       0       0      0      0       0       0      0      0
ram2       0      0       0       0      0      0       0       0      0      0
ram3       0      0       0       0      0      0       0       0      0      0
ram4       0      0       0       0      0      0       0       0      0      0
ram5       0      0       0       0      0      0       0       0      0      0
ram6       0      0       0       0      0      0       0       0      0      0
ram7       0      0       0       0      0      0       0       0      0      0
ram8       0      0       0       0      0      0       0       0      0      0
ram9       0      0       0       0      0      0       0       0      0      0
ram10      0      0       0       0      0      0       0       0      0      0
ram11      0      0       0       0      0      0       0       0      0      0
ram12      0      0       0       0      0      0       0       0      0      0
ram13      0      0       0       0      0      0       0       0      0      0
ram14      0      0       0       0      0      0       0       0      0      0
ram15      0      0       0       0      0      0       0       0      0      0
sda    60002  13714 1226308  225910 697327 7604275 66426068 14031453      0    273
sdb     1595     95    3675    5000 148826 6075320 49793161 23054956      0    161
dm-0   65633      0 1207010  216567 4120182      0 32961456 238369763      0    161
md0        0      0       0       0      0      0       0       0      0      0
loop0      0      0       0       0      0      0       0       0      0      0
loop1      0      0       0       0      0      0       0       0      0      0
loop2      0      0       0       0      0      0       0       0      0      0
loop3      0      0       0       0      0      0       0       0      0      0
loop4      0      0       0       0      0      0       0       0      0      0
loop5      0      0       0       0      0      0       0       0      0      0
loop6      0      0       0       0      0      0       0       0      0      0
loop7      0      0       0       0      0      0       0       0      0      0
dm-1      73      0     584      26      2      0      16       1      0      0
dm-2    1087      0    8690   60825 4183055      0 33464440 354948360      0    131

Latency
to allow you to figure out how much time it is taking between each IO to be completed
The other commands give you a good idea of other disk IO stats this is mainly useful for latency.

Summary Statistics

vmstat -s
     32962112  total memory
     12967472  used memory
      5634344  active memory
      6760352  inactive memory
     19994640  free memory
       494640  buffer memory
     10217728  swap cache
     16777208  total swap
            0  used swap
     16777208  free swap
       190838 non-nice user cpu ticks
         5955 nice user cpu ticks
        84769 system cpu ticks
     34445341 idle cpu ticks
        69688 IO-wait cpu ticks
          283 IRQ cpu ticks
         5219 softirq cpu ticks
            0 stolen cpu ticks
       620127 pages paged in
     58109890 pages paged out
            0 pages swapped in
            0 pages swapped out
     45894069 interrupts
     52365205 CPU context switches
   1353561023 boot time
       137148 forks

Gives you all stats regarding memory and cpu in a single point in time

vmstat -D
           30 disks
            3 partitions
       128390 total reads
        13809 merged reads
      2446267 read sectors
       508328 milli reading
      9149496 writes
     13679663 merged writes
    182646517 written sectors
    630404542 milli writing
            0 inprogress IO
          726 milli spent IO
Gives a disk usage summary at a single point in time

What if I don't have vmstat?

Theres many other tools: including but not limited to: sar, iostat, iometer, top, free or you can get information from the proc directory in some cases.
A couple examples:
cat /proc/meminfo
MemTotal:     32962112 kB
MemFree:      19994888 kB
Buffers:        494656 kB
Cached:       10217736 kB
SwapCached:          0 kB
Active:        5634340 kB
Inactive:      6760392 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     32962112 kB
LowFree:      19994888 kB
SwapTotal:    16777208 kB
SwapFree:     16777208 kB
Dirty:             248 kB
Writeback:           0 kB
AnonPages:     1682296 kB
Mapped:         190864 kB
Slab:           507324 kB
PageTables:      13832 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  33258264 kB
Committed_AS:  4090676 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    272164 kB
VmallocChunk: 34359464959 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB
free
             total       used       free     shared    buffers     cached
Mem:      32962112   12966728   19995384          0     494664   10217736
-/+ buffers/cache:    2254328   30707784
Swap:     16777208          0   16777208

Top in Batch Mode

You can run top in a non-interactive mode so that you can export statistics to a file. This can be useful when you're not quite sure what other command to use to get these statistics or if you just want a summary of activity over a period of time.

In order to run top in batch mode you can use the following example:
top -b -d 10 -n 3

-b batch mode
-d delay in seconds
-n number of iterations


IOStat Statistics

iostat 5
     tty           aacd0            aacd1            aacd2             cpu
tin tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
  0   16 93.90  10  0.96  16.28 315  5.00  45.85 529 23.69  27  0 46  2 25
  0 2439 16.00   1  0.02  16.00 132  2.07  16.94 304  5.03  28  0 47  1 24
  0 4188  0.00   0  0.00  16.00 142  2.21  17.56 362  6.20  27  0 39  1 33
  0 2567  0.00   0  0.00  16.00 139  2.17  16.56 383  6.19  22  0 40  1 37
  0 4489 17.45   4  0.07  16.00 137  2.13  16.40 283  4.53  22  0 37  1 40
  0 2941  0.00   0  0.00  16.00 139  2.17  16.44 285  4.57  17  0 13  1 69
This command will give you useful values like the amount of transactions but also will give the average number of KB/t which is useful as some disk types will be able to handle certain transaction sizes better than others. Small transactions are much heavier on disk than larger ones due to scsi overhead but also when using SSDs small transactions are difficult due to the behavior of writing to the disk and the fact that you need to read the entire sector, store it somewhere, modify the smaller chunk of data, then write it back down.

iostat -zx -w 5
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
aacd0      7.0   3.4   700.3   278.0    0   0.8   1
aacd1      7.2 307.5   195.4  4927.3    0   0.2   5
aacd2    440.8  88.3 21783.3  2477.0    1  10.1  60
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
aacd0      0.0   0.4     0.0     9.6    0   0.1   0
aacd1      0.0 164.6     0.0  2633.3    0   0.1   1
aacd2    292.2  15.8  4674.6   364.4    1   3.2  97
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
aacd1      0.0 126.6     0.0  2025.2    0   0.1   1
aacd2    317.5  15.0  5080.6   353.1    1   2.9  96
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
aacd1      0.0 128.6     0.0  2057.2    0   0.1   1
aacd2    307.5  14.0  4920.6   326.3    1   3.1  97
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
aacd0      0.0   1.0     0.0    16.0    0   0.2   0
aacd1      0.0 142.2     0.0  2274.8    0   0.1   1
aacd2    282.5  12.8  4591.1   296.3    1   3.3  96
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
aacd1      0.0 126.6     0.0  2025.2    0   0.1   1
The above command we can see that there are quite a few reads and writes on the same disk array. The latency is not overly high but the disk is busy 100% of the time meaning that it consistently has IO in queue. The reason for the low latency may be due to array queuing allowing for the transactions to go through in a more sequential manor.

svc_t - latency of disk in ms
%b - percentage of the time the disk has more than 1 transaction in queue

The above is from a database application where aacd2 is the data partition and aacd1 is the log partition.

You should generally separate data and log so that you don't have two different types of disk IO on the same machine to cause more random io and slowdowns.

No comments:

Post a Comment