Friday, October 14, 2005

005 11:01 am Subject: directio

directio
n

I have a running process which does io:

last pid: 22838; load averages: 0.94, 0.91, 0.82
17:54:29
103 processes: 100 sleeping, 1 zombie, 2 on cpu
CPU states: % idle, % user, % kernel, % iowait, %
swap
Memory: 4096M real, 256M free, 7188M swap in use, 3392M swap free

PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
7335 root 2 20 0 1223M 1164M cpu/2 24.7H 49.53%
clemserv_9_0

truss -p 7335

/1: read(15, "020301\0\0\0\0\0\0\0\0\0".., 8192) = 8192
/1: lseek(15, 0x8F48CA00, SEEK_SET) = 0x8F48CA00
/1: read(15, "\0\0\0\0\0\0\0\0\0\0\0\0".., 8192) = 8192
/1: lseek(15, 0x8F48CB20, SEEK_SET) = 0x8F48CB20
/1: read(15, "020301\0\0\0\0\0\0\0\0\0".., 8192) = 8192
/1: lseek(15, 0x8F48CA00, SEEK_SET) = 0x8F48CA00
/1: read(15, "\0\0\0\0\0\0\0\0\0\0\0\0".., 8192) = 8192
/1: lseek(15, 0x8F48CB20, SEEK_SET) = 0x8F48CB20
/1: read(15, "020301\0\0\0\0\0\0\0\0\0".., 8192) = 8192
/1: lseek(15, 0x8F48CA00, SEEK_SET) = 0x8F48CA00
/1: read(15, "\0\0\0\0\0\0\0\0\0\0\0\0".., 8192) = 8192

When i monitor the system , both for directIO and normal - cached io ,
i see the following patterns.I will ve appreciated if someone can
comment in order to eplain this:

The program reads data from the /data filesystem. This is ufs - on emc
disk array . ( i have 1 hba - fibre channel)

mount -o remount,noforcedirectio /data

# sar 30 10000

SunOS verdenfs1 5.9 Generic_117171-12 sun4u 10/05/2005

17:53:42 %usr %sys %wio %idle
17:54:12 14 37 0 49
17:54:42 13 38 0 48
17:55:12 14 39 0 47
17:55:42 13 38 0 49

extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
1.4 0.1 1206.1 0.4 0.0 0.1 0.1 44.8 0 7 c2t16d65
0.1 1.2 1.1 9.5 0.0 0.0 0.0 9.2 0 1 c1t1d0
0.0 1.0 0.0 7.8 0.0 0.0 15.2 35.4 0 0 c1t0d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
1.2 0.1 1183.2 0.4 0.0 0.1 0.1 48.7 0 6 c2t16d65
0.3 1.2 2.7 9.5 0.0 0.0 0.0 7.7 0 1 c1t1d0
0.0 2.4 0.0 19.3 0.1 0.1 29.6 26.9 0 1 c1t0d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
7.2 0.1 1227.7 0.4 0.0 0.1 0.0 11.1 0 8 c2t16d65
0.1 1.2 1.1 9.5 0.0 0.0 0.0 7.4 0 0 c1t1d0
0.0 0.4 0.0 2.3 0.0 0.0 0.0 11.1 0 0 c1t0d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
1.3 0.1 1264.0 0.4 0.0 0.1 0.1 44.9 0 6 c2t16d65
0.3 1.1 2.7 9.5 0.0 0.0 0.0 8.0 0 1 c1t1d0
0.0 0.2 0.0 1.6 0.0 0.0 0.0 11.4 0 0 c1t0d0

verdenfs1@root/tmp #vmstat -p 30
memory page executable anonymous
filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi
fpo fpf
7402800 1789416 73 89 10 0 1 1 0 0 0 1 1 2626
11 9
3474096 262832 158 0 0 0 0 0 0 0 0 0 0 1207
0 0
3474120 262520 187 249 0 0 0 0 0 0 0 0 0 1216
0 0
3474072 262304 166 33 1 0 0 0 0 0 0 0 0 1188
1 1
3474208 262144 159 0 0 0 0 0 0 0 0 0 0 1269
0 0

mount -o remount,forcedirectio /data

# sar 30 10000

SunOS verdenfs1 5.9 Generic_117171-12 sun4u 10/05/2005

%usr %sys %wio %idle
17:57:12 8 24 23 46
17:57:42 2 10 39 49

extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
257.8 0.0 62434.5 0.0 0.0 0.8 0.0 3.1 1 77 c2t16d65
0.1 1.2 1.1 9.4 0.0 0.0 0.0 8.3 0 1 c1t1d0
0.4 0.4 3.2 3.2 0.0 0.0 0.0 8.8 0 0 c1t0d0
extended device statistics
r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
244.4 0.0 64447.9 0.0 0.0 0.8 0.0 3.3 1 78 c2t16d65
0.3 1.2 2.7 9.2 0.0 0.0 0.0 7.7 0 1 c1t1d0
0.7 2.7 4.8 15.6 0.0 0.1 7.1 15.8 0 2 c1t0d0

verdenfs1@root/tmp #vmstat -p 30
memory page executable anonymous
filesystem
swap free re mf fr de sr epi epo epf api apo apf fpi
fpo fpf
3474064 263456 31 15 0 0 0 0 0 0 2 0 0 61924
0 0
3474192 263440 25 8 0 0 0 0 0 0 1 0 0 63965
0 0
3474064 263088 43 88 0 0 0 5 0 0 0 0 0 63966
0 0

1.When i use noforcedirectio , sar reports no wio but 38 %sys , on the
other hand iostat shows about 1MB. read per second with 44.9 msec
service time.This io utilized the disk only 6 or 8 percent. And ,
vmstat -p shows that the system does pagins for file io only for 1MB.

2.But when i force directio , i have %39 wio , io rate grows
significantly (65MB per second) , the disks are utilized at 77 percent
and service time is 3.1 msec. At the same time , i see 65MB. of fpi.

1. Why does the low io rate in 1 (noforcedirectio) create 44.9 msec
service time while the high io rate create only 3.1 msec? Isnt it
logical to expect to see more io more service time.

2. Since the option 2 uses forcedirectio , how can i explain the large
fpi value? (If directio is in use , why does the operating system cache
file data?)

3. comparing the %38 sys with %39 wio , which one of them is better ?

Kind Regards,
tolga
n
I think the output is related with the IO's type of your application.

If IO is through "dd" command, the output is just what you think
right, that is to say,1)using "noforcedirectio" option, IO is using
more cpu power, and cached io(fpi) ,IO thoughput is higher;2)using
"forcedirectio" option, opposite to the former.
"dd" command is using "read/write" system calls.

Rn
Is there any progress about it?

To confirm whether using the filecache or not, you can install the
tool bundle "memtool" to assist you.
http://www.solarisinternals.com/si/tools/memtool/index.php
As a command from "memtool", "memps -m" can tell you which file is in
cache and how much cache the file occupy.

PS, another tool "directiostat" could be also helpful.
http://www.solarisinternals.com/si/tools/directiostat/index.php

n
>Is there any progress about it?

I dont understand, if cache is turned off, physial disk IO increases.
This is natural, and the perfomance problems are to be found elsewhere.

/wfr
n
The applications such as oracle, which has own file IO management
system, will be benefit from directio (file cache disabled);but the
performance of the normal file system IO will get an impact.

n
The applications such as oracle, which has own file IO management
system, will benefit from directio (file cache disabled);but the
performance of the normal file system IO will get an impact.

0 Comments:

Post a Comment

<< Home