Friday, October 28, 2005

httpd processes each with a size of 148M... is the top "size" display directly related to memory

> When I run top on this box, I can see 6 httpd processes each with a
> size of 148M... is the top "size" display directly related to memory?

Yes, it's the amount of virtual memory used by this process. You should
see a similar number (with much greater detail) by doing pmap -x on the
process.

> If it is, how can I possibly be running 6x148M processes just on apache
> alone?

Every page used by the process is not necessarily private to that
process. Read-only portions of the Apache binary may be shared among
all 6, and system libraries (like libC) may be shared by many programs.

The 'pmap -x <pid>' output shows that more explicitly.

Then you might also want to know that there are even a lot more
that offer detailed information regarding system state and performance:
e.g.
sar, vmstat, iostat, trapstat, cpustat, mpstat, cputrack, busstat, kstat

Darren already gave a good answer, but I wanted to elaborate a little.
(Or, after looking back on this after I've written the whole post,
apparently more than a little...)

On a simple computer, there is just a certain amount of RAM available
and every address that a program uses (in a pointer, in an address
register, or whatever) simply corresponds to part of that RAM. And
every program executes in the same address space, which means a given
address refers to the same thing whether it's in the context of one
process or another.

But on Solaris, there is virtual memory, and every process has its
own address space. Using the MMU hardware, the system maps several
different ranges of the process's address space to different things.
Some of the address ranges are private areas that only the process
itself has access to. Any memory you allocate with malloc() will be
in a private address range. Some address ranges correspond to regions
of files on disk. (In Solaris, executables are loaded by setting up
address ranges in the process's address space that correspond to
parts of the executable file. And the same thing is done when an
executable runs against a shared library.) Some address ranges
correspond to other things (sometimes even things like address ranges
that are used to communicate with hardware other than RAM).

Now to make things even a bit more complicated, just the existence
of an address range within a process's address space does not imply
that any RAM is used for that range. For example, if an address
range corresponds to a region of a file and if you've never either
read from or written to any addresses in that range, Solaris doesn't
need to waste time or memory putting that data into RAM. And to
make things yet more complicated, even if Solaris does need to use
RAM for (part of) an address range, if two processes are using the
same region of the same file, Solaris can use the same RAM for
both processes, even if the addresses that would be used within
the processes to access that data aren't the same addresses. And
to make things even yet more complicated, if a process has an
address range that contains private data and that process does a
fork(), then the twin processes that result can both use the same
RAM (or swap space) for that data until the time when one of them
tries to change the data (at which point a copy must be made so
they have their own separate copies).

So, when you run top, and you see the "SIZE" of the process, what
you're seeing is the size of all the address ranges that have been
created for that process. Most (but not all) of these address
ranges could correspond to data in RAM, but even if they do, it
might be data that is shared with another process. So when you
see 148M for an Apache process, that just means that there are 148M
worth of addresses that the Apache process could theoretically
access if it wanted to.

The "RES" column of top's output is a lot closer to the RAM usage
of a process, but it's still not exactly the same thing. The "RES"
column just tells you, of all the addresses that a process could
potentially access, how many of them are currently connected to a
particular spot in physical RAM. That is, how many of those pages
are resident in physical memory. It doesn't say how many of those
are shared with other processes. It's tempting to say that the
"RES" column tells you how much memory the process is using, but
that's not entirely accurate. It's totally possible for a process
to be totally dormant and not running have its resident size increase.
This could happen if the dormant process's address space refers to
some region of a file that some *other* process just accessed and
thus forced into memory.

And in fact, this type of thing partly accounts for why you see such
high numbers, even in the "RES" column. You could have a process
which only accesses a tiny portion of some file that's mapped into
its address space, but if a bunch of other processes also access
tiny portions of the same file, that will make increase the resident
size of all the processes that map the (whole) file bigger. That
may seem a little unlikely, but actually it is quite common because
things like libc.so (the C shared library) are used by a bunch of
processes, and even though each process might only use a few functions,
still when you count the total number of functions that are actively
used from that library, a significant portion of the library ends up
being resident, and that means that it inflates the resident size
numbers of all processes that use it.

> So I learned yesterday that the native stat tool for Solaris is
> prstat.. So I'm guessing from your posting Logan that in prstat, the
> SIZE column is the total amount of memory that each process can use and
> the RSS is the actual amout thats used..

"can use" and "that's used" as a description of memory seem incorrect to
me.

The difference is between Virtual Memory Pages that are actually
resident in RAM (RSS) and those that are allocated (whether currently in
RAM or not (SIZE). I don't think that "in use" and "in RAM" are quite
the same thing...

> "In a virtual memory(1) system, a process' resident set is that part of
> a process' address space which is currently in main memory. If this
> does not include all of the process' working set, the system may
> thrash."
> That makes sense to me except for one thing... If the SIZE is the total
> amount, how come it fluctuates?

Processes allocate memory while they run. Most programs will grow, but
not shrink, but both are possible.

> Whoah.. I just looked at pmap.... thats insane.

Don't try to interpret everything. Most bits are just mappings from
other shared objects. In an average program, the most common place for
it to consume memory directly is in the [heap].

However the breakdown of shared/private/RAM/total can be useful.

If I can't map SIZE and RSS to whats available and whats in use, how
> can I tell when the memory needs to be upgraded?

I was making the distinction between "in use" (which is a little fuzzy
for me when talking about pages) and "in RAM" (which is well-defined).

You can map SIZE and RSS to what pages are in RAM at any one time, which
will probably be all you need.

What "in use" means, and whether or not that has anything to do with RAM
residency is a different question.

--

The top part of the display of top shows memory usage. The 'swap -s' command
shows how much swap is in use. I think both of these include swap as
backed up by the executables and shared images on disk as well as the
swap backed up by the swap file. The 'swap -l' command will show how much
of the actual swap space is in use. A rule of thumb is that when the amount
of swap in use starts to be as big as your memory, you need to add memory.
If you have a lot of mostly inactive programs in use, you can allow more
swap to be used without hurting performance.

One way is to look at the pi and po (page in and page out) columns
in the vmstat output. Assuming you're not starting lts of new programs,
high values here could indicate low memory. You should check out
Adrian Cockcroft's book, Solaris Performance Tuning (aka the Porche
book).

Frequent complaints form users about poor performance can also be an
idicator of too much paging. :-)

4 Comments:

Blogger hamada wady said...

نود ان نشير دائما الى امتلاكنا احدث الخدمات فى مجال الخدمات المنزلية الحديثة وغيرها وبافضل واحدث معدات لازمة وحديثة جا كما انن نود وبشكل متقن على ان تظهر كل خدماتنا الحديثة فى شركة تنظيف شقق بجدة كما اننا نعمل على ان يكون لدينا كل ماهو جديد وتقنى جدا فى كل مايتم ان نريده ونقدم لامور التنظيف للخزانات وكل الملحقات بها حتى ما يقدم فى شركة تنظيف خزانات بجدة

1:21 PM  
Blogger Sowmiya said...

Interesting blog which attracted me more.Spend a worthful time.keep updating more.
MSBI Training in Chennai

12:00 AM  
Blogger Sonal priya said...

i accepting this your blog and so very clearly understand this blog ..thank u so much your post.
Cloud Computing Training in Chennai

1:54 AM  
Blogger Sathya G said...

I have really enjoyed reading your blog posts. I hope you post again soon. photos very nice.
Hadoop Training in Chennai

2:23 AM  

Post a Comment

<< Home