Query on use of GFP_ATOMIC memory

[Brian F. G. Bidulock]

I am having problems with failing to get memory on a system using LiS (2.18.0
plus our patches which are close to 2.19.0) under stress. It looks like each
call to allocb maps down to a kmalloc call with memory type GFP_ATOMIC.
Obviously this is useful if the message is required in interrupt context but a
large part of our Streams code is not running in interrupt context.
What is the background to this?
Could we consider changing so that for instance if we called allocb with
BPRI_HI it used GFP_ATOMIC and otherwise used GFP_KERNEL?

Traditionally SVR4 performs atomic memory allocations for the entire
triplet (message block, data block and data buffer). While it is
possible to use non-atomic memory allocators for the data buffer and
then use esballoc(9) to obtain a message and data block, allocation for
the message and data block will still be atomic, not solving your
problem.

OSF/HP-UX (Mentat) defined another priority flag (BPRI_WAITOK) as did
other STREAMS implementations such as Solaris (BPRI_FT). Linux
Fast-STREAMS implements both these flags (BPRI_WAITOK and BPRI_FT).
They become particularly useful when allocating message blocks from a
blocking process context, such as in the Stream Head, or STREAMS
open procedure (e.g. allocating an stroptions structure for an
M_SETOPTS message).

So, for example, from the STREAMS open procedure (blocking context),
one simply does something like:

while (!(mp = allocb(sizeof(struct stroptions), BPRI_WAITOK))) ;

and one is guaranteed of obtaining a message block or the calling
process will block. Allocation of the entire triplet (message block,
data block and data buffer) is performed at GFP_KERNEL instead of
GFP_ATOMIC. BPRI_WAITOK cannot be used from within a put or service
procedure (which by definition is a non-blocking context), but can be
used from open or close procedures as well as any blocking context
outside of STREAMS (separate kernel thread, user-kernel interface).

For more description of this feature, see:

http://www.openss7.org/man/man2html?9+allocb

That said, if you are failing allocations of message blocks it is likely
because you are exceeding the maximum number of message blocks allocated
in the system. SVR4 had a tunable and many other STREAMS
implementations have tunables that determine at what percentage of
STREAMS dedicated memory message block allocations will fail (it is
normally before the system completely runs out of memory). For Linux
Fast-STREAMS, the sysctl tunable streams.nstrmsgs in
(/proc/sys/streams/nstrmsgs) determines the maximum number of allocated
message blocks. (Currently set to 262144 message blocks.) LiS defaults
to allowing message block allocations up to the limit of memory, but has
to option to set a limit base on memory usage.

That said, if you are failing memory allocations it is likely because
there is no memory (or no memory for STREAMS) left available. The most
common cause of this situation is not GFP_ATOMIC vs. GFP_KERNEL, but a
module or driver with a buffer leak. Another possibility is a system
with just too little memory to be useful.

Hope that helps.

--brian