[Apparmor-general] apparmor-parser gone wild?!

[John Johansen]

Christian Boltz wrote:
> Hello,
> 
> Am Freitag, 5. September 2008 schrieb John Johansen:
>> Christian Boltz wrote:
>>> As you can see, apparmor_parser takes 100% of the cpu and 14.4% of
>>> the available memory (the server has 6 GB RAM!).
>> uhm, yeah this is a little ugly, there is a "bug" in the dfa
>> generation that causes some policies to consume unbelievable amounts
>> of ram and cpu.  It is a problem that I am actively working on now. 
> 
> Bug number?
> 
421077 its an L3 bug against SP2, but its the same problem and exists in
10.3, 11.0, and 11.1 alphas.

The patch for the mem leak, has also been done for the 11.0 parser,
which reduce the memory used when there are hats but not the cpu.

>> While I think this is the problem with your current policy I can't be
>> sure without you sending me your profiles.
> 
> You should have them in the meantime.
> 
> Some of the profiles (especially the apache profile) were originally 
> used (and created) on a 10.2 server, copied to the 11.0 server and then 
> updated using logprof.
> 
well no surprise there, I think most policy is like that

>>> To make it even worse, it doesn't response to kill, not even kill
>>> -9 :-( Since init 6 also doesn't do anything, I had to reboot the
>>> server the hard way.
>> ugh, this should not be the case.  The parser doesn't do anything
>> that should cause it to be unkillable. Can you reliably reproduce 
> 
> Yes, IMHO too easy :-/
> I see the problem at about every third logprof run (given there were 
> profile changes) when the profiles are reloaded. And I'm quite sure it 
> is also reproducable by calling apparmor-parser directly (what command 
> line is used at the end of logprof?)
> 
log prof does

/bin/cat '$filename' | $parser -I$profiledir -r >/dev/null 2>&1"

though you could just as easily do
  apparmor_parser -r <profile name>

perhaps a more useful test however is
  apparmor_parser -S <profile_name> >tmpfile

this causes the parser to compile the policy but instead of loading it
dumps the output to tmpfile.  This lets us decouple creating the policy
and loading it.

The parser in 11.0 is not capable of loading the dumped file but that is
a feature that is coming.

My guess is the unkillable bug, is to do with loading and replacement of
policy, which would look like the apparmor_parser process but would be
in the kernel and thus unkillable.

>>> Is this a known problem? How can I avoid it?
>> The excessive computation and memory consumption is likely a known
>> problem, that it is unkillable is not and should not occur, so it is
>> important to see if we can reliably replicate.
> 
> If needed, I can provide SSH access to the server and a root screen 
> session to attach to. Just send me your SSH key in a signed mail and 
> offer some possible timeframes (late evening in german time is probably 
> the best).
>
we just might do that, but not just yet as I am unsure of what my
internet access will be over the next couple of days.  It is entirely
possible I won't have any access until monday.

> I can already tell you that strace is useless - I got no output after 
> attaching it to the running apparmor-parser process (using "strace -p 
> <pid of apparmor-parser>)
> 
sounding more and more like a kernel bug :(

>>> I can provide my profiles off-list (at least the httpd2-prefork
>>> profile contains customer names, so I don't want to send them on
>>> the list).
>> yes, please send them to me and I will do some testing against your
>> policy.  This should also enable me to make some recommendation about
>> what you can do until I get this nasty bug fixed.
> 
> I'm quite sure the answer will be something like "simplify your apache 
> profile" because this is the largest profile and has lots of hats.
> 
Actually no, I am more of the opinion that the tools need fixed or
improved if policy size is a problem.  It is possible that refactoring
your policy will help however.  Basically there are some rule patterns
that cause problems for the current code and refactoring some rules can
help, but only so much.

I am trying to address the memory/computation problems in two ways.
First by improving the code that computes the dfa, and secondly by
allowing for compiled policy caching, so policy that hasn't changed
doesn't need to be recompiled.

11.0 also has another "bug" (design fault really) in how hats are
handled, that will be affecting the time it takes to load and compute
your profiles.  11.0 introduced the concept children or subprofiles and
implemented hats using them by internally adding change_hat rules
governing which profile change_hat could transition to.

The problem is that because change_hat allows transitions directly
between hats, the number of rules added per hat is = number of hats + 1.
When you have 1000 hats this measurably slows down dfa generation.

This has been fixed in 11.1 by moving to a simple flag on the profile.