[mpm-itk] lots of "closing connections" workers

Pavel Polacek pavel.polacek at ujep.cz
Wed Mar 22 07:23:34 UTC 2017


     Hello,

   I send answer myself.

Apache in "C" state, wait for another apache worker (process 754):
strace -p 754 
Process 754 attached
wait4(771,

Apache process 771:
gdb --pid 771
(gdb) bt
#0  __lll_lock_wait () at ../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00007f032831c479 in _L_lock_909 () from /lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00007f032831c2a0 in __GI___pthread_mutex_lock (mutex=0x7f032ab42420) at ../nptl/pthread_mutex_lock.c:79
#3  0x00007f031fa7ab8f in log4shib::Category::removeAllAppenders() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
#4  0x00007f031fa7b9da in log4shib::HierarchyMaintainer::shutdown() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
#5  0x00007f031fa7bc4c in log4shib::HierarchyMaintainer::~HierarchyMaintainer() () from /usr/lib/x86_64-linux-gnu/liblog4shib.so.1
#6  0x00007f0327f9eb29 in __run_exit_handlers (status=0, listp=0x7f032830c5a8 <__exit_funcs>, run_list_atexit=run_list_atexit at entry=true)
     at exit.c:82
#7  0x00007f0327f9eb75 in __GI_exit (status=<optimized out>) at exit.c:104
#8  0x00007f0325281aea in itk_fork_process () from /usr/lib/apache2/modules/mpm_itk.so
#9  0x00007f0328e81f40 in ap_run_process_connection ()
#10 0x00007f0324c6f7ba in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#11 0x00007f0324c6fa01 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#12 0x00007f0324c70667 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#13 0x00007f0328e5c7ee in ap_run_mpm ()
#14 0x00007f0328e555f3 in main ()

Apache mod_shib module is in suspicion.
So I set logging off in /etc/shibboleth/native.logger, all line comment out.

Now I have 50 workers instead of 1000 after 10 hours of running apache.

I will bugreport mod_shib module in Debian.

     Best regards  Pavel Polacek


On Tue, 21 Mar 2017, Pavel Polacek wrote:

>    Hello,
>
>  I'm using stable Debian Jessie with mpm_itk and php. Lately I see many 
> workers in server-status page, most in "C" state - Closing Connection. Count 
> of apache process never go down, only grow. Workers in "C" state alive to 
> apache restart and looks busy (consuming memory, not CPU, process is in Sl 
> state). I don't understand why. I have another server with oldstable Debian 
> wheezy, where count of apache process grow and decrease by hour-to-hour 
> utilization. I monitor server and I see, that problem is long time, but now 
> it's close to limits in monitoring.
>
> apache2-mpm-itk                2.4.10-10+deb8u8
> libapache2-mod-php5            5.6.30+dfsg-0+deb8u1
> libapache2-mod-shib2           2.5.3+dfsg-2
> libapache2-mpm-itk             2.4.7-02-1.1+deb8u1
>
> <IfModule mpm_prefork_module>
>        StartServers                     5
>        MinSpareServers           5
>        MaxSpareServers          10
>        MaxRequestWorkers         950
>        ServerLimit             950
>        MaxConnectionsPerChild   1000
> </IfModule>
>
> Timeout 60
> KeepAlive On
> MaxKeepAliveRequests 100
> KeepAliveTimeout 5
>
> Apache have sets 1200 limit in monitoring.
>
> /etc/init.d/apache2 restart not kill all "old running" process. I have to 
> kill it by signal 9.
>
> pstree:
> systemd─┬─acpid
>        ├─agetty
>        ├─apache2─┬─82*[apache2─┬─apache2]
>        │         │             └─2*[{apache2}]]
>        │         ├─12*[apache2───2*[{apache2}]]
>        │         └─apache2
>        ├─605*[apache2]
>
>  Do you see similar behaviour with your mpm_itk?
>
>    Thank you  Pavel Polacek


More information about the mpm-itk mailing list