Calendar

[symeon.mattes]

Hi,

I have centos 6.3 on 16GB/4 vCores Virtual Server (hosteurope.de). I'm having currently 3 domains (2 wordpress, 1 drupal). The last few days I receive a lot internal server errors especially on high traffic and I've been trying to find out the root of the problem. The main error I receive is in  /var/log/httpd/error_log:

Code Select Expand

[some_date] [warn] mod_fcgid: process 13442 graceful kill fail, sending SIGKILL
[a_date] [error] FastCGI process 32657 still did not exit, terminating forcefully
[a_date] [warn] mod_fcgid: cleanup zombie process 18283
[a_date] [warn] (105)No buffer space available: connect to listener on 0.0.0.0:80

In the logs of  each website I have also:
domain1 (wordpress), /var/www/vhosts/domain1/logs

Code Select Expand

[a_date] [warn] [client an_ip] mod_fcgid: read data timeout in 70 seconds, referer: myrurl
[a_date] [warn] [client an_ip] mod_fcgid: stderr: \t\t\t\tORDER BY dp_date DESC, cntaccess DESC
[a_date] [warn] [client an_ip] (104)Connection reset by peer: mod_fcgid: error reading data from FastCGI server
[a_date] [error] [client an_ip] Premature end of script headers: wp-cron.php
[a_date] [warn] [client an_ip] (70007)The timeout specified has expired: mod_fcgid: can't get data from http client, referer: a_url


domain2  (wordpress), /var/www/vhosts/domain2/logs

Code Select Expand

[a_date] [error] [client an_ip] Premature end of script headers: wp-comments-post.php, referer: a_url
[a_date] [error] [client an_ip] Premature end of script headers: index.php, referer: a_url
[a_date] [warn] [client an_ip] mod_fcgid: read data timeout in 70 seconds
[a_date] [error] [client an_ip] Premature end of script headers: index.php

domain3  (drupal) /var/www/vhosts/domain3/logs

Code Select Expand

[a_date] [warn] [client an_ip] mod_fcgid: stderr: PHP Warning:  Maximum number of allowable file uploads has been exceeded in Unknown on line 0, referer: a_url
[a_date] [warn] [client an_ip] mod_fcgid: read data timeout in 70 seconds, referer: a_url
[a_date] [error] [client an_ip] Premature end of script headers: index.php, referer: a_url

I was checking around at the beginning how I could handle with the error of the mod_fcgid and I found some answers like increasing some parameters in /var/etc/httpd/fcgid.conf, i.e.

Code Select Expand

FcgidIdleTimeout 40->3600
FcgidProcessLifeTime 30->7200
FcgidMaxProcesses 40->64
FcgidMaxProcessesPerClass 8
FcgidMinProcessesPerClass 0
FcgidConnectTimeout 30->300
FcgidIOTimeout 70->7200
FcgidInitialEnv RAILS_ENV production
FcgidIdleScanInterval 10
IPCCommTimeout  7200
IPCConnectTimeout 180

This has improved just the occurrence of the errors of mod_fcgi but especially on high load there is a lot of latency especially when somebody submits a comment or edit a page.

As a consequence I tried to check whether there is any problem on the performance of the server or something wrong in the website. I tried to check the CPU usage for each website and I found out through top -c that there is high usage (30-100%) on php-cgi processes, especially on the websites that are related to wordpress, i.e.

Code Select Expand

/usr/bin/php-cgi -c /var/www/vhosts/system/domain1/etc/php.ini
/usr/bin/php-cgi -c /var/www/vhosts/system/domain2/etc/php.ini

It seems that there is something wrong with some php script. However I don't know how I could restrict even further the issue. Any ideas?

Thanks in advance

[Mark Greaves (PCNetSpec)]

Have you read this:
http://shiftlock.wordpress.com/2013/06/29/troubleshooting-error-reading-data-from-fastcgi-server/ (http://shiftlock.wordpress.com/2013/06/29/troubleshooting-error-reading-data-from-fastcgi-server/)

and the except it points to on this page:
http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#examples (http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#examples)

Special PHP considerations

By default, PHP FastCGI processes exit after handling 500 requests, and they may exit after this module has already connected to the application and sent the next request. When that occurs, an error will be logged and 500 Internal Server Error will be returned to the client. This PHP behavior can be disabled by setting PHP_FCGI_MAX_REQUESTS to 0, but that can be a problem if the PHP application leaks resources. Alternatively, PHP_FCGI_MAX_REQUESTS can be set to a much higher value than the default to reduce the frequency of this problem. FcgidMaxRequestsPerProcess can be set to a value less than or equal to PHP_FCGI_MAX_REQUESTS to resolve the problem.

PHP child process management (PHP_FCGI_CHILDREN) should always be disabled with mod_fcgid, which will only route one request at a time to application processes it has spawned; thus, any child processes created by PHP will not be used effectively. (Additionally, the PHP child processes may not be terminated properly.) By default, and with the environment variable setting PHP_FCGI_CHILDREN=0, PHP child process management is disabled.

The popular APC opcode cache for PHP cannot share a cache between PHP FastCGI processes unless PHP manages the child processes. Thus, the effectiveness of the cache is limited with mod_fcgid; concurrent PHP requests will use different opcode caches.

and his possible solution:-

One possible solution would be to set the FcgidMaxRequestsPerProcess directive in Apache's configuration to 500, thus instructing FastCGI to terminate the process at the same time as PHP does.

Just a suggestion, as this isn't really my area of expertise

--

[Mad Penguin]

Looks a bit like a system resource issue. Can you give us the output from;

Code Select Expand

free -mtAlso (this may take a minute or so ..);

Code Select Expand

lsof | wc -lAnd;

Code Select Expand

netstat -nat # or netstat -nat|wc -l if it's too long .... while the system is experiencing problems ??

[Mad Penguin]

Incidentally, of you have a high volume site, Apache really isn't your best choice, or in MP speak - avoid like the plague.

Either nginx or lighttpd would be a better solution .. we use lighttpd with spawn-fcgi / php-cli, doesn't produce errors even when hammered ...

# ab -c4 -n10000 http://linux.co.uk/
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking linux.co.uk (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests

Server Software:        lighttpd/1.4.28
Server Hostname:        linux.co.uk
Server Port:            80

Document Path:          /
Document Length:        59079 bytes

Concurrency Level:      4
Time taken for tests:   10.546 seconds
Complete requests:      10000
Failed requests:        0
Write errors:           0
Total transferred:      593949114 bytes
HTML transferred:       590790000 bytes
Requests per second:    948.24 [#/sec] (mean)
Time per request:       4.218 [ms] (mean)
Time per request:       1.055 [ms] (mean, across all concurrent requests)
Transfer rate:          55000.66 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.6      0       7
Processing:     1    4   3.1      3     216
Waiting:        1    4   3.0      3     215
Total:          1    4   3.2      3     216

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      4
  75%      5
  80%      6
  90%      7
  95%      9
  98%     11
  99%     13
100%    216 (longest request)