Stop logging "internal dummy connection" in Apache

Apache 2.x keeps child processes alive by creating internal connections which appear in the log files as "internal dummy connection" on the IP address ::1 or 127.0.0.1. If you ever monitor Apache log files you'll see a lot of these in the log files. This post shows how to prevent logging for these two IP addresses so your log files won't get filled up with these.

What the log lines look like

With IPv6 the log lines come from the IP address ::1 and will look similar to this:

::1 - - [11/Oct/2010:13:02:47 +1300] "OPTIONS * HTTP/1.0" 200 - "-" "Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny9 with Suhosin-Patch mod_ssl/2.2.9 OpenSSL/0.9.8g (internal dummy connection)"

Prevent logging for local requests

The simplest solution is to prevent logging for local requests. Normally these would only be from the Apache server itself, unless you are doing something special which is requesting pages using the local IP address (i.e. ::1 or 127.0.0.1).

Locate the logging section of your main Apache log file. You'll have an entry something along the lines of this, although the exact setting will vary depending on which operating system, distribution and version you are using, or any custom changes you have made:

CustomLog /var/log/apache2/access.log combined

Add this line for IPv4 style IP addresses for local connections (127.0.0.1):

SetEnvIf Remote_Addr "127\.0\.0\.1" dontlog

or this for IPv6 style IP addresses (::1):

SetEnvIf Remote_Addr "::1" dontlog

And then add env=!dontlog to the end of your logging line so it looks like this, using the same example as shown above:

CustomLog /var/log/apache2/access.log combined env=!dontlog

Now restart Apache and any local connections, including those "internal dummy connection" entries, will no longer be logged.

Related posts:

• Viewing live Apache logs with tail, grep and egrep
• Changing Apache log rotation behaviour on CentOS / RHEL
• Howto Restart Apache

Changing Apache log rotation behaviour on CentOS / RHEL

On a default install of CentOS or Red Hat Enterprise Linux, the log rotation script will automatically rotate the Apache log file each day and then reload the httpd service. This post looks at how to prevent this action from occuring automatically, or to change the behaviour to rotate the log files if your naming convention for log files is different from the default.

The cron daemon on a CentOS or Red Hat Enterprise Linux server by default runs the scripts in the directory /etc/cron.daily on a daily basis. This includes running the "logrotate" script which runs /usr/sbin/logrotate passing it the /etc/logrotate.conf configuration file. The default version of this logrotate.conf file is as follows:

# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4



# create new (empty) log files after rotating old ones
create


# uncomment this if you want your log files compressed
#compress


# RPM packages drop log rotation information into this directory
include /etc/logrotate.d


# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    minsize 1M
    create 0664 root utmp
    rotate 1
}


# system-specific logs may be also be configured here.

These default variables are fairly straight forward and give you control over how frequently the log files are rotated, how many lots of previous log files to keep before deleting them, whether or not to compress them and so on. It also tells logrotate to process all the files in the directory /etc/logrotate.d

The file which controls log rotation of Apache on CentOS and Red Hat Enterprise Linux is named /etc/logrotate.d/httpd, and the default contents of this file are as follows:

/var/log/httpd/*log {
    missingok
    notifempty
    sharedscripts
    postrotate
        /sbin/service httpd reload > /dev/null 2>/dev/null || true
    endscript
}

What this file is saying, is to rotate all files that match the pattern /var/log/httpd/*.log. The default access and error log files in Apache on CentOS and RHEL are named "access_log" and "error_log" so they match this pattern.

If you have changed the default names of the log files and they don't match this pattern, then you can have logrotate rotate your log files by making sure the pattern is in this file in place of the /var/log/httpd/*.log line. For example, if your log files were named error.foo and access.foo then /var/log/httpd/*.foo would match them.

If you do not wish the logrotate script to rotate your Apache log files at all, then you can simply delete this file like so, logged in either as root or using sudo:

rm -f /etc/logrotate.d/httpd

The changes will take effect the next time the logrotate script is run.

Viewing live Apache logs with tail, grep and egrep

There are times you may need to monitor what's happening on an Apache web server as is happens. This can be done from the command line using a combination of the tail command, which outputs the last part of a file, and grep or egrep which are used for regular expression pattern matching.

Viewing everything

If the log file to view is at /var/log/apache/myvirtualhost.log the first command below will show the last few lines from the file and then continue to echo to the command line as new lines are entered into the log file i.e. as additional requests are made to the web server.

tail -f /var/log/apache/myvirtualhost.log

The -f flag is what makes the tail command output additional data as it is appended to the log.

Viewing everything from a specific IP address

Tail can be combined with grep to pattern match. To filter the results to only show requests for a specific IP address (in this example 192.168.206.1) pipe the output from tail through grep like so:

tail -f /var/log/apache/myvirtualhost.log | grep 192.168.206.1

This can be useful to only show results from your own requests.

Note that the above example would also match e.g. 192.168.206.10 etc and that dots will match any character not just the period divider; if this is a concern then escape the dots with \ and put the IP address in brackets with a space after the last digit in the IP address like this:

tail -f /var/log/apache/myvirtualhost.log | grep "192\.168\.206\.1 "

Viewing everything excluding a specific IP address

Adding the -v flag to grep excludes the pattern. If you want to exclude requests from your own IP address but show everything else this can be useful:

tail -f /var/log/apache/myvirtualhost.log | grep -v "192\.168\.206\.1 "

Including particular file types only

If you only want to watch for requests for a particular file type, or even a particular file then use the same concept as grepping for the IP address. For example to show only jpg files:

tail -f /var/log/apache/myvirtualhost.log | grep .jpg

And to match a specific file, for example the robots.txt file if perhaps you are looking out for when a search engine bot hits the site:

tail -f /var/log/apache/myvirtualhost.log | grep robots.txt

Excluding particular file types

To show only webpages can be problematic especially if there is no common extension for the files being served, and some might end with / whereas other might end with .html, or there might be query strings at the end of the URL which present issues with pattern matching.

A possible solution is instead to exclude everything that's not a webpage. Multiple exclusions can be entered by separating them with the pipe | character when using egrep instead of grep. To exclude several common file extensions and show hopefully just web pages do this:

tail -f /var/log/apache/myvirtualhost.log | egrep -v "(.gif|.jpg|.png|.swf|.ico|.txt|.xml|.css|.js|.rss)"

Note that because the regular expression contains the pipe character the expression must be contained within quotes. You can adjust the above list of extensions to suit your own conditions.

Related posts:

• Stop logging "internal dummy connection" in Apache
• Get Apache to parse .html files as PHP
• Aliasing images to a remote server with .htaccess
• Changing Apache log rotation behaviour on CentOS / RHEL

Content-type for Javascript with Apache

I was making some changes to an Apache configuration this morning to ensure mod_deflate was compressing CSS and Javascript files and discovered I'd either documented the content-type for Javascript incorrectly, or the Apache version and Linux distro I used at the time served Javascript differently.

text/x-js vs application/javascript content-type

In the above linked post it's documented as being text/x-js. This worked for me at the time on that particular host. When I tried to configure this on a Debian host this morning it had no effect.

To find out what content-type was being served, I used the command line "lynx" web browser with the -head flag to display the headers like so (yes, that .local address is correct - I was testing this on my local development box which had the domain configured like that):

lynx -head -dump http://www.electrictoolbox.local/js/common.min.js

The output from this was:

HTTP/1.1 200 OK
Date: Tue, 19 Jan 2010 18:16:31 GMT
Server: Apache/2.2.9 (Debian) PHP/5.2.6-1+lenny3 with Suhosin-Patch
Last-Modified: Mon, 18 Jan 2010 18:08:20 GMT
ETag: "276030-1c8d3-47d885ad86d00"
Accept-Ranges: bytes
Content-Length: 116947
Vary: Accept-Encoding
Connection: close
Content-Type: application/javascript

Note the last line which shows the content type.

So using the application/javascript content-type I was then able to successfully have mod_deflate compress my Javascript files. I have also seen it in the past served as application/x-javascript

If you are having difficulty configuring mod_deflate and it doesn't seem to be compressing the files, then check the content-type using lynx as shown above and use that content type.

Related posts:

• Compressing files on Apache with mod_deflate

Setting 503 Service Temporarily Unavailable headers with Apache .htaccess

When temporarily taking down a website to perform maintenance, it's a good idea to return a "503 Service Temporarily Unavailable" header so search engines know to come back later. This post shows how to set this header using an Apache .htaccess file, and also how to show a response page to users with PHP so they know to try again later.

.htaccess settings

Create an .htaccess file in the website's root directory containing the following:

RewriteEngine On
RewriteCond %{REMOTE_ADDR} !^111\.111\.111\.111$
RewriteCond %{REQUEST_URI} !\.(css|gif|ico|jpg|js|png|swf|txt)$
RewriteRule .* - [R=503,L]

If you already have an .htaccess file then put the above lines in the top. The L condition means it will be the last rule so the others will be ignored while these lines are present.

Line 2 makes it so the user at the IP address 111.111.111.111 will not have the rewrite rule applied to them and can still view the website. Substitute 111.111.111.111 for your own IP address so you still have access to the website to check it while it's made unavailable.

Line 3 makes it so various media files etc do not have the rules applied. This is especially important when creating a custom error page (see below); without the rule all the image files etc in the error page would also be made unavailable.

Line 4 shows a 503 Service Temporarily Unavailable message using Apache's default page.

Specifying a custom 503 error page with ErrorDocument

In theory, you should be able to add an ErrorDocument directive to show for a 503 error instead of Apache's default page but whenever I tried it I would get a "Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request" error message, even though the path to the file was correct.

If anyone knows the solution to this issue please leave a comment in the comments section below. But for what it's worth, this is what I tried:

ErrorDocument 503 /path/to/503.html

RewriteEngine On
RewriteCond %{REMOTE_ADDR} !^111\.111\.111\.111$
RewriteCond %{REQUEST_URI} !\.(css|gif|ico|jpg|js|png|swf|txt)$
RewriteRule .* - [R=503,L]

Specifying a custom 503 error page with PHP

A scripting language can be used to set the 503 header and also an additional "Retry-After" header so the search engines know when to come back again. This also means you can send a more user-friendly error page to the user to let them know to come back again shortly.

Put this in the .htaccess file:

RewriteEngine On
RewriteCond %{REQUEST_URI} !\.(css|gif|ico|jpg|js|png|swf|txt)$
RewriteCond %{REMOTE_ADDR} !^111\.111\.111\.111$
RewriteRule .* 503.php [L]

And then add this to the start of the PHP page (it can be called whatever you like, but 503.php made sense to me):

header('HTTP/1.1 503 Service Temporarily Unavailable');
header('Retry-After: 600');

The Retry-After header sets the amount of time for the search engines to retry in seconds; in the above example it's 10 minutes. Underneath the PHP code put whatever HTML etc is required to show a friendly error message to the website visitors.

Related posts:

• 410 "Gone" with PHP
• 303 redirect with PHP
• 301 redirect with PHP

How to Restart Apache

If you have made changes to the Apache configuration file httpd.conf or one of the other included configuration files such as the vhosts.d files, you need to reload the Apache service for the changes to take effect. From the command line you do this with the apachectl command. The exact location of this command varies on the Unix or Linux variant you are using (eg Fedora, OSX, FreeBSD, Slackware, Mandrake, SUSE) and the compile time settings, but typically it is accesible at /usr/sbin/apachectl

Gracefully restarting Apache

An example of restarting Apache gracefully is shown below:

/usr/sbin/apachectl graceful

Note that you will either need to be running as root or use the "sudo" command in order to run this command.

If Apache is not already running it will be started. If it is already running then it will reload with the new changes but will not abort active connections, meaning that anyone who is in the middle of downloading something will continue to be able to download it.

Running a configuration test first

Before restarting the Apache service a check will be done on the configuration files to ensure they are valid. If there is an error in them the error will be displayed and the Apache service will continue running using the old settings. You need to correct your settings before attempting to restart again.

You can also just check the settings without restarting Apache like so:

/usr/sbin/apachectl configtest

This will check the httpd.conf file and report whether the syntax of the file is valid or not. A list of errors will be displayed including the line numbers if there are any. This makes it easy to isolate any problems.

Available options for the apachectl command

The following are all the available options that can be passed to the apachectl command. This text is from the apachectl man page.

apachectl start: Start the Apache daemon. Gives an error if it is already running.

apachectl stop: Stops the Apache daemon.

apachectl restart: Restarts the Apache daemon by sending it a SIGHUP. If the daemon is not running, it is started. This command automatically checks the configuration files via configtest before initiating the restart to make sure Apache doesn't die.

fullstatus: Displays a full status report from mod_status. For this to work, you need to have mod_status enabled on your server and a text-based browser such as lynx available on your system. The URL used to access the status report can be set by editing the STATUSURL variable in the script.

apachectl status: Displays a brief status report. Similar to the fullstatus option, except that the list of requests currently being served is omitted.

apachectl graceful: Gracefully restarts the Apache daemon by sending it a SIGUSR1. If the daemon is not running, it is started. This differs from a normal restart in that currently open connections are not aborted. A side effect is that old log files will not be closed immediately. This means that if used in a log rotation script, a substantial delay may be necessary to ensure that the old log files are closed before processing them. This command automatically checks the configuration files via configtest before initiating the restart to make sure Apache doesn't die.

apachectl configtest: Run a configuration file syntax test. It parses the configuration files and either reports Syntax Ok or detailed information about the particular syntax error.

apachectl help: Displays a short help message.

Update March 19th 2007

I wrote this article originally for Apache 1.3 and when I was using Gentoo Linux and OSX to serve pages with Apache. On my current openSUSE machines which run Apache 2.2 there is no longer an apachectl program. Instead of the above command you can run the following very similar command and options instead:

/etc/init.d/apache2 start|stop|reload|restart|configtest

Running /etc/init.d/apache2 on its own outputs a help message to detail the various options as listed below:

start - start httpd

startssl - start httpd with -DSSL

stop - stop httpd (sendign SIGTERM to parent)

try-restart - stop httpd and if this succeeds (i.e. if it was running before), start it again.

status - check whether httpd is running

restart - stop httpd if running; start httpd

reload|graceful - do a graceful restart by sending a SIGUSR1 or start if not running

configtest - do a configuration syntax test

extreme-configtest - try to run httpd as nobody (detects more errors by actually loading the configuration, but cannot read SSL certificates)

probe - probe for the necessity of a reload, give out the argument which is required for a reload. (by comparing conf files with pidfile timestamp)

full-server-status - dump a full status screen; requires lynx or w3m and mod_status enabled

server-status - dump a short status screen; requires lynx or w3m and mod_status enabled

help - this screen

Set PHP configuration options with an Apache .htaccess file

PHP has a large number of configuration option which can be set in the php.ini file, Apache <virtualhost> blocks, .htaccess files and ini_set(). This post looks at how to set PHP configuration options with Apache's .htaccess files.

Note that if an option can be set in an .htaccess file it can usually (but not always) be set using the ini_set() function. I'll cover that function in a later post. The complete list of PHP configuration options can be found in the PHP manual.

Note also that only options labelled PHP_INI_PERDIR or PHP_INI_ALL from that list can be set in .htaccess file.

If for example you wanted to change the setting for auto_prepend_file, which makes a file be included before the PHP script, you would add this to your .htaccess file, where "prepend.inc.php" is the file to be prepended to your script:

php_value auto_prepend_file prepend.inc.php

To change the maximum file size that can be uploaded set the upload_max_filesize value. It defaults to 2M but you can change it to e.g. 5M like so:

php_value upload_max_filesize 5M

Note that although both of the above options can be set using ini_set() they will have no effect because the actions of these options occur before your script is parsed. Putting them in an .htaccess files means they will work.

There are a large number of PHP options that can be configured in an .htaccess file. Read the List of php.ini directivesand Description of core php.ini directives PHP manual pages for more details about each option.

I've written a follow up post which looks at setting PHP_INI_SYSTEM options in an Apache virtualhost settings. Read more here: Set PHP configuration options in an Apache virtualhost

Related posts:

• Get Apache to parse .html files as PHP
• Log PHP errors with log_errors and error_log
• Automatically append or prepend files in a PHP script
• Get a list of all available classes with PHP
• PHP is not showing any error messages