Allowed memory size exhausted with the PHP Simple HTML DOM Parser

The Simple HTML DOM Parser is a useful tool for extracting content from web pages using jQuery like syntax. If multiple pages are loaded into the parser the available memory will eventually be exhausted and a fatal error will occur.

Some example code

The example code below has an already defined array $urls which contains a list of URLs to download content and parse.

require_once('/path/to/simple_html_dom.php');
foreach($urls as $url) {
    $dom = file_get_html($url);
    // do some stuff here
}

Depending on the number of pages downloaded, their size and size of the associated DOM, and how much memory is available to PHP, the available memory may be exhausted and result in a fatal error, halting your script at that point.

The error message 

The error message when memory is exhausted will be similar to this:

PHP Fatal error: Allowed memory size of XYZ bytes exhausted (tried to allocate XYZ bytes) in [path]/simple_html_dom.php on line XYZ

The solution

The Simple HTML DOM Parser does not clean up memory in the DOM each time file_get_html or str_get_html is called so it needs to be done explicity each time you have finished with the current DOM. This is as simple as adding ->clear() at the end of a loop or when you've finished using it.

The revised example with clear() is as follows:

require_once('/path/to/simple_html_dom.php');
foreach($urls as $url) {
    $dom = file_get_html($url);
    // do some stuff here
    $dom->clear();
}

Assuming there is enough memory available to PHP to handle each individual page and associated DOM, the process will now no longer suffer from memory exhaustion.

Download PHP Simple HTML DOM Parser / Examples

The PHP Simple HTML DOM Parser can be downloaded from SourceForge where there are also several examples of extracting content from pages.

Related posts:

• Find all anchor tags in a page with PHP and the Simple HTML DOM Parser
• Extract images from a web page with PHP and the Simple HTML DOM Parser

Find all anchor tags in a page with PHP and the Simple HTML DOM Parser

This post shows how to download a web page and find all the link anchor tags in the page using PHP and the Simple HTML DOM Parser which has a jQuery like syntax selector.

PHP Simple HTML DOM Parser

The PHP Simple HTML DOM Parser makes it easy to find particular elements within an HTML page in a similar way to jQuery. It can be downloaded from http://simplehtmldom.sourceforge.net/ where there are also several examples.

Finding the <a> tags from a web page

First of all include the Simple HTML DOM Parser using either include, require, include_once or require_once:

require_once('/path/to/simple_html_dom.php');

And then load the webpage into the DOM using either the file_get_html() or str_get_html() helper functions. The filename passed to file_get_html() can either be the URL to the web page or the filename of a local file. str_get_html() takes a string instead of a filename.

$dom = file_get_html('http://www.google.com/');
$dom = str_get_html('... some html string ...');

Now do find() on the DOM for 'a' tags as in the following example which echos out the "href" property with a linebreak between each one:

foreach($dom->find('a') as $a) {
    if($a->href) {
        echo $a->href . "\n";
    }
}

Using www.google.com as an example the above would output this:

http://images.google.co.nz/imghp?hl=en&tab=wi
http://maps.google.co.nz/maps?hl=en&tab=wl
http://news.google.co.nz/nwshp?hl=en&tab=wn
http://groups.google.co.nz/grphp?hl=en&tab=wg
http://books.google.co.nz/bkshp?hl=en&tab=wp
http://mail.google.com/mail/?hl=en&tab=wm
http://www.google.co.nz/intl/en/options/
http://scholar.google.co.nz/schhp?hl=en&tab=ws
http://blogsearch.google.co.nz/?hl=en&tab=wb
http://translate.google.co.nz/?hl=en&tab=wT
http://www.youtube.com/?hl=en&tab=w1&gl=NZ
http://www.google.com/calendar/render?hl=en&tab=wc
http://docs.google.com/?hl=en&tab=wo
http://www.google.co.nz/reader/view/?hl=en&tab=wy
http://sites.google.com/?hl=en&tab=w3
http://www.google.co.nz/intl/en/options/
/url?sa=p&pref=ig&pval=3&q=http://www.google.co.nz/ig%3Fhl%3Den%26source%3Diglk&usg=AFQjCNGi5EQv2pmx9Kd5MdCX46heegpxAw
/preferences?hl=en
https://www.google.com/accounts/Login?hl=en&continue=http://www.google.co.nz/
/advanced_search?hl=en
/language_tools?hl=en
http://www.google.co.nz/setprefs?sig=0_Va9MAZW7LCKUpGRFXj4-Xh78Tkc=&hl=mi
/intl/en/ads/
/services/
/intl/en/about.html
http://www.google.com/ncr
/intl/en/privacy.html

Notice that these are the hrefs as they appear in the HTML source, so some are relative to the current document/domain and some are absolute containing a full http:// path.

Resolving the paths

I've posted how to resolve the paths to full http:// URLs using the url_to_absolute library from Nadeau Software Consulting in my earlier post titled "Extract images from a web page with PHP and the Simple HTML DOM Parser"

I will write a standalone post about how to do this later this week, which also deals with a slight issue with the URLs returned as they are partially encoded by default using rawurlencode() which is not really ideal. This later post shows the modification needed to resolve this along with some additional examples.

Related posts:

• Allowed memory size exhausted with the PHP Simple HTML DOM Parser
• Resolving relative URLs to absolute in PHP
• Extract domain, path etc from a full url with PHP
• Extract images from a web page with PHP and the Simple HTML DOM Parser
• Get meta tags from an HTML file with PHP
• Change the user agent string in PHP