Web Basics with LWP
by Sean M. Burke
|
Pages: 1, 2, 3, 4, 5
Adding Other HTTP Request Headers
The most commonly used syntax for requests is $response = $browser->get($url), but in truth, you can add extra HTTP header
lines to the request by adding a list of key-value pairs after the URL, like so:
$response = $browser->get( $url, $key1, $value1, $key2, $value2, ... );
For example, here's how to send more Netscape-like headers, in case you're dealing with a site that would otherwise reject your request:
my @ns_headers = (
'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
'Accept' => 'image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, image/png, */*',
'Accept-Charset' => 'iso-8859-1,*,utf-8',
'Accept-Language' => 'en-US',
);
...
$response = $browser->get($url, @ns_headers);
If you weren't reusing that array, you could just go ahead and do this:
$response = $browser->get($url,
'User-Agent' => 'Mozilla/4.76 [en] (Win98; U)',
'Accept' => 'image/gif, image/x-xbitmap, image/jpeg,
image/pjpeg, image/png, */*',
'Accept-Charset' => 'iso-8859-1,*,utf-8',
'Accept-Language' => 'en-US',
);
If you were only going to change the 'User-Agent' line, you could just change
the $browser object's default line from "libwww-perl/5.65" (or the like)
to whatever you like, using LWP::UserAgent's agent method:
$browser->agent('Mozilla/4.76 [en] (Win98; U)');
Enabling Cookies
A default LWP::UserAgent object acts like a browser with its cookies
support turned off. There are various ways of turning it on, by setting
its cookie_jar attribute. A "cookie jar" is an object representing
a little database of all the HTTP cookies that a browser can know about. It can correspond to a file on disk (the way Netscape uses its cookies.txt file), or it can be just an in-memory object that starts out empty, and whose collection of
cookies will disappear once the program is finished running.
To give a browser an in-memory empty cookie jar, you set its cookie_jar attribute like so:
$browser->cookie_jar({});
To give it a copy that will be read from a file on disk, and will be saved
to it when the program is finished running, set the cookie_jar attribute
like this:
use HTTP::Cookies;
$browser->cookie_jar( HTTP::Cookies->new(
'file' => '/some/where/cookies.lwp',
# where to read/write cookies
'autosave' => 1,
# save it to disk when done
));
That file will be an LWP-specific format. If you want to access the cookies in your Netscape cookies file, you can use the HTTP::Cookies::Netscape class:
use HTTP::Cookies;
# yes, loads HTTP::Cookies::Netscape too
$browser->cookie_jar( HTTP::Cookies::Netscape->new(
'file' => 'c:/Program Files/Netscape/Users/DIR-NAME-HERE/cookies.txt',
# where to read cookies
));
You could add an 'autosave' => 1 line as we did earlier, but at
time of writing, it's uncertain whether Netscape might discard some of the cookies you could be writing back to disk.
Posting Form Data
Many HTML forms send data to their server using an HTTP POST request, which you can send with this syntax:
$response = $browser->post( $url,
[
formkey1 => value1,
formkey2 => value2,
...
],
);
$response = $browser->post( $url,
[
formkey1 => value1,
formkey2 => value2,
...
],
headerkey1 => value1,
headerkey2 => value2,
);
For example, the following program makes a search request to AltaVista (by sending some form data via an HTTP POST request), and extracts from the HTML the report of the number of matches:
use strict;
use warnings;
use LWP 5.64;
my $browser = LWP::UserAgent->new;
my $word = 'tarragon';
my $url = 'http://www.altavista.com/sites/search/web';
my $response = $browser->post( $url,
[ 'q' => $word, # the Altavista query string
'pg' => 'q', 'avkw' => 'tgz', 'kl' => 'XX',
]
);
die "$url error: ", $response->status_line
unless $response->is_success;
die "Weird content type at $url -- ", $response->content_type
unless $response->content_type eq 'text/html';
if( $response->content =~ m{AltaVista found ([0-9,]+) results} ) {
# The substring will be like "AltaVista found 2,345 results"
print "$word: $1\n";
} else {
print "Couldn't find the match-string in the response\n";
}
