Scraping Yahoo! Buzz for a Google Search
A proof of concept tip that scrapes the buzziest items from Yahoo! Buzz and submits them to a Google search.
No web site is an island. Billions of hyperlinks link to billions of documents. Sometimes, however, you want to take information from one site and apply it to another site.
Unless that site has a web service API like Google's, your best bet is scraping. Scraping is where you use an automated program to remove specific bits of information from a web page. Examples of the sorts of elements people scrape include: stock quotes, news headlines, prices, and so forth. You name it and someone's probably scraped it.
There's some controversy about scraping. Some sites don't mind it, while others can't stand it. If you decide to scrape a site, do it gently; take the minimum amount of information you need and, whatever you do, don't hog the scrapee's bandwidth.
So, what are we scraping?
Google has a query popularity page; it's called Google Zeitgeist (https://www.google.com/press/zeitgeist.html). Unfortunately, the Zeitgeist is only updated once a week and contains only a limited amount of scrapable data. That's where Yahoo! Buzz (http://buzz.yahoo.com/) comes in. The site is rich with constantly updated information. Its "Buzz Index" keeps tabs on what's hot in popular culture: celebs, games, movies, television shows, music, and more.
This tip grabs the buzziest of the buzz, top of the "Leaderboard," and searches Google for all it knows on the subject. And to keep things current, only pages indexed by Google within the past few days [Tip #11] are considered.
|
The Code
#!/usr/local/bin/perl
# buzzgle.pl
# Pull the top item from the Yahoo Buzz Index and query the last
# three day's worth of Google's index for it
# Usage: perl buzzgle.pl
# Your Google API developer's key my $google_key='insert key here';
# Location of the GoogleSearch WSDL file my $google_wdsl = "./GoogleSearch.wsdl";
# Number of days back to go in the Google index my $days_back = 3;
use strict;
use SOAP::Lite;
use LWP::Simple;
use Time::JulianDay;
# Scrape the top item from the Yahoo Buzz Index
# Grab a copy of http://buzz.yahoo.com my $buzz_content = get("http://buzz.yahoo.com/")
or die "Couldn't grab the Yahoo Buzz: $!";
# Find the first item on the Buzz Index list my($buzziest) = $buzz_content =~ m!«TR BGCOLOR=white.+?1.+?«a HREF="http://
search.yahoo.com/search\?p=.+?&cs=bz"»(.+?)!i;
die "Couldn't figure out the Yahoo! buzz\n" unless $buzziest;
# Figure out today's Julian date my $today = int local_julian_day(time);
# Build the Google query my $query = "\"$buzziest\" daterange:" . ($today - $days_back) . "-$today";
print
"The buzziest item on Yahoo Buzz today is: $buzziest\n",
"Querying Google for: $query\n",
"Results:\n\n";
# Create a new SOAP::Lite instance, feeding it GoogleSearch.wsdl my $google_search = SOAP::Lite-»service("file:$google_wdsl");
# Query Google my $results = $google_search -»
doGoogleSearch(
$google_key, $query, 0, 10, "false", "", "false",
"", "latin1", "latin1"
);
# No results?
@{$results-»{resultElements}} or die "No results";
# Loop through the results foreach my $result (@{$results-»{'resultElements'}}) {
my $output =
join "\n",
$result-»{title} || "no title",
$result-»{URL},
$result-»{snippet} || 'no snippet',
"\n";
$output =~ s!«.+?»!!g; # drop all HTML tags
print $output;
}
Running the Tip
The script runs from the command line without need of arguments of any kind. Probably the best thing to do is to direct the output to a pager (a command-line application that allows you to page through long output, usually by hitting the spacebar), like so:
% perl buzzgle.pl | more
Or you can direct the output to a file for later perusal:
% perl buzzgle.pl » buzzgle.txt
As with all scraping applications, this code is fragile, subject to breakage if (read: when) HTML formatting of the Yahoo! Buzz page changes. If you find you have to adjust to match Yahoo!'s formatting, you'll have to alter the regular expression match as appropriate:
my($buzziest) = $buzz_content =~ m!«TR BGCOLOR=white.+?1.+?«a href="http ://search.yahoo.com/search\?p=.+?&cs=bz"»(.+?)!i;
Regular expressions and general HTML scraping are beyond the scope of this tutorial. For more information, I suggest you consult O'Reilly's Perl and LWP (http://www.oracle.com/catalog/perllwp/) or Mastering Regular Expressions (http://www.oracle.com/catalog/regex/).
The Results
At the time of this writing - and probably for some time yet to come - musical sensation, Eminem, is all the rage.
% perl buzzgle.pl | less
The buzziest item on Yahoo Buzz today is: Eminem Querying Google for: "Eminem" daterange:2452593-2452596
Results:
Eminem World http://www.eminemworld.com/
Eminem World specializing in Eminem News and Information. With Pictures, Discogr aphy, Lyrics ... your #1 Eminem Resource. Eminem World, ...
Eminem http://www.eminem.com/frameset.asp?PageName=eminem no snippet Eminem Planet - Your Ultimate Resource http://www.eminem-planet.com/
Eminem Planet - A Great Resource about the Real Slim Shady. .:8 Mile .:News .:Bi ography ... More News. ::Order Eminem's tutorial. Click Here to Check ...
...
Tiping the Tip
Here are some ideas for tiping the tip:
- The program as it stands returns 10 results. You could change that to one result and immediately open that result instead of returning a list. Bravo, you've just written I'm Feeling Popular!, as in Google's I'm Feeling Lucky!
- This version of the program searches the last three days of indexed pages. Because there's a slight lag in indexing news stories, I would index at least the last two days' worth of indexed pages, but you could extend it to seven days or even a month. Simply change my $days_back = 3;, altering the value of the $days_back variable.
- You could create a "Buzz Effect" tip by running the Yahoo! Buzz query with and without the date-range limitation. How do the results change between a full search and a search of the last few days?
- Yahoo!'s Buzz has several different sections. This one looks at the Buzz summary, but you could create other ones based on Yahoo!'s other buzz charts (television, http://buzz.yahoo.com/television/, for instance).
« Previous Next »