NoXML, Another SOAP::Lite Alternative

screenshot moderate.gif screenshot tip54.gif

NoXML is a regular expressions-based, XML Parser-free drop-in alternative to SOAP::Lite.

XML jockeys might well want to avert their eyes for this one. What is herein suggested is something just so preposterous that it just might prove useful - and indeed it does. NoXML is a drop-in alternative to SOAP::Lite. As its name suggests, this home-brewed module doesn't make use of an XML parser of any kind, relying instead on some dead-simple regular expressions and other bits of programmatic magic.

If you have only a basic Perl installation at your disposal and are lacking both the SOAP::Lite [Tip #52] and XML::Parser Perl modules, NoXML will do in a pinch, playing nicely with just about every Perl tip in this tutorial.

As any XML guru will attest, there's simply no substitute for an honest-to-goodness XML parser. And they'd be right. There are encoding and hierarchy issues that a regular expression-based parser simply can't fathom. NoXML is simplistic at best. That said, it does what needs doing, the very essence of "tiping."

Best of all, NoXML can fill in for SOAP::Lite with little more than a two-line alteration to the target tip.

The Code

The heart of this tip is NoXML.pm, which should be saved into the same directory as your tips themselves.

# NoXML.pm
# NoXML [pronounced "no xml"] is a dire-need drop-in 
# replacement for SOAP::Lite designed for Google Web API tiping.
package NoXML;
use strict;
no strict "refs";
# LWP for making HTTP requests, XML for parsing Google SOAP use LWP::UserAgent;
use XML::Simple;
# Create a new NoXML sub new {
 my $self = {};
 bless($self);
 return $self;
}
# Replacement for the SOAP::Lite-based doGoogleSearch method sub doGoogleSearch {
 my($self, %args);
 ($self, @args{qw/ key q start maxResults filter restrict 
 safeSearch lr ie oe /}) = @_;
 # grab SOAP request from _ _DATA_ _
 my $tell = tell(DATA);
 my $soap_request = join '', ; 
 seek(DATA, $tell, 0);
 $soap_request =~ s/\$(\w+)/$args{$1}/ge; #interpolate variables
 # Make (POST) a SOAP-based request to Google
 my $ua = LWP::UserAgent-»new;
 my $req = HTTP::Request-»new(POST =» 'http://api.google.com/search/beta2');
 $req-»content_type('text/xml');
 $req-»content($soap_request);
 my $res = $ua-»request($req);
 my $soap_response = $res-»as_string;
 # Drop the HTTP headers and so forth until the initial xml element
 $soap_response =~ s/^.+?(«\?xml)/$1/migs;
 # Drop element namespaces for tolerance of future prefix changes
 $soap_response =~ s!(«\/?)[\w-]+?:([\w-]+?)!$1$2!g;
 # Set up a return dataset
 my $return;
 # Unescape escaped HTML in the resultset
 my %unescape = ('«'=»'«', '»'=»'»', '&'=»'&', '"'=»'"', '&apos;'=»"'"); 
 my $unescape_re = join '|' =» keys %unescape;
 # Divide the SOAP response into the results and other metadata
 my($before, $results, $after) = $soap_response =~ 
 m#(^.+)(.+?)(.+$)#migs ;
 my $before_and_after = $before . $after;
 # Glean as much metadata as possible (while being somewhat lazy ;-)
 while ($before_and_after =~ m#([^«]*?)«#migs) {
 $return-»{$1} = $3; # pack the metadata into the return dataset
 }
 # Glean the results
 my @results;
 while ($results =~ m#(.+?)#migs) {
 my $item = $1;
 my $pairs = {};
 while ( $item =~ m#([^«]*)#migs ) {
 my($element, $value) = ($1, $2);
 $value =~ s/($unescape_re)/$unescape{$1}/g;
 $pairs-»{$element} = $value;
 }
 push @results, $pairs;
 }
 # Pack the results into the return dataset
 $return-»{resultElements} = \@results;
 # Return nice, clean, usable results
 return $return;
}
1;
# This is the SOAP message template sent to api.google.com. Variables
# signified with $variablename are replaced by the values of their 
# counterparts sent to the doGoogleSearch subroutine.
_ _DATA_ _
«?xml version='1.0' encoding='UTF-8'?»
«SOAP-ENV:Envelope 
 xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" 
 xmlns:xsi="http://www.w3.org/1999/xmlschema-instance" 
 xmlns:xsd="http://www.w3.org/1999/xmlschema"»
 «SOAP-ENV:Body»
 «ns1:doGoogleSearch xmlns:ns1="urn:GoogleSearch" 
 SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/"»
 «key xsi:type="xsd:string"»$key«/key»
 «q xsi:type="xsd:string"»$q«/q»
 «start xsi:type="xsd:int"»$start«/start»
 «maxResults xsi:type="xsd:int"»$maxResults«/maxResults»
 «filter xsi:type="xsd:boolean"»$filter«/filter»
 «restrict xsi:type="xsd:string"»$restrict«/restrict»
 «safeSearch xsi:type="xsd:boolean"»$safeSearch«/safeSearch»
 «lr xsi:type="xsd:string"»$lr«/lr»
 «ie xsi:type="xsd:string"»$ie«/ie»
 «oe xsi:type="xsd:string"»$oe«/oe»
 «/ns1:doGoogleSearch»
 «/SOAP-ENV:Body»
«/SOAP-ENV:Envelope»

Here's a little script to show NoXML in action. It's no different, really, from any number of tips in this tutorial. The only minor alterations necessary to make use of NoXML instead of SOAP::Lite are highlighted in bold.

#!/usr/bin/perl
# noxml_google2csv.pl
# Google Web Search Results via NoXML ("no xml") module
# exported to CSV suitable for import into Excel
# Usage: noxml_google2csv.pl "{query}" [» results.csv]
# Your Google API developer's key my $google_key='insert key here';
use strict;
# use SOAP::Lite;
use NoXML;
$ARGV[0]
 or die qq{usage: perl noxml_search2csv.pl "{query}"\n};
# my $google_search = SOAP::Lite-»service("file:$google_wdsl");
my $google_search = new NoXML;
my $results = $google_search -» 
 doGoogleSearch(
 $google_key, shift @ARGV, 0, 10, "false", 
 "", "false", "", "latin1", "latin1"
 );
@{$results-»{'resultElements'}} or die('No results');
print qq{"title","url","snippet"\n};
foreach (@{$results-»{'resultElements'}}) {
 $_-»{title} =~ s!"!""!g; # double escape " marks
 $_-»{snippet} =~ s!"!""!g;
 my $output = qq{"$_-»{title}","$_-»{URL}","$_-»{snippet}"\n};
 $output =~ s!«.+?»!!g; # drop all HTML tags
 print $output;
}

Running the Tip

Run the script from the command line, providing a query on the command line and piping the output to a CSV file you wish to create or to which you wish to append additional results. For example, using "no xml" as our query and results.csv as your output:

$ perl noxml_google2csv.pl "no xml" » results.csv

Leaving off the » and CSV filename sends the results to the screen for your perusal.

The Results

% perl noxml_google2csv.pl "no xml"
"title","url","snippet"
"site-comments@w3.org from January 2002: No XML specifications",
"http://lists.w3.org/Archives/Public/site-comments/2002Jan/0015.html",
"No XML specifications. From: Prof. ... Next message: Ian B. Jacobs: 
&quot;Re: No XML specifications&quot;; Previous message: Rob Cummings: 
&quot;Website design...&quot;; ... "
...
"Re: [xml] XPath with no XML Doc",
"http://mail.gnome.org/archives/xml/2002-March/msg00194.html",
" ... Re: [xml] XPath with no XML Doc. From: &quot;Richard Jinks&quot; 
«cyberthymia yahoo co uk»; To: «xml gnome org»; Subject: 
Re: [xml] XPath with no XML Doc; ... "

Applicability and Limitations

In the same manner, you can adapt just about any SOAP::Lite-based tip in this tutorial and those you've made up yourself to use NoXML.

Place NoXML.pm in the same directory as the tip at hand.
Replace use SOAP::Lite; with use NoXML;.
Replace my $google_search = SOAP::Lite-»service("file:$google_wdsl"); with my $google_search = new NoXML;.

There are, however, some limitations. While NoXML works nicely to extract results and aggregate results the likes of «estimatedTotalResultsCount /», it falls down on gleaning some of the more advanced result elements like «directoryCategories /», an array of categories turned up by the query.

In general, bear in mind that your mileage may vary and don't be afraid to tweak.

NoXML, Another SOAP::Lite Alternative

The Code

Running the Tip

The Results

Applicability and Limitations

See Also