June 2007
Introduction
Sometimes there's no substitute for seeing what goes over the wire. This is especially true when writing software which uses web services like the Google Data APIs, where lots of operations involve making HTTP requests. When all else fails, you can verify that your program is doing what you'd expect by seeing the actual transmitted and received bytes. Many of the client libraries for the Google Data APIs have a debugging mode which displays the HTTP traffic. This is especially useful when you you don't have access to a packet sniffer like WireShark or Fiddler.
I can't count the number of times that I could have sworn my program was correct, only to find upon inspecting a packet trace that there was an extra newline character, or a misnamed HTTP header. Programming against a web service without looking at the HTTP traffic can be like trying to thread a needle with your eyes glued shut.
However, you may find yourself in a situation where a packet sniffer is unavailable or is inadequate to deal with encrypted packets. Never fear-you can get around this limitation by leveraging some in-program logging mechanisms. By utilizing these logging facilities, you can see some, if not all, of the exchanged data, even for encrypted HTTPS data or remote running code.
For this article, I've written sample diagnostic code in 3 languages using the Google Data API client libraries for Java, .NET, and Python. In each example, I turn on logging or debugging, authenticate using client login, and then get a list of my Google Spreadsheets and print out their titles.
Java
You can use the java.util.logging classes to set the logging
levels (and consequently expose traffic data) for a couple of key objects in
the client library. In the example below, I chose to look at the HTTP headers
and the activities of the XML parser to get a complete view of what is
traveling over the wire.
The Google Data Java client library has separate classes to
handle HTTP requests and XML parsing; thus, I need to create two Logger objects,
one for each class:
com.google.gdata.client.http.HttpGDataRequest handles the HTTP
traffic while com.google.gdata.util.XmlParser is responsible for
XML parsing.
The logger instances will record activities for
HttpGDataRequest and XmlParser, and you can control
the level of detail of each logger's output. For
this demonstration, I've chosen to view all of the events produced by the HttpGDataRequest and XmlParser objects.
Once I've created
and configured my Loggers, I need to tell them what to do when they receive an event from their classes. For now, I want to write all logging information out
to the console, so I create a ConsoleHandler and add it to both of my Loggers.
Here's my sample code:
import com.google.gdata.client.spreadsheet.*;
import com.google.gdata.data.spreadsheet.*;
import com.google.gdata.util.*;
import java.io.*;
import java.net.URL;
import java.util.*;
import java.util.logging.*;
public class PrintSpreadsheetsWithLogging {
public static void main(String [] args) throws AuthenticationException,
ServiceException, IOException {
// Configure the logging mechanisms.
Logger httpLogger = Logger.getLogger("com.google.gdata.client.http.HttpGDataRequest");
httpLogger.setLevel(Level.ALL);
Logger xmlLogger = Logger.getLogger("com.google.gdata.util.XmlParser");
xmlLogger.setLevel(Level.ALL);
// Create a log handler which prints all log events to the console.
ConsoleHandler logHandler = new ConsoleHandler();
logHandler.setLevel(Level.ALL);
httpLogger.addHandler(logHandler);
xmlLogger.addHandler (logHandler);
SpreadsheetService service = new SpreadsheetService("testing-loggingExampleApp-1");
service.setUserCredentials(email, password);
// Get a list of your spreadsheets.
URL metafeedUrl = new URL("http://spreadsheets.google.com/feeds/spreadsheets/private/full ");
SpreadsheetFeed feed = service.getFeed(metafeedUrl, SpreadsheetFeed.class);
// Print the title of each spreadsheet.
List spreadsheets = feed.getEntries();
for (int i = 0; i < spreadsheets.size(); i++) {
SpreadsheetEntry entry = (SpreadsheetEntry)spreadsheets.get(i);
System.out.println("\t" + entry.getTitle().getPlainText());
}
}
}
When you run this program, you'll see something like this on the console (I cut out some of the less interesting parts):
Jun 7, 2007 10:24:50 AM ...HttpGDataRequest setPrivateHeader FINER: Authorization: <Not Logged> Jun 7, 2007 10:24:50 AM ...HttpGDataRequest setHeader FINER: User-Agent: ... ... Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute FINE: 200 OK Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute FINER: Date: Thu, 07 Jun 2007 17:25:24 GMT Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute FINER: null: HTTP/1.1 200 OK Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute FINER: Content-Type: application/atom+xml; charset=UTF-8 Jun 7, 2007 10:25:20 AM ...HttpGDataRequest execute FINER: Last-Modified: Thu, 07 Jun 2007 17:25:22 GMT ... Jun 7, 2007 10:25:20 AM ...XmlParser startElement FINE: Start element id Jun 7, 2007 10:25:20 AM ...XmlParser endElement FINE: End element id ... Jun 7, 2007 10:25:20 AM ...XmlParser startElement FINE: Start element title Jun 7, 2007 10:25:20 AM ...XmlParser startElement FINER: Attribute type='text' Jun 7, 2007 10:25:20 AM ...XmlParser endElement FINE: End element title ... Jun 7, 2007 10:25:20 AM ...XmlParser endElement FINE: End element entry ... Jun 7, 2007 10:25:20 AM ...XmlParser endElement FINE: End element feed
These logs can get quite large, so you might want to be more selective in setting the Loggers' levels. You could also
create a FileHandler instead of a ConsoleHandler to allow you to store the log data for later
use.
Of course, if Java isn't your bag, you could try .NET.
.NET
To capture the HTTP traffic in the .NET client library, you can replace the default request factory in the client with a GDataLoggingRequestFactory.
The HTTP requests in the .NET library are created by the GDataRequestFactory which is inside each Service object. The normal request factories don't perform
any logging but the GDataLoggingRequestFactory, which is a subclass of the GDataRequestFactory, has logging built in. You can specify the full path of the
log file by setting CombinedFileName.
After setting up your request factory, you need to replace the request factory in your Service object by setting the RequestFactory of the service object.
Your code might look something like this:
using System;
using Google.GData.Client;
using Google.GData.Extensions;
using Google.GData.Spreadsheets;
namespace LogginTest
{
class Program
{
static void Main(string[] args)
{
SpreadsheetsService service = new SpreadsheetsService("-exampleApp-1");
service.setUserCredentials(email, password);
Google.GData.Client.GDataLoggingRequestFactory factory = new GDataLoggingRequestFactory("wise", "SpreadsheetsLoggingTest");
factory.MethodOverride = true;
factory.CombinedLogFileName = "c:\\temp\\xmllog.log";
Console.WriteLine("Log file name:" + factory.CombinedLogFileName);
service.RequestFactory = factory;
SpreadsheetQuery query = new SpreadsheetQuery();
SpreadsheetFeed feed = service.Query(query);
Console.WriteLine("Your spreadsheets:");
foreach (SpreadsheetEntry entry in feed.Entries)
{
Console.WriteLine(entry.Title.Text);
}
Console.ReadKey();
}
}
}
The resulting log file contains the XML requests and responses. Here's an abbreviated example which I've formatted using tidy.
<?xml version='1.0' encoding='utf-8'?>
<feed xmlns='http://www.w3.org/2005/Atom'
xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/'>
<id>
http://spreadsheets.google.com/feeds/spreadsheets/private/full</id>
<updated>2007-06-07T22:05: 02.674Z</updated>
<link rel='self' type='application/atom+xml'
href='http://spreadsheets.google.com/feeds/spreadsheets/private/full'>
</link>
...
<entry>
<updated>2007-03-28T17:28:57.250Z</updated>
<category scheme=' http://schemas.google.com/spreadsheets/2006'
term='http://schemas.google.com/spreadsheets/2006#spreadsheet'>
<title type='text'>events</title>
<content type='text'>events</content>
...
</entry>
<entry>
<updated>2007-05-25T22:11:08.200Z</updated>
<category scheme=' http://schemas.google.com/spreadsheets/2006'
term='http://schemas.google.com/spreadsheets/2006#spreadsheet'>
</category>
<title type='text'>UnitTest</title>
<content type='text'>UnitTest</content>
...
</entry>
...
</feed>
But perhaps you are really into scripting languages, and you prefer using Python.
Python
To capture the HTTP traffic in the Python client library, you can echo the HTTP header traffic to the console by turning on debug mode in the HTTP client. The service object has a debug member which you can set to True.
Setting debug to true will set the debug flag in the underlying HTTPRequest object which is contained in the service object.
Here's an example which will echo the HTTP headers sent from the spreadsheets server when you ask for a list of your spreadsheets.
#!/usr/bin/python
import gdata.spreadsheet.service
client = gdata.spreadsheet.service.SpreadsheetsService()
client.debug = True
client.ClientLogin(email, password)
feed = client.GetSpreadsheetsFeed()
for entry in feed.entry:
print entry.title.text
And you will see something like this on your console:
reply: 'HTTP/1.1 200 OK\r\n' header: Content-Type: application/atom+xml; charset=UTF-8 header: Last-Modified: Thu, 07 Jun 2007 18:22:35 GMT header: Cache-Control: max-age=0, must-revalidate, private header: Transfer-Encoding: chunked ... header: Date: Thu, 07 Jun 2007 18:22:35 GMT header: Server: GFE/1.3
As you perform additional operations, such as an insert or update, you'll see corresponding request data echoed to your console.
Conclusion
This brief tutorial has illustrated how you can add basic logging functionality into a Java, .NET, or Python program which uses the Google Data API client libraries. These techniques can be useful if you need to debug HTTP exchanges, but don't have access to a packet sniffer. I've only scratched the surface with these examples. Many of the logging mechanisms present in these languages are much more powerful than what is shown here. If you'd like more information on logging or the Google Data APIs, check out the list of resources below.
Client libraries covered in this article can be found on these pages:
Related knowledge base items:
- How do I get HTTP logging information in the Java client library?
- How do I get HTTP logging information in the .NET client library?
- What are some good tools for HTTP debugging?
- What is the Google Spreadsheets API?
Discussion groups: We have quite a few, with more coming as more Google Data APIs are rolled out. We actively monitor the groups.
If you have questions or suggestions, I'd enjoy hearing from you. Hop on the discussion group and start posting.