Using cURL to interact with Google Data services

Warning: This page is about Google's older APIs, the Google Data APIs; it's relevant only to the APIs that are listed in the Google Data APIs directory, many of which have been replaced with newer APIs. For information about a specific new API, see the new API's documentation. For information about authorizing requests with a newer API, see Google Accounts Authentication and Authorization.

Ryan Boyd, Google Data APIs Team
September 2007

Introduction

At heart, Google Data APIs use Atom feeds and entries (XML) as a data format and HTTP as a protocol for data transmission - extending the Atom Publishing Protocol. We publish a number of client libraries to make interacting with Google Data APIs easier. However, you're always welcome to use lower-level tools to work with our services, and it's pretty easy to do with a little guidance.

cURL is a command-line application for performing requests using a variety of protocols including HTTP. cURL is often used by developers to test Google Data services, as it supports the HTTP functionality required to interact with the APIs at a low level.

cURL only provides support for performing the HTTP communication, so knowledge of the Google Data protocol, the service-specific protocol and the XML data format used is a prerequesite for working with the application. Some other tools are mentioned in this article for making these tasks easier.

This article uses examples based upon the Picasa Web Albums data API. However, all these examples can be readily applied to other Google Data APIs.

Obtaining and installing cURL

cURL is commonly available on a default install of many UNIX/Linux platforms. Try typing curl in your favorite shell to see if the tool is installed and is in your PATH. If you don't have the tool installed, visit the download page on the cURL website to obtain the official source or a user-contributed binary package. Note that the command-line tool uses the libcurl library, which may be offered as a separate download package, so, if you're not compiling from the source, be sure to download a 'binary' package instead of a 'libcurl' package. The SSL-enabled packages are required if you wish to use cURL to acquire authentication tokens or to access some Google Data services which require using SSL for requests.

Authenticating to a Google Data Service

Authenticated Google Data requests are performed by adding an HTTP header to the request which contains either a ClientLogin (desktop/mobile apps) or AuthSub (web apps) authentication token. For the purposes of testing using cURL, ClientLogin is the easier method and is documented below. AuthSub authentication headers could be used with cURL, but the more-advanced process of obtaining the tokens is out of scope for this article.

Using ClientLogin

ClientLogin is intended for installed (desktop/mobile) applications. With this method of authentication, the application using Google Data APIs directly handles the username and password of the user.

An authentication request for ClientLogin takes a username, password, and service name as form post variables. These variables are passed as the Email, Passwd, and service arguments, respectively. This request yields a response with several tokens, one of which can be used to make requests to the Google Data service. Note that data arguments passed with curl must be URL-encoded if they contain non-ASCII characters, which often appear in Email and Passwd arguments. You can ask curl to URL-encode these arguments by using the --data-urlencode flag.

Example Request:

curl https://www.google.com/accounts/ClientLogin \
--data-urlencode Email=brad.gushue@example.com --data-urlencode Passwd=new+foundland \
-d accountType=GOOGLE \
-d source=Google-cURL-Example \
-d service=lh2

Example Response:

SID=DQAAAHYBADCv2pSv7nflacDNwz3zEDUGtrSvNVDcpkSfddi77b3U5sEaHmP8YLWhmA36F9rk85mL8J5dqo4apn0T1vKz0fPGI9Xtnuet6cuE2ZzYvrNIwbSC_HjTqF4zudNQnnlDuD2wqZT-g1qXI8KhGAQZV4NexHZoQPlabTsGuRZeIBxj1A
LSID=EUBBBIaBADCl-kNxvRVmcQghpt3cqSMfEooKR9flLOUZqwgP9OrZS83gse-KSdTNeXhxsET7FYenDhceP9lIPOmesH-t9qh-AWUHjjMdZEbUNeF9mWyzln6Z-FajaiG-cVFkqW0ZJ8ZbnCP30xXj6xFK6QxaAcqy_9Pej8jhEnxS9E61ftQGPg
Auth=EUBBIacAAADK-kNxvRVmcQghpt3cqSMfEooLNMflLNIQqwgP9OrZS83gs-KSdTNeXhxsET7FYePWmaD8Vsy1V4LSUGMUP48Je2TO8OcjBj6HgAtPhiZeX-gKDfagZDK44j4n-Tkb44nhOnp2_QPSnBj3Z2vYwOEDjjG3Q53aQVC2132JKOuGh

Please see the ClientLogin documentation for specific information on the parameters used in the above request. In this example, the service we're using is the Picasa Web Albums data API. The service name (service) is lh2. The service names for other Google Data services can be found in the Google Data APIs FAQ page.

The value of the Auth token in the above response is the only value needed for authentication to Google Data services. The value of this token is formed into an HTTP header which is then used for each request to a Google Data service.

curl --silent --header "Authorization: GoogleLogin auth=EUBBIacAAADK\
-kNxvRVmcQghpt3cqSMfEooLNMflLNIQqwgP9OrZS83gs-KSdTNeXhxs\
ET7FYePWmaD8Vsy1V4LSUGMUP48Je2TO8OcjBj6HgAtPhiZeX-gKDfag\
ZDK44j4n-Tkb44nhOnp2_QPSnBj3Z2vYwOEDjjG3Q53aQVC2132JKOuGh" \
"http://picasaweb.google.com/data/feed/api/user/default"

Note: The method of escaping newline characters with backslash characters ('\') above does not work in the Windows command shell, so you must enter the entire command on one line if you're running curl on Windows.


Retrieving feeds and entries

In Google Data APIs, retrieving feeds and entries is done by performing an HTTP GET on a URL, with an optional set of query parameters. Because we are performing a GET request, only the auth header and the URL are required to be passed to curl. The example below will continue using the Picasa Web Albums data API and is used to retrieve a list of albums owned by the authenticated user. Note that we have shortened the auth token to ABCDEFG in this example, but the full token (e.g. EUBBIacA...32JKOuGh from above) should be used in its place.

curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/default"

This will return an unformatted blob of XML:

<?xml version='1.0' encoding='UTF-8'?><feed xmlns='http://www.w3.org/2005/Atom' xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/' xmlns:geo='http://www.w3.org/2003/01/geo/wgs84_pos#' xmlns:gml='http://www.opengis.net/gml' xmlns:georss='http://www.georss.org/georss' xmlns:photo='http://www.pheed.com/pheed/' xmlns:media='http://search.yahoo.com/mrss/' xmlns:batch='http://schemas.google.com/gdata/batch' xmlns:gphoto='http://schemas.google.com/photos/2007'><id>http://picasaweb.google.com/data/feed/base/user/brad.gushue</id><updated>2007-09-13T21:30:21.454Z</updated>...</entry></feed>

There are some decent tools for formatting this output to make it more human-readable, including tidy. The easiest way to use tidy is to pipe the output from the curl command to tidy like the following:

curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/default" | tidy -xml -indent -quiet

This results in a much more readable feed, like the following:

<?xml version='1.0' encoding='utf-8'?>
<feed xmlns='http://www.w3.org/2005/Atom' 
xmlns:openSearch='http://a9.com/-/spec/opensearchrss/1.0/' 
xmlns:geo='http://www.w3.org/2003/01/geo/wgs84_pos#' 
xmlns:gml='http://www.opengis.net/gml' 
xmlns:georss='http://www.georss.org/georss' 
xmlns:photo='http://www.pheed.com/pheed/' 
xmlns:media='http://search.yahoo.com/mrss/' 
xmlns:batch='http://schemas.google.com/gdata/batch' 
xmlns:gphoto='http://schemas.google.com/photos/2007'>
  <id>http://picasaweb.google.com/data/feed/api/user/brad.gushue</id>
  <updated>2007-09-13T21:47:07.337Z</updated>
  <category scheme='http://schemas.google.com/g/2005#kind'
  term='http://schemas.google.com/photos/2007#user' />
  <title type='text'>brad.gushue</title>
  <subtitle type='text'></subtitle>
  <icon>
  http://lh6.google.com/brad.gushue/AAAAj9zigp4/AAAAAAAAAAA/RiMAlXV4MFI/s64-c/brad.gushue</icon>
  <link rel='http://schemas.google.com/g/2005#feed'
  type='application/atom+xml'
  href='http://picasaweb.google.com/data/feed/api/user/brad.gushue' />
  <link rel='alternate' type='text/html'
  href='http://picasaweb.google.com/brad.gushue' />
  <link rel='self' type='application/atom+xml'
  href='http://picasaweb.google.com/data/feed/api/user/brad.gushue?start-index=1&max-results=1000' />
  <author>
    <name>Brad</name>
    <uri>http://picasaweb.google.com/brad.gushue</uri>
  </author>
  <generator version='1.00' uri='http://picasaweb.google.com/'>
  Picasaweb</generator>
  <openSearch:totalResults>8</openSearch:totalResults>
  <openSearch:startIndex>1</openSearch:startIndex>
  <openSearch:itemsPerPage>1000</openSearch:itemsPerPage>
  <gphoto:user>brad.gushue</gphoto:user>
  <gphoto:nickname>Brad</gphoto:nickname>
  <gphoto:thumbnail>
  http://lh6.google.com/brad.gushue/AAAAj9zigp4/AAAAAAAAAAA/RiMAlXV4MFI/s64-c/brad.gushue</gphoto:thumbnail>
  <entry>
    <id>
    http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593</id>
    <published>2007-05-23T04:55:52.000Z</published>
    <updated>2007-05-23T04:55:52.000Z</updated>
    <category scheme='http://schemas.google.com/g/2005#kind'
    term='http://schemas.google.com/photos/2007#album' />
    <title type='text'>Trip To Italy</title>
    <summary type='text'>This was the recent trip I took to
    Italy.</summary>
    <rights type='text'>public</rights>
    <link rel='http://schemas.google.com/g/2005#feed'
    type='application/atom+xml'
    href='http://picasaweb.google.com/data/feed/api/user/brad.gushue/albumid/9810315389720904593' />
    <link rel='alternate' type='text/html'
    href='http://picasaweb.google.com/brad.gushue/TripToItalyV2' />
    <link rel='self' type='application/atom+xml'
    href='http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593' />
    <link rel='edit' type='application/atom+xml'
    href='http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593/123456' />
    <author>
      <name>Brad</name>
      <uri>http://picasaweb.google.com/brad.gushue</uri>
    </author>
    <gphoto:id>9810315389720904593</gphoto:id>
    <media:group>
    ...
    </media:group>
  </entry>
  <entry>
  ...
  </entry>

</feed>

Individual entries can be retrieved in the same way by providing the URL for the entry, as opposed to a feed URL.

Updating entries

Entries in Google Data APIs are updated by doing an HTTP PUT to the edit URL with a new copy of the entry's XML in the body of the request.

  1. Retrieve the entry using the atom:link/@rel='self' URL value
  2. Update the entry locally to make the needed changes
  3. PUT the entry back to the server, using the atom:link/@rel='edit' URL value

1. Retrieving the entry

The entry can be retrieved using one of the two URLs bolded in the feed block above. The URL needed is the href value for the link element with a rel='self'. 

curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593"

2. Updating the entry locally

After you retrieve the entry, the entry needs to be updated using a local text editor or application to make any needed changes to the entry. In the command above to retrieve an entry, we did not pipe the results to tidy as we have done in the previous examples. The result is XML that represents the same data, but has different formatting than the version piped to tidy. For the purposes of hand editing an entry, using tidy can often make working with the XML easier.

Note: Please remember to include all XML namespace definitions which are used as attributes to the atom:entry when you post your new entry. Omitting these will cause parsing exceptions. Also, tidy will replace the spaces between the namespace definitions with newline characters. While this is valid XML, Google Data services do not accept it at this time. If you are using tidy, please be sure to add additional spaces between these attributes on the entry element.

3. Updating the entry on the server

Using the edit URL, you need to PUT a copy of the entry to the service using cURL. A header to indicate the type of content being sent to the server needs to be added. The following snippet assumes that the file with the updated entry is saved in updated_entry.xml.

curl --silent --data-binary "@updated_entry.xml" --request PUT --header "Content-Type: application/atom+xml" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593/123456"

Creating entries

Entries in Google Data APIs are created by doing an HTTP POST to the post URL with a new entry. The atom:id is assigned by the server, and thus not necessary to include in new entries. The easiest way to create a new entry is to take an old entry and modify it. The following example will do just that.

  1. Retrieve a template entry using the atom:link/@rel='self'
  2. Modify the template entry locally to remove unnecessary information and make the needed changes
  3. POST the entry back to the server, using the post URL for the feed. This is either found in the retrieved feed as the href value for the link element with a rel='http://schemas.google.com/g/2005#post', or in the documentation for the service on http://code.google.com.

1. Retrieve a template entry

A single entry can be retrieved using the href value of a link element with a rel='self' in the same way as an entry was retrieved before updating it in the above example.

curl --silent --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593" | tidy -xml -indent -quiet > template_entry.xml

The response, after using tidy will look something like:

<entry xmlns='http://www.w3.org/2005/Atom' 
xmlns:exif='http://schemas.google.com/photos/exif/2007'
xmlns:geo='http://www.w3.org/2003/01/geo/wgs84_pos#'
xmlns:gml='http://www.opengis.net/gml'
xmlns:georss='http://www.georss.org/georss'
xmlns:photo='http://www.pheed.com/pheed/'
xmlns:media='http://search.yahoo.com/mrss/'
xmlns:batch='http://schemas.google.com/gdata/batch'
xmlns:gphoto='http://schemas.google.com/photos/2007'>
  <id>
  http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389741123981</id>
  <published>2007-05-23T04:55:52.000Z</published>
  <updated>2007-05-23T04:55:52.000Z</updated>
  <category scheme='http://schemas.google.com/g/2005#kind'
  term='http://schemas.google.com/photos/2007#album' />
  <title type='text'>Trip To Italy</title>
  <summary type='text'>This was the recent trip I took to
  Italy.</summary>
  <rights type='text'>public</rights>
  <link rel='http://schemas.google.com/g/2005#feed'
  type='application/atom+xml'
  href='http://picasaweb.google.com/data/feed/api/user/brad.gushue/albumid/9810315389741123981' />
  <link rel='alternate' type='text/html'
  href='http://picasaweb.google.com/brad.gushue/TripToItaly' />
  <link rel='self' type='application/atom+xml'
  href='http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389741123981' />
  <link rel='edit' type='application/atom+xml'
  href='http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389741123981/1179896152905401' />
  <author>
    <name>Brad</name>
    <uri>http://picasaweb.google.com/brad.gushue</uri>
  </author>
  <gphoto:id>9810315389741123981</gphoto:id>
  <gphoto:name>TripToItaly</gphoto:name>
  <gphoto:location></gphoto:location>
  <gphoto:access>public</gphoto:access>
  <gphoto:timestamp>1179896152000</gphoto:timestamp>
  <gphoto:numphotos>0</gphoto:numphotos>
  <gphoto:numphotosremaining>500</gphoto:numphotosremaining>
  <gphoto:bytesUsed>0</gphoto:bytesUsed>
  <gphoto:user>brad.gushue</gphoto:user>
  <gphoto:nickname>Brad</gphoto:nickname>
  <gphoto:commentingEnabled>true</gphoto:commentingEnabled>
  <gphoto:commentCount>0</gphoto:commentCount>
  <media:group>
    <media:title type='plain'>Trip To Italy</media:title>
    <media:description type='plain'>This was the recent trip I took
    to Italy.</media:description>
    <media:keywords></media:keywords>
    <media:content url='http://lh5.google.com/brad.gushue/ADFUFKqeA5E/AAAAAAAAABc/V6Sga_Z03WU/TripToItaly.jpg'
    type='image/jpeg' medium='image' />
    <media:thumbnail url='http://lh5.google.com/brad.gushue/ADFUFKqeA5E/AAAAAAAAABc/V6Sga_Z03WU/s160-c/TripToItaly.jpg'
    height='160' width='160' />
    <media:credit>Brad</media:credit>
  </media:group>

</entry>

2. Modify the template entry

We want to create an album called "Curling in Canada" with pictures from our recent curling match. Google Data allows you to drop Atom elements that the server supplies values for, so to create this simple template entry, we'll remove the atom:id, atom:published, atom:updated, atom:author, and the various atom:link elements in the feed. This will give us a stripped-down template entry. The entry then needs to be modified to represent the new album we are creating:

<entry xmlns='http://www.w3.org/2005/Atom' 
xmlns:gphoto='http://schemas.google.com/photos/2007'>
  <category scheme='http://schemas.google.com/g/2005#kind'
  term='http://schemas.google.com/photos/2007#album' />
  <title type='text'>Curling in Canada</title>
  <summary type='text'>Pictures of all my curling matches in Canada.</summary>
  <gphoto:location>Canada</gphoto:location>
  <gphoto:access>public</gphoto:access>
  <gphoto:commentingEnabled>true</gphoto:commentingEnabled>

</entry>

Note: Please remember to include all XML namespace definitions which are used as attributes to the atom:entry when you post your new entry. Omitting these will cause parsing exceptions. Also, tidy will replace the spaces between the namespace definitions and replace them with newline characters. While this is valid XML, Google Data services do not accept it at this time. If you are using tidy, please be sure to add additional spaces between these attributes on the entry element.

3. Posting the new entry to the server

The curl command for posting a new entry to the server is very similar to updating an existing entry except that the URL is different:

curl --silent --request POST --data-binary "@template_entry.xml" --header "Content-Type: application/atom+xml" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/brad.gushue" | tidy -xml -indent -quiet

If the post was successful, the resulting XML output is a copy of the newly-created entry. This entry will include things which the server generated at the time the entry was created, including the values for the atom:id, atom:published, atom:updated, and atom:link elements. The resulting link values can be used to edit or delete the entry, provided no additional changes are made in the interim.

Deleting entries

Deleting entries is very similar to updating entries, except an HTTP DELETE method is used instead of an HTTP PUT and no data is required to be sent. Also like the update request, the edit URL is used as the target of the HTTP request.

curl --silent --request DELETE --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/entry/api/user/brad.gushue/albumid/9810315389720904593/123456"

Uploading media objects

An important feature of the Picasa Web Albums data API and the Documents List data API is the ability to upload binary objects. cURL can easily accomplish uploading binary data and a slug header. However, the Documents List data API currently requires posting the XML along with the binary data as a MIME multipart message. Forming the multipart message is out of scope for this article.

The example below shows how to upload a picture called sweeping_the_rock.png to a Picasa Web Album with the title "Sweeping the rock":

curl --silent --request POST --data-binary "@sweeping_the_rock.png" --header "Slug: Sweeping the rock" --header "Content-Type: image/png" --header "Authorization: GoogleLogin auth=ABCDEFG" "http://picasaweb.google.com/data/feed/api/user/brad.gushue/albumid/5113621341847124417" | tidy -xml -indent -quiet

Other command-line tools

Some developers prefer learning or debugging using other command-line tools.

Popular tools include:

  • telnet, openssl are used for making raw socket connections (plain text and ssl-based, respectively) to web servers and can be used to interact with Google Data services. Note that not all Google Data services may support SSL. Here's how you open up the connections:
    • telnet picasaweb.google.com 80 (Picasa Web Albums data API)
    • openssl s_client -connect www.google.com:443 (Google Calendar data API and other services on www.google.com)
    After the connection is made, a raw HTTP request must be sent. This request includes the HTTP verb, relative path, version, all headers and the body of the request. Here's an example: 
    POST /data/feed/api/user/brad.gushue HTTP/1.1
Host: picasaweb.google.com
Authorization: GoogleLogin auth=ABCDEFG
Content-Length: 493

<entry xmlns='http://www.w3.org/2005/Atom' 
xmlns:gphoto='http://schemas.google.com/photos/2007'>
  <category scheme='http://schemas.google.com/g/2005#kind'
  term='http://schemas.google.com/photos/2007#album' />
  <title type='text'>Curling in Canada</title>
  <summary type='text'>Pictures of all my curling matches in Canada.</summary>
  <gphoto:location>Canada</gphoto:location>
  <gphoto:access>public</gphoto:access>
  <gphoto:commentingEnabled>true</gphoto:commentingEnabled>

</entry>
    When sending raw HTTP data, be aware that the POST and PUT operations will require computing the value for a Content-Length header. You can use the UNIX tool wc to compute this value. Place all the content of the HTTP body into a text file such as template_entry.xml (example used above) and run wc -c template_entry.xml. It is often difficult to debug if you accidentally use an incorrect value for the Content-Length header.
  • wget is typically used to download data from a web server to a local file. However, wget has lots of options which make it capable of performing all the different types of requests needed to interact with Google Data services.. Here's an example of how to use wget to POST a new album entry to Picasa Web Albums: 
    wget --post-file template_entry.xml --header "Authorization: GoogleLogin auth=ABCDEFG" --header "Content-Type: application/atom+xml" "http://picasaweb.google.com/data/feed/api/user/brad.gushue"
  • xsltproc is a tool to apply XSL transformations (XSLT) to XML documents. It can be used to easily extract desired bits of data from an XML entry or feed returned by Google Data APIs, or to generate new or updated entries.

Conclusion

As you've seen, cURL and several other command-line tools can be used to easily interact with Google Data services using raw XML and HTTP. Please join us in the API specific forums if you have any questions about using these tools with your favorite Google Data API.