Structured Data Files (SDFs) are specially-formatted comma-separated value (CSV) files used to retrieve and update data about Display & Video 360 resources in bulk. Through the Display & Video 360 API, you can generate and download customized SDFs, allowing you to retrieve organized, filtered data on your Display & Video 360 resources.
This guide describes how to create a SDF Download operation, track that operation, and download the resulting SDFs.
Information regarding SDF format and versioning can be found in the SDF reference documentation.
Create a task
SDFs are generated by an asynchronous operation, called an sdfdownloadtask
.
When creating this task, you define the parameters regarding your desired SDFs.
This is done via the sdfdownloadtasks.create
method. The
following subsections describe the parameters you can set.
Specify a version
The Structured Data File format is regularly updated independently of the Display & Video 360 API, with new versions released and old versions deprecated regularly. For this reason, it is always recommended that users use the most recent version of SDF.
You set the SDF version of your desired SDF using the
version
field in the request body. If not set or set
to SDF_VERSION_UNSPECIFIED
, the task will use the default SDF version of the
advertiser or partner resource used as the context of the SDF content.
Set the context
You can generate an SDF containing data on any resources available to you, but
any individual SDF can only return content within the context of a single
partner or advertiser. This context is defined in the request body by either
the partnerId
or
advertiserId
field. Exactly one of these two
fields must be set.
Only resources within the given context will be included in the resulting SDF. If you attempt to filter by a resource that isn't owned by the specified partner or advertiser, neither it nor the content under it will be included in the results. If only filtering by these unincluded resources, the resulting files will be empty. Attempting to filter by resources outside the given context will not return an error, so make sure to check that your context is correct.
Choose the right filter
In addition to the context set above, you can further filter the scope of your generated Structured Data Files by specifying the file types you want to generate and the specific resources or family of resources you want to include.
There are three available filters for a sdfdownloadtask
, each one catering to
a particular specification type. You can only assign one for a single
sdfdownloadtask
.
ParentEntityFilter
ParentEntityFilter
is the most broad of the available
filters.
Using the fileType
field, you can list all the desired
file types that you want to generate with your task. This is
required, and if left empty or set to FILE_TYPE_UNSPECIFIED
, your
sdfdownloadtask
will complete in error.
Using the filterType
and
filterIds
fields, you can further refine your results.
filterType
specifies the type of
resources to filter by and filterIds
identifies those resources by their unique id. The resulting SDFs will include
the resources identified by fileType
that are either
the resources or children of the resources identified by
filterType
and filterIds
.
IdFilter
IdFilter
filters your request to only include the resources
identified.
IdFilter
has a field for every SDF type, excluding Inventory
Source. Each one of these fields is a list of unique IDs identifying the
specific resources you wish to include in your generated SDF. The IDs provided
must be within the context set, but they don’t need to be directly related. You
don’t need to request a particular campaign in order to request a line item it
contains, and vice-versa. The only file types generated will be of those
corresponding to the resources identified in the IdFilter
.
InventorySourceFilter
InventorySourceFilter
only allows for the filtering
and downloading of SDFs containing Inventory Source resources. It is the only
filter you can use to get information on your Inventory Source resources.
InventorySourceFilter
has a singular
inventorySourceIds
field where you identify the unique
IDs of the inventory source resources you want to include in your SDF. If the
list provided to inventorySourceIds
is empty, all
inventory sources under the set context will be included in the generated SDF.
Make a request
Once you know the parameters of your desired SDF, you can construct the request
and create the sdfdownloadtask
.
Here's an example of how to create an sdfdownloadtask
using a
ParentEntityFilter
:
Java
// Create the filter structure ParentEntityFilter parentEntityFilter = new ParentEntityFilter(); parentEntityFilter.setFileType(sdf-file-type-list); parentEntityFilter.setFilterType(sdfFilterType); parentEntityFilter.setFilterIds(filter-id-list); // Configure the sdfdownloadtasks.create request Sdfdownloadtasks.Create request = service .sdfdownloadtasks() .create( new CreateSdfDownloadTaskRequest() .setVersion(sdfVersion) .setAdvertiserId(advertiserId) .setParentEntityFilter(parentEntityFilter) ); // Create the sdfdownloadtask Operation operationResponse = request.execute(); System.out.printf("Operation %s was created.\n", operationResponse.getName());
Python
# Configure the sdfdownloadtasks.create request createSdfDownloadTaskRequest = { 'version': sdf-version, 'advertiserId': advertiser-id, 'parentEntityFilter': { 'fileType': sdf-file-type-list, 'filterType': sdf-filter-type, 'filterIds': filter-id-list } } # Create the sdfdownloadtask operation = service.sdfdownloadtasks().create( body=createSdfDownloadTaskRequest).execute(); print("Operation %s was created." % operation["name"])
PHP
// Create the sdfdownloadtasks.create request structure $createSdfDownloadTaskRequest = new Google_Service_DisplayVideo_CreateSdfDownloadTaskRequest(); $createSdfDownloadTaskRequest->setAdvertiserId(advertiser-id); $createSdfDownloadTaskRequest->setVersion(sdf-version); // Create and set the parent entity filter $parentEntityFilter = new Google_Service_DisplayVideo_ParentEntityFilter(); $parentEntityFilter->setFileType(sdf-file-type-list); $parentEntityFilter->setFilterType(sdf-filter-type); if (!empty(filter-id-list)) { $parentEntityFilter->setFilterIds(filter-id-list); } $createSdfDownloadTaskRequest->setParentEntityFilter($parentEntityFilter); // Call the API, creating the SDF Download Task. $operation = $this->service->sdfdownloadtasks->create( $createSdfDownloadTaskRequest ); printf('Operation %s was created.\n', $operation->getName());
Check your request and get download path
When you create an sdfdownloadtask
, an operation object is
returned. This operation represents the status of your asynchronous SDF
generation operation at the time of creation. You can check your operation to
see if it has completed and is ready to download, or has thrown an error, using
the sdfdownloadtasks.operations.get
method.
Upon finishing, the returned operation will have a non-null
done
field. The finished operation will include either
a response
or error
field. If present, the error
field will have a
Status
object containing an error code and
message, which provides details of the error that
occurred. If the response
field is present, it
will have an object with a resourceName
value that identifies the generated
file for download.
Here's an example of how to check your request using exponential backoff:
Java
String operationName = operationResponse.getName(); // Configure the Operations.get request Sdfdownloadtasks.Operations.Get operationRequest = service .sdfdownloadtasks() .operations() .get(operationName); // Configure exponential backoff for checking the status of our operation ExponentialBackOff backOff = new ExponentialBackOff.Builder() .setInitialIntervalMillis(5000) // setting initial interval to five seconds .setMaxIntervalMillis(300000) // setting max interval to five minutes .setMaxElapsedTimeMillis(18000000) // setting max elapsed time to five hours .build(); while (operationResponse.getDone() == null) { long backoffMillis = backOff.nextBackOffMillis(); if (backoffMillis == ExponentialBackOff.STOP) { System.out.printf("The operation has taken more than five hours to complete.\n"); return; } Thread.sleep(backoffMillis); // Get current status of operation operationResponse = operationRequest.execute(); } // Check if the operation finished with an error and return if (operationResponse.getError() != null) { System.out.printf("The operation finished in error with code %s: %s\n", operationResponse.getError().getCode(), operationResponse.getError() .getMessage()); return; } System.out.printf( "The operation completed successfully. Resource %s was created.\n", operationResponse.getResponse().get("resourceName").toString());
Python
# The following values control retry behavior while # the report is processing. # Minimum amount of time between polling requests. Defaults to 5 seconds. min_retry_interval = 5 # Maximum amount of time between polling requests. Defaults to 5 minutes. max_retry_interval = 5 * 60 # Maximum amount of time to spend polling. Defaults to 5 hours. max_retry_elapsed_time = 5 * 60 * 60 # Configure the Operations.get request get_request = service.sdfdownloadtasks().operations().get( name=operation["name"] ) sleep = 0 start_time = time.time() while True: # Get current status of operation operation = get_request.execute() if "done" in operation: if "error" in operation: print("The operation finished in error with code %s: %s" % ( operation["error"]["code"], operation["error"]["message"])) else: print("The operation completed successfully. Resource %s was created." % operation["response"]["resourceName"]) break elif time.time() - start_time > max_retry_elapsed_time: print("Generation deadline exceeded.") sleep = next_sleep_interval(sleep) print("Operation still running, sleeping for %d seconds." % sleep) time.sleep(sleep) def next_sleep_interval(previous_sleep_interval): """Calculates the next sleep interval based on the previous.""" min_interval = previous_sleep_interval or min_retry_interval max_interval = previous_sleep_interval * 3 or min_retry_interval return min(max_retry_interval, random.randint(min_interval, max_interval))
PHP
// The following values control retry behavior // while the task is processing. // Minimum amount of time between polling requests. Defaults to 5 seconds. $minRetryInterval = 5; // Maximum amount of time between polling requests. Defaults to 5 minutes. $maxRetryInterval = 300; // Maximum amount of time to spend polling. Defaults to 5 hours. $maxRetryElapsedTime = 18000; $operationName = $operation->getName(); $sleepInterval = 0; $startTime = time(); while (!$operation->getDone()) { if ($sleepInterval != 0) { printf( 'The operation is still running, sleeping for %d seconds\n', $sleepInterval ); } // Sleep before retrieving the SDF Download Task again. sleep($sleepInterval); // Call the API, retrieving the SDF Download Task. $operation = $this->service->sdfdownloadtasks_operations->get( $operation->getName() ); // If the operation has exceeded the set deadline, throw an exception. if (time() - $startTime > $maxRetryElapsedTime) { printf('SDF download task processing deadline exceeded\n'); throw new Exception( 'Long-running operation processing deadline exceeded' ); } // Generate the next sleep interval using exponential backoff logic. $sleepInterval = min( $maxRetryInterval, rand( max($minRetryInterval, $previousSleepInterval), max($minRetryInterval, $previousSleepInterval * 3) ) ); } // If the operation finished with an error, throw an exception. if($operation->getError() !== null) { $error = $operation->getError(); printf( 'The operation finished in error with code %s: %s\n', $error->getCode(), $error->getMessage() ); throw new Exception($error->getMessage()); } // Print successfully generated resource. $response = $operation->getResponse(); printf( 'The operation completed successfully. Resource %s was ' . 'created. Ready to download.\n', $response['resourceName'] );