Overview

Structured Data Files (SDFs) are specially-formatted comma-separated value (CSV) files used to retrieve and update data about Display & Video 360 resources in bulk. Through the Display & Video 360 API, you can generate and download customized SDFs, allowing you to retrieve organized, filtered data on your Display & Video 360 resources.

This guide describes how to create a SDF Download operation, track that operation, and download the resulting SDFs.

Information regarding SDF format and versioning can be found in the SDF reference documentation.

Create a task

SDFs are generated by an asynchronous operation, called an sdfdownloadtask. When creating this task, you define the parameters regarding your desired SDFs. This is done via the sdfdownloadtasks.create method. The following subsections describe the parameters you can set.

Specify a version

The Structured Data File format is regularly updated independently of the Display & Video 360 API, with new versions released and old versions deprecated regularly. For this reason, it is always recommended that users use the most recent version of SDF.

You set the SDF version of your desired SDF using the version field in the request body. If not set or set to SDF_VERSION_UNSPECIFIED, the task will use the default SDF version of the advertiser or partner resource used as the context of the SDF content.

Set the context

You can generate an SDF containing data on any resources available to you, but any individual SDF can only return content within the context of a single partner or advertiser. This context is defined in the request body by either the partnerId or advertiserId field. Exactly one of these two fields must be set.

Only resources within the given context will be included in the resulting SDF. If you attempt to filter by a resource that isn't owned by the specified partner or advertiser, neither it nor the content under it will be included in the results. If only filtering by these unincluded resources, the resulting files will be empty. Attempting to filter by resources outside the given context will not return an error, so make sure to check that your context is correct.

Choose the right filter

In addition to the context set above, you can further filter the scope of your generated Structured Data Files by specifying the file types you want to generate and the specific resources or family of resources you want to include.

There are three available filters for a sdfdownloadtask, each one catering to a particular specification type. You can only assign one for a single sdfdownloadtask.

ParentEntityFilter

ParentEntityFilter is the most broad of the available filters.

Using the fileType field, you can list all the desired file types that you want to generate with your task. This is required, and if left empty or set to FILE_TYPE_UNSPECIFIED, your sdfdownloadtask will complete in error.

Using the filterType and filterIds fields, you can further refine your results. filterType specifies the type of resources to filter by and filterIds identifies those resources by their unique id. The resulting SDFs will include the resources identified by fileType that are either the resources or children of the resources identified by filterType and filterIds.

IdFilter

IdFilter filters your request to only include the resources identified.

IdFilter has a field for every SDF type, excluding Inventory Source. Each one of these fields is a list of unique IDs identifying the specific resources you wish to include in your generated SDF. The IDs provided must be within the context set, but they don’t need to be directly related. You don’t need to request a particular campaign in order to request a line item it contains, and vice-versa. The only file types generated will be of those corresponding to the resources identified in the IdFilter.

InventorySourceFilter

InventorySourceFilter only allows for the filtering and downloading of SDFs containing Inventory Source resources. It is the only filter you can use to get information on your Inventory Source resources.

InventorySourceFilter has a singular inventorySourceIds field where you identify the unique IDs of the inventory source resources you want to include in your SDF. If the list provided to inventorySourceIds is empty, all inventory sources under the set context will be included in the generated SDF.

Make a request

Once you know the parameters of your desired SDF, you can construct the request and create the sdfdownloadtask.

Here's an example of how to create an sdfdownloadtask using a ParentEntityFilter:

// Create the filter structure
ParentEntityFilter parentEntityFilter = new ParentEntityFilter();
parentEntityFilter.setFileType(sdf-file-type-list);
parentEntityFilter.setFilterType(sdfFilterType);
parentEntityFilter.setFilterIds(filter-id-list);

// Configure the sdfdownloadtasks.create request
Sdfdownloadtasks.Create request =
   service
       .sdfdownloadtasks()
       .create(
           new CreateSdfDownloadTaskRequest()
               .setVersion(sdfVersion)
               .setAdvertiserId(advertiserId)
               .setParentEntityFilter(parentEntityFilter)
       );

// Create the sdfdownloadtask
Operation operationResponse = request.execute();

System.out.printf("Operation %s was created.\n",
   operationResponse.getName());

# Configure the sdfdownloadtasks.create request
createSdfDownloadTaskRequest = {
    'version': sdf-version,
    'advertiserId': advertiser-id,
    'parentEntityFilter': {
        'fileType': sdf-file-type-list,
        'filterType': sdf-filter-type,
        'filterIds': filter-id-list
    }
}

# Create the sdfdownloadtask
operation = service.sdfdownloadtasks().create(
    body=createSdfDownloadTaskRequest).execute();

print("Operation %s was created." % operation["name"])

// Create the sdfdownloadtasks.create request structure
$createSdfDownloadTaskRequest =
    new Google_Service_DisplayVideo_CreateSdfDownloadTaskRequest();
$createSdfDownloadTaskRequest->setAdvertiserId(advertiser-id);
$createSdfDownloadTaskRequest->setVersion(sdf-version);

// Create and set the parent entity filter
$parentEntityFilter = new Google_Service_DisplayVideo_ParentEntityFilter();
$parentEntityFilter->setFileType(sdf-file-type-list);
$parentEntityFilter->setFilterType(sdf-filter-type);
if (!empty(filter-id-list)) {
    $parentEntityFilter->setFilterIds(filter-id-list);
}
$createSdfDownloadTaskRequest->setParentEntityFilter($parentEntityFilter);

// Call the API, creating the SDF Download Task.
$operation = $this->service->sdfdownloadtasks->create(
    $createSdfDownloadTaskRequest
);

printf('Operation %s was created.\n', $operation->getName());

Check your request and get download path

When you create an sdfdownloadtask, an operation object is returned. This operation represents the status of your asynchronous SDF generation operation at the time of creation. You can check your operation to see if it has completed and is ready to download, or has thrown an error, using the sdfdownloadtasks.operations.get method.

Upon finishing, the returned operation will have a non-null done field. The finished operation will include either a response or error field. If present, the error field will have a Status object containing an error code and message, which provides details of the error that occurred. If the response field is present, it will have an object with a resourceName value that identifies the generated file for download.

Here's an example of how to check your request using exponential backoff:

String operationName = operationResponse.getName();

// Configure the Operations.get request
Sdfdownloadtasks.Operations.Get operationRequest =
   service
       .sdfdownloadtasks()
       .operations()
       .get(operationName);

// Configure exponential backoff for checking the status of our operation
ExponentialBackOff backOff = new ExponentialBackOff.Builder()
   .setInitialIntervalMillis(5000) // setting initial interval to five seconds
   .setMaxIntervalMillis(300000)  // setting max interval to five minutes
   .setMaxElapsedTimeMillis(18000000) // setting max elapsed time to five hours
   .build();

while (operationResponse.getDone() == null) {
 long backoffMillis = backOff.nextBackOffMillis();
 if (backoffMillis == ExponentialBackOff.STOP) {
   System.out.printf("The operation has taken more than five hours to
       complete.\n");
   return;
 }
 Thread.sleep(backoffMillis);

 // Get current status of operation
 operationResponse = operationRequest.execute();
}

// Check if the operation finished with an error and return
if (operationResponse.getError() != null) {
 System.out.printf("The operation finished in error with code %s: %s\n",
     operationResponse.getError().getCode(), operationResponse.getError()
         .getMessage());
 return;
}

System.out.printf(
    "The operation completed successfully. Resource %s was created.\n",
    operationResponse.getResponse().get("resourceName").toString());

# The following values control retry behavior while
# the report is processing.
# Minimum amount of time between polling requests. Defaults to 5 seconds.
min_retry_interval = 5
# Maximum amount of time between polling requests. Defaults to 5 minutes.
max_retry_interval = 5 * 60
# Maximum amount of time to spend polling. Defaults to 5 hours.
max_retry_elapsed_time = 5 * 60 * 60

# Configure the Operations.get request
get_request = service.sdfdownloadtasks().operations().get(
  name=operation["name"]
)

sleep = 0
start_time = time.time()
while True:
  # Get current status of operation
  operation = get_request.execute()

  if "done" in operation:
    if "error" in operation:
      print("The operation finished in error with code %s: %s" % (
            operation["error"]["code"],
            operation["error"]["message"]))
    else:
      print("The operation completed successfully. Resource %s was created."
            % operation["response"]["resourceName"])
    break
  elif time.time() - start_time > max_retry_elapsed_time:
    print("Generation deadline exceeded.")

  sleep = next_sleep_interval(sleep)
  print("Operation still running, sleeping for %d seconds." % sleep)
  time.sleep(sleep)

def next_sleep_interval(previous_sleep_interval):
  """Calculates the next sleep interval based on the previous."""
  min_interval = previous_sleep_interval or min_retry_interval
  max_interval = previous_sleep_interval * 3 or min_retry_interval
  return min(max_retry_interval, random.randint(min_interval, max_interval))

// The following values control retry behavior
// while the task is processing.
// Minimum amount of time between polling requests. Defaults to 5 seconds.
$minRetryInterval = 5;
// Maximum amount of time between polling requests. Defaults to 5 minutes.
$maxRetryInterval = 300;
// Maximum amount of time to spend polling. Defaults to 5 hours.
$maxRetryElapsedTime = 18000;

$operationName = $operation->getName();

$sleepInterval = 0;
$startTime = time();

while (!$operation->getDone()) {
    if ($sleepInterval != 0) {
        printf(
            'The operation is still running, sleeping for %d seconds\n',
            $sleepInterval
        );
    }

    // Sleep before retrieving the SDF Download Task again.
    sleep($sleepInterval);

    // Call the API, retrieving the SDF Download Task.
    $operation = $this->service->sdfdownloadtasks_operations->get(
        $operation->getName()
    );

    // If the operation has exceeded the set deadline, throw an exception.
    if (time() - $startTime > $maxRetryElapsedTime) {
        printf('SDF download task processing deadline exceeded\n');
        throw new Exception(
            'Long-running operation processing deadline exceeded'
        );
    }

    // Generate the next sleep interval using exponential backoff logic.
    $sleepInterval = min(
        $maxRetryInterval,
        rand(
            max($minRetryInterval, $previousSleepInterval),
            max($minRetryInterval, $previousSleepInterval * 3)
        )
    );
}

// If the operation finished with an error, throw an exception.
if($operation->getError() !== null) {
    $error = $operation->getError();
    printf(
        'The operation finished in error with code %s: %s\n',
        $error->getCode(),
        $error->getMessage()
    );
    throw new Exception($error->getMessage());
}

// Print successfully generated resource.
$response = $operation->getResponse();
printf(
    'The operation completed successfully. Resource %s was '
        . 'created. Ready to download.\n',
    $response['resourceName']
);