PageSpeed Insights: Mobile Analysis - Manager Account

Ensuring a positive experience for mobile users when they visit landing pages requires that sites are built with mobile in mind. A site well-optimized for mobile will ensure that users remain engaged with your site, making your ads more effective.

Mobile Analysis using PageSpeed Insights provides a report suggesting ways to improve the landing page experience on mobile devices. This solution builds on the single-account version, allowing reports to be produced covering an entire Manager Account structure. The solution also provides the ability to perform desktop analysis, instead of mobile, using a simple settings change.

How it works

When run on a Manager Account, you can schedule this script on an hourly basis: The script will run on a maximum of 50 accounts per run in order to avoid execution limits. Each account will be marked as completed, and once all are completed, a report will be produced. Therefore, if your Manager Account has more than 50 accounts, set this script to run repeatedly on an hourly schedule to ensure that a report is produced.

The script examines URLs from the account - from Ads, Keywords and Sitelinks - using mobile final URLs if available, but defaulting to standard URLs otherwise.

The first stage of the process is to build up a dictionary of as many URLs as time will allow.

Once this has completed, the script looks to identify URLs that are as different as possible - it is possible to imagine the scenario where a large number of URLs in an account return very similar pages. For example:

http://www.example.com/product?id=123 and http://www.example.com/product?id=456

very probably return pages created from the same template. To this end, the difference is classified as follows:

  • Most different - hosts are different - e.g. http://www.example.com/path and http://www.test.com/path.
  • More different - hosts are same, but paths differ - e.g. http://www.example.com/shop and http://www.example.com/blog.
  • Least different - hosts and paths the same, parameters differ - e.g. http://www.example.com/shop?product=1 and http://www.example.com/shop?product=2.

A smaller subset of URLs - with the aim of being as different as possible by the above classification - are then selected and retrieved via the PageSpeed Insights API.

The results are presented in a Google Spreadsheet, which is sent by email.

Setup

Source code

// Copyright 2015, Google Inc. All Rights Reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

/**
 * @fileoverview Mobile Performance from PageSpeed Insights - Manager Accounts
 *
 * Produces a report showing how well landing pages are set up for mobile
 * devices and highlights potential areas for improvement. See :
 * https://developers.google.com/google-ads/scripts/docs/solutions/adsmanagerapp-mobile-pagespeed
 * for more details.
 *
 * @author Google Ads Scripts Team [adwords-scripts@googlegroups.com]
 *
 * @version 1.3.3
 *
 * @changelog
 * - version 1.3.3
 *   - Added guidance for desktop analysis.
 * - version 1.3.2
 *   - Bugfix to improve table sorting comparator.
 * - version 1.3.1
 *   - Bugfix for handling the absence of optional values in PageSpeed response.
 * - version 1.3
 *   - Removed the need for the user to take a copy of the spreadsheet.
 *   - Added the ability to customize the Campaign and Ad Group limits.
 * - version 1.2.1
 *   - Improvements to time zone handling.
 * - version 1.2
 *   - Bug fix for handling empty results from URLs.
 *   - Error reporting in spreadsheet for failed URL fetches.
 * - version 1.1
 *   - Updated the comments to be in sync with the guide.
 * - version 1.0
 *   - Released initial version.
 */

// See "Obtaining an API key" at
// https://developers.google.com/google-ads/scripts/docs/solutions/mobile-pagespeed
var API_KEY = 'INSERT_PAGESPEED_API_KEY_HERE';
var EMAIL_RECIPIENTS = ['INSERT_EMAIL_ADDRESS_HERE'];

// If you wish to add extra URLs to checked, list them here as a
// comma-separated list eg. ['http://abc.xyz', 'http://www.google.com']
var EXTRA_URLS_TO_CHECK = [];

// By default, the script returns analysis of how the site performs on mobile.
// Change the following from 'mobile' to 'desktop' to perform desktop analysis.
var PLATFORM_TYPE = 'mobile';

/**
 * The URL of the template spreadsheet for each report created.
 */
var SPREADSHEET_TEMPLATE =
    'https://docs.google.com/spreadsheets/d/1SKLXUiorvgs2VuPKX7NGvcL68pv3xEqD7ZcqsEwla4M/edit';

var PAGESPEED_URL =
    'https://www.googleapis.com/pagespeedonline/v2/runPagespeed?';

/*
 * The maximum number of Campaigns to sample within the account.
 */
var CAMPAIGN_LIMIT = 50000;

/*
 * The maximum number of Ad Groups to sample within the account.
 */
var ADGROUP_LIMIT = 50000;

/**
 * These are the sampling limits for how many of each element will be examined
 * in each AdGroup.
 */
var KEYWORD_LIMIT = 20;
var SITELINK_LIMIT = 20;
var AD_LIMIT = 30;

/**
 * Specifies the amount of time in seconds required at the end of each parallel
 * execution to collect up the results and return them.
 */
var POST_ITERATION_TIME_SECS = 5 * 60;

/**
 * The actual limit on the number of URLs to fetch from a single account — this
 * is only supposed to be a sample of the performance of URLs in this account,
 * not exhaustive.
 */
var URL_LIMIT = 50;

/**
 * Specified the amount of time after the URL fetches required to write to and
 * format the spreadsheet.
 */
var SPREADSHEET_PREP_TIME_SECS = 4 * 60;

/**
 * Represents the number of retries to use with the PageSpeed service.
 */
var MAX_RETRIES = 3;

/**
 * Naming convention for temporary files, used to store intermediate results
 * between executions.
 */
var TEMP_FILE_PREFIX = 'PageSpeed Temporary File: ';

/**
 * The main entry point for execution.
 */
function main() {
  var ids = [];
  if (!defaultsChanged()) {
    Logger.log('Please change the default configuration values and retry');
    return;
  }
  var managerAccountName = AdsApp.currentAccount().getName();
  intermediateResults = getOrCreateIntermediateResults(managerAccountName);
  var accounts = AdsManagerApp.accounts().get();
  while (accounts.hasNext() && ids.length < 50) {
    var account = accounts.next();
    var customerId = account.getCustomerId();
    if (!intermediateResults.processedAccounts[customerId]) {
      ids.push(customerId);
    }
  }
  AdsManagerApp.accounts().withIds(ids).executeInParallel(
    'processIndividualAccountFromManagerAccount',
    'combineResultsFromManagerAccounts');
}

/**
 * Creates a temporary file on Drive which is used to store the results from
 * each managed account. Once all accounts have been processed (potentially
 * across multiple executions), this temporary file is then used as the source
 * for the resulting spreadsheet that is created and emailed to the user.
 *
 * @param {string} managerAccountName Name of the Manager Account.
 * @return {Object} An object used to store a sample of URLs for each individual
 *     account.
 * @throws Will throw an error if more than one candidate temporary file is
 *     found in the root folder, as there should only ever be zero or one.
 */
function getOrCreateIntermediateResults(managerAccountName) {
  var tempFileName = TEMP_FILE_PREFIX + managerAccountName;
  var intermediateResults = {
    processedAccounts: {},
  };
  var temporaryFiles = DriveApp.getRootFolder().getFilesByName(tempFileName);
  if (!temporaryFiles.hasNext()) {
    DriveApp.createFile(tempFileName, JSON.stringify(intermediateResults));
  } else {
    var temporaryFile = temporaryFiles.next();
    if (!temporaryFiles.hasNext()) {
      intermediateResults = JSON.parse(
          temporaryFile.getBlob().getDataAsString());
    } else {
      throw 'Multiple temporary files detected - please ensure there is only ' +
          'one temporary file per account and try again.';
    }
  }
  return intermediateResults;
}

/**
 * Overwrites the temporary file with the contents of the immediateResults
 * object, which is being used to store PageSpeed results for each execution.
 *
 * @param {string} managerAccountName Name of the Manager Account.
 * @param {Object} intermediateResults The object containing all the PageSpeed
 *     results to overwrite the temporary file with.
 * @throws Will throw an error if more than one candidate temporary file is
 *     found in the root folder, as there should only ever be zero or one.
 */
function updateIntermediateResults(managerAccountName, intermediateResults) {
  var tempFileName = TEMP_FILE_PREFIX + managerAccountName;
  var temporaryFiles = DriveApp.getRootFolder().getFilesByName(tempFileName);
  if (!temporaryFiles.hasNext()) {
    DriveApp.createFile(tempFileName,
        JSON.stringify(intermediateResults));
  } else {
    var temporaryFile = temporaryFiles.next();
    if (!temporaryFiles.hasNext()) {
      temporaryFile.setContent(JSON.stringify(intermediateResults));
    } else {
      throw 'Multiple temporary files detected - please ensure there is only ' +
          'one temporary file per account and try again.';
    }
  }
}

/**
 * Removes any temporary files for this Manager Account.
 *
 * @param {string} managerAccountName Name of the Manager Account.
 */
function removeIntermediateResults(managerAccountName) {
  var files = [];
  var tempFileName = TEMP_FILE_PREFIX + managerAccountName;
  var temporaryFiles = DriveApp.getRootFolder().getFilesByName(tempFileName);
  if (temporaryFiles.hasNext()) {
    var temporaryFile = temporaryFiles.next();
    DriveApp.getRootFolder().removeFile(temporaryFile);
  }
}

/**
 * The principal function that gets called in parallel on each Managed Account.
 *
 * Iterates through all the available Campaigns and AdGroups, to a limit of
 * defined in CAMPAIGN_LIMIT and ADGROUP_LIMIT until the time limit is reached
 * allowing enough time for the post-iteration steps, e.g. fetching and
 * analysing URLs and building results.
 *
 * @return {string} PageSpeed results as JSON.
 */
function processIndividualAccountFromManagerAccount() {
  var urlStore = new UrlStore();
  var campaigns = AdsApp.campaigns()
                      .forDateRange('LAST_30_DAYS')
                      .withCondition('Status = "ENABLED"')
                      .orderBy('Clicks DESC')
                      .withLimit(CAMPAIGN_LIMIT)
                      .get();
  while (campaigns.hasNext() && hasRemainingTimeForAccountIteration()) {
    var campaign = campaigns.next();
    var campaignUrls = getUrlsFromCampaign(campaign);
    urlStore.addUrls(campaignUrls);
  }
  var adGroups = AdsApp.adGroups()
                     .forDateRange('LAST_30_DAYS')
                     .withCondition('Status = "ENABLED"')
                     .orderBy('Clicks DESC')
                     .withLimit(ADGROUP_LIMIT)
                     .get();
  while (adGroups.hasNext() && hasRemainingTimeForAccountIteration()) {
    var adGroup = adGroups.next();
    var adGroupUrls = getUrlsFromAdGroup(adGroup);
    urlStore.addUrls(adGroupUrls);
  }
  var urlSubset = [];
  var urlCount = 0;
  var totalUrlCount = 0;
  for (var url in urlStore) {
    if (urlCount++ < URL_LIMIT) {
      urlSubset.push(url);
    }
    totalUrlCount++;
  }
  return JSON.stringify({
    urls: urlSubset,
    total: totalUrlCount
  });
}

/**
 * The principle function that gets called when each account has finished
 * processing.
 *
 * @param {!Array.<!ExecutionResult>} accountResults A list of results objects
 * from parallel execution.
 */
function combineResultsFromManagerAccounts(accountResults) {
  var managerAccountName = AdsApp.currentAccount().getName();
  var intermediateResults = getOrCreateIntermediateResults(managerAccountName);

  for (var i = 0, lenI = accountResults.length; i < lenI; i++) {
    var customerId = accountResults[i].getCustomerId();
    var error = accountResults[i].getError();
    if (error) {
      Logger.log('Processing of individual account ' + customerId +
          ' resulted in an error: ' + error);
    } else {
      var data = JSON.parse(accountResults[i].getReturnValue());
      if (data) {
        intermediateResults.processedAccounts[customerId] = data;
      } else {
        Logger.log('No data processed for account ' + customerId);
      }
    }
  }
  updateIntermediateResults(managerAccountName, intermediateResults);

  if (areManagedAccountsAllProcessed(intermediateResults)) {
    createAndEmailReport(managerAccountName, intermediateResults);
    removeIntermediateResults(managerAccountName);
  }
}

/**
 * Determine whether all managed accounts have been processed and representative
 * URLs obtained from them and stored.
 *
 * @param {Object} intermediateResults The object representing all accounts
 *     inspected so far.
 * @return {boolean} true if all accounts have been inspected, otherwise false.
 */
function areManagedAccountsAllProcessed(intermediateResults) {
  var accounts = AdsManagerApp.accounts().get();
  while (accounts.hasNext()) {
    var account = accounts.next();
    var customerId = account.getCustomerId();
    if (!intermediateResults.processedAccounts[customerId]) {
      // An account has yet to be processed
      return false;
    }
  }
  return true;
}

/**
 * Combines all results from individual accounts, perform PageSpeed analysis
 * and create spreadsheet on the resulting data.
 *
 * @param {string} managerAccountName Name of the Manager Account.
 * @param {Object} intermediateResults Results object with entries for all
 *     accounts.
 */
function createAndEmailReport(managerAccountName, intermediateResults) {
  var urlStore = new UrlStore(EXTRA_URLS_TO_CHECK);
  var totalUrls = EXTRA_URLS_TO_CHECK.length;
  var accountResults = intermediateResults.processedAccounts;
  var accountResultKeys = Object.keys(accountResults);
  for (var i = 0; i < accountResultKeys.length; i++) {
    var accountResult = accountResults[accountResultKeys[i]];
    totalUrls += accountResult.total;
    urlStore.addUrls(accountResult.urls);
  }
  var result = getPageSpeedResultsForUrls(urlStore);
  var spreadsheet = createPageSpeedSpreadsheet(managerAccountName +
      ': PageSpeed Insights - Mobile Analysis', {
        table: result.table,
        errors: result.errors,
        totalUrls: totalUrls
      });
  spreadsheet.addEditors(EMAIL_RECIPIENTS);
  sendEmail(spreadsheet.getUrl());
}

/**
 * Sends an email to the user with the results of the run.
 *
 * @param {string} url URL of the spreadsheet
 */
function sendEmail(url) {
  var footerStyle = 'color: #aaaaaa; font-style: italic;';
  var scriptsLink = 'https://developers.google.com/google-ads/scripts/';
  var subject = 'Google Ads PageSpeed URL-Sampling Script Results - ' +
      getDateStringInTimeZone('dd MMM yyyy');
  var htmlBody = '<html><body>' +
      '<p>Hello,</p>' +
      '<p>A Google Ads Script has run successfully and the output is ' +
      'available here:' +
      '<ul><li><a href="' + url +
      '">Google Ads PageSpeed URL-Sampling Script Results</a></li></ul></p>' +
      '<p>Regards,</p>' +
      '<span style="' + footerStyle + '">This email was automatically ' +
      'generated by <a href="' + scriptsLink + '">Google Ads Scripts</a>.' +
      '</span></body></html>';
  var body = 'Please enable HTML to view this report.';
  var options = {htmlBody: htmlBody};
  MailApp.sendEmail(EMAIL_RECIPIENTS, subject, body, options);
}

/**
 * Checks to see that placeholder defaults have been changed.
 *
 * @return {boolean} true if placeholders have been changed, false otherwise.
 */
function defaultsChanged() {
  if (API_KEY == 'INSERT_PAGESPEED_API_KEY_HERE' ||
      SPREADSHEET_TEMPLATE == 'INSERT_SPREADSHEET_URL_HERE' ||
      JSON.stringify(EMAIL_RECIPIENTS) ==
      JSON.stringify(['INSERT_EMAIL_ADDRESS_HERE'])) {
    return false;
  }
  return true;
}

/**
 * Creates a new PageSpeed spreadsheet and populates it with result data.
 *
 * @param {string} name The name to give to the spreadsheet
 * @param {Object} pageSpeedResult The result from PageSpeed, and the number of
 *     URLs that could have been chosen from.
 * @return {Spreadsheet} The newly-created spreadsheet
 */
function createPageSpeedSpreadsheet(name, pageSpeedResult) {
  var spreadsheet = SpreadsheetApp.openByUrl(SPREADSHEET_TEMPLATE).copy(name);
  spreadsheet.setSpreadsheetTimeZone(AdsApp.currentAccount().getTimeZone());
  var data = pageSpeedResult.table;
  var activeSheet = spreadsheet.getActiveSheet();
  var rowStub = spreadsheet.getRangeByName('ResultRowStub');
  var top = rowStub.getRow();
  var left = rowStub.getColumn();
  var cols = rowStub.getNumColumns();
  if (data.length > 2) { // No need to extend the template if num rows <= 2
    activeSheet.insertRowsAfter(
          spreadsheet.getRangeByName('EmptyUrlRow').getRow(), data.length);
    rowStub.copyTo(activeSheet.getRange(top + 1, left, data.length - 2, cols));
  }
  // Extend the formulas and headings to accommodate the data.
  if (data.length && data[0].length > 4) {
    var metricsRange = activeSheet
        .getRange(top - 6, left + cols, data.length + 5, data[0].length - 4);
    activeSheet.getRange(top - 6, left + cols - 1, data.length + 5)
        .copyTo(metricsRange);
    // Paste in the data values.
    activeSheet.getRange(top - 1, left, data.length, data[0].length)
        .setValues(data);
    // Move the 'Powered by Google Ads Scripts' to right corner of table.
    spreadsheet.getRangeByName('PoweredByText').moveTo(activeSheet.getRange(1,
      data[0].length + 1, 1, 1));
    // Set summary - date and number of URLs chosen from.
    var summaryDate = getDateStringInTimeZone('dd MMM yyyy');
    spreadsheet.getRangeByName('SummaryDate').setValue('Summary as of ' +
        summaryDate + '. Results drawn from ' + pageSpeedResult.totalUrls +
        ' URLs.');
  }
  // Add errors if they exist
  if (pageSpeedResult.errors.length) {
    var nextRow = spreadsheet.getRangeByName('FirstErrorRow').getRow();
    var errorRange = activeSheet.getRange(nextRow, 2,
        pageSpeedResult.errors.length, 2);
    errorRange.setValues(pageSpeedResult.errors);
  }
  return spreadsheet;
}

/**
 * This function takes a collection of URLs as provided by the UrlStore object
 * and gets results from the PageSpeed service. However, two important things :
 *     (1) It only processes a handful, as determined by URL_LIMIT.
 *     (2) The URLs returned from iterating on the UrlStore are in a specific
 *     order, designed to produce as much variety as possible (removing the need
 *     to process all the URLs in an account.
 *
 * @param {UrlStore} urlStore Object containing URLs to process.
 * @return {Object} An object with two properties: 'table' - the 2d table of
 *     results, and 'errors', a table of errors encountered during analysis.
 */
function getPageSpeedResultsForUrls(urlStore) {
  // Associative array for column headings and contextual help URLs.
  var headings = {};
  var errors = {};
  // Record results on a per-URL basis.
  var pageSpeedResults = {};

  for (var url in urlStore) {
    if (hasRemainingTimeForUrlFetches()) {
      var result = getPageSpeedResultForSingleUrl(url);
      if (!result.error) {
        pageSpeedResults[url] = result.pageSpeedInfo;
        var columnsResult = result.columnsInfo;
        // Loop through each heading element; the PageSpeed Insights API
        // doesn't always return URLs for each column heading, so aggregate
        // these across each call to get the most complete list.
        var columnHeadings = Object.keys(columnsResult);
        for (var i = 0, lenI = columnHeadings.length; i < lenI; i++) {
          var columnHeading = columnHeadings[i];
          if (!headings[columnHeading] || (headings[columnHeading] &&
            headings[columnHeading].length <
            columnsResult[columnHeading].length)) {
            headings[columnHeading] = columnsResult[columnHeading];
          }
        }
      } else {
        errors[url] = result.error;
      }
    }
  }

  var tableHeadings = ['URL', 'Speed', 'Usability'];
  var headingKeys = Object.keys(headings);
  for (var y = 0, lenY = headingKeys.length; y < lenY; y++) {
    tableHeadings.push(headingKeys[y]);
  }

  var table = [];
  var pageSpeedResultsUrls = Object.keys(pageSpeedResults);
  for (var r = 0, lenR = pageSpeedResultsUrls.length; r < lenR; r++) {
    var resultUrl = pageSpeedResultsUrls[r];
    var row = [toPageSpeedHyperlinkFormula(resultUrl)];
    var data = pageSpeedResults[resultUrl];
    for (var j = 1, lenJ = tableHeadings.length; j < lenJ; j++) {
      row.push(data[tableHeadings[j]]);
    }
    table.push(row);
  }
  // Present the table back in the order worst-performing-first.
  table.sort(function(first, second) {
    var f1 = isNaN(parseInt(first[1])) ? 0 : parseInt(first[1]);
    var f2 = isNaN(parseInt(first[2])) ? 0 : parseInt(first[2]);
    var s1 = isNaN(parseInt(second[1])) ? 0 : parseInt(second[1]);
    var s2 = isNaN(parseInt(second[2])) ? 0 : parseInt(second[2]);

    if (f1 + f2 < s1 + s2) {
      return -1;
    } else if (f1 + f2 > s1 + s2) {
      return 1;
    }
    return 0;
  });

  // Add hyperlinks to all column headings where they are available.
  for (var h = 0, lenH = tableHeadings.length; h < lenH; h++) {
    // Sheets cannot have multiple links in a single cell at the moment :-/
    if (headings[tableHeadings[h]] &&
        typeof(headings[tableHeadings[h]]) === 'object') {
      tableHeadings[h] = '=HYPERLINK("' + headings[tableHeadings[h]][0] +
          '","' + tableHeadings[h] + '")';
    }
  }
  table.unshift(tableHeadings);

  // Form table from errors
  var errorTable = [];
  var errorKeys = Object.keys(errors);
  for (var k = 0; k < errorKeys.length; k++) {
    errorTable.push([errorKeys[k], errors[errorKeys[k]]]);
  }
  return {
    table: table,
    errors: errorTable
  };
}

/**
 * Given a URL, returns a spreadsheet formula that displays the URL yet links to
 * the PageSpeed URL for examining this.
 *
 * @param {string} url The URL to embed in the Hyperlink formula.
 * @return {string} A string representation of the spreadsheet formula.
 */
function toPageSpeedHyperlinkFormula(url) {
  return '=HYPERLINK("' +
         'https://developers.google.com/speed/pagespeed/insights/?url=' + url +
         '&tab=' + PLATFORM_TYPE +'","' + url + '")';
}

/**
 * Creates an object of results metrics from the parsed results of a call to
 * the PageSpeed service.
 *
 * @param {Object} parsedPageSpeedResponse The object returned from PageSpeed.
 * @return {Object} An associative array with entries for each metric.
 */
function extractResultRow(parsedPageSpeedResponse) {
  var urlScores = {};
  if (parsedPageSpeedResponse.ruleGroups) {
    var ruleGroups = parsedPageSpeedResponse.ruleGroups;
    // At least one of the SPEED or USABILITY properties will exist, but not
    // necessarily both.
    urlScores.Speed = ruleGroups.SPEED ? ruleGroups.SPEED.score : '-';
    urlScores.Usability = ruleGroups.USABILITY ?
        ruleGroups.USABILITY.score : '-';
  }
  if (parsedPageSpeedResponse.formattedResults &&
      parsedPageSpeedResponse.formattedResults.ruleResults) {
    var resultParts = parsedPageSpeedResponse.formattedResults.ruleResults;
    for (var partName in resultParts) {
      var part = resultParts[partName];
      urlScores[part.localizedRuleName] = part.ruleImpact;
    }
  }
  return urlScores;
}

/**
 * Extracts the headings for the metrics returned from PageSpeed, and any
 * associated help URLs.
 *
 * @param {Object} parsedPageSpeedResponse The object returned from PageSpeed.
 * @return {Object} An associative array used to store column-headings seen
 *     in the response. This can take two forms:
 *     (1) {'heading':'heading', ...} - this form is where no help URLs are
 *     known.
 *     (2) {'heading': [url1, ...]} - where one or more URLs is returned that
 *     provides help on the particular heading item.
 */
function extractColumnsInfo(parsedPageSpeedResponse) {
  var columnsInfo = {};
  if (parsedPageSpeedResponse.formattedResults &&
      parsedPageSpeedResponse.formattedResults.ruleResults) {
    var resultParts = parsedPageSpeedResponse.formattedResults.ruleResults;
    for (var partName in resultParts) {
      var part = resultParts[partName];
      if (!columnsInfo[part.localizedRuleName]) {
        columnsInfo[part.localizedRuleName] = part.localizedRuleName;
      }
      // Find help URLs in the response.
      var summary = part.summary;
      if (summary && summary.args) {
        var argList = summary.args;
        for (var i = 0, lenI = argList.length; i < lenI; i++) {
          var arg = argList[i];
          if ((arg.type) && (arg.type == 'HYPERLINK') &&
              (arg.key) && (arg.key == 'LINK') &&
              (arg.value)) {
            columnsInfo[part.localizedRuleName] = [arg.value];
          }
        }
      }
      if (part.urlBlocks) {
        var blocks = part.urlBlocks;
        var urls = [];
        for (var j = 0, lenJ = blocks.length; j < lenJ; j++) {
          var block = blocks[j];
          if (block.header) {
            var header = block.header;
            if (header.args) {
              var args = header.args;
              for (var k = 0, lenK = args.length; k < lenK; k++) {
                var argument = args[k];
                if ((argument.type) &&
                    (argument.type == 'HYPERLINK') &&
                    (argument.key) &&
                    (argument.key == 'LINK') &&
                    (argument.value)) {
                  urls.push(argument.value);
                }
              }
            }
          }
        }
        if (urls.length > 0) {
          columnsInfo[part.localizedRuleName] = urls;
        }
      }
    }
  }
  return columnsInfo;
}

/**
 * Extracts a suitable error message to display for a failed URL. The error
 * could be passed in in the nested PageSpeed error format, or there could have
 * been a more fundamental error in the fetching of the URL. Extract the
 * relevant message in each case.
 *
 * @param {string} errorMessage The error string.
 * @return {string} A formatted error message.
 */
function formatErrorMessage(errorMessage) {
  var formattedMessage = null;
  if (!errorMessage) {
    formattedMessage = 'Unknown error message';
  } else {
    try {
      var parsedError = JSON.parse(errorMessage);
      // This is the nested structure expected from PageSpeed
      if (parsedError.error && parsedError.error.errors) {
        var firstError = parsedError.error.errors[0];
        formattedMessage = firstError.message;
      } else if (parsedError.message) {
        formattedMessage = parsedError.message;
      } else {
        formattedMessage = errorMessage.toString();
      }
    } catch (e) {
      formattedMessage = errorMessage.toString();
    }
  }
  return formattedMessage;
}

/**
 * Calls the PageSpeed API for a single URL, and attempts to parse the resulting
 * JSON. If successful, produces an object for the metrics returned, and an
 * object detailing the headings and help URLs seen.
 *
 * @param {string} url The URL to run PageSpeed for.
 * @return {Object} An object with pageSpeed metrics, column-heading info
 *     and error properties.
 */
function getPageSpeedResultForSingleUrl(url) {
  var parsedResponse = null;
  var errorMessage = null;
  var retries = 0;

  while ((!parsedResponse || parsedResponse.responseCode !== 200) &&
         retries < MAX_RETRIES) {
    errorMessage = null;
    var fetchResult = checkUrl(url);
    if (fetchResult.responseText) {
      try {
        parsedResponse = JSON.parse(fetchResult.responseText);
        break;
      } catch (e) {
        errorMessage = formatErrorMessage(e);
      }
    } else {
      errorMessage = formatErrorMessage(fetchResult.error);
    }
    retries++;
    Utilities.sleep(1000 * Math.pow(2, retries));
  }
  if (!errorMessage) {
    var columnsInfo = extractColumnsInfo(parsedResponse);
    var urlScores = extractResultRow(parsedResponse);
  }
  return {
    pageSpeedInfo: urlScores,
    columnsInfo: columnsInfo,
    error: errorMessage
  };
}

/**
 * Gets the most representative URL that would be used on a mobile device
 * taking into account Upgraded URLs.
 *
 * @param {Entity} entity A Google Ads entity such as an Ad, Keyword or
 *     Sitelink.
 * @return {string} The URL.
 */
function getMobileUrl(entity) {
  var urls = entity.urls();
  var url = null;
  if (urls) {
    if (urls.getMobileFinalUrl()) {
      url = urls.getMobileFinalUrl();
    } else if (urls.getFinalUrl()) {
      url = urls.getFinalUrl();
    }
  }
  if (!url) {
    switch (entity.getEntityType()) {
      case 'Ad':
      case 'Keyword':
        url = entity.getDestinationUrl();
        break;
      case 'Sitelink':
      case 'AdGroupSitelink':
      case 'CampaignSitelink':
        url = entity.getLinkUrl();
        break;
      default:
        Logger.log('No URL found' + entity.getEntityType());
    }
  }
  if (url) {
    url = encodeURI(decodeURIComponent(url));
  }
  return url;
}

/**
 * Determines whether there is enough remaining time to continue iterating
 * through the account.
 *
 * @return {boolean} Returns true if there is enough time remaining to continue
 *     iterating.
 */
function hasRemainingTimeForAccountIteration() {
  var remainingTime = AdsApp.getExecutionInfo().getRemainingTime();
  return remainingTime > POST_ITERATION_TIME_SECS;
}

/**
 * Determines whether there is enough remaining time to continue fetching URLs.
 *
 * @return {boolean} Returns true if there is enough time remaining to continue
 *     fetching.
 */
function hasRemainingTimeForUrlFetches() {
  var remainingTime = AdsApp.getExecutionInfo().getRemainingTime();
  return remainingTime > SPREADSHEET_PREP_TIME_SECS;
}

/**
 * Work through an ad group's members in the account, but only up to the maximum
 * specified by the SITELINK_LIMIT.
 *
 * @param {AdGroup} adGroup The adGroup to process.
 * @return {!Array.<string>} A list of URLs.
 */
function getUrlsFromAdGroup(adGroup) {
  var uniqueUrls = {};
  var sitelinks =
      adGroup.extensions().sitelinks().withLimit(SITELINK_LIMIT).get();
  while (sitelinks.hasNext()) {
    var sitelink = sitelinks.next();
    var url = getMobileUrl(sitelink);
    if (url) {
      uniqueUrls[url] = true;
    }
  }
  return Object.keys(uniqueUrls);
}

/**
 * Work through a campaign's members in the account, but only up to the maximum
 * specified by the AD_LIMIT, KEYWORD_LIMIT and SITELINK_LIMIT.
 *
 * @param {Campaign} campaign The campaign to process.
 * @return {!Array.<string>} A list of URLs.
 */
function getUrlsFromCampaign(campaign) {
  var uniqueUrls = {};
  var url = null;
  var sitelinks = campaign
      .extensions().sitelinks().withLimit(SITELINK_LIMIT).get();
  while (sitelinks.hasNext()) {
    var sitelink = sitelinks.next();
    url = getMobileUrl(sitelink);
    if (url) {
      uniqueUrls[url] = true;
    }
  }
  var ads = campaign.ads().forDateRange('LAST_30_DAYS')
      .withCondition('Status = "ENABLED"')
      .orderBy('Clicks DESC')
      .withLimit(AD_LIMIT)
      .get();
  while (ads.hasNext()) {
    var ad = ads.next();
    url = getMobileUrl(ad);
    if (url) {
      uniqueUrls[url] = true;
    }
  }
  var keywords = campaign.keywords().forDateRange('LAST_30_DAYS')
      .withCondition('Status = "ENABLED"')
      .orderBy('Clicks DESC')
      .withLimit(KEYWORD_LIMIT)
      .get();
  while (keywords.hasNext()) {
    var keyword = keywords.next();
    url = getMobileUrl(keyword);
    if (url) {
      uniqueUrls[url] = true;
    }
  }
  return Object.keys(uniqueUrls);
}

/**
 * Produces a formatted string representing a given date in a given time zone.
 *
 * @param {string} format A format specifier for the string to be produced.
 * @param {Date} date A date object. Defaults to the current date.
 * @param {string} timeZone A time zone. Defaults to the account's time zone.
 * @return {string} A formatted string of the given date in the given time zone.
 */
function getDateStringInTimeZone(format, date, timeZone) {
  date = date || new Date();
  timeZone = timeZone || AdsApp.currentAccount().getTimeZone();
  return Utilities.formatDate(date, timeZone, format);
}

/**
 * UrlStore - this is an object that takes URLs, added one by one, and then
 * allows them to be iterated through in a particular order, which aims to
 * maximise the variety between the returned URLs.
 *
 * This works by splitting the URL into three parts: host, path and params
 * In comparing two URLs, most weight is given if the hosts differ, then if the
 * paths differ, and finally if the params differ.
 *
 * UrlStore sets up a tree with 3 levels corresponding to the above. The full
 * URL exists at the leaf level. When a request is made for an iterator, a copy
 * is taken, and a path through the tree is taken, using the first host. Each
 * entry is removed from the tree as it is used, and the layers are rotated with
 * each call such that the next call will result in a different host being used
 * (where possible).
 *
 * Where opt_manualUrls are supplied at construction time, these will take
 * precedence over URLs added subsequently to the object.
 *
 * @param {?Array.<string>=} opt_manualUrls An optional list of URLs to check.
 * @constructor
 */
function UrlStore(opt_manualUrls) {
  this.manualUrls = opt_manualUrls || [];
  this.paths = {};
  this.re = /^(https?:\/\/[^\/]+)([^?#]*)(.*)$/;
}

/**
 * Adds a URL to the UrlStore.
 *
 * @param {string} url The URL to add.
 */
UrlStore.prototype.addUrl = function(url) {
  if (!url || this.manualUrls.indexOf(url) > -1) {
    return;
  }
  var matches = this.re.exec(url);
  if (matches) {
    var host = matches[1];
    var path = matches[2];
    var param = matches[3];
    if (!this.paths[host]) {
      this.paths[host] = {};
    }
    var hostObj = this.paths[host];
    if (!path) {
      path = '/';
    }
    if (!hostObj[path]) {
      hostObj[path] = {};
    }
    var pathObj = hostObj[path];
    pathObj[url] = url;
  }
};

/**
 * Adds multiple URLs to the UrlStore.
 *
 * @param {!Array.<string>} urls The URLs to add.
 */
UrlStore.prototype.addUrls = function(urls) {
  for (var i = 0; i < urls.length; i++) {
    this.addUrl(urls[i]);
  }
};

/**
 * Creates and returns an iterator that tries to iterate over all available
 * URLs return them in an order to maximise the difference between them.
 *
 * @return {UrlStoreIterator} The new iterator object.
 */
UrlStore.prototype.__iterator__ = function() {
  return new UrlStoreIterator(this.paths, this.manualUrls);
};

var UrlStoreIterator = (function() {
  function UrlStoreIterator(paths, manualUrls) {
    this.manualUrls = manualUrls.slice();
    this.urls = objectToArray_(paths);
  }
  UrlStoreIterator.prototype.next = function() {
    if (this.manualUrls.length) {
      return this.manualUrls.shift();
    }
    if (this.urls.length) {
      return pick_(this.urls);
    } else {
      throw StopIteration;
    }
  };
  function rotate_(a) {
    if (a.length < 2) {
      return a;
    } else {
      var e = a.pop();
      a.unshift(e);
    }
  }
  function pick_(a) {
    if (typeof a[0] === 'string') {
      return a.shift();
    } else {
      var element = pick_(a[0]);
      if (!a[0].length) {
        a.shift();
      } else {
        rotate_(a);
      }
      return element;
    }
  }

  function objectToArray_(obj) {
    if (typeof obj !== 'object') {
      return obj;
    }

    var a = [];
    for (var k in obj) {
      a.push(objectToArray_(obj[k]));
    }
    return a;
  }
  return UrlStoreIterator;
})();


/**
 * Runs the PageSpeed fetch.
 *
 * @param {string} url
 * @return {Object} An object containing either the successful response from the
 *     server, or an error message.
 */
function checkUrl(url) {
  var result = null;
  var error = null;
  var fullUrl = PAGESPEED_URL + 'key=' + API_KEY + '&url=' + encodeURI(url) +
                '&prettyprint=false&strategy=mobile';
  var params = {muteHttpExceptions: true};
  try {
    var pageSpeedResponse = UrlFetchApp.fetch(fullUrl, params);
    if (pageSpeedResponse.getResponseCode() === 200) {
      result = pageSpeedResponse.getContentText();
    } else {
      error = pageSpeedResponse.getContentText();
    }
  } catch (e) {
    error = e.message;
  }
  return {
    responseText: result,
    error: error
  };
}