Safe Browsing Update API (v4)

Overview

The Update API lets your client applications download hashed versions of the Safe Browsing lists for storage in a local database. URLs can then be checked locally. Only if a match is found in the local database does the client need to send a request to the Safe Browsing servers to verify whether the URL is included on the Safe Browsing lists.

Before using the Update API, you need to set up a local database. Safe Browsing provides a Go package you can use to get going. For more details, see the Database Setup section under Local Databases.

Updating the local database

To stay current, clients are required to periodically update the Safe Browsing lists in their local database. To save bandwidth, clients download the hash prefixes of URLs rather than the raw URLs. For example, if "www.badurl.com/" is on a Safe Browsing list, clients download the SHA256 hash prefix of that URL rather than the URL itself. In the majority of cases the hash prefixes are 4 bytes long, meaning that the average bandwidth cost of downloading a single list entry is 4 bytes before compression.

To update the Safe Browsing lists in the local database, send an HTTP POST request to the threatListUpdates.fetch method:

  • The HTTP POST request includes the names of the lists to be updated along with various client constraints to account for memory and bandwidth limitations.
  • The HTTP POST response returns either a full update or a partial update. The response may also return a minimum wait duration.

Example: threatListUpdates.fetch

HTTP POST request

In the following example, the updates for a single Safe Browsing list are requested.

Request header

The request header includes the request URL and the content type. Remember to substitute your API key for API_KEY in the URL.

POST https://safebrowsing.googleapis.com/v4/threatListUpdates:fetch?key=API_KEY HTTP/1.1
Content-Type: application/json

Request body

The request body includes the client information (ID and version) and the update information (the list name, the list state, and client constraints). For more details, see the threatListUpdates.fetch request body and the explanations that follow the code example.

{
  "client": {
    "clientId":       "yourcompanyname",
    "clientVersion":  "1.5.2"
  },
  "listUpdateRequests": [{
    "threatType":      "MALWARE",
    "platformType":    "WINDOWS",
    "threatEntryType": "URL",
    "state":           "Gg4IBBADIgYQgBAiAQEoAQ==",
    "constraints": {
      "maxUpdateEntries":      2048,
      "maxDatabaseEntries":    4096,
      "region":                "US",
      "supportedCompressions": ["RAW"]
    }
  }]
}
Client information

The clientID and clientVersion fields should uniquely identify a client implementation, not an individual user. (Client information is used in server-side logging. You can choose any name for the client ID but we suggest you choose a name that represents the true identity of the client, such as your company name, presented as all one word, in all-lowercase letters.)

Safe Browsing lists

The threatType, platformType, and threatEntryType fields are combined to identify (name) the Safe Browsing lists. In the example, one list is identified: MALWARE/WINDOWS/URL. Before sending a request, make sure the type combinations you specify are valid (see Safe Browsing Lists).

Client state

The state field holds the current client state of the Safe Browsing list. (Client states are returned in the newClientState field of the threatListUpdates.fetch response.) For initial updates, leave the state field empty.

Size constraints

The maxUpdateEntries field specifies the total number of updates that the client can manage (in the example, 2048). The maxDatabaseEntries field specifies the total number of entries the local database can manage (in the example, 4096). Clients should set size constraints to protect memory and bandwidth limitations and to safeguard against list growth. For more information, (see Update Constraints).

Supported compressions

The supportedCompressions field lists the compression types the client supports. In the example, the client supports only raw, uncompressed data. Safe Browsing, however, supports additional compression types (see Compression).

HTTP POST response

In this example, the response returns a partial update for the Safe Browsing list using the requested compression type.

Response header

The response header includes the HTTP status code and the content type. Clients that receive a status code other than HTTP/200 must enter back-off mode (see Request Frequency).

HTTP/1.1 200 OK
Content-Type: application/json

Response body

The response body includes the update information (the list name, the response type, the additions and removals to be applied to the local database, the new client state, and a checksum). In the example, the response also includes a minimum wait duration. For more details, see the threatListUpdates.fetch response body and the explanations that follow the code example.

{
  "listUpdateResponses": [{
    "threatType":      "MALWARE",
    "threatEntryType": "URL",
    "platformType":    "WINDOWS",
    "responseType" :   "PARTIAL_UPDATE",
    "additions": [{
      "compressionType": "RAW",
      "rawHashes": {
        "prefixSize": 4,
        "rawHashes":  "rnGLoQ=="
      }
    }],
    "removals": [{
      "compressionType": "RAW",
      "rawIndices": {
        "indices": [0, 2, 4]
      }
    }],
    "newClientState": "ChAIBRADGAEiAzAwMSiAEDABEAFGpqhd",
    "checksum": {
      "sha256": "YSgoRtsRlgHDqDA3LAhM1gegEpEzs1TjzU33vqsR8iM="
    }
  }],
  "minimumWaitDuration": "593.440s"
}
Database updates

The responseType field will indicate a partial or full update. In the example, a partial update is returned, so the response includes both additions and removals. There may be multiple sets of additions, but only one set of removals (see Database Updates).

New client state

The newClientState field holds the new client state for the newly updated Safe Browsing list. Clients must save the new client state for subsequent update requests (the state field in the threatListUpdates.fetch request or the clientStates field in the fullHashes.find request).

Checksums

The checksum lets clients verify that the local database has not suffered any corruption. If the checksum does not match, the client must clear the database and reissue an update with an empty state field. However, clients in this situation must still follow the time intervals for updates (see Request Frequency).

Minimum wait durations

The minimumWaitDuration field indicates that the client must wait 593.44 seconds (9.89 minutes) before sending another update request. Note that a wait period may or may not be included in the response (see Request Frequency).

Checking URLs

To check if a URL is on a Safe Browsing list, the client must first compute the hash and hash prefix of the URL (see URLs and Hashing). The client then queries the local database to determine if there is a match. If the hash prefix is not present in the local database, then the URL is considered safe (not on the Safe Browsing lists).

If the hash prefix is present in the local database (a hash prefix collision), the client must send the hash prefix to the Safe Browsing servers for verification. The servers will return all full-length SHA 256 hashes that contain the given hash prefix. If one of those full-length hashes matches the full-length hash of the URL in question, then the URL is considered unsafe. If none of the full-length hashes match the full-length hash of the URL in question, then that URL is considered safe.

At no point does Google learn about the URLs you are examining. Google does learn the hash prefixes of URLs, but the hash prefixes don’t provide much information about the actual URLs.

To check if a URL is on a Safe Browsing list, send an HTTP POST request to the fullHashes.find method:

  • The HTTP POST request can include up to 500 threat entries.
  • The HTTP POST request includes the hash prefixes of the URLs to be checked. Clients are encouraged to batch multiple threat entries into a single request to lower bandwidth usage.
  • The HTTP POST response returns the matching full-length hashes along with the positive and negative cache durations. The response may also return a minimum wait duration.

Example: fullHashes.find

HTTP POST request

In the following example, the names of two Safe Browsing lists and three hash prefixes are sent for comparison and verification.

Request header

The request header includes the request URL and the content type. Remember to substitute your API key for API_KEY in the URL.

POST https://safebrowsing.googleapis.com/v4/fullHashes:find?key=API_KEY HTTP/1.1
Content-Type: application/json

Request body

The request body includes the client information (ID and version), the client states, and the threat information (the list names and the hash prefixes). For JSON requests, hash prefixes must be sent in base64-encoded form. For more details, see the fullHashes.find request body and the explanations that follow the code example.

{
  "client": {
    "clientId":      "yourcompanyname",
    "clientVersion": "1.5.2"
  },
  "clientStates": [
    "ChAIARABGAEiAzAwMSiAEDABEAE=",
    "ChAIAhABGAEiAzAwMSiAEDABEOgH"
  ],
  "threatInfo": {
    "threatTypes":      ["MALWARE", "SOCIAL_ENGINEERING"],
    "platformTypes":    ["WINDOWS"],
    "threatEntryTypes": ["URL"],
    "threatEntries": [
      {"hash": "WwuJdQ=="},
      {"hash": "771MOg=="},
      {"hash": "5eOrwQ=="}
    ]
  }
}
Client information

The clientID and clientVersion fields should uniquely identify a client implementation, not an individual user. (Client information is used in server-side logging. You can choose any name for the client ID, but we suggest you choose a name that represents the true identity of the client, such as your company name, presented as all one word, in all-lowercase letters.)

All client states

The clientStates field holds the client states for all Safe Browsing lists in the client’s local database. (Client states are returned in the newClientState field of the threatListUpdates.fetch response.)

Safe Browsing lists

The threatTypes, platformTypes, and threatEntryTypes fields combine to identify (name) the Safe Browsing lists. In the example, two lists are identified: MALWARE/WINDOWS/URL and SOCIAL_ENGINEERING/WINDOWS/URL. Before sending a request, make sure the type combinations you specify are valid (see Safe Browsing Lists).

Threat hash prefixes

The threatEntries array contains the hash prefixes of the URLs that you want to check. The hash field must contain the exact hash prefix that is present in the local database. For example, if the local hash prefix is 4 bytes long then the threat entry must be 4 bytes long. If the local hash prefix was lengthened to 7 bytes then the threat entry must be 7 bytes long.

In the example, the request includes three hash prefixes. All three prefixes will be compared against each of the Safe Browsing lists to determine if there is a matching full-length hash.

Note: The Update API and the fullHashes.find method should always use the hash field, never the URLfield (see ThreatEntry).

HTTP POST response

In the following example, the response returns the matching data, organized by Safe Browsing list, along with the cache and wait durations.

Response header

The response header includes the HTTP status code and the content type. Clients that receive a status code other than HTTP/200 must back-off (see Request Frequency).

HTTP/1.1 200 OK
Content-Type: application/json

Response body

The response body includes the match information (the list names and the full length hashes, the metadata, if available, and the cache durations). In the example, the response body also includes a minimum wait duration. For more details, see the fullHashes.find response body and the explanations that follow the code example.

{
  "matches": [{
    "threatType":      "MALWARE",
    "platformType":    "WINDOWS",
    "threatEntryType": "URL",
    "threat": {
      "hash": "WwuJdQx48jP-4lxr4y2Sj82AWoxUVcIRDSk1PC9Rf-4="
    },
    "threatEntryMetadata": {
      "entries": [{
        "key": "bWFsd2FyZV90aHJlYXRfdHlwZQ==",  // base64-encoded "malware_threat_type"
        "value": "TEFORElORw=="  // base64-encoded "LANDING"
       }]
    },
    "cacheDuration": "300.000s"
  }, {
    "threatType":      "SOCIAL_ENGINEERING",
    "platformType":    "WINDOWS",
    "threatEntryType": "URL",
    "threat": {
      "hash": "771MOrRPMn6xPKlCrXx_CrR-wmCk0LgFFoSgGy7zUiA="
    },
    "threatEntryMetadata": {
      "entries": []
    },
    "cacheDuration": "300.000s"
  }],
  "minimumWaitDuration": "300.000s",
  "negativeCacheDuration": "300.000s"
}
Matches

The Matches object returns a matching full-length hash for two of the hash prefixes. The URLs corresponding to these hashes are considered unsafe. No match was found for the third hash prefix, so nothing is returned; the URL corresponding to this hash prefix is considered safe.

Note that this example matches one full-length hash to one hash prefix; there could, however, be multiple full hashes that map to the same hash prefix.

Metadata

The threatEntryMetadata field is optional and provides additional information about the threat match. Currently, metadata is available for the MALWARE/WINDOWS/URL Safe Browsing list (see Metadata).

Cache durations

The cacheDuration and negativeCacheDuration fields indicate the amount of time the hashes must be considered either unsafe or safe (see Caching).

Minimum wait durations

The minimumWaitDuration field indicates that the client must wait 300 seconds (5 minutes) before sending another fullHashes request. Note that a wait period may or may not be included in the response (see Request Frequency).