Caching

This document applies to the following methods:

• Lookup API (v4): threatMatches.find
• Update API (v4): fullHashes.find

About caching

To reduce client bandwidth usage and to protect Google from traffic spikes, clients of both the Lookup API and the Update API are required to create and maintain a local cache of threat data. For the Lookup API, the cache is used to reduce the number of threatMatches requests that clients send to Google. For the Update API, the cache is used to reduce the number of fullHashes requests that clients send to Google. The caching protocol for each API is outlined below.

Lookup API

Clients of the Lookup API should cache each returned ThreatMatch item for the duration defined by its cacheDuration field. Clients then need to consult the cache before making a subsequent threatMatches request to the server. If the cache duration for a previously returned ThreatMatch has not yet expired, the client should assume the item is still unsafe. Caching ThreatMatch items may reduce the number of API requests made by the client.

Example: threatMatches.find

Click the request and response links in the table header for complete examples.

"threatEntries": [
 {"url": "http://www.urltocheck.org/"}
]

"matches": [{
 "threat": {"url": "http://www.urltocheck.org/"},
 "cacheDuration": "300.000s"
}]

Update API

To reduce the overall number of fullHashes requests sent to Google using the Update API, clients are required to maintain a local cache. The API establishes two types of caching, positive and negative.

Positive caching

To prevent clients from repeatedly asking about the state of a particular unsafe full hash, each returned ThreatMatch contains a positive cache duration (defined by the cacheDuration field), which indicates how long the full hash is to be considered unsafe.

Negative caching

To prevent clients from repeatedly asking about the state of a particular safe full hash, each fullHashes response defines a negative cache duration for the requested prefix (defined by the negativeCacheDuration field). This duration indicates how long all full hashes with the requested prefix are to be considered safe for the requested lists, except for those returned by the server as unsafe. This caching is particularly important as it prevents traffic overload that could be caused by a hash prefix collision with a safe URL that receives a lot of traffic.

Consulting the cache

When the client wants to check the state of a URL, it first computes its full hash. If the full hash’s prefix is present in the local database, the client should then consult its cache before making a fullHashes request to the server.

First, clients should check for a positive cache hit. If there exists an unexpired positive cache entry for the full hash of interest, it should be considered unsafe. If the positive cache entry expired, the client must send a fullHashes request for the associated local prefix. Per the protocol, if the server returns the full hash, it is considered unsafe; otherwise, it’s considered safe.

If there are no positive cache entries for the full hash, the client should check for a negative cache hit. If there exists an unexpired negative cache entry for the associated local prefix, the full hash is considered safe. If the negative cache entry expired, or it doesn’t exist, the client must send a fullHashes request for the associated local prefix and interpret the response as normal.

Updating the cache

The client cache should be updated whenever a fullHashes response is received. A positive cache entry should be created or updated for the full hash per the cacheDuration field. The hash prefix’s negative cache duration should also be created or updated per the response’s negativeCacheDuration field.

If a subsequent fullHashes request does not return a full hash that is currently positively cached, the client is not required to remove the positive cache entry. This is not cause for concern in practice, since positive cache durations are typically short (a few minutes) to allow for quick correction of false positives.

Example scenario

In the following example, assume h(url) is the hash prefix of the URL and H(url) is the full-length hash of the URL. That is, h(url) = SHA256(url).substr(4), H(url) = SHA256(url).

Now, assume a client (with an empty cache) visits example.com/ and sees that h(example.com/) is in the local database. The client requests the full-length hashes for hash prefix h(example.com/) and receives back the full-length hash H(example.com/) together with a positive cache duration of 5 minutes and a negative cache duration of 1 hour.

The positive cache duration of 5 minutes tells the client how long the full-length hash H(example.com/) must be considered unsafe without sending another fullHashes request. After 5 minutes the client must issue another fullHashes request for that prefix h(example.com/) if the client visits example.com/ again. The client should reset the hash prefix’s negative cache duration per the new response.

The negative cache duration of 1 hour tells the client how long all the other full-length hashes besides H(example.com/) that share the same prefix of h(example.com/) must be considered safe. For the duration of 1 hour, every URL such that h(URL) = h(example.com/) must be considered safe, and therefore not result in a fullHashes request (assuming that H(URL) != H(example.com/)).

If the fullHashes response contains zero matches and a negative cache duration is set, then the client must not issue any fullHashes requests for any of the requested prefixes for the given negative cache duration.

If the fullHashes response contains one or more matches, a negative cache duration is still set for the entire response. In that case, the cache duration of a single full hash indicates how long that particular full-length hash must be assumed unsafe by the client. After the ThreatMatch cache duration elapses, the client must refresh the full-length hash by issuing a fullHashes request for that hash prefix if the requested URL matches the existing full-length hash in the cache. In that case the negative cache duration does not apply. The response’s negative cache duration only applies to full-length hashes that were not present in the fullHashes response. For full-length hashes that are not present in the response, the client must refrain from issuing any fullHashes requests until the negative cache duration is elapsed.

Example: fullHashes.find

Click the request and response links in the table header for complete examples.

"threatEntries": [
  {"hash": "0xaaaaaaaa"}
]

"matches": [],
"negativeCacheDuration": "3600.000s"

"threatEntries": [
  {"hash": "0xbbbbbbbb"}
]

"matches": [
 "threat": {"hash": "0xbbbbbbbb0000..."}
 "cacheDuration": "600.000s",
],
"negativeCacheDuration": "300.000s"

"threatEntries": [
  {"hash": "0xcccccccc"}
]

"matches": [
 "threat": {"hash": "0xccccccccdddd..."},
 "cacheDuration": "600.000s"
],
"negativeCacheDuration": "3600.000s"