Deploy a CSV Connector

This guide is intended for Google Cloud Search CSV (comma-separated values) connector administrators, that is, anyone who is responsible for downloading, configuring, running, and monitoring the connector.

This guide includes instructions for performing key tasks related to CSV connector deployment:

• Download the Google Cloud Search CSV connector software
• Configure the connector for use with a specific CSV data source
• Deploy and run the connector

To understand the concepts in this document, you should be familiar with the fundamentals of Google Workspace, CSV files, and Access Control Lists (ACLs).

Overview of the Google Cloud Search CSV connector

The Cloud Search CSV connector works with any comma-separated values (CSV) text file. A CSV file stores tabular data, and each line of the file is a data record.

Google Cloud Search's CSV Connector extracts individual rows from a CSV file and indexes them into Cloud Search via Cloud Search's Indexing API. Once successfully indexed, individual rows from CSV files are searchable through Cloud Search's clients or Cloud Search's Query API. The CSV connector also supports controlling users' access to content in the search results, by using ACLs.

Google Cloud Search CSV connector can be installed on Linux or Windows. Before you deploy the Google Cloud Search CSV connector, ensure that you have the following required components:

• Java JRE 1.8 installed on a computer that runs the Google Cloud Search CSV connector

• Google Workspace information required to establish relationships between Google Cloud Search and the data source:

  • Google Workspace private key (which contains the service account ID)
  • Google Workspace data source ID

  Typically, the Google Workspace administrator for the domain can supply these credentials for you.

Deployment steps

To deploy the Google Cloud Search CSV connector follow these steps:

1. Install the Google Cloud Search CSV connector software
2. Specify the CSV connector configuration
3. Configure access to the Google Cloud Search data source
4. Configure CSV file access
5. Specify columns names to index, unique key columns, and datetime columns
6. Specify columns to use in clickable search result URLs
7. Specify metadata information, column formats
8. Schedule data traversal
9. Specify Access Control List (ACL) options

1. Install the SDK

Install the SDK into your local Maven repository.

1. Clone the SDK repository from GitHub.

   $ git clone https://github.com/google-cloudsearch/connector-sdk.git
   $ cd connector-sdk/csv

2. Check out the desired version of the SDK:

   $ git checkout tags/v1-0.0.3

3. Build the connector:

   $ mvn package

4. Copy the connector zip file to your local installation directory:

   $ cp target/google-cloudsearch-csv-connector-v1-0.0.3.zip installation-dir
   $ cd installation-dir
   $ unzip google-cloudsearch-csv-connector-v1-0.0.3.zip
   $ cd google-cloudsearch-csv-connector-v1-0.0.3

2. Specify the CSV connector configuration

As the connector administrator, you control the CSV connector's behavior and attributes defining parameters in the connector's configuration file. Configurable parameters include:

• Access to a data source
• Location of the CSV file
• CSV column definitions
• Column(s) that define a unique id
• Traversal options
• ACL options to restrict data access

For the connector to properly access a CSV file and index the relevant content, you must first create its configuration file.

To create a configuration file:

1. Open a text editor of your choice and name the configuration file.
   Add key=value pairs to the file contents as described in the following sections.
2. Save and name the configuration file.
   Google recommends that you name the configuration file connector-config.properties so no additional command line parameters are required to run connector.

Because you can specify the configuration file path on the command line, a standard file location is not necessary. However, keep the configuration file in the same directory as the connector to simplify tracking and running the connector.

To ensure the connector recognizes your configuration file, specify its path on the command line. Otherwise, the connector uses connector-config.properties in your local directory as the default file name. For information about specifying the configuration path on the command-line, see Run the Cloud Search CSV connector.

3. Configure access to the Google Cloud Search data source

The first parameters every configuration file must specify are the ones necessary to access the Cloud Search data source, as shown in the following table. Typically, you will need the Data source ID, service account ID, and the path to the service account's private key file in order to configure the connector's access to Cloud Search. The steps required to set up a data source are described in Manage third-party data sources

4. Configure CSV file parameters

Before the connector can traverse a CSV file and extract data from it for indexing, you must identify the path to the file. You can also specify the file format and type of file encoding. Add the following parameters to specify the CSV file properties in the configuration file.

5. Specify column names to index and unique key columns

For the connector to access and index CSV files, you must provide information about column definitions in the configuration file. If the configuration file does not contain the parameters that specify the column names to index and unique key columns, default values are used.

6. Specify columns to use in clickable search result URLs

When a user searches using Google Cloud Search, it responds by showing a results page that includes clickable URLs for each result. To enable this feature, you must add the parameter shown in the following table to the configuration file.

7. Specify metadata information, column formats, search quality

You can add parameters to the configuration file that specify:

• Metadata Configuration Parameters
• Column formats
• Search quality

Metadata Configuration Parameters

Metadata Configuration Parameters describes the CSV columns used for populating item metadata. If the configuration file does not contain these parameters, default values are used. The following table shows these parameters.

Datetime formats

Datetime formats specify the formats expected in metadata attributes. If the configuration file does not contain this parameter, default values are used. The following table shows this parameter.

Column formats

Column formats specify information about the column(s) that should be a part of the searchable content. If the configuration file does not contain these parameters, default values are used. The following table shows these parameters.

Search quality

The Cloud Search CSV connector allows automatic HTML formatting for data fields. Your connector defines the data fields at the beginning of connector execution, and then uses a content template to format each data record before uploading it to Cloud Search.

The content template defines the importance of each field value for searching. The title field is required and is defined as the highest priority. You can designate search quality importance levels for all the other content fields: high, medium or low. Any content field not defined in a specific category defaults to low priority. The following table shows these parameters.

8. Schedule data traversal

Traversal is the connector's process for discovering content from the data source, in this case, a CSV file. As the CSV connector runs, it will traverse the rows of a CSV file, and index each row to Cloud Search via the Indexing API.

Full traversal indexes all columns in the file. Incremental traversal only indexes columns that are added or modified since the previous traversal. The CSV connector only performs full traversals. It does not perform incremental traversals.

The scheduling parameters determine how often the connector waits between traversals. If the configuration file does not contain scheduling parameters, default values are used. The following table shows these parameters.

9. Specify Access Control List (ACL) options

Google Cloud Search CSV connector supports permissions through ACLs to control access to the content of the CSV file in search results. There are multiple ACL options available to allow you to protect user access to indexed records.

If your repository has individual ACL information associated with each document, upload all ACL information to control document access within Cloud Search. If your repository provides partial or no ACL information, you can supply default ACL information in the following parameters, which the SDK provides to the connector.

The connector relies on default ACLs being enabled in the configuration file. To enable default ACLs, set defaultAcl.mode to any mode other than none and configure it with defaultAcl.*

Schema Definition

Cloud Search allows indexing and serving of structured and unstructured content. In order to support structured data queries on your data, you need to setup Schema for your datasource.

Once defined, CSV Connector can refer defined schema to build indexing requests. To provide an illustrative example, let's consider a CSV file containing information about Movies.

Let's assume, input CSV file has following content.

1. movieId
2. movieTitle
3. description
4. year
5. releaseDate
6. actors (multiple values separated by comma (,))
7. genre (multiple values)
8. ratings

Based on above structure of data, you can define schema for a datasource under which you want to index data from CSV file.

{
  "objectDefinitions": [
    {
      "name": "movie",
      "propertyDefinitions": [
        {
          "name": "actors",
          "isReturnable": true,
          "isRepeatable": true,
          "isFacetable": true,
          "textPropertyOptions": {
            "operatorOptions": {
              "operatorName": "actor"
            }
          }
        },
        {
          "name": "releaseDate",
          "isReturnable": true,
          "isRepeatable": false,
          "isFacetable": false,
          "datePropertyOptions": {
            "operatorOptions": {
              "operatorName": "released",
              "lessThanOperatorName": "releasedbefore",
              "greaterThanOperatorName": "releasedafter"
            }
          }
        },
        {
          "name": "movieTitle",
          "isReturnable": true,
          "isRepeatable": false,
          "isFacetable": false,
          "textPropertyOptions": {
            "retrievalImportance": {
              "importance": "HIGHEST"
            },
            "operatorOptions": {
              "operatorName": "title"
            }
          }
        },
        {
          "name": "genre",
          "isReturnable": true,
          "isRepeatable": true,
          "isFacetable": true,
          "enumPropertyOptions": {
            "operatorOptions": {
              "operatorName": "genre"
            },
            "possibleValues": [
              {
                "stringValue": "Action"
              },
              {
                "stringValue": "Documentary"
              },
              {
                "stringValue": "Drama"
              },
              {
                "stringValue": "Crime"
              },
              {
                "stringValue": "Sci-fi"
              }
            ]
          }
        },
        {
          "name": "userRating",
          "isReturnable": true,
          "isRepeatable": false,
          "isFacetable": true,
          "integerPropertyOptions": {
            "orderedRanking": "ASCENDING",
            "maximumValue": "10",
            "operatorOptions": {
              "operatorName": "score",
              "lessThanOperatorName": "scorebelow",
              "greaterThanOperatorName": "scoreabove"
            }
          }
        }
      ]
    }
  ]
}

Example configuration file

The following example configuration file shows the parameter key=value pairs that define an example connector's behavior.

# data source access
api.sourceId=1234567890abcd
api.serviceAccountPrivateKeyFile=./PrivateKey.json

# CSV data structure
csv.filePath=./movie_content.csv
csv.csvColumns=movieId,movieTitle,description,releaseYear,genre,actors,ratings,releaseDate
csv.skipHeaderRecord=true
url.format=https://mymoviesite.com/movies/{0}
url.columns=movieId
csv.datetimeFormat.releaseDate=yyyy-mm-dd
csv.multiValueColumns=genre,actors
csv.multiValue.genre=;
contentTemplate.csv.title=movieTitle

# metadata structured data and content
itemMetadata.title.field=movieTitle
itemMetadata.createTime.field=releaseDate
itemMetadata.contentLanguage.defaultValue=en-US
itemMetadata.objectType.defaultValue=movie
contentTemplate.csv.quality.medium=description
contentTemplate.csv.unmappedColumnsMode=IGNORE

#ACLs
defaultAcl.mode=fallback
defaultAcl.public=true

For detailed descriptions of each parameter, see the Configuration parameters reference.

Run the Cloud Search CSV connector

To run the connector from the command line, type the following command:

$ java -jar google-cloudsearch-csv-connector-v1-0.0.3.jar -Dconfig=my.config

By default, connector logs are available on standard output. You can log to files by specifying logging.properties.