Compass Solution
Build an agent-powered travel planning app with Generative AI
App architecture
The following diagram shows a high-level overview of how you can combine best-in-class Google developer services to build AI-driven applications.
Build beautiful, AI-powered apps for mobile and web
You can use Flutter and Firebase Genkit to build multi-platform apps that can seamlessly integrate with AI.
Seamlessly call Firebase Genkit from your Flutter app
You can use Genkit to enable your app to confidently consume data from LLMs by defining a schema that can validate the LLM's output. In Flutter, you can use Dart to serialize the request and deserialize the response to match the Genkit schema using standard HTTP requests.
Get started with Flutter in IDX
With IDX, you can create a Flutter app without needing to install any software. This makes it possible to develop and test your android app and web app in the browser.
Connect your database with Firebase Data Connect
Firebase Data Connect is a relational database service for mobile and web apps that lets you build and scale using a fully-managed PostgreSQL database powered by Cloud SQL. It provides a secure schema and query and mutation management using GraphQL that integrates well with Firebase Authentication. Data Connect includes SDK support for Kotlin Android and web.
Build an agent with Firebase Genkit
The agent uses an AI orchestration framework to receive user input and to generate a response.
--- model: googleai/gemini-1.5-flash-latest config: temperature: 1.0 safetySettings: - category: HARM_CATEGORY_HATE_SPEECH threshold: BLOCK_LOW_AND_ABOVE - category: HARM_CATEGORY_DANGEROUS_CONTENT threshold: BLOCK_ONLY_HIGH - category: HARM_CATEGORY_HARASSMENT threshold: BLOCK_LOW_AND_ABOVE - category: HARM_CATEGORY_SEXUALLY_EXPLICIT threshold: BLOCK_LOW_AND_ABOVE input: schema: request: string, The users request for where they want to travel to. place: string, The place that closely represents the users request. placeDescription: string, A description of that place. activities(array, a stringify list of activities that can be found at the specified place): string restaurants?(array, a stringify list of all the restaurants found at that location): string output: schema: place: string, The place the user is traveling to. itineraryName: string, a catchy itinerary name that encapsulates the spirit of the trip and includes the place name startDate: string, the start date of the trip endDate: string, the end date of the trip tags(array, relevant tags for the trip): string itinerary(array): day: number date: string planForDay(array): activityRef: string, the reference value for the activity - this comes from the available activities JSON. If no value is present use a ref value of restaurant. activityTitle: string, a catchy title for the activity activityDesc: string, a six word description of the activity photoUri?: string, set the photo uri value for restaurants only. googleMapsUri?: string, if this is a restaurant include the googleMapsUri --- Generate an itinerary for a tourist planning on traveling to the location specified based in their request. If there is something that does not exist within the list of activities, do not include it in your answer. Feel free to relate the activitiy to the request in a meaningful way. In the plan for day array, put activities as a travel brouchure might do. Come up with a catchy name for the itinerary. Pick three activities per day, minimum of three day trip unless otherwise specified in the request. Output schema should not include the properties type or object. Pick a date after 2024-05-14 but before 2024-12-31. The output date must be in the format year-month-day. Give each activity a unique title and description. Limit activity descriptions to 6 words. If no restaurants are supplied, do not recommend any restaurants to eat at. {{#if restaurants}} Find a restaurant to eat at each day. Include a restaurant to visit in the itinerary for each day from the available restaurants. The restaurant should be the only activity with a photoUri. The photoUri for the restaurant should be from the photoUri property from the restaurant array. If there are no restaurants to pick from, do not include it in the list. The photoUri from the restaurantFinder should be in the format of places/${placeId}/photos/${photoId} Each restaurant should be unique to the overall itinerary. Each restaurant must contain a photoUri in their output JSON schema. Each restaurant must also include an activitiyRef, activityTitle, and activityDesc in their output {{/if}} Output must be in JSON format. REQUEST : {{request}} PLACE : {{place}} PLACE DESCRIPTION : {{placeDescription}} AVAILABLE ACTIVITIES : {{activities}} RESTAURANTS : {{restaurants}}
Create your first flow in Firebase Genkit
When working with generative AI, it's important to craft effective prompts so that the model returns high-quality responses. Firebase Genkit provides the Dotprompt plugin and text format to help you write and organize your generative AI prompts. The format encapsulates the prompt, input and output schema, model, and configuration all in a single file.
The following code example shows a Dotprompt file used in the travel app. The schema is based on the information the user provides when they describe their dream trip.
The following code example shows a Dotprompt file used in the travel app. The schema is based on the information the user provides when they describe their dream trip.
Dotprompt
Dotprompt is designed around the premise that prompts are code. You write and maintain your prompts in specially-formatted files called dotprompt files, track changes to them using the same version control system that you use for your code, and you deploy them along with the code that calls your generative AI models.
Flows
Flows are functions that are strongly typed, streamable, locally and remotely callable, and fully observable. Firebase Genkit provides a command-line interface and developer UI for working with flows, such as running or debugging flows.
import {defineTool} from '@genkit-ai/ai/tool'; ... { name: 'restaurantFinder', description: `Used when needing to find a restaurant based on a users location. The location should be used to find nearby restaurants to a place. You can also selectively find restaurants based on the users preferences, but you should default to 'Local' if there are no indications of restaurant types in the users request. `, inputSchema: z.object({ place: z.string(), typeOfRestaurant: z.string().optional() }), outputSchema: z.unknown(), }, ... async (input) => { if (input.typeOfRestaurant == undefined) { input.typeOfRestaurant = "Local"; } const geocodeEndpoint = "https://places.googleapis.com/v1/places:searchText"; const textQuery = {textQuery: `${input.typeOfRestaurant} restaurants in ${input.place}`}; const response = await axios.post( geocodeEndpoint, JSON.stringify(textQuery), { headers: { "Content-Type": "application/json", "X-Goog-Api-Key": MAPS_API_KEY, "X-Goog-FieldMask": "places.displayName,places.formattedAddress,places.priceLevel,places.photos.name,places.editorialSummary,places.googleMapsUri" } } ); console.log(response.data); let data = (response.data as PlaceResponse); for(let i = 0; i < data.places.length; i++) { if (data.places[i].photos) { data.places[i].photos = [data.places[i].photos[0]]; } } return data as PlaceResponse; }
Extend the agent's functionality by adding tools that interface with external services
You can use function calling in Genkit to extend the agent's functionality so that the agent can further refine responses and complete additional tasks. The travel app defines a tool that can return restaurant information from the Places API based on the user's desired trip. The code uses Zod to define the input and output schema so that the query results can be validated.
... export const textRefinement = defineFlow( { name: 'textRefinement', inputSchema: z.string(), outputSchema: z.unknown(), }, async (userRequest) => { const refinementPrompt = await prompt('textRefinement') const result = await refinementPrompt.generate({ input: { request: userRequest }, }); return result.output(); });
Refine the search query
To help give users a more tailored search experience, after the user describes their dream trip, Gemini determines if more information is needed based on the prompts that the travel app provides, and signals to the app if it thinks more information is needed. The app then prompts the user for that information and appends it to the request on the backend.
import 'package:http:http.dart' as http; ... Future<List<Trip>> generateTrips(String description, List<Image> images) async { final uri = Uri.parse('.../generateTrips'); final request = http.MultipartRequest('POST', uri); request.fields['description'] = description; request.files.add(http.MultipartFile.fromData( images.name, images.bytes, contentType: MediaType('image', 'png'), )); var response = await request.send(); if (response.statusCode == 200) { final body = await response.body.text(); final items = jsonDecode(body) as List<dynamic>; return items.map(Trip.fromJson).toList(); } ... import { imagen2, geminiProVision } from '@genkit-ai/vertexai'; import { generate } from '@genkit-ai/ai'; const imageResult = await generate({ model: imagen2, prompt: 'Generate an image of a very specific historical time and place.', }); const generatedImage = imageResult.media(); const descriptionResult = await generate({ model: geminiProVision, prompt: [ { text: 'What is the historical time and place represented in this picture?', }, { media: generatedImage }, ], }); console.log(descriptionResult.text()); }
Generate list of suggested trip overviews
The travel app asks the user to define their dream trip by using text input or by tapping the microphone button to activate speech-to-text. The user can also optionally upload images.
The app leverages a Dart package from pub.dev to integrate with native speech-to-text capabilities for each platform, and uses the Gemini API inside of Firebase Genkit to handle the multi-modal inputs such as images or videos. The Gemini API uses Retrieval-augmented generation (RAG) to return a set of suggested trips using Firebase Data Connect and embeddings to perform a nearest neighbor search.
The app leverages a Dart package from pub.dev to integrate with native speech-to-text capabilities for each platform, and uses the Gemini API inside of Firebase Genkit to handle the multi-modal inputs such as images or videos. The Gemini API uses Retrieval-augmented generation (RAG) to return a set of suggested trips using Firebase Data Connect and embeddings to perform a nearest neighbor search.
Scale your app for production
Follow production guidelines for all your cloud services
Firebase Hosting integrates with popular modern web frameworks including Flutter. Using Firebase Hosting and Cloud Functions for Firebase with these frameworks, you can develop apps and microservices in your preferred framework environment and then deploy them in a managed, secure server environment. Before going to production, understand the security and performance of all the services in your app. See the Firebase launch checklist for more information.
Vertex AI
The travel app uses Google AI to quickly iterate test data, and is a good choice for apps with minimal AI use cases that don't need to scale. Vertex AI has higher quota to help scale production applications and stronger privacy policies to protect user data. Genkit has built-in functionality for easily switching models so you don't need to rewrite your prompts or API calls.