Cloud Build Notifications with Cloud Run and C++
Carlos O’Ryan (Google)
Greg Miller (Google)
We are moving many of our continuous integration builds for google-cloud-cpp to GCB (Google Cloud Build), as it offers high throughput, simple configuration, and secure integration with other GCP services. While we like most features in GCB, we were missing a way to notify us when a full build (as opposed to a pull request build) fails. This article describes how we solved this issue using Google Cloud services and C++ client libraries. The actual working code for everything described here is available in our GitHub repo.
A few observations helped us put this solution together:
- GCB sends Pub/Sub notifications for every interesting event in a build lifecycle (start, complete with success, cancelled, etc.)
- We can use the Functions Framework for C++ to capture the Cloud Build status changes, as these are sent via Cloud Pub/Sub
- We can send an HTTP POST to a specific URL that will post a message in a Google Chat room to alert us to failures (and only failures).
This also seemed like fun, as we would be putting together functionality we needed using our own Functions Framework.
The C++ Function
The main entrypoint function in this code is SendBuildAlerts() (see below for how to configure this function’s name). This function is invoked automatically by the Functions Framework when receiving a new CloudEvent. The first thing it needs to do is get the secret URL (called a webhook) where the Chat messages will be posted. We inject this URL into the process’s environment using Google Secret Manager.
void SendBuildAlerts(google::cloud::functions::CloudEvent event) {
static auto const webhook = [] {
std::string const name = "GCB_BUILD_ALERT_WEBHOOK";
auto const* env = std::getenv(name.c_str());
if (env) return std::string{env};
throw std::runtime_error("Missing environment variable: "
+ name);
}();
… … …
}Next we do some parsing to extract the JSON object representing the build results from the Cloud Pub/Sub messages. You can see the full details on GitHub, but it is pretty straightforward, with the only non-obvious bit being that the Pub/Sub message has a message field containing a base64-encoded representation of GCB’s Build Resource.
void SendBuildAlerts(google::cloud::functions::CloudEvent event) {
… … …
auto const bs = ParseBuildStatus(std::move(event));
… … …
}Then we filter out events that do not merit an alert, such as successful builds or manually started builds:
void SendBuildAlerts(google::cloud::functions::CloudEvent event) {
… … …
if (bs.status != "FAILURE") return;
auto const substitutions = bs.build["substitutions"];
auto const trigger_type = substitutions.value(
"_TRIGGER_TYPE", "");
auto const trigger_name = substitutions.value("TRIGGER_NAME", "");
// Skips PR invocations and manually invoked builds (no trigger
// name).
if (trigger_type == "pr" || trigger_name.empty()) return;
… … …
}Now it is just a matter of making the POST request to the webhook URL:
void SendBuildAlerts(google::cloud::functions::CloudEvent event) {
… … …
auto const chat = MakeChatPayload(bs);
std::cout << nlohmann::json{{"severity", "INFO"}, {"chat" : chat}}
<< "\n";
HttpPost(webhook, chat.dump());
}The chat message itself is a JSON object:
nlohmann::json MakeChatPayload(BuildStatus const& bs) {
auto const trigger_name = bs.build["substitutions"].value("TRIGGER_NAME", "");
auto const log_url = bs.build.value("logUrl", "");
auto text = fmt::format(
"Build failed: *{}* {}", trigger_name, log_url);
return nlohmann::json{{"text", std::move(text)}};
}And the HTTP POST is implemented using libcurl:
void HttpPost(std::string const& url, std::string const& data) {
static constexpr auto kContentType =
"Content-Type: application/json";
using Headers =
std::unique_ptr<curl_slist, decltype(&curl_slist_free_all)>;
auto const headers = Headers{
curl_slist_append(nullptr, kContentType), curl_slist_free_all};
using CurlHandle =
std::unique_ptr<CURL, decltype(&curl_easy_cleanup)>;
auto curl = CurlHandle(curl_easy_init(), curl_easy_cleanup);
if (!curl) {
throw std::runtime_error("Failed to create CurlHandle");
}
curl_easy_setopt(curl.get(), CURLOPT_URL, url.c_str());
curl_easy_setopt(curl.get(), CURLOPT_HTTPHEADER, headers.get());
curl_easy_setopt(curl.get(), CURLOPT_POSTFIELDS, data.c_str());
CURLcode code = curl_easy_perform(curl.get());
if (code != CURLE_OK) {
throw std::runtime_error(curl_easy_strerror(code));
}
}Getting and Storing the Webhook
To set up an incoming webhook for your Google Chat room you can follow this How To. Once you’ve done that, you’ll have the URL to post to from your Cloud Function (shown above as the contents of the GCB_BUILD_ALERT_WEBHOOK environment variable). You can send a test POST to this URL using curl(1) with a command like:
curl -X POST -sSL "<YOUR_WEBHOOK_URL>" \
--data-binary "{'text': 'hello world'}"This URL should be kept private since it allows anyone to post to your Google Chat room — treat it like a password. We stored this URL in Google Secret Manager following the instructions in this Secret Manager Quickstart Guide. Next we needed that secret to be securely injected into our function’s environment. We configured this by configuring Google Cloud Run to use this secret. Voilà!
Note: We used a Google Chat Webhook here, but we believe a similar approach should work with Slack Webhooks or Discord Webhooks.
Deploying the C++ Function
The functions framework for C++ integrates with Cloud Run, this is a managed serverless platform, where you deploy your code as a container. The platform invokes your code in response to web requests or Pub/Sub events. The platform takes care of starting up the servers, scaling up the number of servers as the load increases, and redelivering messages when the processing fails. All we need to do is write the SendBuildAlerts() function and wrap it into a Docker image.
Here we will show each command one-by-one, but we have a small shell script to run them all.
Conveniently, the Functions Framework for C++ is supported by Google’s build pack, so creating the Docker image is a single command:
readonly IMAGE="gcr.io/${GOOGLE_CLOUD_PROJECT}/send-build-alerts"
pack build --builder gcr.io/buildpacks/builder:latest \
--env "GOOGLE_FUNCTION_SIGNATURE_TYPE=cloudevent" \
--env "GOOGLE_FUNCTION_TARGET=SendBuildAlerts" \
--path "function" "${IMAGE}"This will detect the dependencies for our function, download, compile and install them, then compile our function into an HTTP service. Furthermore, the dependencies are cached (locally, on your workstation), so additional builds are reasonably fast.
Once the image is created, we can push it to GCR (Google Container Registry):
docker push “${IMAGE}:latest”And deploy the code:
gcloud run deploy send-build-alerts \
--project="${GOOGLE_CLOUD_PROJECT}" \
--image="${IMAGE}:latest" \
--region="us-central1" \
--platform="managed" \
--no-allow-unauthenticatedNow that our server is “running”, we need to setup a trigger to send the Cloud Build messages to it, this is done via Eventarc:
PROJECT_NUMBER=$(gcloud projects list \
--filter="project_id=${GOOGLE_CLOUD_PROJECT}" \
--format="value(project_number)" \
--limit=1)gcloud beta eventarc triggers create send-build-alerts-trigger \
--project="$GOOGLE_CLOUD_PROJECT" \
--location="us-central1" \
--destination-run-service="send-build-alerts" \
--destination-run-region="us-central1" \
--transport-topic="cloud-builds" \
--matching-criteria="type=google.cloud.pubsub.topic.v1.messagePublished" \
--service-account="${PROJECT_NUMBER}-compute@developer.gserviceaccount.com"
Note that in this example we use the default Google Compute service account to send the notifications to the function. You may want to use a specific service account for this purpose, following the principle of least privilege, we just did not want to clutter this post with the creation of service accounts and permission assignments.
Next Steps
Once you have code that runs after each build you can really get creative:
- Maybe capture all the build results in a database and track flakiness
- Create dashboards to see long term trends in your builds (build times growing?)
- Consider automatically filing GitHub issues for the failed/flaky builds
Let us know if you find this useful in the Google Cloud Community Slack (#cpp).
