Features

• Add a set wrapper around bitset for storing uint32 values
• Add a thread safe wrapper around hash map
• Add b&a e2e test env
• Add data loading support for uint32 sets
• Add health check to AWS mesh.
• Add hook for running set query using uint32 sets as input
• Add interestGroupNames to V1 API
• Add latency metrics for cache uint32 sets functions
• Add latency without custom code execution metric
• Add option to use existing network on AWS.
• Add padding to responses
• Add request log context to request context
• Add runsetqueryint udf hook
• Add set operation functions for bitsets
• Add support for int32_t sets to key value cache
• Add support for reading and writing int sets to csv files
• Add udf hook for running int sets set query (local lookup)
• Allow pas request to pass consented debug config and log context
• Implement sharded RunSetQueryInt rpc for lookup client
• Implement uint32 sets sharded lookup support
• Load consented debug token from server parameter
• Pass LogContext and ConsentedDebugConfig to internal lookup server in sharded case
• Plumb the safe path log context in the cache update execution path
• Set verbosity level for PS_VLOG
• Simplify thread safe hash map and use a single map for node storage
• Support uint32 sets for query parsing and evaluation
• Support uint32 sets in InternalLookup rpc
• Switch absl log for PS_LOG and PS_VLOG for unsafe code path
• Switch absl log to PS_LOG for safe code path
• Switch absl vlog to PS_VLOG for safe code path
• Update AWS coordinators public prod endpoint from GG to G3P

Bug Fixes

• Add missing include/library deps
• Augment UDF loading info message
• Correct copts build config.
• Correct verbosity flag for gcp validator.
• Effectively lock the key in the set map cleanup
• Fix detached head of continuous e2e branch.
• Properly initialize runSetQueryInt hook
• Remove ignore interestGroupNames from envoy
• Remove test filter to allow all unit tests run in the build
• Simplify request context and pass it as shared pointer to the hooks
• Upgrade common repo version
• Use kms_binaries tar target from common repo
• Use structured initializer for clarity

Dependencies

• deps: Upgrade build-system to 0.62.0
• deps: Upgrade data-plane-shared-libraries to 52239f15 2024-05-21
• deps: Upgrade pre-commit hooks

GCP: Features

• GCP: Switch to internal lb for the otlp collector
• GCP: Switch to internal lb for the otlp collector with bug fixes

Documentation

• Add debugging playbook
• Correct commands for sample_word2vec getting_started example
• KV onboarding guide
• Update to the ads retrieval explainer
• Update word2vec example
• Use aws_platform bazel config
• Use local_{platform,instance} bazel configs

Image digests and PCR0s

GCP: sha256:d09d5a6d340a8829df03213b71b74d4b431e4d5a138525c77269c347a367b004
AWS: {"PCR0":"1e28ac4b72600ea40d61e1756e14f453a3d923a1bf94c360ae48d9777bff0714923d9322ed380823591859e357d2f825"}

Features

• Add cache hit or miss metrics
• Add coorindator specific terraform parameters
• Add data loading prefix allowlist parameter
• Add default PAS UDF
• Add E2E latency for GetKeyValues and GetKeyValueSet in sharded lookup
• Add file groups and file group reader logic
• Add go fmt to pre-commit
• Add key prefix support to blob storage client
• Add LogContext and ConsentedDebugConfiguration proto to v2 API and internal lookup API
• Add prod and nonprod build flag
• Add request context to wrap metrics context
• Add support for configuring directory allowlist
• Add wiring for prefix allowlist (actual impl in follow up cl)
• Allow overrides for coordinators endpoints in nonprod mode
• Allow to disable v1 key not found entry in response
• Create separate metrics context map for internal lookup server
• Deprecate metrics recorder for internal lookup
• Deprecate metrics recorder for internal server
• Deprecate metrics recorder for sharded lookup
• deprecate metrics recorder for V1 server and handler
• Deprecate metrics recorder from cache
• Enable simulation system send realtime udpates
• Enable TCMalloc for KV Server and benchmarks
• Explicitly enable core dumps
• Implement deletion cutoff max timestamp per directory
• Load data files and allow notifications from configured prefix
• Load prefix files on startup and handle prefix blob notifications
• Log common request metrics
• Migrate from glog to absl log
• Partition data loading metrics by delta file name
• Pass request context from hooks to downstream components
• Pass request context to udf hooks
• Read telemetry config from cloud parameter
• Revamp AWS metrics dashboard
• Revamp GCP metrics dashboard
• Set udf_min_log_level from parameter store.
• Support content type proto for v2 api
• Support content type proto for v2 api response
• Update cache interface and blob data location to pass prefix
• Update start_after to use a map from prefix to start_after
• Use file groups for loading snapshots
• Write logs to an Otel endpoint

Bug Fixes

• Actually load all files in a snapshot file group
• AWS: Filter out unavailable zones.
• Correct an error in kokoro_release.
• Correct format for image tag.
• Correct typo for internal dev's service_mesh_address.
• Correct typos in GCP deployment guide.
• Crash server if default UDF fails to load.
• Delete non-active certificate before creating a new one.
• Fix filtering logic for prefixed blobs
• Fix permissions for data-loading-blob-prefix-allowlist
• Make GCP nat optional.
• Parse delta filename from notification before validating it
• Remove glog dependency for record_utils
• Remove temp dir only if it's successfully created.
• Rename class to ThreadManager
• Set retain_initial_value_of_delta_metric flag for aws metrics exporter
• Update a outdated hyperlink.
• Update common repo to pick up the AWS metrics dimension fix
• Update GCP Terraform with ability to delete unhealthy instance.
• Update tf variables to use gorekore instead of kelvingorekore
• Use blob key instead of prefixed basename

GCP: Fixes

• GCP: Make sure server is connected to otel collector before reaching to ready state

GCP: Features

• GCP: Applying Terraform pulls docker image with new tag.
• GCP: Make service mesh address configurable.
• GCP: Make subnet ip cidr configurable.
• GCP: Make xlb/envoy optional.

Documentation

• Add ad retrieval explainer.
• Add docs for directory support
• Add PA and PAS folders
• Add PAS developer guide
• Add public docs for file groups
• Ads retreival explainer update.

Dependencies

• deps: Add clang-tidy bazel config
• deps: Add cpp_nowarn bazel config
• deps: Upgrade bazel to 6.5.0
• deps: Upgrade build-system to 0.55.1
• deps: Upgrade build-system to 0.55.2
• deps: Upgrade build-system to 0.57.0
• deps: Upgrade data-plane-shared repo
• deps: Upgrade data-plane-shared repo to 1684674 2024-02-09
• deps: Upgrade data-plane-shared-libraries to 1fbac46
• deps: Upgrade pre-commit hooks

Bug: N/A

Features

• Add an ability to add default tags
• Add AWS cpu utilization and memory utilization dashboards
• Add bazel configs prod and non_prod
• Add change notifier error count to the AWS metrics dashboard
• Add delta based realtime updates publisher
• Add delta record limited file writer
• Add GCP system metrics dashboard for kv server
• Add gcp_project_id flag when --platform=gcp is specified
• add go script for pushing gcp docker image
• Add optional logical_commit_time param to wasm macros.
• Add sharded realtime message batcher
• Add sharding data locality terraform parameters
• Add sharding support for publishing engine
• Add system metrics to log cpu and memory utilization
• Add UDF timeout parameter value to terraform
• Allow prefix path segments for gRPC http paths.
• Allow setting logging verbosity level through a parameter
• Data locality support
• Enable prettier pre-commit hook for JavaScript
• Expose http_api_paths tf variable to allow prefix matching
• Log error metrics for the change notifier
• Log metrics in the retries with new Telemetry API
• Migrate realtime metrics to new Telemetry API
• Remove envoy health checks
• Remove metrics recorder from blob storage client
• Remove metrics recorder from data orchestrator
• Remove metrics recorder from data reader
• Set UDF timeout from parameters
• Support passing in shard metadata via data_cli flags
• Update default tretyakov aws env settings
• Update v1 to return status per key

Bug Fixes

• Add all record values to switch cases
• add data-bucket-id back as a flag suffix.
• Add run_all_tests bazel config
• bazel config for emscripten no longer needs tool chain resolution
• Check grpc channel connection before sending requests
• deps: Upgrade data-plane-shared-libraries to 44d1d64 2024-01-08
• envoy.yaml permission and pre-commit fix
• Integrate RomaService changes from common repo
• Make some cache logging more precise.
• Missing absl deps
• Prevent Config from being copied into RomaService
• rectify kv_service_port in terraform.
• Remove jaeger as it has been deprecated
• Remove local platform for common repo and fix CloudPlatform import
• Update 'X-allow-fledge' header to 'Ad-Auction-Allowed' according to the spec.
• update GCP tentb environment
• Update UDFs to fix breakages due to FunctionBindingObjectV2
• Upgrade otel collector to the latest version

Dependencies

• deps: Upgrade data-plane-shared to commit f0d0b89 2023-11-14

GCP: Features

• GCP: Add external load balancer and envoy.
• GCP: store Terraform state remotely in GCS bucket

Documentation

• [Sharding] Add info about NUMA clusters
• Add docs for server cpu and memory profiling
• Add info on default wasm macro linkopts and memory limits.
• Add state transition sequence diagrams for aws/gcp sharded/nonsharded configurations
• Data locality
• Doc for aws private communication setup
• Sharding -- add a link to a tool that sets shard id in the delta metadata
• Specify --init flag for docker run commands

Bug: N/A

Features

• [API breaking change] Rename subkey to hostname.
• [Sharding] Add a tool to validate records
• [Sharding] Generate shard specific delta files
• Add base64 encoding to CSV delta cli tool
• Add safe metric definitions
• Add unsafe metric definitions
• Fully statically link the server binary
• Simplified UDF tester.

Bug Fixes

• Add error message when key not found.
• Add proxy visibility back to aws_artifacts
• allow internal ingress for otlp
• Clear errors in driver prior to parsing.
• Fix bash script flag for -v
• Fully qualify RegisterBenchmark calls.
• Package proxify layer on container_image
• Remove unused import httpbody.proto
• Update visibility target to public target in common repo
• Use set for cache key lookups

GCP: Features

• GCP: add capability to use existing service_mesh

Documentation

• Add a top level getting_started directory.
• Add docs about AWS AMI structure
• Add instructions on how to call UDF APIs from C++ WASM.
• Update sharding docs with GCP-specific info

Dependencies

• deps: Update data-plane-shared to b463f16
• deps: Utilize rules_closure deps via data-plane-shared

Bug: N/A

Features

• Add bazel config for code coverage
• Add proto definition for UDF input.
• Add string set support in udf delta tester.
• Add support to output graphviz dot file when playing with queries
• Move server_docker_image.tar to dist/
• Set bazel version to 6.3.2
• Set shard id label for metrics
• Update to latest PublicKeyFetcher

Bug Fixes

• Add seccomp flag to functional tests for local runs
• Allow changelog file to contain non-ascii chars
• bump the terraform version to v1.2.3
• Correct bazel config_setting visibility
• Fix blob_storage_client_gcp's polling frequency
• Fix excessive logging of unmock methods which cause flaky test in local
• Move server_docker_image.tar into dist dir

GCP: Features

• GCP: add GCP blob storage client.

Dependencies

• deps: Upgrade build-system to 0.45.0

Documentation

• [Realtime updates] Update docs to reflect GCP details
• Modify some sentences that I found hard to read.
• Sharding
• Sharding -- metrics

Bug: N/A

Features

• [Coordinators] moving /server/bin to /
• [GCP] Add realtime notifier tool
• [sharding] Add metrics for set query
• A delta based request generator that creates KV requests from delta files
• A generic grpc client that sends a request and returns a response
• Add ability to point to real coordinators through parameters
• Add bazel configs for roma legacy vs sandboxed
• Add bazel macro to generate UDF delta file from wasm binary and JS
• Add delta based request generator to the simulation system
• Add delta file notifier and loader to the request simulation system
• Add Dependencies section to release notes
• Add functions for serializing and deserializing shard mapping records
• Add GCP features and fixes sections in release notes
• add GCP platform and instance
• Add GCP realtime notifier
• Add GCP terraform config for metrics collector
• Add logical sharding config schema and constants
• Add metrics collector endpoint to the parameter
• Add metrics collector to periodically print and publish metrics
• Add NAT gateway for public internet access
• Add OSSF Scorecard badge to top-level README
• Add OSSF Scorecard GitHub Action
• add service mesh to GCP platform
• Add support to execute queries over sharded sets.
• Add tool to convert C++ to JS with inline WASM
• add unit tests to gcp parameter client
• Add version to UdfConfig.
• Adding GCP version of the message service
• Build AMI for request simulation system
• Coordinators: Add missing permission
• Deploy otel with request simulation system to AWS
• Enable metrics and tune the performance for request simulation system
• GCP terraform and parameter client refactor
• Generate synthetic requests at fixed rate
• grpc client worker to send requests at configurable QPS
• Integrating kv value server with the KeyFetcherManagerInterface
• Register get values hook for string and binary output format
• Run request simulation system in local
• Skip delta and snapshot files that belong to other shards
• Support writing shard mapping configs from csv
• Upgrade build-system to v0.33.0
• Upgrade data-plane-shared-libraries to 2023-07-12 commit.
• Upgrade data-plane-shared-libraries to 2023-07-21 commit.
• Upgrade data-plane-shared-libraries to 2023-07-26 commit.
• Upgrade data-plane-shared-libraries to 2023-08-16 commit.
• use local parameter client for gcp platform's local instance

Bug Fixes

• Acquire read lock on the set before iterating over it.
• Add action_env for asan bazel config
• Add noexcept bazel config
• Add seccomp-unconfined flag to build_and_test_all_in_docker
• Check that fb strings are not nullptr.
• Check the metrics collector connection during telemetry initialization
• Do not pass metrics collector endpoint for local and aws instance
• Don't move references in sample udf code.
• Explicitly cast return values of set operations to r-value references
• fix another asan error in the test
• Fix data loading num threads in param client local test
• Fix grpc client error
• Fix the error messages printed in the unit test logs
• Fix UDF function handler name
• logMessage should set an output string.
• make the response outlive grpc client call
• Rearrange bazel config for clarity
• Reduce the number of client workers to 2 in the unit tests to limit the number of threads created
• remove GRPC 4mb payload limit
• remove local variant of component tools.
• Remove redundant docker security-opt
• Set bazel workspace name
• Write data record for set data in the delta test file generator

Dependencies

• deps: Upgrade build-system to 0.42.1
• deps: Upgrade build-system to v0.41.1
• deps: Upgrade emscripten to 3.1.44

GCP: Features

• GCP: Add realtime thread pool manager

Documentation

• Add docs on getValuesBinary API
• Realtime directory is not optional for local dev
• Update docker run instructions to include security-opt flag
• Update inline WASM docs with instructions on how to test it
• Update screenshot of delta file

Bug: N/A

Bug: b/293901782

Features

• [Breaking change] Use UserDefinedFunctionsConfig instead of KVs for loading UDFs.
• [Sharding] Add hpke for s2s communication
• [Sharding] Allow for partial data lookups
• [Sharding] Making downstream requests in parallel
• Add bazel build flag --announce_rc
• Add bool parameter to allow routing V1 requests through V2.
• Add buf format pre-commit hook
• Add build time directive for reentrant parser.
• Add functions to retrieve instance information.
• Add internal run query client and server.
• Add JS hook for set query.
• Add lookup client and server for communication with shards
• Add MessageQueue for the request simulation system
• Add query grammar and interface for set queries.
• Add rate limiter for the request simulation system
• Add second map to store key value set and add set value update interfaces
• Add shard metadata for supporting sharded files
• Add simple microbenchmarks for key value cache
• Add UDF support for format data command.
• Add unit tests for query lexer.
• Adding cluster mappings manager
• Adding padding
• Apply custom lockings on the cache
• Connect InternalRunQuery to the parser
• Extend and simplify collect-logs to capture test outputs
• Extend use of scp deps via data-plane-shared repo
• Implement shard manager
• Move sharding function to public so it's available for file sharding
• Register a logging hook with the UDF.
• Register run query hook with udf framework.
• Sharding - realtime updates
• Sharding read flow fixes
• Simplify work done in set operations. Set operations can be passed by
• Snapshot files support UDF configs.
• Support reading and writing set queries to data files.
• Support reading and writing set values for csv files
• Support reading/writing DataRecords. Requires new DELTA format.
• Support writing sharded files
• Update data_loading.fb to support UDF code updates.
• Update pre-commit hook versions
• Update shard manager mappings continuously
• Upgrade build-system to release-0.28.0
• Upgrade build-system to v0.30.1
• Upgrade scp to 0.72.0
• Use Unix domain socket for internal lookup server.
• Utilize AWS deps via data-plane-shared repo

Bug Fixes

• Add internal lookup client deadline.
• Catch error if insufficient args specified
• Fix aggregation logic for set values.
• Fix ASAN potential deadlock errors in key_value_cache_test
• Proper memory management of callback hook wrappers.
• Specify 2 workers for UDF execution.
• Upgrade pre-commit hooks
• Use shared pointer for UDF absl::Notification.

Build Tools: Fixes

• build: Add scope-based sections in release notes

Documentation

• Add docs for data loading capabilities
• Add explanation that access control is managed by IAM for writes.
• Point readme to a new sharding public explainer

Bug: 290798418

Features

• Add --fast option for running functional tests
• Add a multi-KV SUT
• Add baseline for SSP
• Add bazel macro to generate OpenSSL certificates
• Add brotli compression support
• Add envoy for KV server in baseline SUT
• Add envoy frontend in baseline SUT
• Add optional filename arg to tools/collect-logs
• Add SUT without UDF
• Add terraform support for sharding
• Drop data meant for other shards
• Include tink_cc repository
• Set timestamp for generated delta files for SUTs
• Support selection of single SUT for testing
• Upgrade gRPC to v1.52.1
• Upgrade to functionaltest-system 0.5.1
• Use 443 for http1.1 and 8443 for http2.
• Use scp deps exposed by data-plane-shared-libraries repo
• Use shared control plane skylark functions for transitive dependencies

Bug Fixes

• Address linter issues in production dir
• Address linter issues in tools
• Avoid running prettier on PCR0 files
• Capture test logs into zip archive
• Collect logs from functional testing
• Configure docker compose to pull images quietly
• Correct container deps in multiple-kv-servers
• Defer to data-plane-shared cpp deps
• Remove exposed ports as not needed within docker network
• Remove unused code in tools/get_workspace_status
• Remove workspace parent dirs from logs archive
• Upgrade pre-commit hooks

Documentation

• Add Protected Audience API rename banner

Bug: 280464254

Features

• Add a total realtime QPS metric
• Add aws supplied e2e latency
• Add basic UDF functional tests for v2
• Add error counters for realtime updates
• Add functional test stubs for v2
• Add target to generate delta for sample udf.js
• Add test data artifacts to dist/test_data/deltas
• Add UDF delta file generator tool.
• Add UDF delta file upload through terraform config.
• Add udf.js delta file to test set
• Upgrade to build-system 0.22.0 and functionaltest-system 0.2.0

Bug Fixes

• Add a dashboard for environments
• Add documentation for editing dashboards
• Change envoy log level to debug
• Check that recovery function is valid before calling it.
• Enable docker network cleanup
• Ensure changelog notes use specific version
• ignore interestGroupNames argument
• MetricsRecorder no longer a singleton.
• MetricsRecorder now optional for retry templates.
• Return missing key error status from internal lookup server.
• Upgrade gRPC and make lookup client a singleton
• Use dynamic_cast to get metric_sdk::MeterProvider provider.

Terraform

• Add us-west-1 terraform

Documentation

• Add documentation on roma child processes.
• Add instructions on realtime updates
• Add note to use grpcurl for v2 since http has a bug.
• Add v2 request JSON schema
• AWS realtime update capabilities
• Correct udf target name
• Update documentation for building data cli.
• Update realtime metrics querying docs

Bug: 277075104