SAMZA-2434: Fix the coordinator steam creation workflow #1252
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Symptom: Currently the coordinator kafka topic of a samza job is created with cleanup.policy set to 'delete' instead of 'compact' for certain cases. This potentially leads to losing all the metadata(JobModel/Config) stored in coordinator stream for some samza jobs.
Cause: The control-flow in KafkaSystemAdmin.createStream to build the coordinator stream spec swallows the essential kafka-topic configurations and passes along the empty configuration bag to the kafka-broker when creating the coordinator stream. This issue was introduced in SAMZA-1868.
Changes: Fix the topic-creation control-flow for the coordinator topics and generate the correct topic-configurations.
Tests: Added unit tests to validate that the expected topic configuration bag was generated for the coordinator stream topics.
API Changes: None
Upgrade Instructions: None
Usage Instructions: None