When using multi binder setup in Kafka Streams binder, there is an issue
in which the binder health indicator is not getting bootstapped due to a
ConditionalOnBean is unable to find a match for KafkaStreamsRegistry bean.
Fixing this issue by using an ObjectProvider instead of ConditionalOnBean.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1006
Introducing the capability of routing records based on
event types. If a header in the incoming record contains
the event type set on the binding, then the function
associated with that binding gets invoked.
Adding test/docs.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1003
Fixed typo `springc` to `spring` in two lines:
`spring.cloud.stream.function.bindings.process-in-0=users`
`spring.cloud.stream.function.bindings.process-in-0=regions`
* KafkaBinderMetrics NoopGauge/filtering changes
Fixing the problem of KafkBinderMetrics scheduled task
for finding the offset lag gets triggered, even when the
guage is registered as a NoopGague in the MeterRegistry.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/995
* Addressing PR review comments
* Fix typo
Add a new producer extended property for allowNonTransactional.
When set to true, records published to this output binding will
not be run in a transaction, unless one is already in process.
By default, all ouput bindings associated with a transactional
binder publishes in a new transaction. This new property can be
used to override this behavior.
Addressing PR review comments.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/990
When there are multiple functions present with different outbound target types,
there is an issue of one function overriding the target type of a previous function
in the catalogue where the binder stores the target type information.
This causes problems for the binder initiated Serde inference. Addressing the issue.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/994
Altering existing topic configurations based on a new binder property.
Existing topic configurations are only modified if `autoAlterTopics` is
enabled. By default, this is disabled.
Adding tests to verify.
Docs.
Logging level changes.
Checkstyle fixes
* Copy cert files from classpath to file-system
If `ssl.truststore.location` and `ssl.keystore.location` are
provided as classpath resources, convert them to absolute paths
on the filesystem. This is because of a restriction in the Kafka client
in which it does not allow certificates to be read from the classpath.
See these issues for more details:
https://issues.apache.org/jira/browse/KAFKA-7685https://cwiki.apache.org/confluence/display/KAFKA/KIP-398%3A+Support+reading+trust+store+from+classpath
This commit allows the Spring Cloud Stream application to provide the
cert files as classpath: reosources, but the binder internally move
them to a locations on the local filesystem and then use that absolute
path as the value for cert locations. Adding properties for optional paths
to move the files to. If no values are provided for these properties,
then use the system's /tmp directory.
Adding tests and docs.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/985
* Addressing PR review comments
* Addressing further PR review comments
* Consolidate keystore/truststore filesystem locations into a single property
* Addressing PR review
The property spring.cloud.stream.binder.configuration.num.stream.threads does not work and is silently ignored.
The property spring.cloud.stream.kafka.streams.binder.configuration.num.stream.threads works fine and is already covered by tests at MultipleFunctionsInSameAppTests#125
resolves#987
* Allow retries in Kafka Streams binder
Provide applications the capability to retry critical sections of the
business logic. This is accomplished through a new API using which
critical path can be wrapped inside a Callable.
Adding tests and docs.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/945
* Reworking the PR
Remove the binder API that was added before for retrying.
Reuse binding provided RetryTemplate.
Tests and docs
* Cleanup
* Addressing PR review comments
* Fix typo
* Custom DLQ Destination Resolver
Allow applications to provide a custom DLQ destination resolver
implementaiton by providing a new interface DlqDestinationResolver
as part of binder's public contract. This interface is a BiFunction
extension using which the applications can provide more fine grained
control over where to route records in error.
Adding test to verify.
Adding docs.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/966
* Add DlqDestinationResolver to MessageChannel based binder.
Tests and docs
Spring Kafka -> 2.6.3-SNAPSHOT
Spring Integration Kafka -> 5.4.0-SNAPSHOT
Kafka version -> 2.6.0
Use Kafka_2.13 for tests
Ungignore the Jaas security tests.
Unignore a few Kafka Streams binder tests.
If the user has not explicitly set the `SreamsConfig.REPLICATION_FACTOR_CONFIG`,
set it from the binder property.
This is used for infrastructure topics (change logs and repartition topics).
If both boot and binder level config for bootstrap servers are present, the boot
one always wins currently regardless of any binder settings, unless the boot one
evaluates to the default (localhost:9092). This is especially a problem in a
multi binder scenario. Addressing this issue by simplifying the evaluation and
always gives the binder config the highest precedence.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/967
* Kafka binder metrics improvements
KafkaBinderMetrics has a blocking call in which it waits for the default timeout of
60 seconds if Kafka broker is down. This happens for each topic within a consumer
group. Refactor this code, so that we have this check performed in a periodic task
and if the runtime check fails to return within a smaller timewindow (5 seconds),
return immediately by providing the latest value from the periodic task results.
Periodic task for computing the lags is run every 60 seconds.
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/809
* Addressing PR review
