904 Commits

Author SHA1 Message Date
buildmaster
c5d108ef89 Update SNAPSHOT to 3.1.2 v3.1.2 2021-03-16 13:51:50 +00:00
Soby Chacko
47e7ca07a8 KTable event type routing
Introduce event type based routing for KTable types. This is
already available for KStream types.
See this commit: 386a361a66

Adding a new DeserialiationExceptionHander to address event type routing
use case for GlobalKTable.
See this comment: https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1003#issuecomment-799847819

Upgrade Spring Kafka to 2.6.6
Upgrade Spring Integration to 5.4.4
2021-03-15 20:23:16 -04:00
Soby Chacko
7cc001ac4c Support KStream lifecycle through binding endpoint (#1042)
* Support KStream lifecycle through binding endpoint

Introduce the ability for Kafka Streams application's lifecycle
management through actuator binding endpoints. Kafka Streams
only supports STOP and START operations. PAUSE/RESUME operations
that is available in regular message channel based binders
are not available in Kafka Streams binder.

Adding tests and docs.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1038
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/850

https://stackoverflow.com/questions/60282225/why-doesnt-kstreambinder-have-a-lifecycle-for-defaultbinding

* Addressing PR review comments

* Addressing PR review

* cleanup unused code
2021-03-15 16:14:19 -04:00
Soby Chacko
a7299df63f Cleanup Kafka Streams metrics support
StreamsListener (for Micrometer) is now directly available
in Spring Kafka starting from 2.5.3 (as KafkaStreamsMicrometerListener).
Removing the temporory interface added in the binder.

Addressing PR review comments.

Modifying tests to verify.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1040
2021-03-12 15:14:19 -05:00
Soby Chacko
33aa926940 Fix checkstyle 2021-03-09 17:22:23 -05:00
Soby Chacko
a1fb7f0a2d Kafka Streams function detection improvements (#1033)
* Kafka Streams function detection improvements

Allow Kafka Streams functions defined as Component beans
to be candidates for establishing bindings. Currently, Kafka Streams
functions need to be written as functional beans using @Bean.
Adding this improvement so that if applications prefer to write
the business logic using @Component, then it is possible to do so.

Adding test cases to verify the behavior.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1030

* Kafka Streams functions and bean name overriding

Whn Kafka Streams function bean names are overridden,
there is an issue with scanning it properly for binding.
Addressing this issue.

* Adding docs for Component based model

* Addressing PR review comments
2021-03-09 15:50:10 -05:00
Soby Chacko
e2eca34e4b Cleaning up Kafka producer factories
Calling DisposableBean.destroy() on manually created
Kafka producer factories. This affects both Kafka and
Kafka Streams binders (in the case of Kafka Streams binder,
it only matters when using the DLQ support).

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1034
2021-03-02 17:26:37 -05:00
Soby Chacko
bc02da2900 Docs cleanup
Apply changes from a previous commit to the proper docs file.

See the prior relevant commit at the URL below:

af5778d157

Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1032
2021-03-02 16:42:21 -05:00
Soby Chacko
1f52ec5020 Fix minor issue processing ackMode 2021-02-22 12:22:28 -05:00
Soby Chacko
394b8a6685 Restore autoCommitOffset behavior
Until autoCommitOffset is fully removed (it is deprecated already in 3.1),
honor the old behavior if the application uses the property.

Restore autoCommitOnError from deprecated status to active for polled consumers.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1026
2021-02-09 17:48:56 -05:00
Soby Chacko
32939e30fe AdminClient customizer (#1023)
* AdminClient customizer

Provide the ability for applications to customize AdminClient by
introducing a new interface AdminClientConfigCustomizer.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1014

* Addressing PR review comments
2021-02-03 10:25:03 -05:00
Surendra Galwa
cadb5422cc Fix property name in startOffset 2021-01-27 17:27:25 -05:00
buildmaster
99880ba216 Bumping versions to 3.1.2-SNAPSHOT after release 2021-01-27 17:52:34 +00:00
buildmaster
9e9e0f7ea3 Going back to snapshots 2021-01-27 17:52:34 +00:00
buildmaster
e1648083e6 Update SNAPSHOT to 3.1.1 v3.1.1 2021-01-27 17:51:08 +00:00
Soby Chacko
87491118c3 Update spring-cloud-build parent 2021-01-22 15:59:11 -05:00
Soby Chacko
cf7acb23e8 Update Kafka dependencies 2021-01-22 15:12:29 -05:00
Soby Chacko
1fbb6f250e Make health indicator beans public
Currently, the health indicator beans for both binders are not
customizable due to package visibility issues. Making them public.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1015
2021-01-19 11:01:41 -05:00
Inigo
ae4abe4f33 fix property name in consuming batches 2021-01-06 14:54:08 -05:00
Marco Leichsenring
5d312228db Fix topic name confg in docs 2021-01-06 14:52:44 -05:00
Mehmet
69c5b67126 removed j letter from exception message
Not sure, but it seems like a typo.
2021-01-06 14:51:55 -05:00
Soby Chacko
83cfdfc532 Fix checkstyle issues 2021-01-05 16:33:15 -05:00
Soby Chacko
c6154eecfc Update copyright year for a previous commit 2021-01-05 16:02:56 -05:00
Soby Chacko
f8a4488a0e Update Kafka Streams binder health indicator
Update the health indicator code for Kafka Streams binder
to the latest changes made in the 3.0.x branch.
2021-01-05 15:58:24 -05:00
Soby Chacko
ffde5d35db Kafka Streams binder health indicator improvements
When using multi binder setup in Kafka Streams binder, there is an issue
in which the binder health indicator is not getting bootstapped due to a
ConditionalOnBean is unable to find a match for KafkaStreamsRegistry bean.
Fixing this issue by using an ObjectProvider instead of ConditionalOnBean.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1006
2021-01-05 15:55:40 -05:00
Oleg Zhurakousky
5921d464ed Fix snapshot versioning schema 2020-12-21 15:46:10 +01:00
buildmaster
023d3df7f7 Bumping versions to 3.1.1.SNAPSHOT after release 2020-12-21 12:28:52 +00:00
buildmaster
846534fe84 Going back to snapshots 2020-12-21 12:28:52 +00:00
buildmaster
2eedc8a218 Update SNAPSHOT to 3.1.0 v3.1.0 2020-12-21 12:27:25 +00:00
buildmaster
49d263eafc Going back to snapshots 2020-12-11 15:12:31 +00:00
buildmaster
94f398a3e4 Update SNAPSHOT to 3.1.0-RC1 v3.1.0-RC1 2020-12-11 15:11:13 +00:00
Soby Chacko
7cba801bef Checkstyle cleanup 2020-12-10 21:35:16 -05:00
Soby Chacko
db14154398 Cleanup docs in Kafka Streams binder
- Add StreamListener deprecation notice in docs

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1005
2020-12-10 21:30:57 -05:00
Soby Chacko
386a361a66 Event type based routing in Kafka Streams binder
Introducing the capability of routing records based on
event types. If a header in the incoming record contains
the event type set on the binding, then the function
associated with that binding gets invoked.

Adding test/docs.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1003
2020-12-10 21:18:00 -05:00
hemeda3
27cb6f4b5f Fixed typo springc to spring
Fixed typo `springc` to `spring` in two lines: 
`spring.cloud.stream.function.bindings.process-in-0=users`
`spring.cloud.stream.function.bindings.process-in-0=regions`
2020-12-09 14:23:05 -05:00
Soby Chacko
58ac92ec71 Fix broken documentation links
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/986
2020-12-04 18:09:32 -05:00
Soby Chacko
d1c62bbc26 Kafka Streams binder producer/consumerProperties
Fix the issue of Kafka Streams binder level producerProperties
and consumerProperties (...binder.producerProperties and
...binder.consumerProperties), are not taken into consideration.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/997
2020-12-04 17:54:54 -05:00
Soby Chacko
afe13cc045 Adding docs for Meter filtering
KafkaBinderMetrics docs for meter fitlering.
Refact METRIC_NAME to OFFSET_LAG_METRIC_NAME and make it public.
2020-12-04 16:52:20 -05:00
Soby Chacko
42c9af019e Prodcer/Consumer config customizer changes
* Provide binding and destination names to the configure method
  in ProducerConfigCustomizer and ConsumerConfigCustomizer
  so that those can be used in the customization.
* Modify tests and docs

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/996
2020-12-04 16:16:10 -05:00
Soby Chacko
675c2e4940 KafkaBinderMetrics NoopGauge/filtering changes (#1001)
* KafkaBinderMetrics NoopGauge/filtering changes

Fixing the problem of KafkBinderMetrics scheduled task
for finding the offset lag gets triggered, even when the
guage is registered as a NoopGague in the MeterRegistry.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/995

* Addressing PR review comments

* Fix typo
2020-12-04 16:04:09 -05:00
Soby Chacko
a1b31e67c4 Support allowNonTransactional config
Add a new producer extended property for allowNonTransactional.
When set to true, records published to this output binding will
not be run in a transaction, unless one is already in process.

By default, all ouput bindings associated with a transactional
binder publishes in a new transaction. This new property can be
used to override this behavior.

Addressing PR review comments.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/990
2020-12-04 11:49:13 -05:00
Soby Chacko
5a7cc9f257 Fix Serde inference issues in Kafka Streams binder
When there are multiple functions present with different outbound target types,
there is an issue of one function overriding the target type of a previous function
in the catalogue where the binder stores the target type information.
This causes problems for the binder initiated Serde inference. Addressing the issue.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/994
2020-12-03 19:43:07 -05:00
Soby Chacko
fc184ba422 Setters for fields that need customization
See this commit in core fore details:

442c72b37c
2020-11-20 14:38:36 -05:00
Taras Danylchuk
526627854a Altering existing topics only allowed if opt-in
Altering existing topic configurations based on a new binder property.
Existing topic configurations are only modified if `autoAlterTopics` is
enabled. By default, this is disabled.

Adding tests to verify.

Docs.

Logging level changes.

Checkstyle fixes
2020-11-19 15:36:09 -05:00
Soby Chacko
6be541c070 Copy cert files from classpath to file-system (#989)
* Copy cert files from classpath to file-system

If `ssl.truststore.location` and `ssl.keystore.location` are
provided as classpath resources, convert them to absolute paths
on the filesystem. This is because of a restriction in the Kafka client
in which it does not allow certificates to be read from the classpath.

See these issues for more details:
https://issues.apache.org/jira/browse/KAFKA-7685
https://cwiki.apache.org/confluence/display/KAFKA/KIP-398%3A+Support+reading+trust+store+from+classpath

This commit allows the Spring Cloud Stream application to provide the
cert files as classpath: reosources, but the binder internally move
them to a locations on the local filesystem and then use that absolute
path as the value for cert locations. Adding properties for optional paths
to move the files to. If no values are provided for these properties,
then use the system's /tmp directory.

Adding tests and docs.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/985

* Addressing PR review comments

* Addressing further PR review comments

* Consolidate keystore/truststore filesystem locations into a single property

* Addressing PR review
2020-11-17 13:49:04 -05:00
buildmaster
1afb22d65f Going back to snapshots 2020-11-17 16:04:44 +00:00
buildmaster
097fb89d9e Update SNAPSHOT to 3.1.0-M4 v3.1.0-M4 2020-11-17 16:03:30 +00:00
Oleg Zhurakousky
0ad0a31b4e Disabled JAAS test 2020-11-17 16:53:27 +01:00
Oleg Zhurakousky
19e97dd9c4 Update versions of spring kafka and spring integration kafka 2020-11-17 16:26:07 +01:00
gustavomonarin
676764b923 Fix documentation for kafka streams concurrency on binder level
The property spring.cloud.stream.binder.configuration.num.stream.threads does not work and is silently ignored.

The property spring.cloud.stream.kafka.streams.binder.configuration.num.stream.threads works fine and  is already covered by tests at MultipleFunctionsInSameAppTests#125

resolves #987
2020-11-11 13:55:35 -05:00