Commit Graph

779 Commits

Author SHA1 Message Date
buildmaster
5adeea2acb Bumping versions to 3.1.4-SNAPSHOT after release 2021-05-26 15:50:17 +00:00
buildmaster
65664a2d4c Going back to snapshots 2021-05-26 15:50:17 +00:00
buildmaster
f5e12a29c1 Update SNAPSHOT to 3.1.3 v3.1.3 2021-05-26 15:48:31 +00:00
Soby Chacko
416580e607 Version updates 2021-05-26 10:38:33 -04:00
aleksevi
a81093734e GH-1081 Suspicious multiplication of ScheduledExecutorService in the method "bindTo" of class "KafkaBinderMetrics"
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1081

The wrong consequence from the absense of scheduler.shutdown():
1) First of all we created the pool with 1 thread.
2) After we lost the reference on it and created the pool with 2 threads.
3) But, the first pool is not yet collected by GC and now you have 3 threads together. And so on.

Each thread does nothing, but it takes system memory and takes part in the scheduling process.
After 30 topics, for example, we potentially have (30+1)*15=465 threads.
It is already serious additional load on the switching contexts and the native memory.

Removed waiting after scheduler stop request.

checkstyle fixes
2021-05-26 10:37:42 -04:00
Soby Chacko
5f4e3eebc5 Removing deprecated setAckOnError method call (#1082)
Going forward, binder will use the default ack on error settings
in Spring Kafka, which is true by default. If the applicaitons
need to change this behavior, then a custom error handler must
be provided through ListenerContainerCustomizer.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1079
2021-05-24 16:46:26 -04:00
Taras Danylchuk
cb42e80dac gh-1059 : Added health indicator for kafka messages listener containers created via kafka binder
renamed variables, return unknown on empty listener list

added tests, fixed PR comments

checkstyle fix
2021-05-17 15:20:23 -04:00
Soby Chacko
4fb5037fd7 Use dlqProducerProperties for DLQ topics
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1077
2021-05-14 11:54:01 -04:00
Soby Chacko
d8b8c8d9fd Resetting offsets docs clarification 2021-05-06 18:16:47 -04:00
Soby Chacko
9e4a1075d4 Native compilation changes for Kafka Streams binder
This comnmit effectively removes changes introduced by a4ad9e2c0b.
Removing spring.factories mechanism of registering binders in preference to using spring.binders.
2021-04-29 19:35:15 -04:00
Soby Chacko
79497ad264 Native compilation changes
This commit effectively reverts 4e6881830a

Remove spring.factories changes introduced in the commit mentiond above.
Revert the binder to use spring.binders mechanism to register binder configurations.
2021-04-29 19:08:01 -04:00
Oleg Zhurakousky
7e48afa005 Merge pull request #1067 from sobychacko/kafka-streams-native-1
Native changes required by Kafka Streams binder
2021-04-28 19:13:52 +02:00
Oleg Zhurakousky
87ac491230 Merge pull request #1066 from sobychacko/kafka-binder-native-1
More changes required for native compilation
2021-04-28 19:13:43 +02:00
Soby Chacko
a4ad9e2c0b Native changes required by Kafka Streams binder 2021-04-28 12:26:53 -04:00
Soby Chacko
4e6881830a More changes required for native compilation
These are Kafka binder changes required for native applications.
2021-04-28 12:22:51 -04:00
Soby Chacko
829ce1cf7e Native compilation changes
This commit is a counter part to the commit below in core:

ae017e222f
2021-04-22 11:35:08 -04:00
buildmaster
c5289589c1 Bumping versions 2021-03-27 08:52:52 +00:00
Soby Chacko
dd607627ed Fix destination as pattern issues in Kafka Streams (#1054)
When destination-is-pattern property is enabled and native deserialization
is used, then Kafka Streams binder does not use the correct Serde types.
Fixing this issue by providing the proper Serdes to the StreamsBuilder.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1051
2021-03-25 16:20:33 -04:00
Gary Russell
0ea4315af8 GH-1043: Support Custom BatchMessageConverter
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1043
2021-03-24 13:23:53 -04:00
Soby Chacko
f25dbff2b7 Kafka Streams component bean scanning enhancements
When functional beans are discovered from libraries in the classpath,
it causes issues when Kafka Streams functions are scanned and bootstrapped.
Binder expects the users to provide function definition property although the
application does not directly include these functional beans or is aware of it.
Fixing this issue by excluding non kafka streams function from scanning.

Null check around adding micrometer listener to the StreamsBuilder.

Addressing issues raised by the comments here:
https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1030#issuecomment-804039087
2021-03-22 19:06:07 -04:00
Gary Russell
f8b290844b GH-1046: Fix Out of Order Offset Commit with DLQ
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1046

When using ack mode `MANUAL_IMMEDIATE`, we must acknowledge the delivery on
the container thread, otherwise the commit will be queued for later processing,
possibly causing out of order commits, incorrectly reducing the committed offset.

Wait for the send to complete before acknowledging; use a timeout slightly larger
than the configured producer delivery timeout to avoid premature timeouts.

Tested with user-provided test case.

Fix missing timeout buffer.

More context for this issue: https://gitter.im/spring-cloud/spring-cloud-stream?at=6050f43595e23446e43cacd1
2021-03-19 15:36:29 -04:00
Gary Russell
a7e025794c Fix typo in doc 2021-03-18 13:29:21 -04:00
Gary Russell
634a73c9ff Doc Polishing for resetOffsets (#1045)
* Doc Polishing for resetOffsets

* Fix typos.
2021-03-18 13:13:18 -04:00
buildmaster
bc2f692964 Bumping versions to 3.1.3-SNAPSHOT after release 2021-03-16 13:53:15 +00:00
buildmaster
7eefe6c567 Going back to snapshots 2021-03-16 13:53:15 +00:00
buildmaster
c5d108ef89 Update SNAPSHOT to 3.1.2 v3.1.2 2021-03-16 13:51:50 +00:00
Soby Chacko
47e7ca07a8 KTable event type routing
Introduce event type based routing for KTable types. This is
already available for KStream types.
See this commit: 386a361a66

Adding a new DeserialiationExceptionHander to address event type routing
use case for GlobalKTable.
See this comment: https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1003#issuecomment-799847819

Upgrade Spring Kafka to 2.6.6
Upgrade Spring Integration to 5.4.4
2021-03-15 20:23:16 -04:00
Soby Chacko
7cc001ac4c Support KStream lifecycle through binding endpoint (#1042)
* Support KStream lifecycle through binding endpoint

Introduce the ability for Kafka Streams application's lifecycle
management through actuator binding endpoints. Kafka Streams
only supports STOP and START operations. PAUSE/RESUME operations
that is available in regular message channel based binders
are not available in Kafka Streams binder.

Adding tests and docs.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1038
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/850

https://stackoverflow.com/questions/60282225/why-doesnt-kstreambinder-have-a-lifecycle-for-defaultbinding

* Addressing PR review comments

* Addressing PR review

* cleanup unused code
2021-03-15 16:14:19 -04:00
Soby Chacko
a7299df63f Cleanup Kafka Streams metrics support
StreamsListener (for Micrometer) is now directly available
in Spring Kafka starting from 2.5.3 (as KafkaStreamsMicrometerListener).
Removing the temporory interface added in the binder.

Addressing PR review comments.

Modifying tests to verify.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1040
2021-03-12 15:14:19 -05:00
Soby Chacko
33aa926940 Fix checkstyle 2021-03-09 17:22:23 -05:00
Soby Chacko
a1fb7f0a2d Kafka Streams function detection improvements (#1033)
* Kafka Streams function detection improvements

Allow Kafka Streams functions defined as Component beans
to be candidates for establishing bindings. Currently, Kafka Streams
functions need to be written as functional beans using @Bean.
Adding this improvement so that if applications prefer to write
the business logic using @Component, then it is possible to do so.

Adding test cases to verify the behavior.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1030

* Kafka Streams functions and bean name overriding

Whn Kafka Streams function bean names are overridden,
there is an issue with scanning it properly for binding.
Addressing this issue.

* Adding docs for Component based model

* Addressing PR review comments
2021-03-09 15:50:10 -05:00
Soby Chacko
e2eca34e4b Cleaning up Kafka producer factories
Calling DisposableBean.destroy() on manually created
Kafka producer factories. This affects both Kafka and
Kafka Streams binders (in the case of Kafka Streams binder,
it only matters when using the DLQ support).

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1034
2021-03-02 17:26:37 -05:00
Soby Chacko
bc02da2900 Docs cleanup
Apply changes from a previous commit to the proper docs file.

See the prior relevant commit at the URL below:

af5778d157

Fixes https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1032
2021-03-02 16:42:21 -05:00
Soby Chacko
1f52ec5020 Fix minor issue processing ackMode 2021-02-22 12:22:28 -05:00
Soby Chacko
394b8a6685 Restore autoCommitOffset behavior
Until autoCommitOffset is fully removed (it is deprecated already in 3.1),
honor the old behavior if the application uses the property.

Restore autoCommitOnError from deprecated status to active for polled consumers.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1026
2021-02-09 17:48:56 -05:00
Soby Chacko
32939e30fe AdminClient customizer (#1023)
* AdminClient customizer

Provide the ability for applications to customize AdminClient by
introducing a new interface AdminClientConfigCustomizer.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1014

* Addressing PR review comments
2021-02-03 10:25:03 -05:00
Surendra Galwa
cadb5422cc Fix property name in startOffset 2021-01-27 17:27:25 -05:00
buildmaster
99880ba216 Bumping versions to 3.1.2-SNAPSHOT after release 2021-01-27 17:52:34 +00:00
buildmaster
9e9e0f7ea3 Going back to snapshots 2021-01-27 17:52:34 +00:00
buildmaster
e1648083e6 Update SNAPSHOT to 3.1.1 v3.1.1 2021-01-27 17:51:08 +00:00
Soby Chacko
87491118c3 Update spring-cloud-build parent 2021-01-22 15:59:11 -05:00
Soby Chacko
cf7acb23e8 Update Kafka dependencies 2021-01-22 15:12:29 -05:00
Soby Chacko
1fbb6f250e Make health indicator beans public
Currently, the health indicator beans for both binders are not
customizable due to package visibility issues. Making them public.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1015
2021-01-19 11:01:41 -05:00
Inigo
ae4abe4f33 fix property name in consuming batches 2021-01-06 14:54:08 -05:00
Marco Leichsenring
5d312228db Fix topic name confg in docs 2021-01-06 14:52:44 -05:00
Mehmet
69c5b67126 removed j letter from exception message
Not sure, but it seems like a typo.
2021-01-06 14:51:55 -05:00
Soby Chacko
83cfdfc532 Fix checkstyle issues 2021-01-05 16:33:15 -05:00
Soby Chacko
c6154eecfc Update copyright year for a previous commit 2021-01-05 16:02:56 -05:00
Soby Chacko
f8a4488a0e Update Kafka Streams binder health indicator
Update the health indicator code for Kafka Streams binder
to the latest changes made in the 3.0.x branch.
2021-01-05 15:58:24 -05:00
Soby Chacko
ffde5d35db Kafka Streams binder health indicator improvements
When using multi binder setup in Kafka Streams binder, there is an issue
in which the binder health indicator is not getting bootstapped due to a
ConditionalOnBean is unable to find a match for KafkaStreamsRegistry bean.
Fixing this issue by using an ObjectProvider instead of ConditionalOnBean.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1006
2021-01-05 15:55:40 -05:00