Commit Graph

846 Commits

Author SHA1 Message Date
Pommerening, Nico
f9dfbe09f7 GH-1161: InteractiveQueryService improvements
This PR safe guards state store instances in case there are multiple KafkaStreams
instances present that have distinct application IDs but share State Store Names.

Change is backwards compatible: In case no KafkaStreams association of the thread
can be found, all local state stores are queried as before.

In case an associated KafkaStreams Instance is found, but required StateStore is
not found in this instance, a warning is issued but backwards compatibility is
preserved by looking up all state stores.

Store within KafkaStreams instance of thread is preferred over "foreign" store with same name.

Warning is issued if requested store is not found within KafkaStreams instance of thread.

The main benefit here is to get rid of randomly selecting stores across all KafkaStreams instances
in case a store is contained within multiple streams instances with same name.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1161
2022-01-03 18:59:15 -05:00
Oleg Zhurakousky
ba122bd39d Changes related to GH-2245 from core 2021-11-16 14:52:58 +01:00
Soby Chacko
486469da51 Kafka binder test migration
- EnableBinding to functional
2021-11-11 16:34:22 -05:00
Soby Chacko
ed98f1129d Kafka Streams binder deprecated component removals 2021-11-09 14:07:23 -05:00
Soby Chacko
c32be995f6 4.0.x changes for Kafka Streams tests
Migrating StreamListener based Kafka Streams binder tests to
use the funcitonal model
2021-11-08 19:21:49 -05:00
buildmaster
d37cbc79d7 Going back to snapshots 2021-11-03 09:32:29 +00:00
buildmaster
7a03eeed02 Update SNAPSHOT to 3.2.0-RC1 v3.2.0-RC1 2021-11-03 09:31:02 +00:00
Oleg Zhurakousky
3a88839a5f Disable 'testGlobalStartOffsetWithLatestAndIndividualBindingWthEarliest' temporary 2021-11-03 10:17:03 +01:00
Soby Chacko
c9a07729dd Version updates and compile fixes
Spring Kafka: 2.8.0-RC1
Spring Integration: 5.5.5
Kafka Clients: 3.0.0

Remove schema registry references.
Updates for removed classes/deprecations in Kafka Streams client.
2021-11-01 22:11:06 -04:00
Soby Chacko
07f10f6eb5 Cleaning up
* Disconnect spring-cloud-scheam-registry-client from Kafka Streams binder
  that is used for testing
* Deprecate MessageConverterDelegateSerde
* Remove MessageConverterDelegateSerdeTest
2021-10-28 19:46:46 -04:00
Gary Russell
6fdc663349 GH-1031: Retry/DLQ in Binder or Container
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1031

Provide a mechanism to provide retry/dlq configuration when adding a custom error handler,
effectively moving this functionality from the binder to the container.

Address PR comments; fix test to use embedded broker.
2021-10-25 15:10:21 -04:00
buildmaster
f3a954fad7 Going back to snapshots 2021-10-19 09:29:32 +00:00
buildmaster
7be0f1be23 Update SNAPSHOT to 3.2.0-M3 v3.2.0-M3 2021-10-19 09:28:11 +00:00
Oleg Zhurakousky
0b687ad0ab Update SK to 2.8.0-RC1 2021-10-19 11:16:55 +02:00
Soby Chacko
c0bece64bd README cleanup
The overview doc gets unncessarily copied to the GitHub
repository root README. This only needs to reside in the
reference manual docs. Cleaning the process that generates
the root README for the repository.
2021-10-12 17:20:33 -04:00
Gary Russell
2efd29fb27 GH-1138: HealthIndicator Improvements
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1138

Don't report DOWN if a container is stopped normally.
This is a valid state when containers are not auto-startup or are stopped while
the app remains running.

Containers are stopped abnormally when
- a listener throws an `Error`
- a `CommonContainerStoppingErrorHandler` (or similar) is configured to stop the
  container after an error.
2021-10-04 17:13:01 -04:00
Soby Chacko
82a3306cb9 GH-1157: Issues with Kafka Streams and Kotlin
Kafka Streams binder erroneously tries to parse regular
non Kafka streams Kotlin function registrations. Ignore
function beans ending in _registration in Kafka Streams binder.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1157
2021-10-01 11:09:58 -04:00
buildmaster
af0485e241 Going back to snapshots 2021-10-01 14:13:49 +00:00
buildmaster
ea8912b011 Update SNAPSHOT to 3.2.0-M2 v3.2.0-M2 2021-10-01 14:12:23 +00:00
Oleg Zhurakousky
d76d916970 Upgrade spring-kafka 2021-10-01 13:48:08 +02:00
Soby Chacko
ac0e462ed2 Doc clarification for Kafka Streams binder prefix
Polishing the docs

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1134
2021-09-28 18:07:52 -04:00
Soby Chacko
bd1b49222c GH-1156: Kafka Streams binder composition issues
When both regular Kafka and Kafka Streams functions are present,
the code that was added recently for function composition in
Kafka Streams binder was accidentally creating a binadable proxy
factory bean for non Kafka Streams functions. Resolving this issue.

Resovles https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1156
2021-09-28 15:43:06 -04:00
Soby Chacko
9fd16416d6 GH-1149: Kafka Streams global config issues
When there are multiple functions, streamConfigGlobalProperties are overriden
for subsequent functions, after a binding specific config takes effect in a function.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1149
2021-09-24 10:01:26 -04:00
bono007
a4c38b3453 GH-1152: Property binding in Kafka Streams binder
Add default mappings provider for Kafka Streams (move kafka streams default mapping to new provider)

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1152
2021-09-23 13:19:35 -04:00
Soby Chacko
a8c948a6b2 GH-1148: Native changes required
KafkaBinderConfiguration class needs more pruning for native compilation

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1148
2021-09-21 18:22:33 -04:00
Soby Chacko
d57091d791 GH-1145: Remove destroying producer factory
Remove the un-ncessary call to destroy the producer when checking for partitions.
This way, the producer is cached and reused at the first time data is produced.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1145
2021-09-16 14:46:37 -04:00
Gary Russell
2b7f6ecb96 Fix Doc Typos for KafkaBindingRebalanceListener 2021-09-15 17:35:17 -04:00
Soby Chacko
53d32c2332 GH-1140: CommonErrorHandler per consumer binding (#1143)
* GH-1140: CommonErrorHandler per consumer binding

Setting CommonErrorHandler on consumer binding through its bean name.
If present, binder will resolve this bean and assign it on the listener
container.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1140

* Addressing PR review
2021-09-14 10:17:32 -04:00
Soby Chacko
b7ebc185e7 GH-1141: Streams cleanup docs
Clarify streams cleanup default in the docs.
Effective from Spring Kafka 2.7, no cleanup will be performed on shutdown.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1141
2021-09-13 16:35:36 -04:00
Soby Chacko
56e25383f8 Fix wrong property in docs
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1137
2021-09-03 15:38:50 -04:00
Soby Chacko
b4c7c36229 GH-1135: Disable container retries when no DLQ set (#1136)
* GH-1135: Disable container retries when no DLQ set

Disable default container retries when binding retries are enabled
(maxAttempts > 1) and no DLQ set.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1135

* Address PR review comments

* Addressing PR review
2021-09-01 13:29:36 -04:00
Soby Chacko
6eed115cc9 Kafka Streams binder tests cleanup 2021-08-27 20:03:56 -04:00
Soby Chacko
8e6d07cc7b Update Kafka Streams branching docs/tests
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1133
2021-08-27 18:16:42 -04:00
Soby Chacko
adde49aab3 Refactoring Kafka Streams join tests
Coalesce the stream join tests in Kafka Streams binder around the functional model.
Remove duplicating the same join tests using the StreamListener model.
2021-08-25 15:45:38 -04:00
Soby Chacko
e500138486 Consumer/Producer prefix in Kafka Streams binder (#1131)
* Consumer/Producer prefix in Kafka Streams binder

Kafka Streams allows the applications to separate the consumer and producer
properties using consumer/producer prefixes. Add these prefixes automatically
if they are missing from the application.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1065

* Addressing PR review comments
2021-08-25 13:57:42 -04:00
Soby Chacko
970bac41bb Update pom.xml
Make the maven configuraiton consistent with core Spring Cloud Stream
2021-08-24 16:54:19 -04:00
Soby Chacko
afe39bf78a Kafka Streams binding lifecycle changes
Start Kafka Streams bindgings only when they are not running.
Similarly, stop them only if they are running.

Without these guards in the bindings for KStream, KTable and GlobalKTable,
it may cause NPE's due to the backing concurrent collections in KafkaStreamsRegistry
not finding the proper KafkaStreams object, especially when the StreamsBuilderFactory
bean is already stopped through the binder provided manager.
2021-08-24 16:53:23 -04:00
Derek Eskens
c1ad3006e9 Fixing consumer config typo in overview.adoc
ConsumerConfigCustomizer is the correct name of the interface.
2021-08-23 19:33:15 -04:00
Soby Chacko
ba2c3a05c9 GH-1129: Kafka Binder Metrics Improvements
Avoid blocking committed() call in KafkaBinderMetrics in a loop for
each topic partition.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1129
2021-08-23 16:14:45 -04:00
Soby Chacko
739b499966 Honoring auto startup in Kafka Streams binder (#1127)
* Honoring auto startup in Kafka Streams binder

When using Kafka Streams bineder, the processors are started
unconditionally, i.e. autoStartup is always true by default.
If spring.kafka.streams.auto-startup is set, then honor that
as the auto-startup flag in the binder.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1126

* Addressing PR review comments

Auto startup flag is honored per individual consumer bindings as well.

* Addressing PR review comments

* Making KafkaStreamsRegistry#kafkaStreams set to use a concurrent Set.
2021-08-23 14:18:05 -04:00
Soby Chacko
1b26f5d629 GH-1112: Fix OOM in DlqSender.sendToDlq
https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1112
2021-08-20 14:57:15 -04:00
Soby Chacko
99c323e314 Document Kafka Streams and Sleuth integration
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1102
2021-08-16 20:45:08 -04:00
Soby Chacko
f4d3715317 Recycle KafkaStreams Objects
In the event Kafka Streams bindings are restarted (stop/start)
using the actuator bindings endpoints, the underlying KafkaStreams
objects are not recycled. After restarting, it still sees the previous
KafkaStreams object. Addressing this issue.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1119
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1120
2021-08-12 15:47:40 -04:00
Soby Chacko
d7907bbdcc Docs updates for Kakfa Streams skipAndContinue 2021-08-12 14:43:22 -04:00
buildmaster
9b5e735f74 Going back to snapshots 2021-07-30 15:12:01 +00:00
buildmaster
201668542b Update SNAPSHOT to 3.2.0-M1 v3.2.0-M1 2021-07-30 15:10:38 +00:00
Soby Chacko
a5f01f9d6f GH-1110: Kafka Streams state machine changes
Restore Kafka Streams error state behavior in the binder, equivalent
to Kafka Streams prior to 2.8. Starting with 2.8, users can customize
the way uncaught errors are interpreted via an UncaughtExceptionHandler
in the applicaiton. Binder now sets a default handler that shuts down
the Kafka Streams client if thre is an uncaught error.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1110
2021-07-29 13:35:08 -04:00
Soby Chacko
912c47e3ac Fix failing tests in Kafka Streams binder
Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1109
2021-07-28 18:41:22 -04:00
Soby Chacko
001882de4e Upgrade versions
Spring Kafka: 2.8.0-M1
Spring Integration Kafka: 5.5.2
Kafka: 2.8.0

Ignore a few Kafka Streams binder tests temporarily.
2021-07-27 20:01:08 -04:00
Soby Chacko
54ac274ea3 GH-1085: Allow KTable binding on the outbound
At the moment, Kafka Streams binder only allows KStream bindings on the outbound.
There is a delegation mechanism in which we stil can use KStream for output binding
while allowing the applications to provide a KTable type as the function return type.

Update docs.

Resolves https://github.com/spring-cloud/spring-cloud-stream-binder-kafka/issues/1085

Resolves #1105
2021-07-16 09:46:04 +02:00