mindol1004/spring-cloud-stream-samples
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Refactoring samples

Browse Source 
Restructuring the samples repository
Add more docker support
Add acceptance tests for the apps
Adding sensor average processor sample
Remove aggregate samples
This commit is contained in:
Soby Chacko
2018-03-08 16:06:52 -05:00
parent b2cecb08bc
commit e94de0d535
368 changed files with 4428 additions and 3470 deletions
Download Patch File Download Diff File Expand all files Collapse all files

71
README.adoc
View File

@@ -1,44 +1,69 @@
== Samples
== Spring Cloud Stream Sample Applications
There are several samples, most running on the RabbitMQ transport (so you need RabbitMQ running locally to test them).
This repository contains a collection of applications written using Spring Cloud Stream. All the applications are self contained.
They can be run against either Kafka or RabbitMQ middleware technologies.
You have the option of running the samples against local or Docker containerized versions of Kafka and Rabbit.
For convenience, `docker-compose.yml` files are provided as part of each application wherever it is applicable.
For this reason, Docker Compose is required and it’s recommended to use the https://docs.docker.com/compose/install/[latest version].
These compose files bring up the middleware (kafka or Rabbit) and other necessary components for running each app.
If you bring up Kafka or RabbitMQ in Docker containers, please make sure that you bring them down while in the same sample directory.
You can read the README that is part of each sample and follow along the instructions to run them.
To build the samples do:
You can build the entire samples by going to the root of the repository and then do: `./mvnw clean package`
However, the recommended approach to build them is to pick the sample that you are interested in and go to that particular app and follow the instructions there in the README for that app.
```
./mvnw clean build
```
=== Following is the list of various sample applications provided
==== Source samples
* `double` is an example of an aggregate application, the Source and Sink are combined into one single application.
* Sample JDBC source using MySQL - MariaDB variant - (Kafka and Rabbit)
* `dynamic-source` publishes messages to dynamically created destinations.
* Source with dynamic destinations (Kafka and Rabbit)
* `kinesis-produce-consume` An example application using spring-cloud-stream-binder-aws-kinesis. Presents a web endpoint to send Orders, these are placed on a Kinesis stream and then consumed by the application from that stream.
==== Sink samples
* `multi-io` shows how to use configure multiple input/output channels inside a single application.
* Simple JDBC sink using MariaDB (Kafka and Rabbit)
* `multibinder-differentsystems` shows how an application could use same binder implementation but different configurations for its channels. In this case, a processor's input/output channels connect to same binder implementation but with two separate broker configurations.
==== Processor samples
* `multibinder` shows how an application could use multiple binders. In this case, the processor's input/output channels connect to different brokers using their own binder configurations.
* Basic StreamListener sample (Kafka and Rabbit)
* Transformer sample (Kafka and Rabbit)
* Reactive processor sample (Kafka and Rabbit)
* `non-self-contained-aggregate-app` shows how to write a non self-contained aggregate application.
==== Multi IO sample
* `reactive-processor-kafka` shows how to create a reactive Apache Kafka processor application.
* Sample with multiple input/output bindings (Kafka and Rabbit)
* `rxjava-processor` shows how to create an RxJava processor application.
==== Multi Binder samples
* `sink` A simple sink that logs the incoming payload. It has no options (but some could easily be added), and just logs incoming messages at INFO level.
* Multi binder - Input with Kafka and output with Rabbit
* Multi binder - Same binder type but different clusters (Kafka only, but can be extended for Rabbit as well)
* `source` A simple time source example. It has a "fixedDelay" option (in milliseconds) for the period between emitting messages.
==== Kinesis
* `stream-listener` shows how to use StreamListener support to enable message mapping and automatic type conversion.
* Kinesis produce consume sample
* `test-embedded-kafka` is a sample that shows how to test with an embedded Apache Kafka broker.
We generally recommend testing with the http://docs.spring.io/spring-cloud-stream/docs/current/reference/htmlsingle/#_testing[TestSupportBinder] but if you have a need for testing with an embedded broker, you can use the techniques in this sample.
==== Kafka Streams samples
* `transform` is a simple pass through logging transformer (just logs the incoming message and passes it on).
A collection of various applications in stream processing using Spring Cloud Stream support for Kafka Streams binding.
* `kstream` is a collection of applications that demonstrate the capabilities of the Spring Cloud Stream support for Kafka Streams
* Kafka Streams word count
* Kafka Streams branching
* Kafka Streams DLQ
* Kafka Streams aggregation
* Kafka Streams Interactive query basic
* Kafka Streams Interactive query advanced
* Kafka Streams product tracker
* Kafka Streams KTable join
* Kafka Streams and normal Kafka binder together
* `testing` is a bunch of applications and tests for them to demonstrate the capabilities of testing for the the Spring Cloud Stream applications.
==== Testing samples
* Sample with embedded Kafka
* General testing patterns in Spring Cloud Stream
==== Samples Acceptance Tests
This module is strictly used as an end to end acceptance test framework for the samples in this repo.
By default, the tests are not run as part of a normal build.
Please see the README in the acceptance test module for more details.

94
dynamic-destination-source/pom.xml
View File

@@ -1,94 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>dynamic-destination-source</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>dynamic-destination-source</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.0.BUILD-SNAPSHOT</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<spring-cloud.version>Finchley.BUILD-SNAPSHOT</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>http://repo.spring.io/libs-snapshot-local</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>false</enabled>
</releases>
</repository>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>http://repo.spring.io/libs-milestone-local</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
</project>

2
dynamic-destination-source/start-kafka-shell.sh
View File

@@ -1,2 +0,0 @@
#!/bin/bash
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka /bin/bash

115
jdbc-sink/pom.xml
View File

@@ -1,115 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>jdbc-sink</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>sample-jdbc-sink</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.0.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<spring-cloud.version>Finchley.BUILD-SNAPSHOT</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.mariadb.jdbc</groupId>
<artifactId>mariadb-java-client</artifactId>
<version>1.1.9</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>http://repo.spring.io/libs-snapshot-local</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>false</enabled>
</releases>
</repository>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>http://repo.spring.io/libs-milestone-local</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
</project>

2
jdbc-sink/start-kafka-shell.sh
View File

@@ -1,2 +0,0 @@
#!/bin/bash
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka /bin/bash

115
jdbc-source/pom.xml
View File

@@ -1,115 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>jdbc-source</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>sample-jdbc-source</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.0.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<spring-cloud.version>Finchley.BUILD-SNAPSHOT</spring-cloud.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.mariadb.jdbc</groupId>
<artifactId>mariadb-java-client</artifactId>
<version>1.1.9</version>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-test-support</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>http://repo.spring.io/libs-snapshot-local</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
<releases>
<enabled>false</enabled>
</releases>
</repository>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>http://repo.spring.io/libs-milestone-local</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
</repositories>
</project>

2
jdbc-source/start-kafka-shell.sh
View File

@@ -1,2 +0,0 @@
#!/bin/bash
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka /bin/bash

0
kafka-streams/kafka-streams-aggregate/.gitignore → kafka-streams-samples/kafka-streams-aggregate/.gitignore vendored
View File

0
kafka-streams/kafka-streams-aggregate/.mvn/wrapper/maven-wrapper.jar → kafka-streams-samples/kafka-streams-aggregate/.mvn/wrapper/maven-wrapper.jar vendored
View File

0
kafka-streams/kafka-streams-aggregate/.mvn/wrapper/maven-wrapper.properties → kafka-streams-samples/kafka-streams-aggregate/.mvn/wrapper/maven-wrapper.properties vendored
View File

7
kafka-streams/kafka-streams-aggregate/README.adoc → kafka-streams-samples/kafka-streams-aggregate/README.adoc
View File

@@ -6,12 +6,11 @@ The application simply aggregates a string for a particular key.
=== Running the app:
*Make the appropriate changes in application.yml if need be.
Go to the root of the repository and do:
`spring.cloud.stream.kstream.binder.brokers=<Broker IP Address>` +
`spring.cloud.stream.kstream.binder.zkNodes=<Zookeeper IP Address>`
`docker-compose up -d`
Go to the root of the repository and do: `./mvnw clean package`
`./mvnw clean package`
`java -jar target/kafka-streams-aggregate-0.0.1-SNAPSHOT.jar`

2
kafka-streams/kafka-streams-word-count/docker/docker-compose.yml → kafka-streams-samples/kafka-streams-aggregate/docker-compose.yml
View File

@@ -1,4 +1,4 @@
version: '2'
version: '3'
services:
kafka:
image: wurstmeister/kafka

0
kafka-streams/kafka-streams-aggregate/mvnw → kafka-streams-samples/kafka-streams-aggregate/mvnw vendored
View File

0
kafka-streams/kafka-streams-aggregate/mvnw.cmd → kafka-streams-samples/kafka-streams-aggregate/mvnw.cmd vendored
View File

42
kafka-streams-samples/kafka-streams-aggregate/pom.xml Normal file
View File

@@ -0,0 +1,42 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>kafka-streams-aggregate</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>kafka-streams-aggregate</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>spring-cloud-stream-samples-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../..</relativePath>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

30
non-self-contained-aggregate-app/src/main/java/config/processor/ProcessorApplication.java → kafka-streams-samples/kafka-streams-aggregate/src/main/java/kafka/streams/table/join/DomainEvent.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2017 the original author or authors.
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -14,14 +14,30 @@
* limitations under the License.
*/
package config.processor;
import org.springframework.boot.autoconfigure.SpringBootApplication;
package kafka.streams.table.join;
/**
* @author Marius Bogoevici
* @author Soby Chacko
*/
@SpringBootApplication
public class ProcessorApplication {
public class DomainEvent {
String eventType;
String boardUuid;
public String getEventType() {
return eventType;
}
public void setEventType(String eventType) {
this.eventType = eventType;
}
public String getBoardUuid() {
return boardUuid;
}
public void setBoardUuid(String boardUuid) {
this.boardUuid = boardUuid;
}
}

16
kafka-streams/kafka-streams-aggregate/src/main/java/kafka/streams/table/join/KafkaStreamsAggregateSample.java → kafka-streams-samples/kafka-streams-aggregate/src/main/java/kafka/streams/table/join/KafkaStreamsAggregateSample.java
View File

@@ -1,3 +1,19 @@
/*
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package kafka.streams.table.join;
import com.fasterxml.jackson.databind.ObjectMapper;

16
kafka-streams/kafka-streams-aggregate/src/main/java/kafka/streams/table/join/Producers.java → kafka-streams-samples/kafka-streams-aggregate/src/main/java/kafka/streams/table/join/Producers.java
View File

@@ -1,3 +1,19 @@
/*
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package kafka.streams.table.join;
import com.fasterxml.jackson.databind.ObjectMapper;

0
kafka-streams/kafka-streams-aggregate/src/main/resources/application.yml → kafka-streams-samples/kafka-streams-aggregate/src/main/resources/application.yml
View File

0
kafka-streams/kafka-streams-aggregate/src/main/resources/logback.xml → kafka-streams-samples/kafka-streams-aggregate/src/main/resources/logback.xml
View File

0
kafka-streams/kafka-streams-aggregate/src/test/java/kafka/streams/table/join/KafkaStreamsAggregateSampleTests.java → kafka-streams-samples/kafka-streams-aggregate/src/test/java/kafka/streams/table/join/KafkaStreamsAggregateSampleTests.java
View File

0
kafka-streams/kafka-streams-branching/.gitignore → kafka-streams-samples/kafka-streams-branching/.gitignore vendored
View File

0
kafka-streams/kafka-streams-branching/.mvn/wrapper/maven-wrapper.jar → kafka-streams-samples/kafka-streams-branching/.mvn/wrapper/maven-wrapper.jar vendored
View File

0
kafka-streams/kafka-streams-branching/.mvn/wrapper/maven-wrapper.properties → kafka-streams-samples/kafka-streams-branching/.mvn/wrapper/maven-wrapper.properties vendored
View File

39
kafka-streams-samples/kafka-streams-branching/README.adoc Normal file
View File

@@ -0,0 +1,39 @@
== What is this app?
This is an example of a Spring Cloud Stream processor using Kafka Streams branching support.
The example is based on the word count application from the https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/main/java/io/confluent/examples/streams/WordCountLambdaExample.java[reference documentation].
It uses a single input and 3 output destinations.
In essence, the application receives text messages from an input topic, filter them by language (Englihs, French, Spanish and ignoring the rest), and computes word occurrence counts in a configurable time window and report that in the output topics.
This sample uses lambda expressions and thus requires Java 8+.
By default native decoding and encoding are disabled and this means that any deserializaion on inbound and serialization on outbound is performed by the Binder using the configured content types.
=== Running the app:
Go to the root of the repository and do:
`docker-compose up -d`
`./mvnw clean package`
`java -jar target/kafka-streams-branching-0.0.1-SNAPSHOT.jar --spring.cloud.stream.kafka.streams.timeWindow.length=60000`
Issue the following commands:
`docker exec -it kafka-branch /opt/kafka/bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic words`
On another terminal:
`docker exec -it kafka-branch /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic english-counts`
On another terminal:
`docker exec -it kafka-branch /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic french-counts`
On another terminal:
`docker exec -it kafka-branch /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic spanish-counts`
Enter text ("English", "French", "Spanish" - case doesn't matter) in the console producer and watch the output in the respective console consumer.
The word "english" goes to topic english-counts, "french" goes to topic french-counts and "spanish" goes to spanish-counts.

19
kafka-streams-samples/kafka-streams-branching/docker-compose.yml Normal file
View File

@@ -0,0 +1,19 @@
version: '3'
services:
kafka:
image: wurstmeister/kafka
container_name: kafka-branch
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
- KAFKA_ADVERTISED_HOST_NAME=zookeeper

0
kafka-streams/kafka-streams-branching/mvnw → kafka-streams-samples/kafka-streams-branching/mvnw vendored
View File

0
kafka-streams/kafka-streams-branching/mvnw.cmd → kafka-streams-samples/kafka-streams-branching/mvnw.cmd vendored
View File

40
kafka-streams-samples/kafka-streams-branching/pom.xml Normal file
View File

@@ -0,0 +1,40 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>kafka-streams-branching</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>kafka-streams-branching</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>spring-cloud-stream-samples-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../..</relativePath>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

16
kafka-streams/kafka-streams-branching/src/main/java/kafka/streams/branching/KafkaStreamsBranchingSample.java → kafka-streams-samples/kafka-streams-branching/src/main/java/kafka/streams/branching/KafkaStreamsBranchingSample.java
View File

@@ -1,3 +1,19 @@
/*
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package kafka.streams.branching;
import org.apache.kafka.streams.KeyValue;

0
kafka-streams/kafka-streams-branching/src/main/resources/application.yml → kafka-streams-samples/kafka-streams-branching/src/main/resources/application.yml
View File

0
kafka-streams/kafka-streams-branching/src/main/resources/logback.xml → kafka-streams-samples/kafka-streams-branching/src/main/resources/logback.xml
View File

19
self-contained-aggregate-app/src/test/java/demo/ModuleApplicationTests.java → kafka-streams-samples/kafka-streams-branching/src/test/java/kafka/streams/branching/KafkaStreamsBranchingSampleTests.java
View File

@@ -1,5 +1,5 @@
/*
* Copyright 2015 the original author or authors.
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
@@ -14,23 +14,20 @@
* limitations under the License.
*/
package demo;
package kafka.streams.branching;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.test.annotation.DirtiesContext;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.web.WebAppConfiguration;
import org.springframework.test.context.junit4.SpringRunner;
@RunWith(SpringJUnit4ClassRunner.class)
@SpringBootTest(classes = AggregateApplication.class)
@WebAppConfiguration
@DirtiesContext
public class ModuleApplicationTests {
@RunWith(SpringRunner.class)
@SpringBootTest
public class KafkaStreamsBranchingSampleTests {
@Test
@Ignore
public void contextLoads() {
}

0
kafka-streams/kafka-streams-dlq-sample/.gitignore → kafka-streams-samples/kafka-streams-dlq-sample/.gitignore vendored
View File

0
kafka-streams/kafka-streams-dlq-sample/.mvn/wrapper/maven-wrapper.jar → kafka-streams-samples/kafka-streams-dlq-sample/.mvn/wrapper/maven-wrapper.jar vendored
View File

0
kafka-streams/kafka-streams-dlq-sample/.mvn/wrapper/maven-wrapper.properties → kafka-streams-samples/kafka-streams-dlq-sample/.mvn/wrapper/maven-wrapper.properties vendored
View File

41
kafka-streams-samples/kafka-streams-dlq-sample/README.adoc Normal file
View File

@@ -0,0 +1,41 @@
== What is this app?
This is an example of a Spring Cloud Stream processor using Kafka Streams support.
This is a demonstration of deserialization errors and DLQ in Kafka Streams binder.
The example is based on the word count application from the https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/main/java/io/confluent/examples/streams/WordCountLambdaExample.java[reference documentation].
It uses a single input and a single output.
In essence, the application receives text messages from an input topic and computes word occurrence counts in a configurable time window and report that in an output topic.
This sample uses lambda expressions and thus requires Java 8+.
=== Running the app:
`docker-compose up -d`
Go to the root of the repository and do: `./mvnw clean package`
`java -jar target/kstream-word-count-0.0.1-SNAPSHOT.jar`
The default application.yml file demonstrates native decoding by Kafka.
The default value serializer is set to IntegerSerde to force a deserialization errors.
Issue the following commands:
`docker exec -it kafka-dlq /opt/kafka/bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic words`
On another terminal:
`docker exec -it kafka-dlq /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic counts`
On another terminal:
`docker exec -it kafka-dlq /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic words-count-dlq`
On the console producer, enter some text data.
You will see that the messages produce deserialization errors and end up in the DLQ topic - words-count-dlq.
You will not see any messages coming to the regular destination counts.
There is another yaml file provided (by-framework-decoding.yml).
Use that as application.yml to see how it works when the deserialization done by the framework.
In this case also, the messages on error appear in the DLQ topic.

19
kafka-streams-samples/kafka-streams-dlq-sample/docker-compose.yml Normal file
View File

@@ -0,0 +1,19 @@
version: '3'
services:
kafka:
image: wurstmeister/kafka
container_name: kafka-dlq
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
- KAFKA_ADVERTISED_HOST_NAME=zookeeper

0
kafka-streams/kafka-streams-dlq-sample/mvnw → kafka-streams-samples/kafka-streams-dlq-sample/mvnw vendored
View File

0
kafka-streams/kafka-streams-dlq-sample/mvnw.cmd → kafka-streams-samples/kafka-streams-dlq-sample/mvnw.cmd vendored
View File

42
kafka-streams-samples/kafka-streams-dlq-sample/pom.xml Normal file
View File

@@ -0,0 +1,42 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>kafka-streams-dlq-sample</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>kafka-streams-dlq-sample</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>spring-cloud-stream-samples-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../..</relativePath>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

16
kafka-streams/kafka-streams-dlq-sample/src/main/java/kafka/streams/dlq/sample/KafkaStreamsDlqSample.java → kafka-streams-samples/kafka-streams-dlq-sample/src/main/java/kafka/streams/dlq/sample/KafkaStreamsDlqSample.java
View File

@@ -1,3 +1,19 @@
/*
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package kafka.streams.dlq.sample;
import org.apache.kafka.common.serialization.Serdes;

0
kafka-streams/kafka-streams-dlq-sample/src/main/resources/application.yml → kafka-streams-samples/kafka-streams-dlq-sample/src/main/resources/application.yml
View File

0
kafka-streams/kafka-streams-dlq-sample/src/main/resources/by-framework-decoding.yml → kafka-streams-samples/kafka-streams-dlq-sample/src/main/resources/by-framework-decoding.yml
View File

0
kafka-streams/kafka-streams-dlq-sample/src/main/resources/logback.xml → kafka-streams-samples/kafka-streams-dlq-sample/src/main/resources/logback.xml
View File

0
kafka-streams/kafka-streams-dlq-sample/src/test/java/kafka/streams/dlq/sample/KafkaStreamsDlqExampleTests.java → kafka-streams-samples/kafka-streams-dlq-sample/src/test/java/kafka/streams/dlq/sample/KafkaStreamsDlqExampleTests.java
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/.gitignore → kafka-streams-samples/kafka-streams-interactive-query-advanced/.gitignore vendored
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/.mvn/wrapper/maven-wrapper.jar → kafka-streams-samples/kafka-streams-interactive-query-advanced/.mvn/wrapper/maven-wrapper.jar vendored
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/.mvn/wrapper/maven-wrapper.properties → kafka-streams-samples/kafka-streams-interactive-query-advanced/.mvn/wrapper/maven-wrapper.properties vendored
View File

26
kafka-streams-samples/kafka-streams-interactive-query-advanced/README.adoc Normal file
View File

@@ -0,0 +1,26 @@
== What is this app?
This is an example of a Spring Cloud Stream processor using Kafka Streams support.
This example is a Spring Cloud Stream adaptation of this Kafka Streams sample: https://github.com/confluentinc/kafka-streams-examples/tree/4.0.0-post/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic
This sample demonstrates the concept of interactive queries in kafka streams.
There is a REST service provided as part of the application that can be used to query the store interactively.
=== Running the app:
1. `docker-compose up -d`
2. Start the confluent schema registry: The following command is based on the confluent platform.
`./bin/schema-registry-start ./etc/schema-registry/schema-registry.properties`
3. Go to the root of the repository and do: `./mvnw clean package`
4. `java -jar target/kafka-streams-interactive-query-0.0.1-SNAPSHOT.jar`
5. Run the stand-alone `Producers` application to generate data and start the processing.
Keep it running for a while.
6. Go to the URL: http://localhost:8080/charts/top-five?genre=Punk
keep refreshing the URL and you will see the song play count information changes.

19
kafka-streams-samples/kafka-streams-interactive-query-advanced/docker-compose.yml Normal file
View File

@@ -0,0 +1,19 @@
version: '3'
services:
kafka:
image: wurstmeister/kafka
container_name: kafka-iq-advanced
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
- KAFKA_ADVERTISED_HOST_NAME=zookeeper

0
kafka-streams/kafka-streams-interactive-query-advanced/mvnw → kafka-streams-samples/kafka-streams-interactive-query-advanced/mvnw vendored
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/mvnw.cmd → kafka-streams-samples/kafka-streams-interactive-query-advanced/mvnw.cmd vendored
View File

67
kafka-streams/kafka-streams-interactive-query-advanced/pom.xml → kafka-streams-samples/kafka-streams-interactive-query-advanced/pom.xml
View File

@@ -3,26 +3,21 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>kafka.streams.interactive.query</groupId>
<artifactId>kafka-streams-interactive-query-advanced</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>kafka-streams-interactive-query-advanced</name>
<description>Demo project for Spring Boot</description>
<description>Spring Cloud Stream sample for KStream interactive queries</description>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.0.BUILD-SNAPSHOT</version>
<relativePath/> <!-- lookup parent from repository -->
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>spring-cloud-stream-samples-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../..</relativePath>
</parent>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<java.version>1.8</java.version>
<spring-cloud.version>Finchley.BUILD-SNAPSHOT</spring-cloud.version>
<confluent.version>4.0.0</confluent.version>
<avro.version>1.8.2</avro.version>
</properties>
@@ -49,26 +44,10 @@
<artifactId>kafka-schema-registry-client</artifactId>
<version>${confluent.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
<version>2.1.3.BUILD-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
<version>2.0.0.BUILD-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
@@ -76,24 +55,8 @@
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
@@ -112,27 +75,15 @@
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
<repositories>
<repository>
<id>spring-snapshots</id>
<name>Spring Snapshots</name>
<url>http://repo.spring.io/libs-snapshot-local</url>
<snapshots>
<enabled>true</enabled>
</snapshots>
</repository>
<repository>
<id>spring-milestones</id>
<name>Spring Milestones</name>
<url>http://repo.spring.io/libs-milestone-local</url>
<snapshots>
<enabled>false</enabled>
</snapshots>
</repository>
<repository>
<id>confluent</id>
<url>http://packages.confluent.io/maven/</url>

16
kafka-streams/kafka-streams-interactive-query-advanced/src/main/java/kafka/streams/interactive/query/KafkaStreamsInteractiveQuerySample.java → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/main/java/kafka/streams/interactive/query/KafkaStreamsInteractiveQuerySample.java
View File

@@ -1,3 +1,19 @@
/*
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package kafka.streams.interactive.query;
import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig;

16
kafka-streams/kafka-streams-interactive-query-advanced/src/main/java/kafka/streams/interactive/query/Producers.java → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/main/java/kafka/streams/interactive/query/Producers.java
View File

@@ -1,3 +1,19 @@
/*
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package kafka.streams.interactive.query;
import io.confluent.kafka.serializers.AbstractKafkaAvroSerDeConfig;

16
kafka-streams/kafka-streams-interactive-query-advanced/src/main/java/kafka/streams/interactive/query/SongPlayCountBean.java → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/main/java/kafka/streams/interactive/query/SongPlayCountBean.java
View File

@@ -1,3 +1,19 @@
/*
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package kafka.streams.interactive.query;
import java.util.Objects;

0
kafka-streams/kafka-streams-interactive-query-advanced/src/main/resources/application.yml → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/main/resources/application.yml
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/src/main/resources/avro/kafka/streams/interactive/query/playevent.avsc → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/main/resources/avro/kafka/streams/interactive/query/playevent.avsc
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/src/main/resources/avro/kafka/streams/interactive/query/song.avsc → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/main/resources/avro/kafka/streams/interactive/query/song.avsc
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/src/main/resources/avro/kafka/streams/interactive/query/songplaycount.avsc → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/main/resources/avro/kafka/streams/interactive/query/songplaycount.avsc
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/src/main/resources/logback.xml → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/main/resources/logback.xml
View File

0
kafka-streams/kafka-streams-interactive-query-advanced/src/test/java/kafka/streams/interactive/query/KafkaStreamsInteractiveQueryTests.java → kafka-streams-samples/kafka-streams-interactive-query-advanced/src/test/java/kafka/streams/interactive/query/KafkaStreamsInteractiveQueryTests.java
View File

0
kafka-streams/kafka-streams-interactive-query-basic/.jdk8 → kafka-streams-samples/kafka-streams-interactive-query-basic/.jdk8
View File

0
kafka-streams/kafka-streams-interactive-query-basic/.mvn/wrapper/maven-wrapper.jar → kafka-streams-samples/kafka-streams-interactive-query-basic/.mvn/wrapper/maven-wrapper.jar vendored
View File

0
kafka-streams/kafka-streams-interactive-query-basic/.mvn/wrapper/maven-wrapper.properties → kafka-streams-samples/kafka-streams-interactive-query-basic/.mvn/wrapper/maven-wrapper.properties vendored
View File

35
kafka-streams-samples/kafka-streams-interactive-query-basic/README.adoc Normal file
View File

@@ -0,0 +1,35 @@
== What is this app?
This is an example of a Spring Cloud Stream processor using Kafka Streams support.
The example is based on a contrived use case of tracking products by interactively querying their status.
The program accepts product ID's and track their counts hitherto by interactively querying the underlying store. \
This sample uses lambda expressions and thus requires Java 8+.
=== Running the app:
Go to the root of the repository and do:
`docker-compose up -d`
`./mvnw clean package`
`java -jar target/kafka-streams-interactive-query-basic-0.0.1-SNAPSHOT.jar --app.product.tracker.productIds=123,124,125`
The above command will track products with ID's 123,124 and 125 and print their counts seen so far every 30 seconds.
Issue the following commands:
`docker exec -it kafka-iq-basic /opt/kafka/bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic products`
Enter the following in the console producer (one line at a time) and watch the output on the console (or IDE) where the application is running.
```
{"id":"123"}
{"id":"124"}
{"id":"125"}
{"id":"123"}
{"id":"123"}
{"id":"123"}
```

19
kafka-streams-samples/kafka-streams-interactive-query-basic/docker-compose.yml Normal file
View File

@@ -0,0 +1,19 @@
version: '3'
services:
kafka:
image: wurstmeister/kafka
container_name: kafka-iq-basic
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
- KAFKA_ADVERTISED_HOST_NAME=zookeeper

0
kafka-streams/kafka-streams-interactive-query-basic/mvnw → kafka-streams-samples/kafka-streams-interactive-query-basic/mvnw vendored
View File

41
kafka-streams-samples/kafka-streams-interactive-query-basic/pom.xml Normal file
View File

@@ -0,0 +1,41 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>kafka-streams-interactive-query-basic</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>kafka-streams-interactive-query-basic</name>
<description>Spring Cloud Stream sample for KStream interactive queries</description>
<parent>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>spring-cloud-stream-samples-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../..</relativePath>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

0
kafka-streams/kafka-streams-interactive-query-basic/src/main/java/kafka/streams/product/tracker/KafkaStreamsInteractiveQueryApplication.java → kafka-streams-samples/kafka-streams-interactive-query-basic/src/main/java/kafka/streams/product/tracker/KafkaStreamsInteractiveQueryApplication.java
View File

0
kafka-streams/kafka-streams-interactive-query-basic/src/main/resources/application.yml → kafka-streams-samples/kafka-streams-interactive-query-basic/src/main/resources/application.yml
View File

0
kafka-streams/kafka-streams-interactive-query-basic/src/main/resources/logback.xml → kafka-streams-samples/kafka-streams-interactive-query-basic/src/main/resources/logback.xml
View File

0
kafka-streams/kafka-streams-message-channel/.gitignore → kafka-streams-samples/kafka-streams-message-channel/.gitignore vendored
View File

0
kafka-streams/kafka-streams-message-channel/.mvn/wrapper/maven-wrapper.jar → kafka-streams-samples/kafka-streams-message-channel/.mvn/wrapper/maven-wrapper.jar vendored
View File

0
kafka-streams/kafka-streams-message-channel/.mvn/wrapper/maven-wrapper.properties → kafka-streams-samples/kafka-streams-message-channel/.mvn/wrapper/maven-wrapper.properties vendored
View File

39
kafka-streams-samples/kafka-streams-message-channel/README.adoc Normal file
View File

@@ -0,0 +1,39 @@
== What is this app?
This is an example of a Spring Cloud Stream processor using Kafka Streams support.
The example is based on the word count application from the https://github.com/confluentinc/examples/blob/3.2.x/kafka-streams/src/main/java/io/confluent/examples/streams/WordCountLambdaExample.java[reference documentation].
The application has two inputs one as a KStream and another one with the binding for a message channel.
=== Running the app:
Go to the root of the repository and do:
`docker-compose up -d`
`./mvnw clean package`
`java -jar target/kafka-streams-message-channel-0.0.1-SNAPSHOT.jar --spring.cloud.stream.kafka.streams.timeWindow.length=60000`
Issue the following commands:
`docker exec -it kafka-mc /opt/kafka/bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic words`
On another terminal:
`docker exec -it kafka-mc /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic counts`
Enter some text in the console producer and watch the output in the console consumer.
Also watch the console for logging statements from the regular sink StreamListener method.
The default time window is configured for 5 seconds and you can change that using the following property.
`spring.cloud.stream.kafka.streams.timeWindow.length` (value is expressed in milliseconds)
In order to switch to a hopping window, you can use the `spring.cloud.stream.kafka.streams.timeWindow.advanceBy` (value in milliseconds).
This will create an overlapped hopping windows depending on the value you provide.
Here is an example with 2 overlapping windows (window length of 10 seconds and a hop (advance) by 5 seconds:
`java -jar target/kafka-streams-message-channel-0.0.1-SNAPSHOT.jar --spring.cloud.stream.kafka.streams.timeWindow.length=10000 --spring.cloud.stream.kafka.streams.timeWindow.advanceBy=5000`

19
kafka-streams-samples/kafka-streams-message-channel/docker-compose.yml Normal file
View File

@@ -0,0 +1,19 @@
version: '3'
services:
kafka:
image: wurstmeister/kafka
container_name: kafka-mc
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
- KAFKA_ADVERTISED_HOST_NAME=zookeeper

0
kafka-streams/kafka-streams-message-channel/mvnw → kafka-streams-samples/kafka-streams-message-channel/mvnw vendored
View File

0
kafka-streams/kafka-streams-message-channel/mvnw.cmd → kafka-streams-samples/kafka-streams-message-channel/mvnw.cmd vendored
View File

46
kafka-streams-samples/kafka-streams-message-channel/pom.xml Normal file
View File

@@ -0,0 +1,46 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>kafka-streams-message-channel</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>kafka-streams-message-channel</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>spring-cloud-stream-samples-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../..</relativePath>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

16
kafka-streams/kafka-streams-message-channel/src/main/java/kafka/streams/message/channel/KafkaStreamsWordCountApplication.java → kafka-streams-samples/kafka-streams-message-channel/src/main/java/kafka/streams/message/channel/KafkaStreamsWordCountApplication.java
View File

@@ -1,3 +1,19 @@
/*
* Copyright 2018 the original author or authors.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package kafka.streams.message.channel;
import org.apache.kafka.common.serialization.Serdes;

0
kafka-streams/kafka-streams-message-channel/src/main/resources/application.yml → kafka-streams-samples/kafka-streams-message-channel/src/main/resources/application.yml
View File

0
kafka-streams/kafka-streams-message-channel/src/main/resources/logback.xml → kafka-streams-samples/kafka-streams-message-channel/src/main/resources/logback.xml
View File

0
kafka-streams/kafka-streams-message-channel/src/test/java/kafka/streams/message/channel/KafkaStreamsWordCountApplicationTests.java → kafka-streams-samples/kafka-streams-message-channel/src/test/java/kafka/streams/message/channel/KafkaStreamsWordCountApplicationTests.java
View File

0
kafka-streams/kafka-streams-product-tracker/.jdk8 → kafka-streams-samples/kafka-streams-product-tracker/.jdk8
View File

0
kafka-streams/kafka-streams-product-tracker/.mvn/wrapper/maven-wrapper.jar → kafka-streams-samples/kafka-streams-product-tracker/.mvn/wrapper/maven-wrapper.jar vendored
View File

0
kafka-streams/kafka-streams-product-tracker/.mvn/wrapper/maven-wrapper.properties → kafka-streams-samples/kafka-streams-product-tracker/.mvn/wrapper/maven-wrapper.properties vendored
View File

25
kafka-streams/kafka-streams-product-tracker/README.adoc → kafka-streams-samples/kafka-streams-product-tracker/README.adoc
View File

@@ -7,24 +7,12 @@ Although contrived, this type of use cases are pretty common in the industry whe
In essence, the application receives product information from input topic and count the interested products in a configurable time window and report that in an output topic.
This sample uses lambda expressions and thus requires Java 8+.
==== Starting Kafka in a docker container
* Skip steps 1-3 if you already have a non-Docker Kafka environment.
1. Go to the docker directory in this repo and invoke the command `docker-compose up -d`.
2. Ensure that in the docker directory and then invoke the script `start-kafka-shell.sh`
3. cd $KAFKA_HOME
4. Start the console producer: +
Assuming that you are running kafka on a docker container on mac osx. Change the zookeeper IP address accordingly otherwise. +
`bin/kafka-console-producer.sh --broker-list 192.168.99.100:9092 --topic products`
5. Start the console consumer: +
Assuming that you are running kafka on a docker container on mac osx. Change the zookeeper IP address accordingly otherwise. +
`bin/kafka-console-consumer.sh --bootstrap-server 192.168.99.100:9092 --key-deserializer org.apache.kafka.common.serialization.IntegerDeserializer --property print.key=true --topic product-counts`
=== Running the app:
Go to the root of the repository and do:
`docker-compose up -d`
`./mvnw clean package`
`java -jar target/kafka-streams-product-tracker-0.0.1-SNAPSHOT.jar --app.product.tracker.productIds=123,124,125 --spring.cloud.stream.kafka.streams.timeWindow.length=60000 --spring.cloud.stream.kafka.streams.timeWindow.advanceBy=30000`
@@ -32,12 +20,13 @@ Go to the root of the repository and do:
The above command will track products with ID's 123,124 and 125 every 30 seconds with the counts from the last minute.
In other words, every 30 seconds a new 1 minute window is started.
Issue the following commands:
* By default we use the docker container IP (mac osx specific) in the `application.yml` for Kafka broker and zookeeper.
Change it in `application.yml` (which requires a rebuild) or pass them as runtime arguments as below.
`docker exec -it kafka-prod-tracker /opt/kafka/bin/kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic products`
`spring.cloud.stream.kstream.binder.brokers=<Broker IP Address>` +
`spring.cloud.stream.kstream.binder.zkNodes=<Zookeeper IP Address>`
On another terminal:
`docker exec -it kafka-prod-tracker /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --key-deserializer org.apache.kafka.common.serialization.IntegerDeserializer --property print.key=true --topic product-counts`
Enter the following in the console producer (one line at a time) and watch the output on the console consumer:

19
kafka-streams-samples/kafka-streams-product-tracker/docker-compose.yml Normal file
View File

@@ -0,0 +1,19 @@
version: '3'
services:
kafka:
image: wurstmeister/kafka
container_name: kafka-prod-tracker
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
- KAFKA_ADVERTISED_HOST_NAME=zookeeper

0
kafka-streams/kafka-streams-product-tracker/mvnw → kafka-streams-samples/kafka-streams-product-tracker/mvnw vendored
View File

42
kafka-streams-samples/kafka-streams-product-tracker/pom.xml Normal file
View File

@@ -0,0 +1,42 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>kafka-streams-product-tracker</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>kafka-streams-product-tracker</name>
<description>Demo project for Spring Boot</description>
<parent>
<groupId>spring.cloud.stream.samples</groupId>
<artifactId>spring-cloud-stream-samples-parent</artifactId>
<version>0.0.1-SNAPSHOT</version>
<relativePath>../..</relativePath>
</parent>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</build>
</project>

0
kafka-streams/kafka-streams-product-tracker/src/main/java/kafka/streams/product/tracker/KafkaStreamsProductTrackerApplication.java → kafka-streams-samples/kafka-streams-product-tracker/src/main/java/kafka/streams/product/tracker/KafkaStreamsProductTrackerApplication.java
View File

0
kafka-streams/kafka-streams-product-tracker/src/main/resources/application.yml → kafka-streams-samples/kafka-streams-product-tracker/src/main/resources/application.yml
View File

0
kafka-streams/kafka-streams-product-tracker/src/main/resources/logback.xml → kafka-streams-samples/kafka-streams-product-tracker/src/main/resources/logback.xml
View File

0
kafka-streams/kafka-streams-table-join/.gitignore → kafka-streams-samples/kafka-streams-table-join/.gitignore vendored
View File

0
kafka-streams/kafka-streams-table-join/.mvn/wrapper/maven-wrapper.jar → kafka-streams-samples/kafka-streams-table-join/.mvn/wrapper/maven-wrapper.jar vendored
View File

0
kafka-streams/kafka-streams-table-join/.mvn/wrapper/maven-wrapper.properties → kafka-streams-samples/kafka-streams-table-join/.mvn/wrapper/maven-wrapper.properties vendored
View File

22
kafka-streams-samples/kafka-streams-table-join/README.adoc Normal file
View File

@@ -0,0 +1,22 @@
== What is this app?
This is an example of a Spring Cloud Stream processor using Kafka Streams support.
This example is a Spring Cloud Stream adaptation of this Kafka Streams sample: https://github.com/confluentinc/kafka-streams-examples/blob/4.0.0-post/src/test/java/io/confluent/examples/streams/StreamToTableJoinIntegrationTest.java
The application uses two inputs - one KStream for user-clicks and a KTable for user-regions.
Then it joins the information from stream to table to find out total clicks per region.
=== Running the app:
Go to the root of the repository.
`docker-compose up -d`
`./mvnw clean package`
`java -jar target/kafka-streams-table-join-0.0.1-SNAPSHOT.jar`
`docker exec -it kafka-join /opt/kafka/bin/kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic output-topic --key-deserializer org.apache.kafka.common.serialization.StringDeserializer --value-deserializer org.apache.kafka.common.serialization.LongDeserializer --property print.key=true --property key.separator="-"`
Run the stand-alone `Producers` application to generate some data and watch the output on the console consumer above.

19
kafka-streams-samples/kafka-streams-table-join/docker-compose.yml Normal file
View File

@@ -0,0 +1,19 @@
version: '3'
services:
kafka:
image: wurstmeister/kafka
container_name: kafka-join
ports:
- "9092:9092"
environment:
- KAFKA_ADVERTISED_HOST_NAME=127.0.0.1
- KAFKA_ADVERTISED_PORT=9092
- KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
depends_on:
- zookeeper
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
environment:
- KAFKA_ADVERTISED_HOST_NAME=zookeeper

Some files were not shown because too many files have changed in this diff Show More