Connecting 7 databases in 15 minutes using Kafka

This talk tries to solve the problem of companies that have different databases in the organisation without a way of consuming all the data in real time. The talk will highlight the art of the possible with bringing the data together with just configurations and enriching it in real-time to create business events.

The talk will start with giving audiences a crash course into the open source distributed messaging system aka Kafka and it’s friends aka Zookeeper, Schema Registry ,KSQL DB and connect. The crash course will cover the basics of all the technologies and how they talk to each other.

This will be followed by a Live demo of deployment of Kafka on AWS using open source Confluent. The live demo will then onboard 7 different databases in real time onto Kafka using Connect.

At the end of the talk the audience will see live data streaming form 6 different databases i.e SQL, Oracle, MongoDB, Amazon Kinesis, SQLite3, csv files or Cassandra (audiences choice) into Kafka. The data will be combined from all the 6 sources and will be synced to Amazon S3. This will include creating small business events from the sourced data using KSQL DB.

Hence, proving audience complete end to end processing of data in real time.