I gave a presentation recently on Real-time streaming and data pipelines with Apache Kafka.
A correction in the talk (~ 22 minutes in) : I said that you have to have all your topic data fit on one server. That is not true, you can’t span logs so you have to have all of your data for a partition fit on one server. Kafka will spread your partitions around for you within topics.
For that presentation I put together sample code for producing and consuming with an Apache Kafka broker using Scala.
To get up and running, use vagrant.
Your entry point is the test file
On the producer side I have started to look more into using Akka. The prototype for this implementation is in the test case above…
View original post 44 more words