Leverage on open source to build up a data product

摘要：

Nowadays, standing on the shoulders of giants, there are more and more open source projects enable people to build up any kind of application, service or product without spending much efforts. In this talk, we will introduce how we construct a fully open source based data product which supports Lambda Architecture from scratch to analyze large volumes of logs. We will explain how to use Fluentd, Kafka, Cassandra and Akka to do data wrangling, how to use Spark to do data analysis and how to use Grafana, InfluxDB and cAdvisor to do system and data flow monitoring. Finally, we will talk about using Docker to Dockerize above components to facilitate composing a standalone or cluster mode data product.

Although leveraging on open source projects could accelerate product development, it still has some side effects. Hence, in the end, we will share some challenges we faced and some lessons learned during developing the data product.

講者簡介：

Darren Chen (Yu-Lun Chen) is Sr. software engineer of Trend Micro. He is enthusiast in big data and cloud computing technologies. Aside engineering stuff, he is also an enthusiast in giving talks to share software innovations and cutting edge technologies.

• ← TenMax 網路廣告的架構以及經驗分享
• Streaming process with Kafka Connect & Kafka Streams →