Leverage on open source to build up a data product

講者:陳煜倫 / Sr. Engineer @ 趨勢科技
地點:1101 會議廳 (11F)
講題:Leverage on open source to build up a data product


Nowadays, standing on the shoulders of giants, there are more and more open source projects enable people to build up any kind of application, service or product without spending much efforts. In this talk, we will introduce how we construct a fully open source based data product which supports Lambda Architecture from scratch to analyze large volumes of logs. We will explain how to use Fluentd, Kafka, Cassandra and Akka to do data wrangling, how to use Spark to do data analysis and how to use Grafana, InfluxDB and cAdvisor to do system and data flow monitoring. Finally, we will talk about using Docker to Dockerize above components to facilitate composing a standalone or cluster mode data product.

Although leveraging on open source projects could accelerate product development, it still has some side effects. Hence, in the end, we will share some challenges we faced and some lessons learned during developing the data product.


Darren Chen (Yu-Lun Chen) is Sr. software engineer of Trend Micro. He is enthusiast in big data and cloud computing technologies. Aside engineering stuff, he is also an enthusiast in giving talks to share software innovations and cutting edge technologies.

Tagged on: