This is an error I encountered when I have a Spark Streaming job running on Databricks 6.1. Consider the case I have to update a running streaming query. Databricks recommends to always start (and restart too?) a streaming query on a new dedicated cluster. However, in some scenario you …
read moreNew Work on atacmonitor.com
My side project atacmonitor features a new guise. Data is now being collected for all bus and tram lines in Rome. Data pull is achieved via Python functions running on AWS Lambda. Data is then stored in MongoDB hosted in MongoDB Atlas. Atlas also provides the charts in the page …
read moreThe Pragmatic Programmer [Highlights]
read moreRather than construction, software is more like gardening— it is more organic than concrete. You plant many things in a garden according to an initial plan and conditions. Some thrive, others are destined to end up as compost. [...] You constantly monitor the health of the garden, and make adjustments (to …
6 Take-Aways after Reading "The Signal and The Noise"
The Signal and The Noise by Nate Silver is a must-read book for those interested in predictions. It is not a technical book. You will not learn any algorithm. However, it presents a series of real-world scenarios when predictions did work and where predictions did not work. The book is …
read moreMy Talk about Superset [Python Milano Meetup]
Yesterday, I gave a talk Python Milano Meetup. The Meetup was designed as Python pills: three 20-minutes talks in a row. The talks:
- Superset: data visualization at AirBnB - Marco Santoni
- Java Vs Python - Cesare Placanica
- pdb in action - Lorenzo Mele
read moreVery nice talk of @Airbnb #Superset with @MrSantoni at #PythonMilano …
Manufacturing. When data is not a commodity
What does it mean to work as a data scientist in manufacturing? What is the value behind data? Data science has gained popularity in domains like internet, but the industrial production domain has specific requirements.
I gave a talk at Data Driven Innovation about the specific challenges when doing data …
read moreWeighted Random Sampling with PostgreSQL [Follow-up]
I received valuable feedbacks by Jim Nasby regarding the post about weighted random sampling with PostgreSQL. I will report here Jim's email.
Sadly, Common Table Expressions (CTE)s are insanely expensive, because each one must be fully materialized. So in your example, you're essentially creating 5 temp tables (one for …
read moreMonitoring Bus Frequencies in Rome
I have just launched atacmonitor. It is a website providing information about the waiting time at bus stops in Rome.
Overview
The datasource is live data about bus waiting time of ATAC, Rome's public transport company. The transport office provides public API with real-time data.
I have implemented a simple …
read moreBlog Migrated to Pelican on GitHub Pages
I have migrated my blog. It is built under Pelican, a static site generator. It allows me to write posts as plain markdown or even Jupyter notebooks. I then use GitHub Pages to version and publish the blog. I am continuing to use Aruba as domain provider. It is sufficient …
read moreInsights from IEEE Big Data 16
I have attended the IEEE Big Data 16 conference in Washington DC. I thank my company for sponsoring the trip. The conference included a special symposium dedicated to manufacturing. The symposium hosted some participants of the Bosch Production Line Performance competition from Kaggle.
read more2016 IEEE International Conference on Big Data …