The Future of pandas

Abstract

There are an increasing number of packages that are built on top of or compatible with pandas. Some of these like sklearn-pandas integrate with other packages like scikit-learn to utilize DataFrames for machine learning. Others, like Plotly, provide interactive plotting capabilities and online collaboration. pandas has recently made the push in the last couple years to branch out into other languages. There is now a pandas.js package and a ruby wrapper that enables ruby users to call into the Python pandas API. There is also a push to optimize data analysis at a more global scale using an up-and-coming LLVM called Weld. It takes an approach similar to NumExpr but on an even larger scale. The idea is to combine all the data analysis operations together lazily and only run them when an actual result is needed. This allows the operations to be optimized for parallel compute and loading of memory on a much grander scale.