pyarrow

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

The flexibility of Python with the scale and performance of modern SQL.

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Exploring Chicago crimes dataset with Jupyter notebooks, DuckDB, Malloy and new Panel/PyScript data and dashboard tools.

Converts a whole subdirectory with a big (or small) volume of PDF documents to a dataset (pandas DataFrame) with error tracking and choice of features

(PoC) A very memory-efficient way to read data from PostgreSQL

A web application for viewing Apache Parquet files . This is a Python + Flask application

Poor mans simple python api for creating a local or remote datalake based on several (pyarrow) datasets using duckdb

highspeed timeseries pandas dataframe database

↻ 一个 Mongodb 数据库转换为表格文件的库

Organize neuroimaging data derivatives into parquet tables

Dremio Arrow Flight Client

Concise interface to cache numpy arrays and pandas dataframes

ibm_db extension to load a pyarrow table to db2

Code examples / snippets for website news post

A small cast tookit class drived from _ParquetDatasetV2 to support cast in filters argument

↻ 一个 MySQL 数据库转换为表格文件的库

Dockerfile and Python 3.9 wheel for PyArrow 3.0.0 built on Alpine 3.14 (does not include Plasma or Parquet)

Inspect your lakehouse data by using PyArrow

En este repositorio se va a compartir todo el material relacionado con la charla "Como compartir grandes Datasets entre procesos sin perder la salud mental" de la Pycones 2021