Create HTML profiling reports from pandas DataFrame objects

Always know what to expect from your data.

🚚 Agile Data Science Workflows made easy with Pyspark

Visualize and compare datasets, target values and associations, with one line of code.

🚕 A GUI for dataprep that works over Optimus(pandas, dask, cudf, dask-cufd and PySpark)

Monitor the stability of a pandas or spark dataframe ⚙︎

a set of scripts to pull meta data and data profiling metrics from relational database systems

Dataset search engine, discovering data from a variety of sources, profiling it, and allowing advanced queries on the index

A Node.js tool to examine the correctness of Open Data Metadata and build custom dataset profiles

R package to simplify the usage of the RDS REST API and provide convenience in accessing data and metadata.

Data cleaning tool.

Open Data Profiling, Quality and Analysis on NYC OpenData dataset with semantic profiling using fuzzy ratio, Levenshtein distance and regex

HPCC Systems ECL bundle that provides some basic data profiling and research tools to an ECL programmer

Automated exploration of files in a folder structure to extract metadata and potential usage of information.

Simplify usage of the RDS API for TypeScript/JavaScript developers

Data profiler is an attempt to model the behavior of a given operator for a set of datasets.

The program compares two files at a time and does the following 1.Gathering metadata on the individual tables(column count,record count,list of columns with datatype etc) 2.Identifying matching columns between tables based on names as well as data. Using machine learning, we are handling syntactic as well as semantic variations of column names for accurate matching. 3. Finding duplicate columns in single table with the option to deduplicate if required 4. Finding columns with missing data/null values.

TypeScript/JavaScript example code using the RDS API

Distributable UCC Discovery Algorithm based on Akka

Profile. Generate data profiles in the browser (work in progress)

Sandbox to test out ideas for profiling document data

Map naturally-occurring inter-subreddit content sharing patterns on Reddit by analyzing how posts are “cross-posted" between subreddits based on 2.5 million posts across the top 2,500 subreddits. Uses ECL and HPCC Systems.

Python function to generate a mask analysis

Always know what to expect from your data.

Fork of Sato for easy deployment as a Python package

Identified data types for each distinct column value on 1900 data sets. For each column, summarized semantic types present in the column, using Fuzzy Logic, Levenshtein distance. Identified & derived inference the 3 most frequent 311 complaint types by borough.

MetricDoc is an interactive visual exploration environment for assessing data quality

🚚 Agile Data Science Workflows made easy with Pyspark

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets.

Homework for exploring function dependencies in data sets