Location of the documentation

https://pandera.readthedocs.io/en/stable/checks.html

Documentation problem

An important piece of functionality is that ignore_na=True in a Check will drop elements with nulls (for columns) and rows with any nulls (for dataframe checks)

Suggested fix for documentation

Add documentation for this in the user guide so that it's highlighted

Write unit test coverage for SafeDataset and SafeDataLoader, along with the functions in utils.py.

The exception in subject is thrown by the following code:

from datetime import date
from pysparkling.sql.session import SparkSession
from pysparkling.sql.functions import collect_set

spark = SparkSession.Builder().getOrCreate()

dataset_usage = [
    ('steven', 'UUID1', date(2019, 7, 22)),
]
dataset_usage_schema = 'id: string, datauid: string, access_date: date'

df = spa

• Run read and write ends of the conduits concurrently.
• When reading a Partition created by <> operations, consume the smaller partitions in parallel.