Skip to content
Prev Previous commit
Next Next commit
chore: document how schemas are unioned in the analyzer
  • Loading branch information
wiedld committed Aug 19, 2025
commit f2ea8e53bea1dfbbeb90ecacfd626a327300336d
37 changes: 37 additions & 0 deletions datafusion/optimizer/src/analyzer/type_coercion.rs
Original file line number Diff line number Diff line change
Expand Up @@ -957,6 +957,43 @@ fn coerce_case_expression(case: Case, schema: &DFSchema) -> Result<Case> {
///
/// This method presumes that the wildcard expansion is unneeded, or has already
/// been applied.
///
/// ## Schema and Field Handling in Union Coercion
///
Comment on lines +960 to +962

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that here in the analyzer, we handle union coercion differently than at construction.

/// **Processing order**: The function starts with the base schema (first input) and then
/// processes remaining inputs sequentially, with later inputs taking precedence in merging.
///
/// **Schema-level metadata merging**: Later schemas take precedence for duplicate keys.
///
/// **Field-level metadata merging**: Later fields take precedence for duplicate metadata keys.
///
/// **Type coercion precedence**: The coerced type is determined by iteratively applying
/// `comparison_coercion()` between the accumulated type and each new input's type. The
/// result depends on type coercion rules, not input order.
///
/// **Nullability merging**: Nullability is accumulated using logical OR (`||`).
/// Once any input field is nullable, the result field becomes nullable permanently.
/// Later inputs can make a field nullable but cannot make it non-nullable.
///
/// **Field precedence**: Field names come from the first (base) schema, but the field properties
/// (nullability and field-level metadata) have later schemas taking precedence.
///
/// **Example**:
/// ```sql
/// SELECT a, b FROM table1 -- a: Int32, metadata {"source": "t1"}, nullable=false
/// UNION
/// SELECT a, b FROM table2 -- a: Int64, metadata {"source": "t2"}, nullable=true
/// UNION
/// SELECT a, b FROM table3 -- a: Int32, metadata {"encoding": "utf8"}, nullable=false
/// -- Result:
/// -- a: Int64 (from type coercion), nullable=true (from table2),
/// -- metadata: {"source": "t2", "encoding": "utf8"} (later inputs take precedence)
/// ```
///
/// **Precedence Summary**:
/// - **Datatypes**: Determined by `comparison_coercion()` rules, not input order
/// - **Nullability**: Later inputs can add nullability but cannot remove it (logical OR)
/// - **Metadata**: Later inputs take precedence for same keys (HashMap::extend semantics)
pub fn coerce_union_schema(inputs: &[Arc<LogicalPlan>]) -> Result<DFSchema> {
coerce_union_schema_with_schema(&inputs[1..], inputs[0].schema())
}
Expand Down