Feature Tools integration

[willsmithorg]

Could FeatureTools be implemented as an automated preprocessor to Autogluon, adding the ability to handle multi-entity problems (i.e. Data split across multiple normalised database tables)? So if you supply Autogluon with a list of Dataframes instead of a single Dataframe it would first invoke FeatureTools:

• take the multiple Dataframes (entities) and try to auto-infer the relationship between them (identify likely primary key per table, guess foreign key relationships based on similar column names and domain of values in primary and foreign key columns)
• run FeatureTools' deep feature synthesis ("DFS") to generate lots of additional features (https://featuretools.alteryx.com/en/stable/getting_started/afe.html)
• if necessary use FeatureTools cutoff time logic (https://featuretools.alteryx.com/en/stable/getting_started/handling_time.html) to make sure we're doing DFS without snooping features from the future
• throwing the resulting flat dataframe into Autogluon as usual

The FeatureTools license is BSD-3-Clause not Apache2. I don't suppose this is a problem. Alteryx is a $5bn company so they're around for a while. The only amber flag I see is their list of feature primitives (https://primitives.featurelabs.com/) has both open source and "premium" primitives. The premium ones aren't that impressive. Companies that try to segment a single product into open and proprietary can cause trouble down the line, they have a conflict of interest if external people want to improve the open and there's overlap with their proprietary.

Their "woodwork" typing system (https://woodwork.alteryx.com/en/stable/index.html) to add/infer more logical and semantic meaning to a set of features is also interesting. Might become a standard if FeatureTools does well.

Would be grateful for your thoughts. If implemented well, this would allow Autogluon to do SOTA automated feature engineering as well as model selection, stacking and ensembling.

[Innixma]

This is certainly a possibility to have FeatureTools be an optional dependency in AutoGluon that can be used to enhance feature preprocessing and expand the set of inputs AutoGluon can handle such as multi-DF. As the tool is quite extensive, I don't have time at present to dive into it personally, so I'm opening this issue up to the community to create a POC that adds value to AutoGluon's users.

Things that need to be addressed in the POC:

• API
• Set of new functionality that wasn't possible prior (multi-table input?)
• Set of enhanced functionality that is integrated with existing APIs (such as new FeatureGenerators that use FeatureTools). These additions will need to be tested against real datasets to ensure they are helpful.
• Resolving redundant functionality that exists in both AutoGluon and FeatureTools (feature primitives / type inference / feature processing pipelines, etc.)

If interested in contributing this functionality, please respond to this issue.