We create multiple jars during our builds to accommodate multiple versions of Apache Spark. In the current approach, the implementation is copied from one version to another and then necessary changes are made.

An ideal approach could create a common directory and extract common classes from duplicate code. Note that even if class/code is exactly the same, you cannot pull out to a common class if it depends on Apache Spark.

Success Criteria:

• PR that refactors all the classes appropriately
• Documentation for all the classes changed/added
• Documentation on upgrading versions (if it doesn't already exist)