The Wayback Machine - https://web.archive.org/web/20210727090635/https://github.com/AlexIoannides/pyspark-example-project/issues/20
Skip to content

/ pyspark-example-project

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setup and Teardown should be @classmethods setUpClass and tearDownClass #20

Open
amrishan opened this issue Sep 16, 2020 · 1 comment
Open

Setup and Teardown should be @classmethods setUpClass and tearDownClass #20

amrishan opened this issue Sep 16, 2020 · 1 comment
Labels
enhancement good first issue

Comments

@amrishan
Copy link

@amrishan amrishan commented Sep 16, 2020

if they are not class methods then the method would be invoked for every test and a session would be created for each of those tests.

`class PySparkTest(unittest.TestCase):
@classmethod
def suppress_py4j_logging(cls):
logger = logging.getLogger('py4j')
logger.setLevel(logging.WARN)

@classmethod
def create_testing_pyspark_session(cls):
    return SparkSession \
                .builder \
                .master('local[*]') \
                .appName("my-local-testing-pyspark-context") \
                .getOrCreate()


@classmethod
def setUpClass(cls):
    cls.suppress_py4j_logging()
    cls.spark = cls.create_testing_pyspark_session()
    cls.test_data_path = "<PATH>"
    cls.df = cls.spark.read.options(header='true', inferSchema='true') \
                .csv(cls.test_data_path)
    cls.df_exepcted = transform_data(cls.df, cls.spark)


@classmethod
def tearDownClass(cls):
    cls.spark.stop()`
@AlexIoannides
Copy link
Owner

@AlexIoannides AlexIoannides commented Sep 16, 2020

Well spotted - thank you 👍
@amrishan amrishan closed this Sep 16, 2020
@amrishan amrishan reopened this Sep 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement good first issue
2 participants
@AlexIoannides @amrishan