etl

Java and Python provide logging frameworks for recording errors and stack traces. Logging can be configured to format those messages, such as by removing newlines from stack traces, adding timestamps, and putting a log level (INFO, WARN, ERROR) on the message. In the future, it may also include json structured logging. Using the logging framew

Issue from the Dagster Slack

[dagster_shell] defer environ copy to solid run time

This issue was generated from the slack conversation at: https://dagster.slack.com/archives/C01U954MEER/p1623181007055200?thread_ts=1623181007.055200&cid=

During migration of old sync code to async discovered that we don't have async versions of some API.

问题描述：关于工程中某个工作流节点单独执行不报错，保存工作流时提示：NotClassDefFoundError：Cloud not initialize class dispatch.Http$

问题原因：linkis-publish微服务关于netty-3.6.2.Final.jar升级包缺失

问题解决：将升级包上传重启linkis-publish微服务即可。

The functions in this file should be factored out to a separate utility lib as they are reused in bitcoin-etl https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/misc_utils.py

if they are not class methods then the method would be invoked for every test and a session would be created for each of those tests.

`class PySparkTest(unittest.TestCase):
@classmethod
def suppress_py4j_logging(cls):
logger = logging.getLogger('py4j')
logger.setLevel(logging.WARN)

@classmethod
def create_testing_pyspark_session(cls):
    return Sp

Bigquery external table API supports auto schema detection for external table source by a boolean property - ref.

If the target field names are the same as the source fields in the Google Spreadsheet, it should be better not to supply any schema rather than sp

Right now, these aren't caught until we try to gob-encode. Consider failing faster in type-checking to avoid too much confusion/loss when it works with local execution.