etl

We should do something like https://blog.questionable.services/article/kubernetes-deployments-configmap-change/ to ensure that if the pod sweeper has a different config map the underlying pod gets rolled.

https://dagster.slack.com/archives/C01U954MEER/p1642163479382400:

Hi, I'm experiencing a bug when trying to write an ALS model from pyspark.ml.recommendation to S3 and reading it back in if this takes place within a dynamically executed graph (i.e. via dynamic mapping). I wrote a custom IO manager using the pattern f's3a://{self.s3_bucket}/{key}' as _uri_for_key similar to the one currently imp

Under the hood, Benthos csv input uses the standard encoding/csv packages's csv.Reader struct.

The current implementation of csv input doesn't allow setting the LazyQuotes field.

We have a use case where we need to set the LazyQuotes field in order to make things work correctly.

I am trying to implement retry logic for Aws Aurora Mysql.

In Linq2db there is retyrpolicy already implemented except there is no way to pass retrypolicy to DataContext.
So I have extended the DataContext as follows.

public class RetryingDataContext : DataContext
{
protected IRetryPolicy _retryPolicy;
public RetryingDataContext(IDataProvider dataProvider, string c

The functions in this file should be factored out to a separate utility lib as they are reused in bitcoin-etl https://github.com/blockchain-etl/ethereum-etl/blob/develop/ethereumetl/misc_utils.py

if they are not class methods then the method would be invoked for every test and a session would be created for each of those tests.

`class PySparkTest(unittest.TestCase):
@classmethod
def suppress_py4j_logging(cls):
logger = logging.getLogger('py4j')
logger.setLevel(logging.WARN)

@classmethod
def create_testing_pyspark_session(cls):
    return Sp

Sending a rest call to delete a job specification throws 404 where as grpc call works fine. Steps to reproduce

curl -X DELETE "http://localhost:9100/v1/project/my-project/namespace/kush/helloworld" -H  "accept: application/json"

Right now, these aren't caught until we try to gob-encode. Consider failing faster in type-checking to avoid too much confusion/loss when it works with local execution.

If i have an input yaml connecting to jdbc source like:

inputs:
  somedb:
    jdbc:
      connectionUrl: jdbc:jtds:redact
      user: someuser
      password: somepass

the log has

2020-12-08 17:01:28,076 [main] INFO  com.yotpo.metorikku.Job - these are the config inputs: Some(Map(.....somepass

ideally this should be printed as ****** in the log