data-engineering
Here are 587 public repositories matching this topic...
Description
Recently, a task_run_name parameter was added to the task constructors, and documentation was written on how to dynamically set task Run Names dynamically based on inputs.
At runtime, Prefect automatically populates kwargs with raw inputs and also prefect context:
https://github.com/PrefectHQ/prefect/blob/1bbefb71
-
Updated
Nov 8, 2020
We are trying to use GE with GCP DataProc clusters. While cluster creation we are installing great-expectations==0.12.4. This installs ruamel.yaml==0.15.35 as dependency. After cluster creation if we try to import great_expectations we get error:
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/default/lib/python3.6/site-packages/great_expectations/_
-
Updated
Nov 8, 2020 - Go
-
Updated
Sep 11, 2020
-
Updated
Oct 19, 2020 - JavaScript
-
Updated
Nov 7, 2020 - Python
-
Updated
Nov 6, 2020 - Jupyter Notebook
-
Updated
Oct 14, 2020 - Jupyter Notebook
-
Updated
Mar 9, 2020 - Python
Instructions for how to install pyjanitor via pipenv
Some folks might use pipenv for environment management. The recent update requires a prerelease dependency (black, as menti
if they are not class methods then the method would be invoked for every test and a session would be created for each of those tests.
`class PySparkTest(unittest.TestCase):
@classmethod
def suppress_py4j_logging(cls):
logger = logging.getLogger('py4j')
logger.setLevel(logging.WARN)
@classmethod
def create_testing_pyspark_session(cls):
return Sp
-
Updated
Nov 8, 2020 - R
-
Updated
Aug 21, 2020 - CSS
-
Updated
Mar 5, 2020 - Python
-
Updated
Nov 29, 2018 - Java
-
Updated
Sep 15, 2020
In SubjectAreaRESTServicesInstance, it hard codes the default page size as 0, this is not correct
public static final String PAGE_SIZE_DEFAULT_VALUE = "0";
it should be changed to
public static final String PAGE_SIZE_DEFAULT_VALUE = "1000";
So it is consistent with OMAGServerConfig default
private static final int defaultMaxPageSize = 1000;
-
Updated
Nov 5, 2020 - TypeScript
-
Updated
Apr 20, 2020 - Python
-
Updated
Nov 8, 2020
-
Updated
Jun 18, 2020 - Python
-
Updated
Sep 24, 2020 - Python
-
Updated
Oct 1, 2020 - Python
-
Updated
Mar 25, 2019
-
Updated
Nov 7, 2020 - Clojure
-
Updated
Aug 7, 2019 - Jupyter Notebook
-
Updated
Aug 24, 2020 - Scala
Improve this page
Add a description, image, and links to the data-engineering topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the data-engineering topic, visit your repo's landing page and select "manage topics."


Screenshot
N/A
Description
Right now whenever users search for queries they are case sensitive. We should remove this to allow users to put in term with any cases
Design input
[describe any input/collaboration you'd like from designers, and
tag accordingly. For design review, add the
label
design:review. If this includes a design proposal,include the label `design:suggest