dbt

What type of re_data dbt macro you would like to add

• validate

What macro should be doing

Return true is string is a valid JSON and can be parsed to JSON

Let's prepare a mixin for interacting with Roles and Policies with the Python client, in case users want to use the API directly.

Do not only have the list, get etc, but also utility methods, such as updating a default role. It should wrap the following logic:

import requests
import json

# Get the ID
data_consumer = requests.get("http://localhost:8585/api/v1/roles/name/DataCo

Support copy into queries

In this PR, I wanted to solve issue #25 by creating a CSV file to list and also help the brand property matching process. This PR including:

• Compile a list of all brand names and operator names from the name-suggestion-index as a CSV with the columns id, display_name, and wiki_data in tmp folder.
  Deadline: 05.01.2022

• Create a PySpark UDF similar to the ones in the osm

faldbt object already has access to the manifest; we need to pass it here in the exec method. We probably want to pass the dbt manifest directly rather than the fal wrapper as we dont want the fal wrapper to be our public api.

https://github.com/fal-ai/fal/blob/8f17b4f32c96f8d3e2fbb1534ccebca3a9276bcf/src/fal/run_scripts.py#L22-L47

Where it makes sense, we should check inputs for datatype, length etc before processing, and if necessary raise an exception via exceptions.raise_compiler_error.
See: https://github.com/calogica/dbt-expectations/blob/b69ac04cacfe1dfaf1de129778908898e666f9e3/macros/schema_tests/multi-column/expect_compound_columns_to_be_unique.sql#L16

I'm new to the idea of Data Vault 2.0 and I'm reading the Dan Linstedt book to understand it better.

To help me get a better perspective of how dbtvault works I would like to know how difficult do you think it would be to add support for BigQuery?

Are there specific features of Snowflake which makes it better for running dbt/dbtvault ?

Thanks,
Jacob

@mhindery asks whether we could use the binary instead as a lot of other common data engineering projects defer to it now and this forces users to use yet another package.

We will have to test it but I think it should be fine.

Initial discussion happened here:
bitpicky/dbt-sugar#20 (comment)

Other adapters (e.g. dbt-spark) have adopted a single-source-of-truth approach to documentation, prefering to document setup and configuration information only on the docs.getdbt website, rather than duplicating it on the docs page and the adapter repo's readme.

I think we should do the same.