Timeline for From Oracle to Apache Parquet : how to handle eventual consistency?
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 22, 2022 at 14:17 | answer | added | sam able | timeline score: 0 | |
| Mar 3, 2020 at 21:58 | vote | accept | Klun | ||
| Feb 3, 2020 at 12:07 | answer | added | Martin K | timeline score: 1 | |
| Feb 2, 2020 at 22:13 | comment | added | Benjamin Erichsen | Also note, a row can't be related to a foreign key that does not exist in a relational database. | |
| Feb 2, 2020 at 22:12 | comment | added | Benjamin Erichsen | No that (missing data in the foreign key table) is not possible unless you are deleting stuff from the foreign key table after 10 oclock. I think you might be missing something. | |
| Feb 2, 2020 at 20:22 | comment | added | Klun | By directly exporting the join between the data table and types metadata table in the CSV each time, instead of two different batchs (one for data, and an other for metadata) ? (But If I do that, the export is likely to take longer and more ressource, as the metadata labels and properties will be repeated each time ?). It is more clear for you ? | |
| Feb 2, 2020 at 20:21 | comment | added | Klun | Thanks. If I see Oracle, there is always a fk_metadata_types_id defined in the data table. The problem is that the regularly dumped CSVs files corresponding to each table, are done asynchonously. At 10 oclock, the data table is being dumped to CSV, and at 12 oclock, the metadata table is dumped to CSV. Even if both data are dumped at the same time, it can happen that in the data CSV there are metadata_types IDs that doesn't exists yet in the metadata CSV. So when I convert the CSVs to Parquet, there are the same eventual consistency problem. So my question is how to get strong consistency ? | |
| Feb 2, 2020 at 18:19 | comment | added | Benjamin Erichsen | I am also a little confused. If data has a foreign key, wouldn't the id always need to be in the metadata_types table? Not sure why there is a consistency problem? | |
| Feb 2, 2020 at 18:17 | comment | added | Benjamin Erichsen | 1. Have you considered materialized views? | |
| Feb 2, 2020 at 18:00 | review | First posts | |||
| Feb 3, 2020 at 17:55 | |||||
| Feb 2, 2020 at 17:55 | history | asked | Klun | CC BY-SA 4.0 |