3,725 questions
1
vote
1
answer
127
views
scala.MatchError: TimestampNTZType (of class org.apache.spark.sql.types.TimestampNTZType$)
We are using Kafka to get avro data with the help of schema registry. After upgrading to Spark 3.3.2 from 2.4 the kafka consumer is failing with error -
scala.MatchError: TimestampNTZType (of class ...
- 897
3
votes
0
answers
69
views
Kafka REST proxy: Produce message with union type
I am using Kafka, SchemaRegistry, KafkaUI and Kafka REST proxy in Docker container. I am now trying to use the REST proxy now more for test purposes, but as soon the schema is a bit more than trivial, ...
- 829
1
vote
1
answer
301
views
How to register an Avro schema with a custom schema ID in Confluent Schema Registry?
I'm working with Confluent Schema Registry and I want to register an Avro schema using a custom schema ID instead of letting the registry auto-assign it. Normally, the schema id will be generated with ...
0
votes
1
answer
60
views
KafkaAvroSerializer UnresolvedUnionException with nullable timestamp-millis field
I'm using confluent's KafkaAvroSerializer with a schema that includes a nullable timestamp-millis field. The field maps to Instant in Java, but when I send a non-null I get an UnresolvedUnionException....
0
votes
1
answer
69
views
Flink dynamic sink routing with confluent schema-registry
I have an Apache Flink app that is using a kafksink with a setTopicSelector
KafkaSink<T>> sink =
KafkaSink.<T>>builder()
.setBootstrapServers(sink_brokers)
...
- 1,054
1
vote
0
answers
70
views
AvroRowDeserializationSchema and AvroRowSerializationSchema not working in PyFlink 2.1.0
I am trying to use AvroRowSerializationSchema with PyFlink 2.1.0, but I keep getting a Py4JError saying that the class does not exist in the JVM, even though I have the right JARs.
Environment:
...
- 11
0
votes
2
answers
209
views
How to pass a Python datetime to BigQuery through Avro?
I have a Python pipeline where I try to:
get some json data from an API response
modify it via duckdb (just in memory)
convert the resulting data to a list of python dicts (1 row: 1 dict)
Add a ...
- 480
1
vote
1
answer
61
views
Deserialization of multiple avro tables in Databricks
I have multiple tables sent to eventhub and they're avro based with apicurio as schema registry but how can I deserialize them as pyspark's from_avro only does 1 table?
- 65
0
votes
0
answers
28
views
ksqldb multiple SchemaRegistry
I have:
DC1
Kafka1
SR1
DC2
Kafka2
SR2
Clone SR1
I have MirrorMaker that copy data from DC1 to DC2. As well as internal SchemaRegistry topic with schemas. So in DC2 I have full copy SR1 to read ...
- 41
0
votes
0
answers
65
views
Handle env specific namespaces with schema evolution using KafkaAvroDeserializer
I am working in a Java Spring Boot app that primarily consumes Kafka events from a non compacted cdc topic, which a Debezium Connector publishes to. I am using basic defaults in KafkaAvroDeserializer ...
- 1,991
0
votes
1
answer
182
views
flink ConfluentRegistryAvroSerializationSchema not respecting registryConfigs
When I use in Apache Flink the KafkaRecordSerializationSchema with settings for the schema registry serialization , the registryConfigs settings are not taken in account
settings like
auto.register....
- 1,054
0
votes
1
answer
233
views
Build IcebergSinkConnector with AWSKafkaAvroConverter for MSK Connect
I'm trying to build a MSK Connect compatible IcebergSinkConnector with AWSKafkaAvroConverter as I'm using the AWS Streaming Schema Registry. The following is my connector configuration:
...
- 362
0
votes
0
answers
152
views
How to successfully implement FULL_TRANSITIVE evolution in schema registry with ENUMs
I have a schema with a set of enums and a schema registry set to full transitive compatibility. I'm seeing that simply adding an enum WITH a default field in both the V1 and V2 versions is returning ...
- 107
1
vote
0
answers
234
views
Apache.Avro.Tools (avrogen) fails to resolve referenced schema in .NET: "Undefined name"
I'm using Apache.Avro.Tools for .NET (avrogen) to generate C# classes from Avro schemas. I have two Avro schema files:
address.avsc (defines a record net.common.events.Address)
sample.avsc (...
- 187
1
vote
0
answers
63
views
How to resolve nested Avro schema references in C# using Apache.Avro NuGet package
I'm working on a project in C# using the [Apache.Avro NuGet package to read and validate Avro schemas. I have a schema (sample.avsc) that references another schema (address.avsc) by namespace-...
- 187