When a user wants to stream data to a date-partitioned BQ table, the way to do this is:

//noinspection ScalaStyle
class DayPartitionFunction()
  extends SerializableFunction[ValueInSingleWindow[TableRow], TableDestination] {
  override def apply(input: ValueInSingleWindow[TableRow]): TableDestination = {
    val partition = DateTimeFormat.forPattern("yyyyMMdd").withZone(DateTimeZo

In v0.9.0 we started offering --keep_intermediate_avro_files. Avro files are a great candidate for long term storage of variant data for several reasons, mainly because they can be loaded into BigQuery fast and free.

The only missing data is information in VCF header. Storing them with AVRO files will enable us to recov