Data Collection in German Language on Sewing Machine Operation for Video Question Answering

script.pdf:

• Instructions about basics of sewing machine operation
• 7 larger tasks split into 35 total sub-steps
• Glossary of sewing machine terms
• German and English translation

dataset.csv:

• Row for every unique script task and step
• Columns:
  • Paths to corresponding video files in third person (tpv_filepath), first person (fpv_filepath) and with eye tracking annotations as circle (fpv_circle_filepath) and as an attention map (fpv_attention_filepath)
  • Path to corresponding audio file (audio_filepath)
  • Audio transcription (transcript)

human_qa.csv:

• Row for every question answer pair formulated by human annotators
• Columns:
  • Human formulated question (question)
  • Human formulated answer (correct_answer)
  • Task (task) and step (step) of script corresponding to the question
  • Unique ID of human annotator (participant_id)
  • Submission time of question and answer (created_at)
  • Annotation whether QA pair is useable (acceptable)
  • Annotation of reasoning type (reasoning):
    • KNOWLEDGE
    • SPATIAL
    • TEMPORAL
    • PERCEPTION
  • Unique ID of QA pair (qa_id)

gemini_qa.csv:

• Row for every answer generated by Gemini 1.5 Pro
• Columns:
  • Task (task) and step (step) of script corresponding to the question
  • Corresponding instructions for task and step (instructions)
  • Paths to corresponding video files in third person (tpv_filepath), first person (fpv_filepath) and with eye tracking annotations as circle (fpv_circle_filepath)
  • Unique ID of human QA pair annotator (participant_id)
  • Annotation whether human QA pair is useable (acceptable)
  • Annotation of reasoning type for human QA pair (reasoning):
  • Human formulated question (question)
  • Human formulated answer (correct_answer)
  • Unique ID of QA pair (qa_id)
  • Model generated answer (gemini_answer)
  • Manual annotation whether model generated answer is acceptable (gemini_acceptable)
  • Type of ablation setting used for prompts (ablation setting):
    • naive baseline: no context beyond question
    • TPV: TPV video as context
    • FPV: FPV video as context
    • FPV_C: FPV video with circle annotation for eye tracking as context
    • FPV_A: FPV video with attention map for eye tracking as context
    • text-only reference model: textual instructions related to question as context
    • multimodal reference model: textual instructions related to question and FPV video as context