0

I am building a storage engine software that allows concurrent data writing, now I have two different choices here:

Method 1. Background Long-Running Thread

  • Multiple user threads write to their own WriteBuffer
  • A background long-running IO thread iterate all write buffers and flush each write buffer one by one. (Need to obtain write buffer's mutex lock)
  • On each iteration, we can do a this_thread::yield() to prevent block other jobs too long.

Method 2. Use User Threads

  • Multiple user threads write to their own WriteBuffer
  • On each user thread writes, we use this user thread to iterate all write buffers and flush buffers. (also need to obtain buffers mutex)

Can anyone tell which one is a better choice and why? Or do you have any other suggestions ? Thanks.

Improve this question
1
  • 1
    What does "concurrent data writing mean"? Is it multiple users writing different files, located on the same HDD, at the same time? Or is it multiple users writing to the same file at the same time? Something else? Commented Mar 1, 2021 at 8:23

1 Answer 1

Reset to default
1

Method 3: Producer-Consumer

  • A user requests a future write (with the corresponding data) by atomically appending the data to a thread-safe queue. (Producer)
  • The long-running background task pulls from this queue atomically a long as the queue is not empty. (Consumer)

If the queue is empty the long-running background task is put to sleep. Therefore, the disk is only hit when there is data to write.

This is used all the time for background logging to a file (particularly in games) and it has no noticeable affect on performance.

Improve this answer

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.