1

Is there a way to capture and write very fast serial data to a file?

I'm using a 32kSPS external ADC and a baud rate of 2000000 while printing in the following format: adc_value (32bits) \t millis()

This results in ~15 prints every 1 ms. Unfortunately every single soulution I have tried fails to capture and store real time data to a file. This includes: Processing sketches, TeraTerm, Serial Port Monitor, puTTY and some Python scripts. All of them are unable to log the data in real time. Arduino Serial Monitor on the other hand is able to display real time serial data, but it's unable to log it in a file, as it lacks this function.

Here's a printscreen of the serial monitor in Arduino with the incoming data:

enter image description here

Improve this question
2
  • this is not the answer to question, but by looking at the image attached, you should use micros() instead of millis(). Because for same timestamp you have more than one adc values. Commented Feb 5, 2018 at 10:20
  • Thanks for the tip! That's a good one :) Commented Feb 5, 2018 at 10:42

2 Answers 2

Reset to default
3

One problematic thing is probably that you try to do a write each time you receive a new record. That will waste a lot of time writing data.

Instead try to collect the data into buffers, and as a buffer is about to overflow write the whole buffer in a single and as low-level as possible write call.

And to not stop the receiving of the data to much, you could use threads and double-buffering: Receive data in one thread, write to a buffer. When the buffer is about to overflow signal a second thread and switch to a second buffer. The other thread takes the full buffer and writes it to disk, and waits for the next buffer to become full.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

You're right about that: every time a new value is acquired I call Serial.print() for sending it to Serial console. Maybe that's also why I'm unable to achieve the full potential of 32kSPS of the ADC. Buffering it might be a solution. Any idea on how to do that? Here's a short video to give you an idea of the serial print speed: drive.google.com/open?id=1e7PJlYmFzU9SzVccUb5BCliL6GoXFiH1
By the way, I still don't understand why all the serial capture tools I've used aren't able to handle the stream of data.
0

After trying more than 10 possible solutions for this problem including dedicated serial capture software, python scripts, Matlab scripts, and some C projects alternatives, the only one that kinda worked for me proved to be MegunoLink Pro.

It does not achieve the full 32kSPS potential of the ADC, rather around 12-15kSPS, but it is still much better than anything I've tried.

Not achieving the full 32kSPS might also be limited by the Serial.print() method that I'm using for printing values to the serial console. By the way, the platform I've been using is ESP32.

Later edit: don't forget to edit MegunoLinkPro.exe.config file in the MegunoLink Pro install directory in order to add further baud rates, like 1000000 or 2000000. By default it is limited to 500000.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.