I've got a huge (about half a GiB, impossible to use a usual text editor on) CSV file with fields enclosed in double quotes like "abc","def" but need a file without quotes (I am sure this is not going to break the file consistency - a comma is never used inside the values in it).
How to remove all the quotes (without introducing spaces on their places)?
tr can do that:
tr -d \" < infile > outfile
You could also use sed:
sed 's/"//g' < infile > outfile
< infile > outfile? IMHO it was more informative.
< before a couple of times (to import SQL scripts into MySQL and SQLite) It would be immediately obvious for me that I should use < in this case. I think it would be better to return the full command line example back for further reference of those who may need it.
< infile > outfile, hope that's OK.
Another version of sed command:
sed -i s/\"//g file.txt
sed stream editor
-i in-place (edit file in place)s the substitute command/replacement_from_reg_exp/replacement_to_text/ statement\" quotes preceded by backslash (replacement_from_reg_exp)g global (for replace all occurrence in line)file.txt the file name
You don't need to remove the double quotes if you read the data using a CSV-aware library. The rest of this answer shows how you may remove the unneeded double quotes from a generic CSV file, for the case where you rely on a more simplistic approach to parsing the data later.
Miller (mlr) would be able to read the CSV data and output it again using just
mlr --csv cat file
(Use with -I to do "in-place" editing.)
Miller would by default only quote fields that require quoting, e.g. fields containing embedded commas, quotes, or newlines.
In the same fashion, csvformat from csvkit would similarly only by default quote fields that actually require quoting:
csvformat file
The csvformat utility can't do "in-place" editing, so you would need to redirect the output to a new name.
