How to use awk for joining two csv using a key column and write a another csv

Question

I have two CSV with following data,

file1.csv:-

path(S1),filename,timestamp(S1),memory(S1)  
xx/APS50CC2,APS50CC2,23-09-2019,45  
yy/APS50CC3,APS50CC3,18-09-2019,25

file2.csv :-

path(S2),filename,timestamp(S2),memory(S2)
mm/APS03CR1,APS03CR1,23-09-2019,200
zz/APS50CC2,APS50CC2,18-09-2019,55

Expected output
output.csv :-

filename,path(S1),timestamp(S1),memory(S1),path(S2),timestamp(S2),memory(S2)
APS50CC2,xx/APS50CC2,23-09-2019,45,zz/APS50CC2,18-09-2019,55
APS50CC3,yy/APS50CC3,18-09-2019,25,Not found,Not found,Not found
APS03CR1,Not found,Not found,Not found,mm/APS03CR1,23-09-2019,200

I have tried join command and have run,it is working fine for smaller set of input. But when I run for about 500000+ records, it is giving errors, hence I do not want to use join, I want to try it out using awk...

I am kind of new to awk, kindly help me out with the syntax. The key here for joining is column 2 i.e. filename.

Answer 1

Using Miller (https://github.com/johnkerl/miller) and running

mlr --csv join --ul --ur -j filename -f file1.csv \
then unsparsify --fill-with "Not found" \
then reorder -f "filename","path(S1)","timestamp(S1)","memory(S1)","path(S2)","timestamp(S2)","memory(S2)" file2.csv

you will have

+----------+-------------+---------------+-------------+---------------+------------+--------------+
| filename | path(S1)    | timestamp(S1) | path(S2)    | timestamp(S2) | memory(S2) | memory(S1)   |
+----------+-------------+---------------+-------------+---------------+------------+--------------+
| APS03CR1 | Not found   | Not found     | mm/APS03CR1 | 23-09-2019    | 200        | Not found    |
| APS50CC2 | xx/APS50CC2 | 23-09-2019    | zz/APS50CC2 | 18-09-2019    | 55         | 45           |
| APS50CC3 | yy/APS50CC3 | 18-09-2019    | Not found   | Not found     | Not found  | 25           |
+----------+-------------+---------------+-------------+---------------+------------+--------------+

I know it's not awk but Miller is another great tool for structured text