I was trying to download data using Yahoo Finance with wget but ran into some issues with getting cookies and crumb. I used BTC-USD in this case and I saved the crumb data in a file named "crumb.store".

wget --no-check-certificate --save-cookies=cookie.txt "https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD" -O crumb.store

From there I wanted to extract the crumb string and insert it into the quote link shown below ($crumb). The time period is from 2017-01-01 to 2020-01-06 in this case.

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/BTC-USD?period1=1483265281&period2=1578305313&interval=1d&events=history&crumb=$crumb" -O BTC-USD_price.csv

In order to see how the "crumb.store" file is structured, I ran the following code with grep.

grep -o ".\{0,50\}CrumbStore.\{0,50\}" crumb.store

I was able to see what I needed to get was the series of letters and numbers stored in "CrumbStore":{"crumb":"XXXXXXXXXXX"}.

Therefore, the question is how to use sed or awk to extract only the crumb string from the "crumb.store" file.