Return to Question

I was trying to download data using Yahoo Finance with wget but ran into some issues with getting cookies and crumb. I used BTC-USD in this case and I saved the crumb data in a file named "crumbstore.txt".

wget --no-check-certificate --save-cookies=cookie.txt "https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD" -O crumbstore.txt

From there I wanted to extract the crumb string and insert it into the quote link shown below ($crumb). The time period is from 2017-01-01 to 2020-01-06 in this case.

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/BTC-USD?period1=1483265281&period2=1578305313&interval=1d&events=history&crumb=$crumb" -O BTC-USD_price.csv

In order to see how the "crumbstore.txt" file is structured, I ran the following code with grep.

grep -o ".\{0,50\}CrumbStore.\{0,50\}" crumbstore.txt

I was able to see what I needed to get was the series of letters and numbers stored in "CrumbStore":{"crumb":"XXXXXXXXXXX"}.

Here is a snapshot of what the crumbstore.txt looks like and the snippet needed is "BdpohsJbyt" in the file.

Therefore, the question is how to use sed or awk to extract only the crumb string from the "crumbstore.txt" file.

I was trying to download data using Yahoo Finance with wget but ran into some issues with getting cookies and crumb. I used BTC-USD in this case and I saved the crumb data in a file named "crumbstore.txt".

wget --no-check-certificate --save-cookies=cookie.txt "https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD" -O crumbstore.txt

From there I wanted to extract the crumb string and insert it into the quote link shown below ($crumb). The time period is from 2017-01-01 to 2020-01-06 in this case.

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/BTC-USD?period1=1483265281&period2=1578305313&interval=1d&events=history&crumb=$crumb" -O BTC-USD_price.csv

In order to see how the "crumbstore.txt" file is structured, I ran the following code with grep.

grep -o ".\{0,50\}CrumbStore.\{0,50\}" crumbstore.txt

I was able to see what I needed to get was the series of letters and numbers stored in "CrumbStore":{"crumb":"XXXXXXXXXXX"}.

Here is a snippet of crumbstore.txt and the essential task is to extract "BdpohsJbyt." from the text.

"secondary_accounts":{"list":"Pos(r) M(0) P(0) List(n)","list_item":"Bdt(asMenuBorder)","anchor":"Td(n) Td(n):h Py(16px) D(b) Cf C(#000) Bgc(asMenuHoverBgc):h"}}}}},"CrumbStore":{"crumb":"BdpohsJbyt."},"StreamStore":{"articleCategory":{"term":"","label":""}

I was trying to download data using Yahoo Finance with wget but ran into some issues with getting cookies and crumb. I used BTC-USD in this case and I saved the crumb data in a file named "crumb.store".

wget --no-check-certificate --save-cookies=cookie.txt "https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD" -O crumb.store

From there I wanted to extract the crumb string and insert it into the quote link shown below ($crumb). The time period is from 2017-01-01 to 2020-01-06 in this case.

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/BTC-USD?period1=1483265281&period2=1578305313&interval=1d&events=history&crumb=$crumb" -O BTC-USD_price.csv

In order to see how the "crumb.store" file is structured, I ran the following code with grep.

grep -o ".\{0,50\}CrumbStore.\{0,50\}" crumb.store

I was able to see what I needed to get was the series of letters and numbers stored in "CrumbStore":{"crumb":"XXXXXXXXXXX"}.

Therefore, the question is how to use sed or awk to extract only the crumb string from the "crumb.store" file.

I was trying to download data using Yahoo Finance with wget but ran into some issues with getting cookies and crumb. I used BTC-USD in this case and I saved the crumb data in a file named "crumbstore.txt".

wget --no-check-certificate --save-cookies=cookie.txt "https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD" -O crumbstore.txt

From there I wanted to extract the crumb string and insert it into the quote link shown below ($crumb). The time period is from 2017-01-01 to 2020-01-06 in this case.

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/BTC-USD?period1=1483265281&period2=1578305313&interval=1d&events=history&crumb=$crumb" -O BTC-USD_price.csv

In order to see how the "crumbstore.txt" file is structured, I ran the following code with grep.

grep -o ".\{0,50\}CrumbStore.\{0,50\}" crumbstore.txt

I was able to see what I needed to get was the series of letters and numbers stored in "CrumbStore":{"crumb":"XXXXXXXXXXX"}.

Here is a snapshot of what the crumbstore.txt looks like and the snippet needed is "BdpohsJbyt" in the file.

Therefore, the question is how to use sed or awk to extract only the crumb string from the "crumbstore.txt" file.

I was trying to download data using Yahoo Finance with wget but ran into some issues with getting cookies and crumb. I used BTC-USD in this case and I saved the crumb data in a file named "crumb.store".

wget --no-check-certificate --save-cookies=cookie.txt "https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD" -O crumb.store

From there I wanted to extract the crumb string and insert it into the quote link shown below ($crumb). The time period is from 2017-01-01 to 2020-01-06 in this case.

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/BTC-USD?period1=1483265281&period2=1578305313&interval=1d&events=history&crumb=$crumb" -O BTC-USD_price.csv

In order to see how the "crumb.store" file is structured, I ran the following code with grep.

grep -o ".\{0,50\}CrumbStore.\{0,50\}" crumb.store

I was able to see what I needed to get was the series of letters and numbers stored in "CrumbStore":{"crumb":"XXXXXXXXXXX"}.

Therefore, the question is how to use sed or awk to extract only the crumb string from the "crumb.store" file.

Please let me know what y'all think

I was trying to download data using Yahoo Finance with wget but ran into some issues with getting cookies and crumb. I used BTC-USD in this case and I saved the crumb data in a file named "crumb.store".

wget --no-check-certificate --save-cookies=cookie.txt "https://finance.yahoo.com/quote/BTC-USD/history?p=BTC-USD" -O crumb.store

From there I wanted to extract the crumb string and insert it into the quote link shown below ($crumb). The time period is from 2017-01-01 to 2020-01-06 in this case.

wget --no-check-certificate --load-cookies=cookie.txt "https://query1.finance.yahoo.com/v7/finance/download/BTC-USD?period1=1483265281&period2=1578305313&interval=1d&events=history&crumb=$crumb" -O BTC-USD_price.csv

In order to see how the "crumb.store" file is structured, I ran the following code with grep.

grep -o ".\{0,50\}CrumbStore.\{0,50\}" crumb.store

I was able to see what I needed to get was the series of letters and numbers stored in "CrumbStore":{"crumb":"XXXXXXXXXXX"}.

Therefore, the question is how to use sed or awk to extract only the crumb string from the "crumb.store" file.