grep to extract a substring from a huge string

Question

I am extracting certain information from a website using curl command. I need to extract a certain subsection of the string using grep command. For example, using curl command I am getting the HTML page and storing in a variable as below.

page="$(curl http://www.somewebsite.com)"

My extracted HTML page is as below.

<html>
<body>
--------
--------
--------
<div class="section" id="id1" style="display:block">
<div class="header">
      <h2>Sub heading here
      </h2>
</div>
<div class="content">
<ol class="class1" id="id1">
<li class="">
<span class="jellybean">
</div>
----------
----------
</html>

Now, I need to extract the subsection of the entire div with id="id1". I tried the below command.

output="$(curl http:/somesite.com | grep "id="id1")"

It just gets me the 2 lines where I have id=id1. I need to extract the entire contents between the and tag.

Careful: you're going to make the unholy child weep the blood of virgins!. — Joseph R.
– Joseph R., Commented Feb 8, 2014 at 0:47
I found w3m command to scrap out HTML contents in a page. Not explored it completely. But seems interesting. Will update the answer if I found something useful. — Ramesh
– Ramesh, Commented Feb 8, 2014 at 1:00

Stéphane Chazelas · Accepted Answer · 2014-02-08 12:39:41Z

2

If the file were correct xml, you could use an xml parsing tool.

Otherwise, if there were no other (nested) div section inside that section, you could have done:

pcregrep -Mo '(?s)<div[^>]*id="id1".*?</div>' the-file.html

Here, you could try something like:

awk -vRS='<' '
  inside || /^div[^>]*id="id1"/ {
    inside = 1
    if (/^div/)
      n++
    else if (/^\/div>/ && !--n) {
      $0="/div>\n"
      inside=0
    }
    printf "<%s", $0
  }' the-file.html

edited Feb 8, 2014 at 12:39

answered Feb 8, 2014 at 11:45

Stéphane Chazelas

585k96 gold badges1.1k silver badges1.7k bronze badges

Add a comment |

Stack Exchange Network

grep to extract a substring from a huge string

1 Answer 1

You must log in to answer this question.

Hot Network Questions

grep to extract a substring from a huge string

1 Answer 1

You must log in to answer this question.

Related

Hot Network Questions