selenium webdriver in python

Question

So basically I'm trying to make script in python that uses selenium webdriver to open and login to a website called marketsworlds and retrieve the market value of a stock. I have the script able to open and login into the page. I just cant figure out how to capture/get the market value and have it print out. I used inspect element to find the class:

<p class="market_value"> 

</p>

in between the open and close brackets inspect element displays the market value, which constantly changes. I tried setting driver.find_element_by_class("market_value") to a variable and printing the variable. I get a print out of " at object 0x" and what ever comes after the x. any way to return what it actually displays?

share the url of webpage plz

John Dene
– John Dene

2015-06-25 16:09:34 +00:00
Commented Jun 25, 2015 at 16:09 — John Dene
– John Dene, Commented Jun 25, 2015 at 16:09
the url is [link]marketsworld.com

dvitt90
– dvitt90

2015-06-25 16:11:46 +00:00
Commented Jun 25, 2015 at 16:11 — dvitt90
– dvitt90, Commented Jun 25, 2015 at 16:11

That1Guy · Accepted Answer · 2015-06-25 16:12:19Z

If you have to use Selenium for navigation, such as on JavaScript-heavy sites, It would suggest acquiring the page source and using an HTML parser to extract the data you want.

BeautifulSoup is a great choice of parser. For example:

html = driver.page_source
soup = BeautifulSoup(html)

# Get *all* 'p' tags with the specified class attribute.
p_tags = soup.findAll('p',{'class':'market_value'})
for p in p_tags:
    print p.text

This should print to screen the text contained in the <p> tags with the class market_value. Its hard to give specifics without knowing the exact page source, however.

However, if you're determined to use strictly Selenium, you can find these elements by:

# Get *all* 'p' tags with the specified class attribute.
elements = driver.find_elements_by_class_name('market_value')
for element in elements:
    print element.text

# or

# Get a *single* 'p' tag with the specified class attribute.
element = driver.find_element_by_class_name('market_value')
print element.text

thank you for the help. I am kinda new to using python for extracting info from webpages. i decided to use selenium as a start. i would much rather use a different way if you have any suggestions that would achieve the same thing.
Take a look at urllib2. it might help shed some light on when Selenium is and is not required for scraping. Also take a look at the BeautifulSoup documentation I linked to in the answer. It will hopefully show you how easy HTML parsing is when done properly.

Collectives™ on Stack Overflow

selenium webdriver in python

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related