Crawling Selenium Python

Question

I try to crawl this page of crédit suisse (It's an exercice for me).

So I made this script, but I don't know how to get the data. I thought it is an iframe problem, but, it's not. Then I thought it is an AngularJS website, but I think it's not.

So my code is :

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Firefox()
driver.get("https://tas-creditsuisse.taleo.net/careersection/campus/moresearch.ftl?lang=fr-FR")

iframe = driver.find_element_by_xpath("//iframe[@id='ftlhiddenframe']")

thepage = driver.switch_to.frame(iframe)

webpage = ""
webpage = driver.page_source

webpage = "<body>"+webpage+"</body>"

import os
helloFile = open('C:\\Users\\Thie\\Desktop\\cs.html', 'w')
helloFile.write(webpage)
helloFile.close()

driver.close()

Somebody can give me the way to get this webpage? Thanks a lot for your help.

The solution would be to make like the firefox inspector like on the picture below (select the TBody and save it in a file):

It is not clear what is wrong and what you want to get. Also are you getting an error, if yes - which? — Andrei
– Andrei, Commented Sep 11, 2019 at 9:13
I try to get all joboffer of this page, but I can't see the HTML text to get all joboffer link. Do you have a solution ? the solution would be to make the same as Firefox inspector — cretthie
– cretthie, Commented Sep 11, 2019 at 9:17

Andrei · Accepted Answer · 2019-09-11 10:19:18Z

1

First of all you don't need to switch to iframe. Delete this code:

iframe = driver.find_element_by_xpath("//iframe[@id='ftlhiddenframe']")
thepage = driver.switch_to.frame(iframe)

Second, if I right understand you, you want to get all links to the jobs on the page. You can use this selector:

//span[@class = 'titlelink']/a

in code it will be something like this:

listOfAllJobLinksOnThePage = driver.find_element_by_xpath("//span[@class = 'titlelink']/a")

PS remember, you will get only 25 links, if you want more you have to go to other pages and do the same

edited Sep 11, 2019 at 10:19

answered Sep 11, 2019 at 9:25

Andrei

5,6675 gold badges27 silver badges51 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

cretthie Over a year ago

Thanks ;) it works ! Thanks ! You're right, it was not clear :) I need the link to go later on the job offer page and then get and crawl the job offer.

Andrei Over a year ago

If you are satisfied with my help, you can upvote and/or accept my answer by clicking on the check button below upvote button

Collectives™ on Stack Overflow

Crawling Selenium Python

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related