Python Phantomjs Loading Webpage Not Correct

I have an issue where extracting from this link http://www.bursamalaysia.com/market/listed-companies/company-announcements/#/?category=FA&sub_category=FA1&alphabetical=All&

Solution 1:

When you are saving the source the page is not completely loaded with your submitted post so try to wait for a couple of second before fetching the page source:

defgetURLS(url):
driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true'])
driver.get(url) # load the web page
time.sleep(5)# waiting for 5 seconds before fetching the source
src = driver.page_source
#Get text and split it
soup = BeautifulSoup(src, 'html5lib')

print soup

To perform dropdown select you have import the Select class as follow : from selenium.webdriver.support.ui import Select and then you have to select the dropdown element like that:

category_select =Select(driver.find_element_by_xpath('//*[@id="bm_announcement_types"]'))
category_select.select_by_visible_text('Financial Results')

In my example I've done it for the -Category- dropdown, follow the exact steps for every category. Note that selecting the dropdown by xpath is the best way and you can achieve this by using Google Chrome -> righ click on the element -> Inspect-> right click on the <select> in the right menu that appeared -> Copy -> Copy Xpath

When you`ve selected all the element you have to click the Submit and wait for a couple of seconds to load and after that you will fetch the source code.

Let me know if my answer helped you.