How to scrape AJAX based website data

Question

I am unable to get data from https://www.propertyfinder.ae/en/search?bf=1&bt=2&c=2&fu=1&l=39&ob=pa&rp=y&t=1 

It will take only 25 first full records and prices for all records. After 25 records it miss the names and URL only prices will extract. Need your expert opinion on this matter. I am using UiPath Version 2019.10.0. There are 3 pages record it only take all details which I selected for first 25 records only. After that it will pick only prices. Here is the screenshot of csv file.

Also share some ajax based web scraping. And what it I want to save all image files from scraping?

Thanks.

Comments

Hey @Adnan, could you share the script that you have used that is fetching these records?

Answer 1

Hi Adnan,

Before you scrape any website/webpage, you must check if the administrator of that site has blocked scrapping Administrators block it because crawling affects the performance of the website

. In case you continue to scrape such a website, your ISP may block (blacklist)you.  I checked the website you mentioned and  they have blocked scrapping. Hence trying any method might just end up your ISP to block you(blacklist) or worse legal action for scrapping.

read this link for further reference: https://www.scrapehero.com/how-to-prevent-getting-blacklisted-while-scraping/

https://www.propertyfinder.ae/robots.txt as per this link, ajax scrapping is not allowed. Please talk to the respective service provider to seek permission to scrape/crawl their content

Once you seek their permission, you can use once of the methods mentioned in this link. 

https://www.codementor.io/codementorteam/how-to-scrape-an-ajax-website-using-python-qw8fuitvi

Regards,

RRR

PS:

If my post has answered your question, please mark it as answer. Feel free to post your thoughts. !!