Requests - Fetch Data From Api-based Website

I want to get all the review from this site. at first, I use this code: import requests from bs4 import BeautifulSoup r = requests.get( 'https://www.traveloka.com/hotel/singap

Solution 1:

Look at the request payload at the network tab. There is a part where skip:8 and top:8 and you will see those numbers increment by 8 when you click on the right arrow to get the next page of reviews.

You can duplicate that request and scrape the results the same way

Edit:

Open your page with chrome and hit f12. Go to Network tab, scroll down at the bottom of your page where you can advance to the next batch of reviews. As soon as you hit the right arrow the network tab will be populated. Find the second hotelReviewAggregate and click on it. Under the headers tab you will find Request Payload. Open the data dict and find skip and top. Advance the next batch of reviews and see how those numbers change. You can simulate this behavior to get to the other pages.

Then what you need to do is to prepare your payload where you can increment the values and make GET requests and use the response objects to scrape the data with BeautifulSoup.

Requests here

Quick Example from the tutorial:

payload = {'key1': 'value1', 'key2': 'value2'} r = requests.get('http://httpbin.org/get', params=payload)

I don't know why people decided to give a negative value to my answer without an explanation. But ohh well, If you find this useful and answers your question, please accept it.