Most web scraping activities aim to fetch data as quickly as possible. In this article, we will talk about 5 tips you can follow to overcome blocking. Nevertheless, there are still ways to fight against blocking. More and more web owners have equipped their sites with all kinds of anti-scraping techniques to block scrapers, which makes web scraping more difficult. A straightforward example is when web scraping overloads a web server and leads to a server breakdown. While various web scraping tools, like Octoparse, are getting popular around and benefits people substantially in all fields, they come with a price for web owners. Set up a wait time in 'Advanced Options' for steps like 'Click Item' or 'Extract Data' can effectively avoid data skip and make the crawling process more human-like! (Usually 2-5 seconds would work well).Web scraping is a technique often employed for automating human’s browsing behavior for the purpose of retrieving large amounts of data from webpage efficiently. Data fields extracted will be displayed in 'Data field' pane next to the workflow designer. Notice an 'Extract data' step gets auto-generated and added to the workflow. Select 'Extract data' from 'Action Tips' to complete the extraction action when you finish selecting.Click on target data fields such as title, review, price, etc.Now, set up an extraction template by designating the specific data fields to capture from the page Octoparse will apply this template to the other item pages. Once the 'Loop Item' is completed, Octoparse will load the first item page in the built-in browser. So when you select an item with URL, the selected tag would be 'A', which stands for anchor that usually links one page to another.Ģ) Select details on the item page to extract Octoparse automatically identifies tags of selected items. To loop click through items on the list, it is important that you select the anchor texts. Notice a 'Loop Item' for the clicking action is being auto-generated and added to the workflow. Select 'Loop click each URL' from 'Action Tips'.

Click on the second product title containing the URL.The selected item will be highlighted in green while items with the same layout will be highlighted in red. Click on the first product title that contains the URL to access the item page.To do this, we will create a 'Loop Item' to loop click each product link on the result page. ġ) Select the links to click into individual item page Let’s see how it is done with an example.

Octoparse loop wait