This is mandatory.
Take the list of urls used as the starting point.

// add the urls as the starting point
scaper.startWithURLs(['www.googl.com','www.bing.com'])
scaper.startWithURLs('www.googl.com')

Takes a non async callback function as argument,url added to processing queue only if the function return true value.
This is optional. By default is accept all urls added to processing queue.

// accept url contains www.google.com
scraper.allowIfMatches(function(url) {  
  return url.indexOf('www.google.com')>-1 
})

This is optional setting.
This will save your progress in the file and you can stop and start the scraper from the previous state.
The file is a sqlite db file you can modify the content using sqllite clients.
If no file specified the stored in memory..

// state stored in state.db file
scraper.saveProgressInFile("./state.db")

This will allow the scraper to automatically download all the links form the page and add to processing queue.
Note the urls will be filtered if allowIfMatches function return 'false'.

scraper.enableAutoCrawler(true)

Time delay between each page load in milliseconds

//wait for 90 milliseconds between page load
scraper.waitBetweenPageLoad(90)

Final callback when scarping is completed

scraper.callbackOnFinish(function(result){
  console.log(result)
})

This is the main function.Your scarping logic to be defined in the function.
This called for each page in the processing queue.
Called with pupetter page object as input.
The page object input got addtional methods to support scraping

scraper.waitBetweenPageLoad(90)

To start the scraping process. callbackOnFinish function is called once the scraping is completed.

scraper.start()

Download image from url and save to local disk

scraper.callbackOnPageLoad(async function(page){
var img = await page.$('img')
var img_src = await page.evaluate(img => img.getAttribute("src"), img);
 page.download_image(img_src,"usr/test/profile.png")
})

Save the text result ,this will returned as input to callbackOnFinish function
Each url can store one result

scraper.callbackOnPageLoad(async function(page){
  var article = await page.$eval('article', tag => tag.innerText);
  page.saveResult(article)
})

Write text content to local file

scraper.callbackOnPageLoad(async function(page){
  var article = await page.$eval('article', tag => tag.innerText);
  page.download_image(article,"usr/test/article.txt")
});

Add the url to processing queue

scraper.callbackOnPageLoad(async function(page){
   var a = await page.$('a')
   var url = await page.evaluate(a => a.getAttribute("href"), a);
   page.add_url_to_queue(url)
});