Code examples

Common tasks in cURL, Python and TypeScript.

The examples use the official SDKs (pip install scrapecrate, npm install scrapecrate) and read the key from the SCRAPECRATE_API_KEY environment variable. Every call works the same way with plain HTTP: a GET request to https://api.scrapecrate.com/v1 with the key in the X-API-Key header.

Every page of a list (cursor pagination)

Example: all reviews of a Google Maps place, 20 per page (3 credits per page).

cursor=""
while :; do
  page=$(curl -s "https://api.scrapecrate.com/v1/maps/places/ChIJ2eUgeAK6j4ARbn5u_wAGqWA/reviews?limit=20&sort=newest${cursor:+&cursor=$cursor}" \
    -H "X-API-Key: $SCRAPECRATE_API_KEY")
  echo "$page" | jq -c '.data[]'
  cursor=$(echo "$page" | jq -r '.pagination.next_cursor // empty')
  [ -z "$cursor" ] && break
done
import os
from scrapecrate import ScrapeCrate

client = ScrapeCrate(api_key=os.environ["SCRAPECRATE_API_KEY"])

# iterate() follows pagination.next_cursor for you
for review in client.maps.iterate("place_reviews", place_id="ChIJ2eUgeAK6j4ARbn5u_wAGqWA", sort="newest", limit=20):
    print(review["rating"], review["text"])
import { ScrapeCrate } from 'scrapecrate';

const client = new ScrapeCrate({ apiKey: process.env.SCRAPECRATE_API_KEY! });

for await (const review of client.maps.iterate('placeReviews', { place_id: 'ChIJ2eUgeAK6j4ARbn5u_wAGqWA', sort: 'newest', limit: 20 })) {
  console.log(review.rating, review.text);
}

Each page is one call and costs the same credits, so the total cost depends on how many pages the place has. Stop early (break out of the loop) when you have enough, for example when you reach reviews older than the last run: with sort=newest that makes a daily sync cost only the pages with new reviews.

Keep an eye on credits

Every response says what it cost (meta.credits_used) and what is left (meta.credits_remaining).GET /v1/account returns the balance for free.

curl "https://api.scrapecrate.com/v1/account" -H "X-API-Key: $SCRAPECRATE_API_KEY"

Handle errors

Errors arrive as {"error": {"code", "message", "request_id", "details"}} and are never billed. The SDKs retry rate limits and temporary 5xx errors for you, then raise a typed error; catch the ones your code can act on and log the request id of the others (see errors).

from scrapecrate import ScrapeCrate, NotFoundError, InsufficientCreditsError, ApiError

client = ScrapeCrate()  # reads SCRAPECRATE_API_KEY
try:
    company = client.trustpilot.company(domain="example.com").data
except NotFoundError:
    company = None          # not billed
except InsufficientCreditsError as e:
    print("Top up needed:", e.details)
except ApiError as e:
    print(e.status, e.code, e.request_id)

Combine APIs with one key

One key and one balance work on every API, so a single script can join the data: here, restaurants found on Google Maps with the Trustpilot score of their website.

client = ScrapeCrate()

# Restaurants in Milan with their Trustpilot score, when they have a website
places = client.maps.places_search(q="restaurants in Milan", limit=20).data
for place in places:
    website = place.get("website")
    if website:
        domain = website.split("//")[-1].split("/")[0].removeprefix("www.")
        try:
            score = client.trustpilot.company(domain=domain).data["trust_score"]
        except Exception:
            score = None
        print(place["name"], place["rating"], score)