Use a proxy with Scrapy

Get your details

Copy the host, port, username and password for your proxy from the dashboard. Where this page writes <host>, <port>, <username> or <password>, paste your own values.

Set it in the spider

With credentials in the URL, the middleware adds the Proxy-Authorization header for you.

import os
import scrapy


class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        proxy = "http://{}:{}@{}:{}".format(
            os.environ["PROXY_USER"],
            os.environ["PROXY_PASS"],
            os.environ["PROXY_HOST"],
            os.environ["PROXY_PORT"],
        )
        yield scrapy.Request(
            "https://api.ipify.org",
            meta={"proxy": proxy},
        )

    def parse(self, response):
        yield {"ip": response.text}

Pace the crawl

Set DOWNLOAD_DELAY and CONCURRENT_REQUESTS_PER_DOMAIN so no single address sends more than a person would.