I have list of 100 proxies list where should i configure in Selfhostable instance?

[ranaharoon3222]

I have list of 100 proxies list with each one has a string of ip and username and password i want to use it all at once so each url hit with unique proxy ip

where should i configure in Selfhostable instance? in configuiration file? if yes how?

or in frontend application if yes how?

please guide me thanks a lot for this great project

[ranaharoon3222]

Thanks a lot can you let me know documentation for this please?

[us]

Hey @ranaharoon3222 — thanks again for the clear write-up and for using crw! 🙏

Great news: proxy list + rotation just shipped (#144), and it does exactly what you asked — give crw your pool and it rotates so each request egresses from a different IP.

Your case: 100 proxies, a unique IP per request

In config.toml:

[crawler]
proxy_list = [
  "http://user:pass@ip1:port",
  "http://user:pass@ip2:port",
  # … all 100 …
]
proxy_rotation = "round_robin"   # cycle through the pool, one per request

Or via env (handy for Docker) — comma-separated, JSON array, or TOML array all work:

CRW_CRAWLER__PROXY_LIST="http://user:pass@ip1:port,http://user:pass@ip2:port"
CRW_CRAWLER__PROXY_ROTATION="round_robin"

You can also send a pool per request (proxy_list + proxy_rotation in the body) on /v1 and /v2.

A few things worth knowing

• Strategies: sticky_per_host (default — pins each host to one IP, best against anti-bot/session checks), round_robin (what you want for "unique IP per request"), and random.
• Coverage: rotation applies to the HTTP path and the JS/Chrome path, for scrape, crawl, and map.
• Fail-closed: a bad proxy is a hard error — crw never silently falls back to your real IP.
• SOCKS5 works on the HTTP path (auth too). On the JS/Chrome path use http/https or a credential-less SOCKS proxy (Chrome can't authenticate SOCKS).

📖 Full guide: https://docs.fastcrw.com/proxies

It'll be in the next release; until then it's on main (build from source or ghcr.io/us/crw once the image updates). If you hit anything or want a strategy that isn't there yet, reply here or drop into Discord — happy to help. Thanks again for pushing on this! 🚀

[us]

Update: this is now live in v0.16.0 🎉 — cargo install crw-server, or pull ghcr.io/us/crw once the release image finishes building. Docs: https://docs.fastcrw.com/proxies. Enjoy, and thanks again for the request! 🚀