How do I use proxies with the CROWler?

[pzaino]

A common question I usually receive is: How do I use/configure proxy services (for example like smartproxy.com) with the CROWler?

[pzaino]

The answer to this question is, you need to configure the proxy service you use (doesn't matter if it's Data center IPs, Residential IPs etc.) in the VDI section of your config.yaml:

vdi:
  - name: vdi_japan_01
    location: Japan
    type: chrome
    host: ${VDI_JAPAN_HOST_01}
    path: ""
    port: 4444
    proxy_url: "http://your_proxy_server:port"

Where:

• You can use both http and https for your proxy
• your_proxy_server is either the IP or the FQDN to your proxy (for example with smartproxy.com it should be something like dc.smartproxy.com for the Data Center proxies/IPs)
• port is the port number, for example 8080

Please note(1): Because the CROWler also uses Selenium and Selenium does NOT support user and password for the proxy configuration, at the moment I reccomend to use whitelisted your server IP on your proxy service. Given the proxy is used in multiple places in the CROWler architecture, this limitation is only on the selenium side, NOT on other areas of the CROWler, however to keep your configuration consistent, you probably want to just use a white listed IP.

Please note (2): If you are using a version of the CROWler before release 1.0.0 then the vdi: section is called selenium: on the pre-releases. This is because at the beginning the VDI image was only a selenium image, but over time it gained a lot more tools and services and selenium became just an API + fallback mechanism for when the new tools like Rbee, tshark etc. do not work or are not available.

HTH