定时抓取免费代理网站,简易可扩展。(非翻墙)

崔庆才丨静觅 74ea5816d7 Merge pull request #54 from muyun001/feature/修改example 5 years ago
examples a98c2b7a95 修改example举例 5 years ago
proxypool e2675c0a3f Update db.py 6 years ago
.gitignore 4483901afc fix pop 8 years ago
LICENSE a0593a31ae add free proxy 8 years ago
README.md 9212ceca3a Update README.md 7 years ago
importer.py 133b4e0890 update 8 years ago
proxy provider.txt e48c88a3a8 added a note about proxy providers 7 years ago
requirements.txt 2cafcfdf61 add password 8 years ago
run.py 133b4e0890 update 8 years ago

README.md

ProxyPool

安装

安装Python

至少Python3.5以上

安装Redis

安装好之后将Redis服务开启

配置代理池

cd proxypool

进入proxypool目录,修改settings.py文件

PASSWORD为Redis密码,如果为空,则设置为None

安装依赖

pip3 install -r requirements.txt

打开代理池和API

python3 run.py

获取代理

利用requests获取方法如下

import requests

PROXY_POOL_URL = 'http://localhost:5555/random'

def get_proxy():
    try:
        response = requests.get(PROXY_POOL_URL)
        if response.status_code == 200:
            return response.text
    except ConnectionError:
        return None