环境
依赖安装
pip install flask-cors flask selenium
安装chromedriver
mac下安装selenium+phantomjs+chromedriver
实现代码
1、hook.js
监听 XMLHttpRequest 请求
// 打开链接,复制代码到这里 // https://unpkg.com/ajax-hook@2.0.3/dist/ajaxhook.min.js // https://unpkg.com/axios/dist/axios.min.js ah.proxy({ //请求成功后进入 onResponse: (response, handler) => { if (response.config.url.startsWith('/api/movie')) { axios.post('http://localhost:5000/receiver/movie', { url: window.location.href, data: response.response }) console.log(response.response) handler.next(response) } } })
2、main.py
驱动chrome
# -*- coding: utf-8 -*- from selenium import webdriver import time browser = webdriver.Chrome() browser.get('https://dynamic2.scrape.center/') browser.execute_script(open('hook.js').read()) time.sleep(2) for index in range(3): print('current page', index) btn_next = browser.find_element_by_css_selector('.btn-next') btn_next.click() time.sleep(2) browser.close() browser.quit()
3、server.py
接收数据的服务,可以进一步将数据存入数据库
# -*- coding: utf-8 -*- import json from flask import Flask, request, jsonify from flask_cors import CORS app = Flask(__name__) CORS(app) @app.route('/receiver/movie', methods=['POST']) def receive(): content = json.loads(request.data) print(content) # to something return jsonify({'status': True}) if __name__ == '__main__': app.run(host='0.0.0.0', port=5000, debug=True)