由于Scrapy该库在Windows下会存在兼容问题,下面介绍的是在Linux系统进行安装。
1、关于pip3命令的报错
报错代码:
error: externally-managed-environment × This environment is externally managed ╰─> To install Python packages system-wide, try apt install python3-xyz, where xyz is the package you are trying to install. If you wish to install a non-Debian-packaged Python package, create a virtual environment using python3 -m venv path/to/venv. Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make sure you have python3-full installed. If you wish to install a non-Debian packaged Python application, it may be easiest to use pipx install xyz, which will manage a virtual environment for you. Make sure you have pipx installed. See /usr/share/doc/python3.11/README.venv for more information. note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages. hint: See PEP 668 for the detailed specification.
解决方法:在原有命令后面追加参数 --break-system-packages
因为scrapy可以直接当做命令使用,所以我们也可以在终端输入不存在的命令,它会提示你安装
有时候pip3不行,我们也可以换pipx,或者apt install python3-文件名来进行安装
pipx install scrapy
apt install python3-scrapy
2、执行scrapy报错(Python3下的OpenSSL模块出错)
报错代码:
Traceback (most recent call last): File "/usr/bin/scrapy", line 33, in <module> sys.exit(load_entry_point('Scrapy==2.10.0', 'console_scripts', 'scrapy')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/bin/scrapy", line 25, in importlib_load_entry_point return next(matches).load() ^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 202, in load module = import_module(match.group('module')) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.11/importlib/__init__.py", line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "<frozen importlib._bootstrap>", line 1206, in _gcd_import File "<frozen importlib._bootstrap>", line 1178, in _find_and_load File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed ... AttributeError: module 'lib' has no attribute 'SSL_CTX_set_ecdh_auto'
这个是Python3下的OpenSSL模块出错,我们需要卸载它并重装
解决方法:
pip3 uninstall pyopenssl pip3 install pyopenssl
尝试了一些命令发现不行
3、卸载pyopenssl时报错
报错代码:
Found existing installation: pyOpenSSL 21.0.0 Not uninstalling pyopenssl at /usr/lib/python3/dist-packages, outside environment /usr Can't uninstall 'pyOpenSSL'. No files were found to uninstall. Requirement already satisfied: pyopenssl in /usr/lib/python3/dist-packages (21.0.0)
解决办法:
找到该文件夹所在路径,直接将pyOpenSSL-21.0.0目录删掉,再重新安装
该文件夹在 /usr/lib/python3/dist-packages/ 下面
删掉整个文件夹
重新使用命令安装pyopenssl
pip3 install pyopenssl --break-system-packages
安装成功后输入scrapy命令
如下图,则代表该库已安装成功
我们也可以进入ipython3进行验证:
ipython是一个python的交互式shell,比默认的python shell好用很多,支持变量自动补全,自动缩进,支持bash shell命令,内置了许多很有用的功能和函数
只要可以成功导入该库,没有报错,证明已经安装成功