构建快速部署的python pip环境及pypi本地源环境-阿里云开发者社区

开发者社区> 技术小胖子> 正文

构建快速部署的python pip环境及pypi本地源环境

简介:
+关注继续查看

前言:

大家用pip安装模块有没有遇见过这样的情况:

161732587.jpg

对的,这就是抽风的timeout

昨天看到一个朋友慢腾腾给所有节点搞tornado,他的源是官方的源,官方的源时不时的会抽风,会慢的。所以我推荐他用国内的源,或者是连接我们自己搭建的源。

由此

我就跟大家扯一下如何构建快速的python 模块的环境 ~


其实国内很多大公司都有自己的pypi源的,只是好多都私有环境的

下面都是国内速度比较快的节点


指定pypi源的方法:


1
pip install tornado -i http://pypi.sdutlinux.org/simple


143728730.jpg



也可以是全局的模式


在unix和macos,配置文件为:$HOME/.pip/pip.conf

在windows上,配置文件为:%HOME%\pip\pip.ini


需要在配置文件内加上:


1
2
[global]
index-url=http://mirrors.tuna.tsinghua.edu.cn/pypi/simple



以前我都是用豆瓣的源,现在豆瓣的源貌似有问题啦。。。哎,可惜啦

提示:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
Downloading/unpacking tornado
  Getting page http://pipy.douban.com/simple/tornado/
  Could not fetch URL http://pipy.douban.com/simple/tornado/: <urlopen error [Errno -2] Name or service not known>
  Will skip URL http://pipy.douban.com/simple/tornado/ when looking for download links for tornado
  Getting page http://pipy.douban.com/simple/
  Could not fetch URL http://pipy.douban.com/simple/: <urlopen error [Errno -2] Name or service not known>
  Will skip URL http://pipy.douban.com/simple/ when looking for download links for tornado
  Cannot fetch index base URL http://pipy.douban.com/simple/
  URLs to search for versions for tornado:
  * http://pipy.douban.com/simple/tornado/
  Getting page http://pipy.douban.com/simple/tornado/
  Could not fetch URL http://pipy.douban.com/simple/tornado/: <urlopen error [Errno -2] Name or service not known>
  Will skip URL http://pipy.douban.com/simple/tornado/ when looking for download links for tornado
  Could not find any downloads that satisfy the requirement tornado



想用更多的源,到这里看看 www.pypi-mirrors.org


144458553.jpg


可以看到好多的节点的运行情况和速度。


我们的监控系统是python开发的,客户端当然也是python,时常需要大量的部署,最开始我们用的是反向代理做缓存,效果还是很不错的。

我们在用puppet批量部署监控客户端的时候,会让pip指定我们已经反向缓存的pypi的地址、版本库  


比如:


pip install -i https://10.2.20.66/qinghua


意思就是从清华那里搞到包。我想大家都应该熟悉缓存代理吧,大家自己加个location指定proxy_pass就行啦~


一个例子:

(我这里就不详细说明意思了,大家可以看看我写过的nginx反向代理的文章。。。)


install


1
2
3
4
5
6
7
8
9
10
11
ulimit -SHn 65535
yum install pcre pcre-devel -y 安装pcre
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              
wget http://labs.frickle.com/files/ngx_cache_purge-1.4.tar.gz
tar zxvf ngx_cache_purge-1.4.tar.gz
wget http://nginx.org/download/nginx-1.0.11.tar.gz
tar zxvf nginx-1.0.11.tar.gz
cd nginx-1.0.11/
./configure --user=www --group=www --add-module=../ngx_cache_purge-1.4 --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module
make && make install
cd ../


关键配置


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
proxy_cache_path  /var/lib/nginx/cache/ levels=1:1:2 inactive=24000h keys_zone=cache:100m;
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
server {
        listen   8000 default;
        server_name  localhost;
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        access_log  /var/log/nginx/localhost.access.log;
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        #中间省略部分默认配置
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
        location /guanfang {
                proxy_pass http://pypi.python.org/simple;
        proxy_cache cache;
        proxy_cache_valid  any 2400h;
        }
        location /douban {
            proxy_pass http://pipy.douban.com/simple;
            proxy_cache cache;
            proxy_cache_valid  any 24000h;
            proxy_redirect off;
            proxy_set_header Host $host;
            proxy_cache cache_one;
            proxy_cache_valid 200 302 1h;
            proxy_cache_valid 301 1d;
            expires 30d;
    }
    location /taiwan {
            proxy_pass http://mirrors.tuna.tsinghua.edu.cn/pypi/simple;
            proxy_cache cache;
            proxy_cache_valid  any 24000h;
            proxy_redirect off;
            proxy_set_header Host $host;
            proxy_cache cache_one;
            proxy_cache_valid 200 302 1h;
            proxy_cache_valid 301 1d;
            expires 30d;
        }
    location /jiaoyu {
            proxy_pass http://mirrors.tuna.tsinghua.edu.cn/pypi/simple;
            proxy_cache cache;
            proxy_cache_valid  any 24000h;
            proxy_redirect off;
            proxy_set_header Host $host;
            proxy_cache cache_one;
            proxy_cache_valid 200 302 1h;
            proxy_cache_valid 301 1d;
            expires 30d;
        }
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    
}


我们讲了用国内国外的pypi节点来快速部署你的环境,来说下,我们线上的部署方法:

个人觉得这个适用于需要大批量部署,又有模块定制,而有些节点不能联外网情况下适用。

要是大家的环境不大,推荐用 -i 的方式。 毕竟搭建私有pypi服务是很折腾的事~


安装pypimirror

1
pip install z3c.pypimirror


要是安装不上的话,可以用官方的方式:



1
2
3
4
5
6
$ git clone https://github.com/macagua/macagua.buildout.pypimirror.git
$ virtualenv .
$ source ./bin/activate
$ python bootstrap.py
$ ./bin/buildout -vvvN
$ deactivate




安装完毕就要填写配置文件,需要填写的参数主要有:

mirror_file_path 下载的包的存放路径

base_url 服务器地址,这个注意要和Apache上的一致!

create_indexes 布尔类型,用来在下载的每个包目录下创建索引

package_matches 这个是用户自定义的,PyPI包无数,使用正则表达式有选择地下载,要不然硬盘要爆了

lock_file_name 设置运行时锁状态文件存放位置

log_filename 设置日志文件存放位置


生成配置索引

1
pypimirror -c -v -I pypimirror.cfg


这里很慢的,就算你用国内的源,也会让你气的抽风,骂的吐血~


021103810.jpg

我是在公司线上机房做的测试,居然还是同步出问题,想看看抽风不?

021519688.jpg


也可以修改配置文件


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
DEFAULT]
# the root folder of all mirrored packages.
if necessary it will be created for you
mirror_file_path = /home/pypimirror/paquetes
# where's your mirror on the net?
base_url = http://pypi.python.jp/
# lock file to avoid duplicate runs of the mirror script
lock_file_name = /home/pypimirror/pypi-poll-access.lock
# Pattern for package files, only those matching will be mirrored
filename_matches =
    *.zip
    *.tgz
    *.egg
    *.tar.gz
    *.tar.bz2
# Pattern for package names; only packages having matching names will
# be mirrored
package_matches =
#   zope.*
#   plone.*
#   Products.*
#   collective.*
   *.*
# remove packages not on pypi (or externals) anymore
cleanup = True
# create index.html files
create_indexes = True
# be more verbose
verbose = True
# resolve download_url links on pypi which point to files and download
# the files from there (if they match filename_matches).
# The filename and filesize (from the download header) are used
# to find out if the file is already on the mirror. Not all servers
# support the content-length header, so be prepared to download
# a lot of data on each mirror update.
# This is highly experimental and shouldn't be used right now.
#
# NOTE: This option should only be set to True if package_matches is not
set to '*' - otherwise you will mirror a huge amount of data. BE CAREFUL
# using this option!!!
external_links = False
# similar to 'external_links' but also follows an index page if no
# download links are available on the referenced download_url page
# of a given package.
#
# NOTE: This option should only be set to True if package_matches is not
set to '*' - otherwise you will mirror a huge amount of data. BE CAREFUL
# using this option!!!
follow_external_index_pages = False
# logfile
log_filename = /home/pypimirror/pypimirror.log



更新

1
pypimirror -c -v -i -U pypimirror.cfg


参数说明:

-i 创建索引

-U 更新镜像。



假设前面设置的下载文件存放路径为/data/pypi/files,下面把它链接到/var/www/目录下:

ln -s /data/pypi/files /var/www/pypi

重启Apache,访问http://xiaorui.cc/pypi/ 应该就可以了。这是最简单的配置,这时的base_url就是 http://xiaorui.cc/pypi/ 。


让他定期更新下

003618269.jpg


安装pypimirror遇到的问题:


1
Traceback (most recent call last):  File "/usr/local/lib/python2.6/dist-packages/bottle.py", line 764in _handle    return route.call(**args)  File "/usr/local/lib/python2.6/dist-packages/bottle.py", line 1575in wrapper    rv = callback(*a, **ka)  File "hydenMain.py", line 39in static    return static_file(filename, root='{}/static'.format('.'))ValueError: zero length field name in format


大家把python升级到2.7就行了

004625758.jpg


需要说明的是,本地的pypi安装的时候,会出现各种各样的问题,请大家都尝试和搜下问题所在。 我们可以把问题都统计下来,好让也遇到这样的问题的人,能更好的定位和解决问题。



 本文转自 rfyiamcool 51CTO博客,原文链接:http://blog.51cto.com/rfyiamcool/1290456,如需转载请自行联系原作者



版权声明:本文内容由阿里云实名注册用户自发贡献,版权归原作者所有,阿里云开发者社区不拥有其著作权,亦不承担相应法律责任。具体规则请查看《阿里云开发者社区用户服务协议》和《阿里云开发者社区知识产权保护指引》。如果您发现本社区中有涉嫌抄袭的内容,填写侵权投诉表单进行举报,一经查实,本社区将立刻删除涉嫌侵权内容。

相关文章
阿里云服务器怎么设置密码?怎么停机?怎么重启服务器?
如果在创建实例时没有设置密码,或者密码丢失,您可以在控制台上重新设置实例的登录密码。本文仅描述如何在 ECS 管理控制台上修改实例登录密码。
10061 0
Python获取本机所有IP地址
import socket # 查看当前主机名 print('当前主机名称为 : ' + socket.gethostname()) # 根据主机名称获取当前IP print('当前主机的IP为: ' + socket.
1601 0
阿里云服务器如何登录?阿里云服务器的三种登录方法
购买阿里云ECS云服务器后如何登录?场景不同,大概有三种登录方式:
2959 0
Python 图像处理篇-利用opencv库和numpy库读取包含中文路径下的本地图片实例演示
Python 图像处理篇-利用opencv库和numpy库读取包含中文路径下的本地图片实例演示
17 0
阿里云服务器端口号设置
阿里云服务器初级使用者可能面临的问题之一. 使用tomcat或者其他服务器软件设置端口号后,比如 一些不是默认的, mysql的 3306, mssql的1433,有时候打不开网页, 原因是没有在ecs安全组去设置这个端口号. 解决: 点击ecs下网络和安全下的安全组 在弹出的安全组中,如果没有就新建安全组,然后点击配置规则 最后如上图点击添加...或快速创建.   have fun!  将编程看作是一门艺术,而不单单是个技术。
10882 0
使用OpenApi弹性释放和设置云服务器ECS释放
云服务器ECS的一个重要特性就是按需创建资源。您可以在业务高峰期按需弹性的自定义规则进行资源创建,在完成业务计算的时候释放资源。本篇将提供几个Tips帮助您更加容易和自动化的完成云服务器的释放和弹性设置。
12070 0
Python 技术篇-用os库实现本地文件重命名实例演示
Python 技术篇-用os库实现本地文件重命名实例演示
6 0
21119
文章
0
问答
文章排行榜
最热
最新
相关电子书
更多
《2021云上架构与运维峰会演讲合集》
立即下载
《零基础CSS入门教程》
立即下载
《零基础HTML入门教程》
立即下载