本文介绍通过离线的方式安装airflow 1.10.9,将安装包上传至服务器。
1. 安装必要的组件
1.1安装gcc
进入dependencies目录,依次执行安装(注意顺序)
1. rpm -ivh kernel-headers-3.10.0-1062.12.1.el7.x86_64.rpm
2. rpm -ivh glibc-headers-2.17-292.el7.x86_64.rpm
3. rpm -ivh glibc-devel-2.17-292.el7.x86_64.rpm
4. rpm -ivh mpfr-3.1.1-4.el7.x86_64.rpm
5. rpm -ivh libmpc-1.0.1-3.el7.x86_64.rpm
6. rpm -ivh cpp-4.8.5-39.el7.x86_64.rpm
7. rpm -ivh gcc-4.8.5-39.el7.x86_64.rpm
1.2安装openssl-devel, 依次执行安装(注意顺序)
1. rpm -ivh libverto-devel-0.2.5-4.el7.x86_64.rpm
2. rpm -ivh keyutils-libs-devel-1.5.8-3.el7.x86_64.rpm
3. rpm -ivh libcom_err-devel-1.42.9-16.el7.x86_64.rpm
4. rpm -ivh libsepol-devel-2.5-10.el7.x86_64.rpm
5. rpm -ivh krb5-libs-1.15.1-37.el7_7.2.x86_64.rpm --force
6. rpm -ivh libkadm5-1.15.1-37.el7_7.2.x86_64.rpm
7. rpm -ivh pcre-devel-8.32-17.el7.x86_64.rpm
8. rpm -ivh libselinux-devel-2.5-14.1.el7.x86_64.rpm
9. rpm -ivh krb5-devel-1.15.1-37.el7_7.2.x86_64.rpm
10. rpm -ivh zlib-devel-1.2.7-18.el7.x86_64.rpm
11. rpm -ivh openssl-devel-1.0.2k-19.el7.x86_64.rpm
1.3安装sqlite-devel, 依次执行安装(注意顺序)
1. rpm -ivh sqlite-3.7.17-8.el7_7.1.x86_64.rpm --force
2. rpm -ivh sqlite-devel-3.7.17-8.el7_7.1.x86_64.rpm
1.4安装gcc-c++, 依次执行安装(注意顺序)
1. rpm -ivh libstdc++-devel-4.8.5-39.el7.x86_64.rpm
2. rpm -ivh gcc-c++-4.8.5-39.el7.x86_64.rpm
1.5安装mysql-devel gcc gcc-devel python-devel, 依次执行安装(注意顺序)
1. rpm -ivh python-srpm-macros-3-32.el7.noarch.rpm
2. rpm -ivh python-rpm-macros-3-32.el7.noarch.rpm
3. rpm -ivh python2-rpm-macros-3-32.el7.noarch.rpm
4. rpm -ivh python-devel-2.7.5-86.el7.x86_64.rpm
5. rpm -ivh mariadb-devel-5.5.64-1.el7.x86_64.rpm
1.6安装sasl, 依次执行安装(注意顺序)
1. rpm -ivh cyrus-sasl-2.1.26-23.el7.x86_64.rpm
2. rpm -ivh cyrus-sasl-devel-2.1.26-23.el7.x86_64.rpm
2. 安装python3.6.5
2.1编译安装zlib
解压python-3.6.5.tgz,进入Modules/zlib目录
1. #解压
2. tar -zxvf Python-3.6.5.tgz
3. #进入python-3.6.5/Modules/zlib
4. cd Python-3.6.5/Modules/zlib
5. #编译安装zlib
6. ./configure
7. make install
2.2编译安装python 3.6.5
将python3.6.5安装至/usr/local/python3目录
1. #建立目录
2. mkdir /usr/local/python3
3. #进入Python-3.6.5目录,执行编译安装
4. cd Python-3.6.5
5. ./configure --prefix=/usr/local/python3 --enable-shared --with-ssl
6. make && make install
2.3建立python3和pip3的软链
1. #创建python3的软链
2. ln -s /usr/local/python3/bin/python3.6 /usr/bin/python3
3. #创建pip3 的软链
4. ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3
2.4添加环境变量
1. #编辑.bash_profile文件
2. vi ~/.bash_profile
3. #加入以下内容
4. #环境变量,否则会报找不到libpython3.6m.so.1.0
5. export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/python3/lib
6. #配置airflow需要的环境变量,
7. export AIRFLOW_HOME=~/airflow
8. export AIRFLOW_GPL_UNIDECODE=yes
9. #添加python3的目录,否则无法识别airflow命令
10. export PATH=$PATH:/usr/local/python3/bin
11. #保存退出,加载环境变量
12. source ~/.bash_profile
测试pip3和python3命令,出现如下版本号则安装成功
1. [root@localhost Python-3.6.5]# pip3 -V
2. pip 9.0.3 from /usr/local/python3/lib/python3.6/site-packages (python 3.6)
3. [root@localhost Python-3.6.5]# python3
4. Python 3.6.5 (default, Feb 28 2020, 22:31:30)
5. [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
6. Type "help", "copyright", "credits" or "license" for more information.
7. >>>
2.5升级pip3的版本
如上图,pip3版本号为9.0.3,版本太低,如不升级会报pip当前版本无效的错误
进入安装包的pip20.0.2目录,执行以下语句:
1. #进入安装包pip20.0.2目录
2. cd pip20.0.2
3. #安装pip新版本
4. pip3 install pip-20.0.2-py2.py3-none-any.whl
5. #再次查看版本,已更新为20.0.2
6. pip3 -V
3. 安装airflow
3.1安装airflow
进入安装包airflow_1.10.9_all,执行安装
1. #进入安装包airflow_1.10.9_all目录
2. cd airflow_1.10.9_all
3. #执行安装
4. pip3 install apache-airflow[all] --no-index -f ./
安装时间会比较长,如出现版本不兼容的报错,可忽略,如出现报错,参考以下方法解决:
报错:
1. distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pytest-runner')
解决方法:
1. #执行以下命令后,重新执行安装
2. pip3 install pytest_runner-5.2-py2.py3-none-any.whl
报错:
1. gcc: error trying to exec 'cc1plus': execvp: No such file or directory
解决方法:
进入安装包的dependencies目录,按照1.4的方法重新安装gcc-c++,再返回airflow_1.10.9_all目录重新执行安装
报错:
1. distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('Cython!=0.25,<0.30,>=0.20')
解决方法:
1. #执行以下命令后,重新执行安装
2. pip3 install Cython-0.29.15-cp36-cp36m-manylinux1_x86_64.whl
注:如还出现其他类似的缺少包的提示,都可使用pip3 install进行安装,所有依赖的包都在airflow_1.10.9目录里,最终出现Successfully installed,表示安装成功。
3.2初始化数据库
1. #初始化数据库
2. airflow initdb
3.3启动airflow
1. #启动airflow web
2. #指定端口
3. airflow webserver -p xxx
4. #不指定端口,默认为8080,启动成功后,通过http://x.x.x.x:8080访问
5. airflow webserver
6. #使用以下命令可使airflow不随窗口关闭而关闭,并将日志输出(追加)到指定目录
7. nohup airflow webserver >> /airflow.log &
8. #启动airflow scheduler,必须启动scheduler,否则web界面不更新新的dag等内容,
9. airflow scheduler
3.4更改数据库为mysql
Airflow默认数据库为SQLite,如需更改数据库为mysql,请修改$AIRFLOW_HOME/airflow.cfg文件,默认在~/airflow/airflow.cfg,加入以下内容
1. #加入以下语句,格式如下:
2. #mysq://[db_username]:[db_password]@[db_host]:[db_port]/[db_name]
3. #例:
4. sql_alchemy_conn = mysql://root:root@localhost:3306/airflow?charset=utf8
注:配置完后,重新使用airflow initdb初始化数据库,请提前准备好数据库及设置访问权限
3.5 中文化
进入airflow_1.10.9_all目录,安装Flask-BabelEx和PyMySQL
1. pip3 install Flask_BabelEx-0.9.4-py3-none-any.whl
2. pip3 install PyMySQL-0.9.3-py2.py3-none-any.whl
修改配置文件,~/airflow/airflow.cfg
1. #修改时区
2. default_timezone = Asia/Shanghai
3. #不加载示例
4. load_examples = False
将汉化包里的文件替换/usr/local/python3/lib/python3.6/site-packages/airflow目录里的文件,再重新始初化数据库,重新启动服务