airflow安装教程（local模式）

基本环境介绍

OS：centos 7.6

python：3.6.8

pip：21.2.4

具体安装

pip配置阿里云清华源

mkdir -p $HOME/.config/pip/

vim $HOME/.config/pip/pip.conf

新增如下内容：

[global]

index-url = https://pypi.tuna.tsinghua.edu.cn/simple

执行安装

export AIRFLOW_HOME=/data/airflow

AIRFLOW_VERSION=2.2.2

export PYTHON_VERSION=3.6

CONSTRAINT_URL="https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"

pip install "apache-airflow==${AIRFLOW_VERSION}" --constraint "${CONSTRAINT_URL}"

pip install virtualenv

启动airflow

airflow standalone

报错

[root@hadoop1 ~]# airflow standalone
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/usr/local/lib/python3.6/site-packages/airflow/__init__.py", line 34, in <module>
    from airflow import settings
  File "/usr/local/lib/python3.6/site-packages/airflow/settings.py", line 35, in <module>
    from airflow.configuration import AIRFLOW_HOME, WEBSERVER_CONFIG, conf  # NOQA F401
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 1129, in <module>
    conf.validate()
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 224, in validate
    self._validate_config_dependencies()
  File "/usr/local/lib/python3.6/site-packages/airflow/configuration.py", line 278, in _validate_config_dependencies
    f"error: sqlite C library version too old (< {min_sqlite_version}). "
airflow.exceptions.AirflowConfigException: error: sqlite C library version too old (< 3.15.0). See https://airflow.apache.org/docs/apache-airflow/2.2.2/howto/set-up-database.html#setting-up-a-sqlite-database

解决方法：

参考：Set up a Database Backend — Airflow Documentation (apache.org)，重新配置sqlite

wget https://www.sqlite.org/src/tarball/sqlite.tar.gz
tar xzf sqlite.tar.gz
cd sqlite/
export CFLAGS="-DSQLITE_ENABLE_FTS3 \
    -DSQLITE_ENABLE_FTS3_PARENTHESIS \
    -DSQLITE_ENABLE_FTS4 \
    -DSQLITE_ENABLE_FTS5 \
    -DSQLITE_ENABLE_JSON1 \
    -DSQLITE_ENABLE_LOAD_EXTENSION \
    -DSQLITE_ENABLE_RTREE \
    -DSQLITE_ENABLE_STAT4 \
    -DSQLITE_ENABLE_UPDATE_DELETE_LIMIT \
    -DSQLITE_SOUNDEX \
    -DSQLITE_TEMP_STORE=3 \
    -DSQLITE_USE_URI \
    -O2 \
    -fPIC"
export PREFIX="/usr/local"
LIBS="-lm" ./configure --disable-tcl --enable-shared --enable-tempstore=always --prefix="$PREFIX"
make
make install

将

export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

新增到/etc/profile中，并执行source /etc/profile生效

再次启动airflow

airflow standalone

可以看到：

 webserver | ____________       _____________
 webserver | ____    |__( )_________  __/__  /________      __
 webserver | ____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
 webserver | ___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 webserver | _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
 webserver | [2021-12-03 03:36:04,583] {dagbag.py:500} INFO - Filling up the DagBag from /dev/null
 webserver | [2021-12-03 03:36:04,851] {manager.py:512} WARNING - Refused to delete permission view, assoc with role exists DAG Runs.can_create Admin
 webserver | [2021-12-03 03:36:07 -0500] [5428] [INFO] Starting gunicorn 20.1.0
 webserver | [2021-12-03 03:36:07 -0500] [5428] [INFO] Listening at: http://0.0.0.0:8080 (5428)
 webserver | [2021-12-03 03:36:07 -0500] [5428] [INFO] Using worker: sync
 webserver | [2021-12-03 03:36:07 -0500] [5433] [INFO] Booting worker with pid: 5433
 webserver | [2021-12-03 03:36:07 -0500] [5434] [INFO] Booting worker with pid: 5434
 webserver | [2021-12-03 03:36:07 -0500] [5435] [INFO] Booting worker with pid: 5435
 webserver | [2021-12-03 03:36:07 -0500] [5436] [INFO] Booting worker with pid: 5436

进入$AIRFLOW_HOME查看admin密码

cat $AIRFLOW_HOME/standalone_admin_password.txt

但是请注意，SQLite只适合测试场景，不能使用其作为生产环境部署使用，生产环境部署建议使用Postgres或者MySQL。