1.安装部署
TensorFlow安装部署
Alluxio安装部署
SSH免密登陆
2.使用
1).创建alluxio根目录
[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs mkdir /training-data
Successfully created directory /training-data
2).创建本地目录，并挂载到alluxio根目录
a).创建本地目录
mkdir -p /home/bigdata/data
b).挂载到alluxio根目录
[bigdata@carbondata alluxio-2.0.0]$ ./integration/fuse/bin/alluxio-fuse mount /home/bigdata/data /training-data
Starting alluxio-fuse on local host.
Successfully mounted Alluxio path "/training-data" to /home/bigdata/data.
See /home/bigdata/alluxio-2.0.0/logs/fuse.log for logging messages
c).验证挂载状态
[bigdata@carbondata alluxio-2.0.0]$ ./integration/fuse/bin/alluxio-fuse stat
pid mount_point alluxio_path
11074 /home/bigdata/data /training-data
3).准备测试数据
a).下载测试数据
inception-2015-12-05
wget http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz
b).测试数据上传至alluxio
## 创建数据目录
[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs mkdir /training-data/imagenet
Successfully created directory /training-data/imagenet
## 上传数据
[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs copyFromLocal /home/bigdata/inception-2015-12-05.tgz /training-data/imagenet
Failed to cache: There is no worker with enough space for a new block of size 536,870,912
[bigdata@carbondata alluxio-2.0.0]$ ./bin/alluxio fs copyFromLocal /home/bigdata/inception-2015-12-05.tgz /training-data/imagenet
Copied file:///home/bigdata/inception-2015-12-05.tgz to /training-data/imagenet
4).图像识别测试
下载脚本
curl -o classify_image.py -L https://raw.githubusercontent.com/tensorflow/models/master/tutorials/image/imagenet/classify_image.py
运行脚本
(tensorflow) [bigdata@carbondata tensorflow_data]$ python classify_image.py --model_dir /home/bigdata/data/imagenet
WARNING:tensorflow:From classify_image.py:227: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.
WARNING:tensorflow:From classify_image.py:139: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.
W0829 21:25:37.012851 139730996795200 deprecation_wrapper.py:119] From classify_image.py:139: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.
WARNING:tensorflow:From classify_image.py:141: __init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
W0829 21:25:39.488382 139730996795200 deprecation.py:323] From classify_image.py:141: __init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
WARNING:tensorflow:From classify_image.py:125: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
W0829 21:25:41.580853 139730996795200 deprecation_wrapper.py:119] From classify_image.py:125: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
2019-08-29 21:26:31.840733: W tensorflow/core/framework/op_def_util.cc:357] Op BatchNormWithGlobalNormalization is deprecated. It will cease to work in GraphDef version 9. Use tf.nn.batch_normalization().
WARNING:tensorflow:From classify_image.py:146: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
W0829 21:27:04.229350 139730996795200 deprecation_wrapper.py:119] From classify_image.py:146: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2019-08-29 21:27:04.889623: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-08-29 21:27:05.865288: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2019-08-29 21:27:05.888049: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x557e570 executing computations on platform Host. Devices:
2019-08-29 21:27:05.888110: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2019-08-29 21:27:07.301608: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.
2019-08-29 21:27:07.357415: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.
2019-08-29 21:27:07.359446: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.
2019-08-29 21:27:07.431136: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.
2019-08-29 21:27:07.593467: W tensorflow/core/framework/allocator.cc:107] Allocation of 8257536 exceeds 10% of system memory.
2019-08-29 21:27:08.113432: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412](One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From classify_image.py:85: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
W0829 21:27:11.583446 139730996795200 deprecation_wrapper.py:119] From classify_image.py:85: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
giant panda, panda, panda bear, coon bear, Ailuropoda melanoleuca (score = 0.89107)
indri, indris, Indri indri, Indri brevicaudatus (score = 0.00779)
lesser panda, red panda, panda, bear cat, cat bear, Ailurus fulgens (score = 0.00296)
custard apple (score = 0.00147)
earthstar (score = 0.00117)
