报错信息:
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1558, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1627, in _build
build_save=build_save, build_restore=build_restore)
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1188, in _build_internal
restore_sequentially, reshape)
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 783, in _AddShardedRestoreOps
name="restore_shard"))
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 752, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 278, in restore
self.op.get_shape().is_fully_defined())
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/ops/state_ops.py", line 236, in assign
validate_shape=validate_shape)
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_state_ops.py", line 62, in assign
use_locking=use_locking, name=name)
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(args, *kwargs)
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3401, in create_op
op_def=op_def)
File "/worker/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1771, in init
self._traceback = tf_stack.extract_stack()
InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Assign requires shapes of both tensors to match. lhs shape= [700,8] rhs shape= [660,8]
[node save/Assign_7 (defined at /worker/tensorflow_jobs/easy_rec/python/model/easy_rec_estimator.py:74) = Assign[T=DT_FLOAT, _class=["loc:@attr_value_names_embedding/embedding_weights"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
很明显可以看到是Restoring from checkpoint failed ,从ckpt恢复模型出错,出错原因呢是现在的模型和ckpt的模型中attr_value_names的参数不一样。