开发者社区> 问答> 正文

天池提交内存总是报错

天池最新“抱大腿” 大赛提交老是遇到这个报错: ==> /root/tianchi_occlum/image/opt/workflow_tianchi_antispam.tianchi_main.1630773450960_project/tianchi_aiflow/tianchi_main/train_job/logs/train_job_stderr.log <== train(train_path, model_dir, save_name) File "/opt/workflow_tianchi_antispam.tianchi_main.1630773450960_project/tianchi_aiflow/workflows/tianchi_main/tf_main.py", line 62, in train callbacks=[checkpoint], File "/opt/python-occlum/lib/python3.7/site-packages/keras/engine/training.py", line 1148, in fit steps_per_execution=self._steps_per_execution) File "/opt/python-occlum/lib/python3.7/site-packages/keras/engine/data_adapter.py", line 1383, in get_data_handler return DataHandler(*args, **kwargs) File "/opt/python-occlum/lib/python3.7/site-packages/keras/engine/data_adapter.py", line 1150, in init model=model) File "/opt/python-occlum/lib/python3.7/site-packages/keras/engine/data_adapter.py", line 230, in init x, y, sample_weights = _process_tensorlike((x, y, sample_weights)) File "/opt/python-occlum/lib/python3.7/site-packages/keras/engine/data_adapter.py", line 1031, in _process_tensorlike inputs = tf.nest.map_structure(_convert_numpy_and_scipy, inputs) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 869, in map_structure structure[0], [func(*x) for x in entries], File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/util/nest.py", line 869, in structure[0], [func(*x) for x in entries], File "/opt/python-occlum/lib/python3.7/site-packages/keras/engine/data_adapter.py", line 1026, in convert_numpy_and_scipy return tf.convert_to_tensor(x, dtype=dtype) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/util/dispatch.py", line 206, in wrapper return target(*args, **kwargs) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1431, in convert_to_tensor_v2_with_dispatch value, dtype=dtype, dtype_hint=dtype_hint, name=name) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1441, in convert_to_tensor_v2 as_ref=False) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/profiler/trace.py", line 163, in wrapped return func(*args, **kwargs) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1566, in convert_to_tensor ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/framework/tensor_conversion_registry.py", line 52, in default_conversion_function return constant_op.constant(value, dtype, name=name) ./run_bash.sh: line 78: 2444 Terminated /opt/python-occlum/bin/python3.7 push_kafka.py --input_filename=/tcdata/predict.csv --server localhost:9092 --input_topic tianchi_input${time} --output_topic tianchi_output${time} > push_kafka.log File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py", line 272, in constant allow_broadcast=True) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl return _constant_eager_impl(ctx, value, dtype, shape, verify_shape) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py", line 308, in _constant_eager_impl t = convert_to_eager_tensor(value, ctx, dtype) File "/opt/python-occlum/lib/python3.7/site-packages/tensorflow/python/framework/constant_op.py", line 106, in convert_to_eager_tensor return ops.EagerTensor(value, ctx.device_name, dtype) numpy.core._exceptions.MemoryError: Unable to allocate 257. MiB for an array with shape (900000, 75) and data type float32

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/opt/python-occlum/lib/python3.7/site-packages/ai_flow_plugins/job_plugins/python/python_run_main.py", line 94, in run_project(l_graph_file, l_working_dir) File "/opt/python-occlum/lib/python3.7/site-packages/ai_flow_plugins/job_plugins/python/python_run_main.py", line 89, in run_project raise Exception(str(e)) Exception: Unable to allocate 257. MiB for an array with shape (900000, 75) and data type float32 Error: 256

按照doc 增加swap 空间,输入到swapon 的时候老是报invalid argument , 根本加不了,绝望,怎么解决?

展开
收起
小有本竹竹喵 2021-09-05 02:41:17 763 0
0 条回答
写回答
取消 提交回答
问答排行榜
最热
最新

相关电子书

更多
内存取证与IaaS云平台恶意行 为的安全监控 立即下载
云服务器ECS内存增强型实例re6全新发布 立即下载
十分钟上线-使用函数计算构建支付宝小程序服务 立即下载