微调训练gpt3-1.3B 卡在compiling and loading fused kernel
2023-02-27 16:57:30,329 - modelscope - INFO - ===========================Training Config End=========================== 2023-02-27 16:57:30,330 - modelscope - INFO - ===========================Training Config End=========================== 2023-02-27 16:57:30,330 - modelscope - INFO - initialize model from /root/.cache/modelscope/hub/damo/nlp_gpt3_text-generation_1.3B 2023-02-27 16:57:30,330 - modelscope - INFO - initialize model from /root/.cache/modelscope/hub/damo/nlp_gpt3_text-generation_1.3B using world size: 2, data-parallel-size: 1, tensor-model-parallel size: 2, pipeline-model-parallel size: 1 using torch.float32 for parameters ...
initializing torch distributed ... initializing tensor model parallel with size 2 initializing pipeline model parallel with size 1 setting random seeds to 42 ... initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 2760 and data parallel seed: 42 compiling and loading fused kernels ...
