ModelScope中，新的swift在lora调baichuan时有报错呢？请问是为什么？

ModelScope中，新的swift在lora调baichuan时有报错呢，我把训练记录传上请帮忙看下是为什么？
nohup: ignoring input
WARNING:torch.distributed.run:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

2023-09-23 09:47:01,255 - modelscope - INFO - PyTorch version 1.13.1 Found.
2023-09-23 09:47:01,256 - modelscope - INFO - PyTorch version 1.13.1 Found.
2023-09-23 09:47:01,256 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2023-09-23 09:47:01,257 - modelscope - INFO - Loading ast index from /root/.cache/modelscope/ast_indexer
2023-09-23 09:47:01,295 - modelscope - INFO - Loading done! Current index file version is 1.9.1, with md5 d853f65341d4aaa0a71d3e47db72b836 and a total number of 924 components indexed
2023-09-23 09:47:01,295 - modelscope - INFO - Loading done! Current index file version is 1.9.1, with md5 d853f65341d4aaa0a71d3e47db72b836 and a total number of 924 components indexed
[INFO:swift] Using DDP + MP(device_map)
device_count: 8
rank: 1, local_rank: 1, world_size: 2, local_world_size: 2
[INFO:swift] args: SftArguments(model_type='baichuan2-7b-chat', sft_type='lora', tuner_bankend='swift', template_type='baichuan', output_dir='output/baichuan2-7b-chat', ddp_backend='nccl', seed=42, resume_from_ckpt=None, dtype='fp16', ignore_args_error=False, dataset='sharegpt-zh', dataset_split_seed=42, dataset_test_ratio=0.01, train_dataset_sample=-1, system='you are a helpful assistant!', max_length=4096, quantization_bit=0, bnb_4bit_comp_dtype='fp16', bnb_4bit_quant_type='nf4', bnb_4bit_use_double_quant=True, lora_target_modules=['ALL'], lora_rank=8, lora_alpha=32, lora_dropout_p=0.0, gradient_checkpointing=False, batch_size=1, eval_batch_size=1, num_train_epochs=1, max_steps=-1, optim='adamw_torch', learning_rate=0.0001, weight_decay=0.0, gradient_accumulation_steps=8, max_grad_norm=0.5, predict_with_generate=False, lr_scheduler_type='cosine', warmup_ratio=0.03, eval_steps=10, save_steps=40, only_save_model=False, save_total_limit=2, logging_steps=1, dataloader_num_workers=1, push_to_hub=False, hub_model_id='baichuan2-7b-chat-lora', hub_private_repo=True, hub_strategy='every_save', hub_token='your-sdk-token', test_oom_error=False, use_flash_attn='auto', max_new_tokens=1024, do_sample=True, temperature=0.9, top_k=20, top_p=0.9, repetition_penalty=1.0)
device_count: 8
rank: 0, local_rank: 0, world_size: 2, local_world_size: 2
[INFO:swift] Global seed set to 42
[INFO:swift] Using DDP + MP(device_map)
[INFO:modelscope] Use user-specified model revision: v1.0.1
[INFO:swift] model_config: BaichuanConfig {
"_from_model_config": true,
"_name_or_path": "/root/.cache/modelscope/hub/baichuan-inc/Baichuan2-7B-Chat",
"architectures": [
"BaichuanForCausalLM"
],
"auto_map": {
"AutoConfig": "configuration_baichuan.BaichuanConfig",
"AutoModelForCausalLM": "modeling_baichuan.BaichuanForCausalLM"
},
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 11008,
"max_position_embeddings": 4096,
"model_max_length": 4096,
"model_type": "baichuan",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"pad_token_id": 0,
"rms_norm_eps": 1e-06,
"tie_word_embeddings": false,
"torch_dtype": "float16",
"transformers_version": "4.33.2",
"use_cache": true,
"vocab_size": 125696,
"z_loss_weight": 0
}

Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.
Xformers is not installed correctly. If you want to use memory_efficient_attention to accelerate training use the following command to install Xformers
pip install xformers.

Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 0%| | 0/2 [00:00<?, ?it/s]
Loading checkpoint shards: 50%|█████ | 1/2 [00:17<00:17, 17.64s/it]
Loading checkpoint shards: 50%|█████ | 1/2 [00:19<00:19, 19.60s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:26<00:00, 12.63s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:26<00:00, 13.38s/it]
[INFO:swift] Setting lora_target_modules: ['W_pack', 'up_proj', 'gate_proj', 'o_proj', 'down_proj']

Loading checkpoint shards: 100%|██████████| 2/2 [00:29<00:00, 13.77s/it]
Loading checkpoint shards: 100%|██████████| 2/2 [00:29<00:00, 14.64s/it]
[INFO:swift] lora_config: LoRAConfig(swift_type='LORA', r=8, target_modules=['W_pack', 'up_proj', 'gate_proj', 'o_proj', 'down_proj'], lora_alpha=32, lora_dropout=0.0, merge_weights=True, use_merged_linear=False, enable_lora=None, fan_in_fan_out=False, bias='none')
[INFO:swift] [model.model.embed_tokens.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.0.self_attn.W_pack.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.0.self_attn.W_pack.loramodule_default.lora_A]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.self_attn.W_pack.loramodule_default.lora_B]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.self_attn.o_proj.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.0.self_attn.o_proj.loramodule_default.lora_A]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.self_attn.o_proj.loramodule_default.lora_B]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.gate_proj.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.gate_proj.loramodule_default.lora_A]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.gate_proj.loramodule_default.lora_B]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.down_proj.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.down_proj.loramodule_default.lora_A]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.down_proj.loramodule_default.lora_B]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.up_proj.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.up_proj.loramodule_default.lora_A]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.mlp.up_proj.loramodule_default.lora_B]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] [model.model.layers.0.input_layernorm.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.0.post_attention_layernorm.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.1.self_attn.W_pack.weight]: requires_grad=False, dtype=torch.float16, device=cuda:0
[INFO:swift] [model.model.layers.1.self_attn.W_pack.loramodule_default.lora_A]: requires_grad=True, dtype=torch.float32, device=cuda:0
[INFO:swift] ...
[INFO:swift] SwiftModel: 7523.8646M Params (17.8913M Trainable [0.2378%]), 0.0000M Buffers.
[INFO:swift] SwiftModel(
(model): BaichuanForCausalLM(
(model): BaichuanModel(
(embed_tokens): Embedding(125696, 4096, padding_idx=0)
(layers): ModuleList(
(0): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(1): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(2): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(3): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(4): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(5): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(6): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(7): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(8): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(9): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(10): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(11): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(12): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(13): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(14): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(15): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(16): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(17): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(18): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(
(gate_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(down_proj): Linear(
in_features=11008, out_features=4096, bias=False
(loramodule_default): Linear(in_features=11008, out_features=4096, bias=False)
)
(up_proj): Linear(
in_features=4096, out_features=11008, bias=False
(loramodule_default): Linear(in_features=4096, out_features=11008, bias=False)
)
(act_fn): SiLUActivation()
)
(input_layernorm): RMSNorm()
(post_attention_layernorm): RMSNorm()
)
(19): DecoderLayer(
(self_attn): Attention(
(W_pack): Linear(
in_features=4096, out_features=12288, bias=False
(loramodule_default): Linear(in_features=4096, out_features=12288, bias=False)
)
(o_proj): Linear(
in_features=4096, out_features=4096, bias=False
(loramodule_default): Linear(in_features=4096, out_features=4096, bias=False)
)
(rotary_emb): RotaryEmbedding()
)
(mlp): MLP(![lQLPJxnF