app.py运行问题

[cjx336]

您好，我运行app.py之后，模型的回答是一堆乱码，模型路径都已经设置过了，请问可能是哪里出问题了？

[ZebangCheng]

不同json文件中 模型配置的路径 您发我看看。
以及启动demo时的log也可以发我看看。

[cjx336]

非常感谢您，下面是log，结尾是几个yaml文件
(llama) chenjiaxin@7bj0cg2pdc3rn-0:~/Emotion-LLaMA$ python app.py
/zengjinkai/home/chenjiaxin/anaconda3/envs/llama/lib/python3.9/site-packages/accelerate/utils/torch_xla.py:18: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

Initializing Chat
model_config: {'arch': 'minigpt_v2', 'image_size': 448, 'drop_path_rate': 0, 'use_grad_checkpoint': False, 'vit_precision': 'fp16', 'freeze_vit': True, 'prompt': '', 'llama_model': '/zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/llama-2-7b', 'lora_r': 64, 'lora_alpha': 16, 'model_type': 'pretrain', 'max_txt_len': 500, 'end_sym': '', 'low_resource': True, 'prompt_template': '[INST] {} [/INST]', 'ckpt': '/zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/save_checkpoint/Emotion_LLaMA.pth'}
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████ 3/3 [00:19<00:00, 6.34s/it]
Some weights of LlamaForCausalLM were not initialized from the model checkpoint at /zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/llama-2-7b and are newly initialized: ['model.layers.14.self_attn.rotary_emb.inv_freq', 'model.layers.5.self_attn.rotary_emb.inv_freq', 'model.layers.7.self_attn.rotary_emb.inv_freq', 'model.layers.2.self_attn.rotary_emb.inv_freq', 'model.layers.11.self_attn.rotary_emb.inv_freq', 'model.layers.27.self_attn.rotary_emb.inv_freq', 'model.layers.1.self_attn.rotary_emb.inv_freq', 'model.layers.10.self_attn.rotary_emb.inv_freq', 'model.layers.21.self_attn.rotary_emb.inv_freq', 'model.layers.8.self_attn.rotary_emb.inv_freq', 'model.layers.4.self_attn.rotary_emb.inv_freq', 'model.layers.19.self_attn.rotary_emb.inv_freq', 'model.layers.29.self_attn.rotary_emb.inv_freq', 'model.layers.13.self_attn.rotary_emb.inv_freq', 'model.layers.24.self_attn.rotary_emb.inv_freq', 'model.layers.3.self_attn.rotary_emb.inv_freq', 'model.layers.28.self_attn.rotary_emb.inv_freq', 'model.layers.17.self_attn.rotary_emb.inv_freq', 'model.layers.30.self_attn.rotary_emb.inv_freq', 'model.layers.18.self_attn.rotary_emb.inv_freq', 'model.layers.9.self_attn.rotary_emb.inv_freq', 'model.layers.22.self_attn.rotary_emb.inv_freq', 'model.layers.0.self_attn.rotary_emb.inv_freq', 'model.layers.12.self_attn.rotary_emb.inv_freq', 'model.layers.6.self_attn.rotary_emb.inv_freq', 'model.layers.26.self_attn.rotary_emb.inv_freq', 'model.layers.25.self_attn.rotary_emb.inv_freq', 'model.layers.15.self_attn.rotary_emb.inv_freq', 'model.layers.16.self_attn.rotary_emb.inv_freq', 'model.layers.23.self_attn.rotary_emb.inv_freq', 'model.layers.31.self_attn.rotary_emb.inv_freq', 'model.layers.20.self_attn.rotary_emb.inv_freq']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
loraconfig: LoraConfig(peft_type=<PeftType.LORA: 'LORA'>, base_model_name_or_path=None, task_type='CAUSAL_LM', inference_mode=False, r=64, target_modules=['q_proj', 'v_proj'], lora_alpha=16, lora_dropout=0.05, merge_weights=False, fan_in_fan_out=False, enable_lora=None, bias='none', modules_to_save=None)
trainable params: 33554432 || all params: 6771970048 || trainable%: 0.49548996469513035
Position interpolate from 16x16 to 32x32
Load Minigpt-4-LLM Checkpoint: /zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/save_checkpoint/Emotion_LLaMA.pth
/zengjinkai/home/chenjiaxin/Emotion-LLaMA/app.py:596: GradioDeprecationWarning: 'scale' value should be an integer. Using 0.5 will cause issues.
with gr.Column(scale=0.5):
/zengjinkai/home/chenjiaxin/Emotion-LLaMA/app.py:598: GradioUnusedKwargWarning: You have unused kwarg parameters in Video, please remove them: {'sources': ['upload', 'webcam']}
image = gr.Video(sources=["upload", "webcam"])
/zengjinkai/home/chenjiaxin/Emotion-LLaMA/app.py:696: GradioDeprecationWarning: The enable_queue parameter has been deprecated. Please use the .queue() method instead.
demo.launch(share=True, enable_queue=True)
Running on local URL: http://127.0.0.1:7860

Could not create share link. Please check your internet connection or our status page: https://status.gradio.app.
+++gradio_ask+++
user_message: [detection] face
chatbot: []
chat_state: None
user_message: [detection] face
chat_state: Conversation(system='', roles=('[INST] ', ' [/INST]'), messages=[['[INST] ', ' '], ['[INST] ', '[detection] face']], offset=2, sep_style=<SeparatorStyle.SINGLE: 1>, sep='', sep2=None, skip_next=False, conv_id=None)
---gradio_stream_answer---
isinstance str
MoviePy - Writing audio in audio.wav
MoviePy - Done.
audio faature shape: torch.Size([1, 1, 1024])
video_features torch.Size([1, 3, 1024])
<minigpt4.conversation.conversation.Chat object at 0x7f7f501add60>
stream_answer img shape: torch.Size([1, 260, 4096])
prompt: [INST] [INST] [detection] face [/INST]
streamer: <transformers.generation.streamers.TextIteratorStreamer object at 0x7f7f42503400>
+++gradio_ask+++
user_message: The person in video says: Why are you looking at me like this? It's just a woman, so you have to have something to do with me. [emotion] Determine the emotional state shown in the video, choosing from happy, sad, neutral, angry, worried, or surprise.
chatbot: [['[detection] face', '\<video\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> .']]
chat_state: Conversation(system='', roles=('[INST] ', ' [/INST]'), messages=[['[INST] ', ' '], ['[INST] ', '[detection] face'], [' [/INST]', None]], offset=2, sep_style=<SeparatorStyle.SINGLE: 1>, sep='', sep2=None, skip_next=False, conv_id=None)
user_message: The person in video says: Why are you looking at me like this? It's just a woman, so you have to have something to do with me. [emotion] Determine the emotional state shown in the video, choosing from happy, sad, neutral, angry, worried, or surprise.
chat_state: Conversation(system='', roles=('[INST] ', ' [/INST]'), messages=[['[INST] ', ' '], ['[INST] ', "The person in video says: Why are you looking at me like this? It's just a woman, so you have to have something to do with me. [emotion] Determine the emotional state shown in the video, choosing from happy, sad, neutral, angry, worried, or surprise."]], offset=2, sep_style=<SeparatorStyle.SINGLE: 1>, sep='', sep2=None, skip_next=False, conv_id=None)
output: <video> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <featureHere> <
+++gradio_ask+++
user_message: The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise.
chatbot: [['[detection] face', '\<video\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<featurehere\> \<.']]
chat_state: Conversation(system='', roles=('[INST] ', ' [/INST]'), messages=[['[INST] ', ' '], ['[INST] ', '[detection] face'], [' [/INST]', '']], offset=2, sep_style=<SeparatorStyle.SINGLE: 1>, sep='', sep2=None, skip_next=False, conv_id=None)
user_message: The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise.
chat_state: Conversation(system='', roles=('[INST] ', ' [/INST]'), messages=[['[INST] ', ' '], ['[INST] ', 'The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise.']], offset=2, sep_style=<SeparatorStyle.SINGLE: 1>, sep='', sep2=None, skip_next=False, conv_id=None)
---gradio_stream_answer---
isinstance str
MoviePy - Writing audio in audio.wav
MoviePy - Done.
audio faature shape: torch.Size([1, 1, 1024])
video_features torch.Size([1, 3, 1024])
<minigpt4.conversation.conversation.Chat object at 0x7f7f501add60>
stream_answer img shape: torch.Size([1, 260, 4096])
prompt: [INST] [INST] The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise. [/INST]
streamer: <transformers.generation.streamers.TextIteratorStreamer object at 0x7f7f424ddfd0>
output:

Emotion-LLaMA_finetune.yaml
demo.yaml
minigpt_v2.yaml

[ZebangCheng]

请在minigpt_v2.yaml文件中修改llama_model的路径试试。
把llama-2-7b修改为Llama-2-7b-chat-hf

[cjx336]

还是不行，模型的回答是重复了prompt：
(llama) chenjiaxin@7bj0cg2pdc3rn-0:~/Emotion-LLaMA$ python app.py
/zengjinkai/home/chenjiaxin/anaconda3/envs/llama/lib/python3.9/site-packages/accelerate/utils/torch_xla.py:18: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources

===================================BUG REPORT===================================
Welcome to bitsandbytes. For bug reports, please submit your error trace to: https://github.com/TimDettmers/bitsandbytes/issues

Initializing Chat
model_config: {'arch': 'minigpt_v2', 'image_size': 448, 'drop_path_rate': 0, 'use_grad_checkpoint': False, 'vit_precision': 'fp16', 'freeze_vit': True, 'prompt': '', 'llama_model': '/zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/Llama-2-7b-chat-hf', 'lora_r': 64, 'lora_alpha': 16, 'model_type': 'pretrain', 'max_txt_len': 500, 'end_sym': '', 'low_resource': True, 'prompt_template': '[INST] {} [/INST]', 'ckpt': '/zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/save_checkpoint/Emotion_LLaMA.pth'}
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:28<00:00, 14.09s/it]
loraconfig: LoraConfig(peft_type=<PeftType.LORA: 'LORA'>, base_model_name_or_path=None, task_type='CAUSAL_LM', inference_mode=False, r=64, target_modules=['q_proj', 'v_proj'], lora_alpha=16, lora_dropout=0.05, merge_weights=False, fan_in_fan_out=False, enable_lora=None, bias='none', modules_to_save=None)
trainable params: 33554432 || all params: 6771970048 || trainable%: 0.49548996469513035
Position interpolate from 16x16 to 32x32
Load Minigpt-4-LLM Checkpoint: /zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/save_checkpoint/Emotion_LLaMA.pth
/zengjinkai/home/chenjiaxin/Emotion-LLaMA/app.py:596: GradioDeprecationWarning: 'scale' value should be an integer. Using 0.5 will cause issues.
with gr.Column(scale=0.5):
/zengjinkai/home/chenjiaxin/Emotion-LLaMA/app.py:598: GradioUnusedKwargWarning: You have unused kwarg parameters in Video, please remove them: {'sources': ['upload', 'webcam']}
image = gr.Video(sources=["upload", "webcam"])
/zengjinkai/home/chenjiaxin/Emotion-LLaMA/app.py:696: GradioDeprecationWarning: The enable_queue parameter has been deprecated. Please use the .queue() method instead.
demo.launch(share=True, enable_queue=True)
Running on local URL: http://127.0.0.1:7860

Could not create share link. Please check your internet connection or our status page: https://status.gradio.app.
+++gradio_ask+++
user_message: The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise.
chatbot: []
chat_state: None
user_message: The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise.
chat_state: Conversation(system='', roles=('[INST] ', ' [/INST]'), messages=[['[INST] ', ' '], ['[INST] ', 'The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise.']], offset=2, sep_style=<SeparatorStyle.SINGLE: 1>, sep='', sep2=None, skip_next=False, conv_id=None)
---gradio_stream_answer---
isinstance str
MoviePy - Writing audio in audio.wav
MoviePy - Done.
audio faature shape: torch.Size([1, 1, 1024])
video_features torch.Size([1, 3, 1024])
<minigpt4.conversation.conversation.Chat object at 0x7fbaa0a3bd90>
stream_answer img shape: torch.Size([1, 260, 4096])
prompt: [INST] [INST] The person in video says: Oh no, my phone and wallet are all in my bag. [emotion] Please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise. [/INST]
streamer: <transformers.generation.streamers.TextIteratorStreamer object at 0x7fbaa0a60550>
output: <video> the person in video says: oh no, my phone and wallet are all in my bag [emotion] please determine which emotion label in the video represents: happy, sad, neutral, angry, worried, surprise

[ZebangCheng]

从日志看，第一次的问题确实有一部分来自 `llama_model` 配错了：demo 需要使用 `Llama-2-7b-chat-hf`，不能用普通的 `llama-2-7b`。

不过你改成 `Llama-2-7b-chat-hf` 后仍然复读 prompt，说明现在底座 LLaMA 已经能正常加载，但 Emotion-LLaMA 的多模态/LoRA checkpoint 很可能没有正确加载或不是对应的 demo 权重。请确认：

1. `minigpt4/configs/models/minigpt_v2.yaml` 中：
   ```yaml
   llama_model: "/zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/Llama-2-7b-chat-hf"

2. eval_configs/demo.yaml 中的 ckpt 必须指向 README 里 Google Drive 下载的 demo 模型：

   ckpt: "/zengjinkai/home/chenjiaxin/Emotion-LLaMA/checkpoints/save_checkpoint/Emoation_LLaMA.pth"

   注意不要把 minigptv2.pth、minigptv2_checkpoint.pth 或训练中间权重当作 demo checkpoint 使用。文件名可以改，但文件内容必须是 Emotion-LLaMA demo 权重。

3. 修改配置后重新启动 python app.py，不要在旧的 Gradio session 里继续测。

如果还不行，可以在 minigpt4/models/minigpt_v2.py 的：

msg = model.load_state_dict(ckpt['model'], strict=False)

后面临时加：

print(msg)

看一下 missing/unexpected keys。重点检查 LoRA 和投影层相关权重是否没有加载成功，例如 llama_proj、feats_llama_proj*、cls_tk_llama_proj、lora_A/lora_B。如果这些缺失很多，基本就是 checkpoint 不对。

[cjx336]

好的，非常感谢您的指导，我之后再试试