Post
619
Hi! I'm having an issue converting a Phi3 model to a torch scripted model. Code, error, and versions below. Any guidance would be super helpful, thanks!
Here is the code:
Here is the error:
torch==2.5.1
transformers==4.51.3
Here is the code:
from transformers import AutoModelForCausalLM
import torch
base_model = "microsoft/Phi-3-mini-4k-instruct"
model = AutoModelForCausalLM.from_pretrained(
base_model,
return_dict=True,
low_cpu_mem_usage=True,
torch_dtype=torch.float16,
device_map="auto",
trust_remote_code=True,
torchscript=True
)
scripted_model = torch.jit.script(model)Here is the error:
File ".../anaconda3/envs/qcomm_phi3.5instruct_aihub_py_310/lib/python3.10/site-packages/torch/jit/frontend.py", line 910, in <listcomp>
args = [build_expr(ctx, py_arg) for py_arg in expr.args]
File ".../anaconda3/envs/qcomm_phi3.5instruct_aihub_py_310/lib/python3.10/site-packages/torch/jit/frontend.py", line 417, in __call__
return method(ctx, node)
File ".../anaconda3/envs/qcomm_phi3.5instruct_aihub_py_310/lib/python3.10/site-packages/torch/jit/frontend.py", line 1255, in build_GeneratorExp
return ExprBuilder.build_ListComp(ctx, stmt)
File ".../anaconda3/envs/qcomm_phi3.5instruct_aihub_py_310/lib/python3.10/site-packages/torch/jit/frontend.py", line 1244, in build_ListComp
raise NotSupportedError(r, "Comprehension ifs are not supported yet")
torch.jit.frontend.NotSupportedError: Comprehension ifs are not supported yet:
File ".../.cache/huggingface/modules/transformers_modules/microsoft/Phi-3-mini-4k-instruct/0a67737cc96d2554230f90338b163bc6380a2a85/modeling_phi3.py", line 1148
next_cache = next_decoder_cache.to_legacy_cache() if use_legacy_cache else next_decoder_cache
if not return_dict:
return tuple(v for v in [hidden_states, next_cache, all_hidden_states, all_self_attns] if v is not None)
return BaseModelOutputWithPast(
last_hidden_state=hidden_statestorch==2.5.1
transformers==4.51.3