transformers - 💡(How to fix) Fix Loading kimik2 is taking forever [4 comments, 4 participants]

Problem Summary Fast loading of large models like kimik2 is slow due to the use of trust_remote_code=True and potential forloop issues.

Root Cause Analysis The root cause is likely due to the large size of the kimik2 model and the use of trust_remote_code=True, which can lead to slow loading times.

Fix Plan To enable fast loading of large models like kimik2, follow these steps:

1. Disable trust_remote_code=True

model = AutoModelForCausalLM.from_pretrained(model_name)

2. Use torch.load to load the model

Instead of using from_pretrained, try loading the model using torch.load:

model_state_dict = torch.load(model_name, map_location='cpu')
model = AutoModelForCausalLM.from_pretrained(model_name)
model.load_state_dict(model_state_dict)

3. Use fp16 or bf16 for faster loading

If you have a GPU with fp16 or bf16 support, you can use fp16 or bf16 to load the model faster:

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16)

4. Use torch.jit.load for faster loading

You can also use torch.jit.load to load the model faster:

model_jit = torch.jit.load(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name)
model.load_state_dict(model_jit.state_dict())

Verification To verify that the fix worked, check the loading time of the model. You can use the time module to measure the loading time:

import time
start_time = time.time()
model = AutoModel