Hi,
thanks for the feedback, would it be possible to send the notebook for easier investigation?
Best,
Jan

Looks like a README bug to me. I reproduced the AttributeError; inputs is a BatchEncoding, and model.generate(**inputs) works. @danielporta , should we update this across all model READMEs? I don’t want to make a blanket change without confirming whether the current form was intentional. The prompt length should then use inputs["input_ids"].shape[-1].
Thanks for bringing it up.

Actually I feel like we do not need an example code at all, as it is already covered by HF? https://huggingface.co/Soofi-Project/Soofi-S-Isar-Preview?library=transformers

Thanks for the quick answer. Unfortunately, it only brought me to the next error.

Skip to main panel
/StableDiffusion/QwenImage2512/

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Soofi-Project/Soofi-S-Isar-Preview"
#model_id = "Soofi-Project/Soofi-S-Rhine-Preview"
tok = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id, trust_remote_code=True, torch_dtype="auto", device_map="auto"
)

The fast path is not available because on of (selective_state_update, causal_conv1d_fn, causal_conv1d_update) is None. Falling back to the naive implementation. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d

Loading weights: 100%
 6243/6243 [02:31<00:00, 50.56it/s]

No system prompt is required (none is injected by default).

print(model.device)
messages = [{"role": "user", "content": "How many r's are in strawberry?"}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(**inputs) # sampling defaults come from generation_config.json

print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

cuda:0

TypeError Traceback (most recent call last)
Cell In[4], line 5
3 messages = [{"role": "user", "content": "How many r's are in strawberry?"}]
4 inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
----> 5 out = model.generate(**inputs) # sampling defaults come from generation_config.json
7 print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))

File /opt/conda/lib/python3.12/site-packages/torch/utils/_contextlib.py:120, in context_decorator..decorate_context(*args, **kwargs)
117 @functools.wraps(func)
118 def decorate_context(*args, **kwargs):
119 with ctx_factory():
--> 120 return func(*args, **kwargs)

File /opt/conda/lib/python3.12/site-packages/transformers/generation/utils.py:2645, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, custom_generate, **kwargs)
2642 model_kwargs["use_cache"] = generation_config.use_cache
2644 # 9. Call generation mode
-> 2645 result = decoding_method(
2646 self,
2647 input_ids,
2648 logits_processor=prepared_logits_processor,
2649 stopping_criteria=prepared_stopping_criteria,
2650 generation_config=generation_config,
2651 **generation_mode_kwargs,
2652 **model_kwargs,
2653 )
2655 return result

File /opt/conda/lib/python3.12/site-packages/transformers/generation/utils.py:2845, in GenerationMixin._sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)
2838 model_forward = (
2839 self.get_compiled_call(generation_config.compile_config)
2840 if self._valid_auto_compile_criteria(model_kwargs, generation_config)
2841 else self.call
2842 )
2844 prefill_consumed = False
-> 2845 outputs = self._prefill(
2846 input_ids,
2847 generation_config,
2848 model_kwargs,
2849 is_first_iteration=not generation_config.is_assistant,
2850 )
2852 while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device):
2853 if prefill_consumed:

File /opt/conda/lib/python3.12/site-packages/transformers/generation/utils.py:3890, in GenerationMixin._prefill(self, input_ids, generation_config, model_kwargs, is_first_iteration)
3888 # Usual prefill
3889 if generation_config.prefill_chunk_size is None:
-> 3890 model_inputs = self.prepare_inputs_for_generation(
3891 input_ids,
3892 next_sequence_length=next_sequence_length,
3893 is_first_iteration=is_first_iteration,
3894 **model_kwargs,
3895 )
3896 return self(**model_inputs, return_dict=True)
3898 # Chunked prefill (for very large contexts)
3899 else:
3900 # Even if we are not compiling the forward, flex is always compiled when used. With chunked prefill, we may
3901 # end up needing just a bit more graphs than the default (which is 8). Doing this avoids very cryptic warnings

File ~/.cache/huggingface/modules/transformers_modules/Soofi_hyphen_Project/Soofi_hyphen_S_hyphen_Rhine_hyphen_Preview/4fe77857340a7af0cbaef5ef677b4c12fa78ded8/modeling_nemotron_h.py:1633, in NemotronHForCausalLM.prepare_inputs_for_generation(self, input_ids, past_key_values, attention_mask, inputs_embeds, cache_position, position_ids, use_cache, **kwargs)
1625 # If we have cache: let's slice input_ids through cache_position, to keep only the unprocessed tokens
1626 # Exception 1: when passing input_embeds, input_ids may be missing entries
1627 # Exception 2: some generation methods do special slicing of input_ids, so we don't need to do it here
1628 # Exception 3: with synced GPUs cache_position may go out of bounds, but we only want dummy token in that case.
1629 # (we can't check exception 3 while compiling)
1630 if not empty_past_kv:
1631 if (
1632 inputs_embeds is not None # Exception 1
-> 1633 or cache_position[-1] >= input_ids.shape[1] # Exception 3
1634 ):
1635 input_ids = input_ids[:, -cache_position.shape[0] :]
1636 elif input_ids.shape[1] != cache_position.shape[0]: # Default case (the "else", a no op, is Exception 2)

TypeError: 'NoneType' object is not subscriptable

from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "Soofi-Project/Soofi-S-Isar-Preview"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
#model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)
messages = [
{"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_dict=True,
return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

TypeError Traceback (most recent call last)
Cell In[5], line 16
5 messages = [
6 {"role": "user", "content": "Who are you?"},
7 ]
8 inputs = tokenizer.apply_chat_template(
9 messages,
10 add_generation_prompt=True,
(...) 13 return_tensors="pt",
14 ).to(model.device)
---> 16 outputs = model.generate(**inputs, max_new_tokens=40)
17 print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

File /opt/conda/lib/python3.12/site-packages/torch/utils/_contextlib.py:120, in context_decorator..decorate_context(*args, **kwargs)
117 @functools.wraps(func)
118 def decorate_context(*args, **kwargs):
119 with ctx_factory():
--> 120 return func(*args, **kwargs)

File /opt/conda/lib/python3.12/site-packages/transformers/generation/utils.py:2645, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, custom_generate, **kwargs)
2642 model_kwargs["use_cache"] = generation_config.use_cache
2644 # 9. Call generation mode
-> 2645 result = decoding_method(
2646 self,
2647 input_ids,
2648 logits_processor=prepared_logits_processor,
2649 stopping_criteria=prepared_stopping_criteria,
2650 generation_config=generation_config,
2651 **generation_mode_kwargs,
2652 **model_kwargs,
2653 )
2655 return result

File /opt/conda/lib/python3.12/site-packages/transformers/generation/utils.py:2845, in GenerationMixin._sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)
2838 model_forward = (
2839 self.get_compiled_call(generation_config.compile_config)
2840 if self._valid_auto_compile_criteria(model_kwargs, generation_config)
2841 else self.call
2842 )
2844 prefill_consumed = False
-> 2845 outputs = self._prefill(
2846 input_ids,
2847 generation_config,
2848 model_kwargs,
2849 is_first_iteration=not generation_config.is_assistant,
2850 )
2852 while self._has_unfinished_sequences(this_peer_finished, synced_gpus, device=input_ids.device):
2853 if prefill_consumed:

File /opt/conda/lib/python3.12/site-packages/transformers/generation/utils.py:3890, in GenerationMixin._prefill(self, input_ids, generation_config, model_kwargs, is_first_iteration)
3888 # Usual prefill
3889 if generation_config.prefill_chunk_size is None:
-> 3890 model_inputs = self.prepare_inputs_for_generation(
3891 input_ids,
3892 next_sequence_length=next_sequence_length,
3893 is_first_iteration=is_first_iteration,
3894 **model_kwargs,
3895 )
3896 return self(**model_inputs, return_dict=True)
3898 # Chunked prefill (for very large contexts)
3899 else:
3900 # Even if we are not compiling the forward, flex is always compiled when used. With chunked prefill, we may
3901 # end up needing just a bit more graphs than the default (which is 8). Doing this avoids very cryptic warnings

File ~/.cache/huggingface/modules/transformers_modules/Soofi_hyphen_Project/Soofi_hyphen_S_hyphen_Rhine_hyphen_Preview/4fe77857340a7af0cbaef5ef677b4c12fa78ded8/modeling_nemotron_h.py:1633, in NemotronHForCausalLM.prepare_inputs_for_generation(self, input_ids, past_key_values, attention_mask, inputs_embeds, cache_position, position_ids, use_cache, **kwargs)
1625 # If we have cache: let's slice input_ids through cache_position, to keep only the unprocessed tokens
1626 # Exception 1: when passing input_embeds, input_ids may be missing entries
1627 # Exception 2: some generation methods do special slicing of input_ids, so we don't need to do it here
1628 # Exception 3: with synced GPUs cache_position may go out of bounds, but we only want dummy token in that case.
1629 # (we can't check exception 3 while compiling)
1630 if not empty_past_kv:
1631 if (
1632 inputs_embeds is not None # Exception 1
-> 1633 or cache_position[-1] >= input_ids.shape[1] # Exception 3
1634 ):
1635 input_ids = input_ids[:, -cache_position.shape[0] :]
1636 elif input_ids.shape[1] != cache_position.shape[0]: # Default case (the "else", a no op, is Exception 2)

TypeError: 'NoneType' object is not subscriptable

Hi,
as mentioned via mail, I think the example code delivered is only compatible with transformers<5.
%pip install --force-reinstall "transformers==4.57.6"
This should fix it for now.

looks like this is the way forward to make it transformers 5 compatible @danielporta @mfromm (ofc i wrote and debugged this all myself and manually) 🤓

No—the snippet itself is compatible with Transformers 5. The incompatibility is in the SOOFI model’s custom implementation.

`trust_remote_code=True` means `modeling_nemotron_h.py` is downloaded from the SOOFI repository and executed. Hugging Face explicitly notes that, for custom models, “the modeling code is not from Transformers.” [Hugging Face documentation](https://huggingface.co/docs/transformers/models)

So:

- Generic Hugging Face snippet: Transformers 5 compatible.
- Current SOOFI remote code: built for Transformers 4.57.6.
- Safest pin: `transformers==4.57.6`, not merely `<5`.
- Supporting v5 requires updating SOOFI’s `prepare_inputs_for_generation()` implementation.

The auto-generated Hugging Face example does not guarantee that third-party remote code supports the newest Transformers version.

Hallo René, ich hoffe es klappt:)

see #2