not found tokenizer_config.json and tokenizer.json
#2
by sinlew - opened
Why can't the tokenizer_config.json and tokenizer.json... be found?
tokenizer = AutoTokenizer.from_pretrained("apple/DCLM-Baseline-7B-8k")
model = AutoModelForCausalLM.from_pretrained("apple/DCLM-Baseline-7B-8k")
DCLM-Baseline-7B-8k ?
I have uploaded the tokenizer files. Note that this is just a GPT-NeoX tokenizer
apf1 changed discussion status to closed