error in tokenizer
#2
by kz919 - opened
File "/import/snvm-sc-scratch1/kaizhaol/miniconda3/envs/coe/lib/python3.10/site-packages/sentencepiece/init.py", line 1038, in _func
raise IndexError('piece id is out of range.')
IndexError: piece id is out of range.
Hi @kz919 , thanks for bringing this to our attention.
Could you please share more information for us to replicate the error?
Thanks.
We are closing this report for now but if you have any updates on it, please feel free to re-open it or reach out to us at research@deepnight.tech
deepnight-research changed discussion status to closed