reseponse_format in OpenAI-compatible vLLM Server
Hey, it's great to finally see a model that natively does transcription+diarization in one pass!
One question regarding the compatibility with the OpenAI transcription API when served with vLLM:
According to https://developers.openai.com/api/reference/resources/audio/subresources/transcriptions/methods/create, the correct response_format for including speaker diarization in the transcription response is "diarized_json". In the model card, you pass "verbose_json" in the curl example. Is this a mistake in the docs, or does it really not match the official API?
The reason this matters to me is that I maintain a service that communicates via the OpenAI transcription API and supports all response_formats defined by it.
The model's response_formats currently seem to mismatch the API's, which prevents me from testing how it performs as the inference backend of my service.
Further; does the model only support transcription+diarization ("diarized_json" in the OpenAI API) or also only transcription ("json" or "verbose_json" in the OpenAI API)?
Thank you very much in advance for the work!
Hi, thank you for your feedback. We only provide the parsing interface in SGLang-Omni; vLLM only returns the raw output. Our documentation may not have been clear enough, which led to this misunderstanding. We sincerely apologize for that.
Did you write the vLLM integration yourself or was it added by someone else? It should be possible in the vLLM model implementation to parse this correctly. But as far as I know, vLLM does not support "diarized_json" in principle; last time I checked, it was hardcoded to abort when "response_format=diarized_json"
Did you write the vLLM integration yourself or was it added by someone else? It should be possible in the vLLM model implementation to parse this correctly. But as far as I know, vLLM does not support "diarized_json" in principle; last time I checked, it was hardcoded to abort when "response_format=diarized_json"
Thanks for reporting this issue! vLLM framework should support verbose_json format but we lost it accidentally in MTD. I will submit a PR to fix it. And yeah, it seems that both of vLLM and SGLang haven't supported diarized_json. Not sure whether they plan to accept it but I will try to make it merged into main branch.
Awesome! Thank you for the effort! As far as I know, they haven't supported it yet due to no single vLLM-supported model providing the capabilities required for diarized_json
I just noticed that there is already a vLLM PR for precisely this!
https://github.com/vllm-project/vllm/pull/48543
I just noticed that there is already a vLLM PR for precisely this!
https://github.com/vllm-project/vllm/pull/48543
Looks good! You could use it by merging this PR locally first.