README.md

---
license: cc-by-sa-4.0
datasets:
- csdc-atl/query-document-retrieval-full
language:
- zh
---

# csdc-atl/doc2query

This is a [doc2query](https://arxiv.org/abs/1904.08375) model based on T5 (also known as [docT5query](https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf)).

It can be used for:
- **Document expansion**: You generate for your paragraphs 20-40 queries and index the paragraphs and the generates queries in a standard BM25 index like Elasticsearch, OpenSearch, or Lucene. The generated queries help to close the lexical gap of lexical search, as the generate queries contain synonyms. Further, it re-weights words giving important words a higher weight even if they appear seldomn in a paragraph. In our [BEIR](https://arxiv.org/abs/2104.08663) paper we showed that BM25+docT5query is a powerful search engine. In the [BEIR repository](https://github.com/beir-cellar/beir) we have an example how to use docT5query with Pyserini.
- **Domain Specific Training Data Generation**: It can be used to generate training data to learn an embedding model. In our [GPL-Paper](https://arxiv.org/abs/2112.07577) / [GPL Example on SBERT.net](https://www.sbert.net/examples/domain_adaptation/README.html#gpl-generative-pseudo-labeling) we have an example how to use the model to generate (query, text) pairs for a given collection of unlabeled texts. These pairs can then be used to train powerful dense embedding models.

## Usage
```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

model_name = 'csdc-atl/doc2query'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

text = "2014年12月9日，于洋转会至中超联赛球队广州富力。2015赛季初，于洋并没有出场机会。韩国中后卫张贤秀受伤后，主教练选择用金洋洋代替。足协杯4比0战胜贵州人和的比赛中，金洋洋打入两球。赛后，中国足协认定金洋洋在庆祝进球时使用侮辱性手势，将他禁赛四场。之后对阵山东鲁能的联赛，于洋迎来出场机会，首次代表广州富力出战正式比赛。从此开始，于洋得到了较为充足的出场时间。2015赛季于洋中超联赛出场17次、亚冠联赛1次，这18次出场中有17次为首发。2016赛季，于洋成为广州富力三后卫体系的主力，还曾担任队长。这个赛季，他在中超联赛出场25次、足协杯出场5次，联赛的25次出场中含22次首发。效力广州富力期间，他于2015年重返中国国家足球队。\n2016年12月30日，广州富力宣布于洋离队，加盟北京国安。有媒体透露，转会费在5000万至6000万元人民币之间。回归北京国安之后，于洋成为中后卫位置上的主力。2018年3月31日北京国安与北京人和的北京德比上，于洋第100次代表北京国安出场。他在比赛中打入一球，助球队4比0获胜。"


def create_queries(para):
    input_ids = tokenizer.encode(para, return_tensors='pt')
    with torch.no_grad():
        # Here we use top_k / top_k random sampling. It generates more diverse queries, but of lower quality
        sampling_outputs = model.generate(
            input_ids=input_ids,
            max_length=64,
            do_sample=True,
            top_p=0.95,
            top_k=10, 
            num_return_sequences=10
            )

    print("Paragraph:")
    print(para)

    print("\nSampling Outputs:")
    for i in range(len(sampling_outputs)):
        query = tokenizer.decode(sampling_outputs[i], skip_special_tokens=True)
        print(f'{i + 1}: {query}')

create_queries(text)

# 1: 于洋在2016年重返中国国家足球队是在哪个球队效力?
# 2: 于洋在2018年3月31日的北京德比上打入了几个球?
# 3: 于洋在哪些比赛中有出场机会?
# 4: 于洋在哪个比赛中打入了两球?
# 5: 于洋在2015赛季中超联赛中出场次数和亚冠联赛中的首发次数分别是多少?
# 6: 于洋在哪个比赛中打入了两球,帮助球队赢了这场比赛?
# 7: 于洋在2018年3月31日北京国安与北京人和的北京德比上打进了几个进球?
# 8: 于洋在2015赛季中超联赛和亚冠联赛中出场次数分别是多少?
# 9: 于洋在广州富力期间曾担任什么职位?
# 10: 于洋在哪些比赛中有出场机会?

```

**Note:** `model.generate()` is non-deterministic for top_k/top_n sampling. It produces different queries each time you run it.

## Training
This model fine-tuned [Langboat/mengzi-t5-base](https://huggingface.co/Langboat/mengzi-t5-base).

The input-text was truncated to 768 word pieces. Output text was generated up to 64 word pieces. 

This model was trained on a (query, positive, negative) from the [CSDC query document retrieval dataset](https://huggingface.co/datasets/csdc-atl/query-document-retrieval-full).