BAAI

/

bge-en-icl

like 136

Follow

Beijing Academy of Artificial Intelligence 3.77k

We upload examples at: https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/llm_dense_retriever/examples

thanks a lot. Is it the same example for every query for the same dataset? Can you share some insights how they were selected? Have you selected them by random?

@nv-bschifferer , yes, we use the same examples for different queries in the same dataset. For the task that has a training split in mteb hf repo, we randomly sample a few examples from the training split. If there is no training split, we use chatgpt to generate some examples for this task.

@Shitao : do you use few-shot prompts during training or fine-tuning or only during evaluation?

Hello, can you share the prompts and examples for reproducing the BEIR results? The link above is no longer valid.

Ok. The new link is here https://github.com/FlagOpen/FlagEmbedding/tree/master/research/llm_dense_retriever/examples/bge-en-icl/MTEB
But what are the prompts used in these tasks? I only saw example queries and responses there.

Hello, can you share the prompts and examples for reproducing the BEIR results? The link above is no longer valid.

Ok. The new link is here https://github.com/FlagOpen/FlagEmbedding/tree/master/research/llm_dense_retriever/examples/bge-en-icl/MTEB
But what are the prompts used in these tasks? I only saw example queries and responses there.

The link is here: https://github.com/FlagOpen/FlagEmbedding/blob/master/FlagEmbedding/evaluation/mteb/prompts.py