-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathmetadata_models.csv
We can make this file beautiful and searchable if this error is corrected: It looks like row 15 should actually have 1 column, instead of 2 in line 14.
41 lines (41 loc) · 9.5 KB
/
metadata_models.csv
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
run_id LLM pipeline
colbertv2 - Colbert-v2 and using pre-trained checkpoint and do continue training on MS MARCO v2 training
cip_run_7 MonoT5/GPT-3.5 Unicoil+doct5query (pyserini)%0A2. MonoT5-3b (pygaggle)%0A3. GPT-3.5-Turbo top-50 prf reranking %0A4. GPT-3.5-Turbo top-40 reranking %0A5. GPT-3.5-Turbo top-30 reranking with different window size%
cip_run_1 MonoT5/GPT-3.5/GPT-4 Unicoil+doct5query (pyserini)%0A2. MonoT5-3b (pygaggle)%0A3. GPT-3.5-Turbo top-50 prf reranking %0A4. GPT-3.5-Turbo top-40 reranking %0A5. GPT-3.5-Turbo top-30 reranking with different window size%0A6. GPT-4 top-20 reranking
cip_run_2 MonoT5/GPT-3.5/GPT-4 Unicoil+doct5query (pyserini)%0A2. MonoT5-3b (pygaggle)%0A3. GPT-3.5-Turbo top-50 prf reranking %0A4. GPT-3.5-Turbo top-40 reranking %0A5. GPT-3.5-Turbo top-30 reranking with different window size%0A6. GPT-4 top-20 reranking%0A7.
cip_run_3 MonoT5/GPT-3.5 Unicoil+doct5query (pyserini)%0A2. MonoT5-3b (pygaggle)%0A3. GPT-3.5-Turbo top-50 prf reranking %0A4. GPT-3.5-Turbo top-40 reranking %0A5. GPT-3.5-Turbo top-30 reranking with different window size (fuse)
cip_run_4 MonoT5/GPT-3.5 Unicoil+doct5query (pyserini)%0A2. MonoT5-3b (pygaggle)%0A3. GPT-3.5-Turbo top-50 prf reranking %0A4. GPT-3.5-Turbo top-40 reranking
cip_run_5 MonoT5/GPT-3.5 Unicoil+doct5query (pyserini)%0A2. MonoT5-3b (pygaggle)%0A3. GPT-3.5-Turbo top-50 prf reranking %0A4. GPT-3.5-Turbo top-40 reranking %0A5. GPT-3.5-Turbo top-30 reranking with different window size
cip_run_6 MonoT5/GPT-3.5 Unicoil+doct5query (pyserini)%0A2. MonoT5-3b (pygaggle)%0A3. GPT-3.5-Turbo top-50 prf reranking
agg-cocondenser - Aggretriever is a dense retriever with semantic and lexical matching.
slim-pp-0shot-uw - SPRINT-https%3A//arxiv.org/pdf/2302.06587.pdf-only doct5query
uot-yahoo_rankgpt35 GPT-3.5 There was no fine-tuning or training of the model. This run utilized the GPT3.5 model to generate the re-ranking results via a List-wise approach. This is a Zero-shot learning approach.
uot-yahoo_rankgpt4 GPT-4 There was no fine-tuning or training of the model. This run utilized the GPT4 model to generate the re-ranking results via a List-wise approach. This is a Zero-shot learning approach.
uot-yahoo_LLMs-blender GPT-3.5/GPT-4 There was no fine-tuning or training of the model. This run uses multiple LLM models to judge candidate document pairs in a pair-wise approach and finally aggregates the judgments of all models to generate the final ranking result. This is a Zero-shot learning approach.
WatS-LLM-Rerank Llama Using the given top 100 passages for each query.:No training.::100:no-prompt-no:A100 80GB, about 8400 minutes. We didn't put effort into optimizing efficiency. This attempt is more like a proof of concept.:Prompting the llama model to assess passages.
naverloo_bm25_RR RankT5 First stage is BM25, second stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro
naverloo-frgpt4 T5/FlanT5/GPT-4 First stage is a very large ensemble of BM25+DOCT5+SPLADEPP+SPLADESD+AGG+SLIM, second stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro%0AWe then do a third step with an ensemble of 3 duo rankers over the top50, duoT5, PRP-FlanT5-3b and PRP-FlanT5-UL2. We finally finish by applying RankGPT4 over the top30 and ensembling with the previous step.
naverloo-rgpt4 T5/FlanT5/GPT-4 First stage is a very large ensemble of BM25+DOCT5+SPLADEPP+SPLADESD+AGG+SLIM, second stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro%0AWe then do a third step with an ensemble of 3 duo rankers over the top50, duoT5, PRP-FlanT5-3b and PRP-FlanT5-UL2. We finally finish by applying RankGPT4 over the top30.
splade_pp_ensemble_distil - Splade++ Ensemble distil available on huggingface
splade_pp_self_distil - Splade++ Self distil available on huggingface
bm25_splades - Ensemble of BM25 + SPLADE++SD + SPLADE++ED
naverloo_fs - Ensemble of BM25 + SPLADE++SD + SPLADE++ED + SLIM + AGGRetriever
naverloo_fs_RR RankT5 First stage is an ensemble of BM25 + SPLADE++SD + SPLADE++ED + SLIM + AGGRetriever. %0ASecond stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro
naverloo_fs_RR_duo T5 First stage is an ensemble of BM25 + SPLADE++SD + SPLADE++ED + SLIM + AGGRetriever. %0ASecond stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro%0AThird step is an ensemble of 3 duo rankers over the top50%3A%0AduoT5 %0APRP-FlanT5-3b%0APRP-FlanT5-UL2
naverloo_bm25_splades_RR RankT5 First stage is an ensemble of BM25 + SPLADE++SD + SPLADE++ED%0ASecond stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro
D_bm25_splades - Ensemble of BM25 + SPLADE++SD + SPLADE++ED
D_naverloo-frgpt4 GPT-4 First stage is an ensemble of BM25 + SPLADE++SD + SPLADE++ED + SLIM + AGGRetriever%0ASecond stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro%0AThird step is an ensemble of 3 duo rankers over the top50%0AduoT5%0APRP-FlanT5-3b%0APRP-FlanT5-UL2%0AFourth step is RankGPT4 over the top30, which is then ensemble with the 3rd stage%0A
D_naverloo_bm25_RR T5 First stage is BM25, second stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro%0A
D_naverloo_bm_splade_RR RankT5 First stage is BM25+SPLADE++ED+SPLADE++SD, second stage is an ensemble of 5 rerankers%3A%0Anaver/trecdl22-crossencoder-albert%0Anaver/trecdl22-crossencoder-debertav2%0Anaver/trecdl22-crossencoder-debertav3%0Anaver/trecdl22-crossencoder-electra%0Anaver/trecdl22-crossencoder-rankT53b-repro
WatS-Augmented-BM25 GPT Pyserini indexing the corpus.:No training.::100:no-prompt-no:A100 80G, about 840 minutes. The code was not optimized for efficiency. This run is for proof of concept.:Prompting a LLM to rewrite queries.
uogtr_dph - Lexical indexing of msmarco-passage-v2; using CPU:None::000:no-trad-yes:DPH; using CPU:Performs DPH on the entire msmarco-passage-v2 inverted index.
uogtr_dph_bo1 - no-trad-yes:DPH; Bo1 expansion; DPH; using CPU:Performs DPH with Bo1 query expansion on the entire msmarco-passage-v2 inverted index.
uogtr_be - no-nnlm-yes:BM25; ELECTRA re-ranking; using CPU and GPU:BM25 retrieval, re-ranked using crystina-z/monoELECTRA_LCE_nneg31
uogtr_se - SPLADE inference and indexing of msmarco-passage-v2; using CPU and GPU:We use publicly available trained models. No new training costs were made for this submission.::010:no-nnlm-yes:SPLADE query encoding; TF; ELECTRA re-ranking; using CPU and GPU:SPLADE retrieval using naver/splade-cocondenser-ensembledistil, re-ranked using crystina-z/monoELECTRA_LCE_nneg31
uogtr_s - no-nnlm-yes:SPLADE query encoding; TF; using CPU and GPU:SPLADE retrieval using naver/splade-cocondenser-ensembledistil
uogtr_se_gb - no-nnlm-no:SPLADE query encoding; TF, BM25 Graph adaptive rerank, ELECTRA re-ranking; using CPU and GPU:SPLADE retrieval using naver/splade-cocondenser-ensembledistil, adaptive reranking using crystina-z/monoELECTRA_LCE_nneg31 with BM25 Graph
uogtr_be_gb - no-nnlm-no:BM25, BM25 Graph adaptive rerank, ELECTRA reranking; Using GPU and CPU:BM25 over entire msmarco-passage-v2 inverted index,adaptive reranking using crystina-z/monoELECTRA_LCE_nneg31 with BM25 Graph
uogtr_qr_be_gb FlanT5 no-prompt-no:FLANT5-XXL Generative Query Reformulation, BM25 Graph adaptive rerank,ELECTRA reranking; Using GPU and CPU:Generative query expansion using google/flant5-xxl (8-bit quantized), BM25 over entire msmarco-passage-v2 inverted index, adaptive reranking using crystina-z/monoELECTRA_LCE_nneg31 with BM25 Graph
uogtr_b_grf_e FlanT5 no-prompt-no:BM25, FLANT5-XXL Generative Relevance Feedback, ELECTRA reranking; Using CPU and GPU:BM25 over entire msmarco-passage-v2 inverted index, Generative relevance feedback using google/flant5-xxl (8 bit quantized), reranking using crystina-z/monoELECTRA_LCE_nneg31
uogtr_qr_be FlanT5 no-prompt-no:FLANT5-XXL Generative Query Reformulation, BM25, ELECTRA reranking; Using GPU and CPU:Generative query expansion using google/flant5-xxl (8-bit quantized), BM25 over entire msmarco-passage-v2 inverted index, reranking using crystina-z/monoELECTRA_LCE_nneg31
uogtr_b_grf_e_gb FlanT5 no-prompt-no:FLANT5-XXL Generative Query Reformulation, BM25 Graph adaptive rerank,ELECTRA reranking; Using GPU and CPU:Generative query expansion using google/flant5-xxl (8-bit quantized), BM25 over entire msmarco-passage-v2 inverted index, adaptive reranking using crystina-z/monoELECTRA_LCE_nneg31 with BM25 Graph