Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error training retriever #208

Open
ShubhamVerma16 opened this issue Nov 30, 2023 · 4 comments
Open

Error training retriever #208

ShubhamVerma16 opened this issue Nov 30, 2023 · 4 comments

Comments

@ShubhamVerma16
Copy link

ShubhamVerma16 commented Nov 30, 2023

Hi,

I am facing the below error while training for retriever. The data for retriever was preprocessed and the training runs for 1 epoch.
The error generates when the training goes for 2nd Epoch.

Error:

`RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/shubham.v/mini/envs/bench/lib/python3.10/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/home/shubham.v/ToolBench/toolbench/retrieval/api_evaluator.py", line 55, in compute_ndcg_for_query
return ndcg_score([true_relevance], [predicted_scores], k)
File "/home/shubham.v/mini/envs/bench/lib/python3.10/site-packages/sklearn/utils/_param_validation.py", line 192, in wrapper
params = func_sig.bind(*args, **kwargs)
File "/home/shubham.v/mini/envs/bench/lib/python3.10/inspect.py", line 3186, in bind
return self._bind(args, kwargs)
File "/home/shubham.v/mini/envs/bench/lib/python3.10/inspect.py", line 3112, in _bind
raise TypeError(
TypeError: too many positional arguments
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last)
│ /home/shubham.v/ToolBench/toolbench/retrieval/train.py:103 in
│ 101 # You may need to modify the .fit() method to ensure all data is moved to the correct de
│ ❱ 103 model.fit(train_objectives=[(train_dataloader, train_loss)],
│ 104 │ evaluator=ir_evaluator,
│ 105 │ epochs=num_epochs,
│ 106 │ warmup_steps=warmup_steps,
│ /home/shubham.v/mini/envs/bench/lib/python3.10/site-packages/sentence_transformers/SentenceTransformer.py:745 in fit
│ 742 │ self._save_checkpoint(checkpoint_path, checkpoint_save_total_limit,

│ ❱ 745 │ │ │ self._eval_during_training(evaluator, output_path, save_best_model, epoch, -
│ 747 │ │ if evaluator is None and output_path is not None: #No evaluator, but output pa
│ 748 │ │ │ self.save(output_path)

│ /home/shubham.v/mini/envs/bench/lib/python3.10/site-packages/sentence_transformers/SentenceTrans
│ former.py:777 in eval_during_training
│ 774 │ os.makedirs(eval_path, exist_ok=True)
│ 776 │ if evaluator is not None:
│ ❱ 777 │ score = evaluator(self, output_path=eval_path, epoch=epoch, steps=steps)
│ 778 │ if callback is not None:
│ 779 │ callback(score, epoch, steps)
│ 780 │ if score > self.best_score:
│ /home/shubham.v/ToolBench/toolbench/retrieval/api_evaluator.py:118 in call
│ 115 │ logger.info("Information Retrieval Evaluation" + out_txt)
│ 117 │ # scores = self.compute_metrices(model)
│ ❱ 118 │ avg_ndcg = self.compute_metrices(model)
│ 120 │ # Write results to disc
│ 121 │ if output_path is not None and self.write_csv:
│ /home/shubham.v/ToolBench/toolbench/retrieval/api_evaluator.py:200 in compute_metrices
│ 197 │ logger.info("Corpus: {}\n".format(len(self.corpus)))
│ 199 │ # Compute scores
│ ❱ 200 │ scores = self.compute_metrics(queries_result_list)
│ 202 │ # Output
│ 203 │ logger.info("Average NDCG@1: {:.2f}".format(scores[0] * 100))
│ /home/shubham.v/ToolBench/toolbench/retrieval/api_evaluator.py:242 in compute_metrics
│ 239 │ with Pool() as p:
│ 240 │ max
= len(query_tuples)
│ 241 │ with tqdm(total=max_) as pbar:
│ ❱ 242 │ for i, _ in tqdm(
│ 243 │ enumerate(p.imap(compute_ndcg_for_query, query_tuples))
│ 244 │ ):
│ 245 │ pbar.update()
│ /home/shubham.v/mini/envs/bench/lib/python3.10/site-packages/tqdm/std.py:1182 in iter
│ 1179 │ time = self._time
│ 1181 │ try:
│ ❱ 1182 │ for obj in iterable:
│ 1183 │ yield obj
│ 1184 │ # Update and possibly print the progressbar.
│ 1185 │ # Note: does not call self.update(1) for speed optimisation.
│ /home/shubham.v/mini/envs/bench/lib/python3.10/multiprocessing/pool.py:873 in next
│ 870 │ success, value = item
│ 871 │ if success:
│ 872 │ return value
│ ❱ 873 │ raise value
│ 875 │ next = next # XXX
│ 876 |
TypeError: too many positional arguments
`

Can you please let me know a resolution for the same.

@Zhang-star-master
Copy link

same error

@Zijian007
Copy link

same error

1 similar comment
@lili1117
Copy link

same error

@songjiechong
Copy link

same error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants