insights on noise in got dataset and fine-tuning issues

[ep0p]

After re-reading the GOT paper, I’d like more insight into how noise or document quality was handled during training. For example, was there any focus on the percentage of pdf documents from Common Crawl that were distorted or noisy?

In my experiments, adding noise to 40% of the documents during fine-tuning still results in hallucinations during inference. Should I increase that margin, or would starting training from scratch be a better approach?