Fine tuning for translations in multiple languages

[Grandnainconnu]

Hi,

I have a question regarding the fine-tuning of a Mixtral8x22b for translation purpose.
Our idea was to train subsets to handle pairs (fr -> en, fr -> it, fr -> es, fr -> ja...) and combine the resulting adapters with the base Mixtral model.

As I saw in the doc, it's possible to use multiple adapters at the same time, so technically it should be a valid solution, but since I know only half of what I am doing, I'd like to make sure.

Once this is done, given all the pairs loaded, I was wondering about cross pairs when it hasn't been directly trained for it, example ja -> it, should that be specified in the prompt, like "First translate to FR and then to IT", or it'll react like this by itself?

We chose Mixtral and not some other models pre-trained for translations due to its capability to handle user-defined functions, and handle HTML perfectly, so we'd like to stick to it.

Thanks in dvance!

[BenjaminBossan]

Our idea was to train subsets to handle pairs (fr -> en, fr -> it, fr -> es, fr -> ja...) and combine the resulting adapters with the base Mixtral model.

I haven't seen this suggested use case yet. What do you mean by "subset"? I assume that you mean one LoRA adapter per language pair.

As I saw in the doc, it's possible to use multiple adapters at the same time, so technically it should be a valid solution, but since I know only half of what I am doing, I'd like to make sure.

Yes, technically it's possible. As you correctly noted, you can lead multiple adapters at the same time and choose which ones you want to be active or inactive.

Once this is done, given all the pairs loaded, I was wondering about cross pairs when it hasn't been directly trained for it, example ja -> it, should that be specified in the prompt, like "First translate to FR and then to IT", or it'll react like this by itself?

I don't think it will work just by prompting, even if you activate both adapters at the same time. From your description, the adapters are trained to always start from a French text, so it would be surprising if this worked.

[Grandnainconnu]

Our idea was to train subsets to handle pairs (fr -> en, fr -> it, fr -> es, fr -> ja...) and combine the resulting adapters with the base Mixtral model.

I haven't seen this suggested use case yet. What do you mean by "subset"? I assume that you mean one LoRA adapter per language pair.

I based my tactic on: https://github.com/hppRC/llm-translator?tab=readme-ov-file

As I saw in the doc, it's possible to use multiple adapters at the same time, so technically it should be a valid solution, but since I know only half of what I am doing, I'd like to make sure.

Yes, technically it's possible. As you correctly noted, you can lead multiple adapters at the same time and choose which ones you want to be active or inactive.

Can't I keep them all active at the same time?

Once this is done, given all the pairs loaded, I was wondering about cross pairs when it hasn't been directly trained for it, example ja -> it, should that be specified in the prompt, like "First translate to FR and then to IT", or it'll react like this by itself?

I don't think it will work just by prompting, even if you activate both adapters at the same time. From your description, the adapters are trained to always start from a French text, so it would be surprising if this worked.

I see, maybe I should try and see what results it gives for that.

[BenjaminBossan]

Can't I keep them all active at the same time?

Yes, that's also possible, assuming you use LoRA. You'll have to use model.base_model.set_adapter([<name0>, <name1>, ...]), calling model.set_adapter([<name0>, <name1>, ...]) will result in an error.

I see, maybe I should try and see what results it gives for that.

Let us know if it works.

[Grandnainconnu]

I will keep you updated when I finish the training! Thank you for your answers