It is not evaluated with eval_dataset.

[Kang812]

trainer = SFTTrainer(
            model = model,
            tokenizer = tokenizer,
            train_dataset = train_dataset,
            eval_dataset = valid_dataset,
            dataset_text_field = "text",
            max_seq_length = args.max_seq_length,
            data_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer),
            dataset_num_proc = 2,
            packing = False, # Can make training 5x faster for short sequences.
            args = TrainingArguments(
                per_device_train_batch_size = args.per_device_train_batch_size,
                gradient_accumulation_steps = args.gradient_accumulation_steps,
                warmup_steps = args.warmup_steps,
                num_train_epochs = args.num_train_epochs, # Set this for 1 full training run.
                save_total_limit = 1,
                #max_steps = 60,
                save_strategy = 'steps',
                eval_strategy = 'steps',
                do_eval = True,
                save_steps = 10,
                eval_steps = 10,
                learning_rate = args.learning_rate,
                fp16 = not is_bfloat16_supported(),
                bf16 = is_bfloat16_supported(),
                logging_steps = 1,
                optim = args.optim,
                weight_decay = args.weight_decay,
                lr_scheduler_type = "linear",
                seed = 3407,
                output_dir = args.output_dir,
                report_to = "none", # Use this for WandB etc
                load_best_model_at_end=True,
                include_inputs_for_metrics = True,
                metric_for_best_model = "eval_loss",
            ),
        )
    
    trainer = train_on_responses_only(
            trainer,
            instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n",
            response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n",)
    
    trainer_stats = trainer.train()

I set the config as follows. However, the evaluation is done with the train dataset, not the validation dataset.
Maybe I'm not doing something right .. ?

[Kang812]

I added per_device_eval_batch_size and eval_accumulation_steps to TrainingArguments and it worked.