TL;DR

The calculation of train_batch_size should consider the split_batches flag to accurately reflect the actual batch size during training.

Guidance

• Review the calculation of train_batch_size to ensure it accounts for the split_batches flag, potentially adjusting the formula to train_batch_size = self.per_device_train_batch_size when split_batches is True.
• Verify the logic by checking the batch size when split_batches is True and False to ensure it aligns with the expected behavior.
• Consider adding a conditional statement to the train_batch_size property to handle the split_batches case, ensuring the batch size calculation is correct in both scenarios.
• Evaluate the impact of split_batches on the training process and batch size to ensure it meets the requirements of the application.

Example

    @property
    def train_batch_size(self) -> int:
        """
        The actual batch size for training.
        """
        if self.accelerator_config.split_batches:
            train_batch_size = self.per_device_train_batch_size
        else:
            train_batch_size = self.per_device_train_batch_size * max(1, self.n_gpu)
        return train_batch_size

Notes

The provided code snippet and log message suggest that the split_batches flag affects the batch size calculation, but the current implementation does not account for this. The proposed solution aims to address this discrepancy.

Recommendation

Apply workaround: Modify the train_batch_size property to conditionally calculate the batch size based on the split_batches flag, as shown in the example code snippet. This ensures the actual batch size during training accurately reflects the configuration.