Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix JSON Serialization Error in TrainerState due to np.float32 #3250

Open
belo-involead opened this issue Feb 27, 2025 · 0 comments
Open

Fix JSON Serialization Error in TrainerState due to np.float32 #3250

belo-involead opened this issue Feb 27, 2025 · 0 comments

Comments

@belo-involead
Copy link

TrainerState.save_to_json fails due to np.float32 values not being JSON serializable, causing crashes when saving training state.

Problem:

The TrainerState.save_to_json method in the transformers library fails when trying to save training state because np.float32 values are not natively serializable in JSON. This issue causes the training process to crash when saving checkpoints.

Steps to Reproduce:

Run the model distillation script using SentenceTransformerTrainer.
The training proceeds normally but fails when saving state due to an np.float32 serialization error.
The error message typically looks like:

TypeError: Object of type float32 is not JSON serializable

Expected Behavior:

The training state should save without errors.
The script should complete training and store checkpoints correctly.

Proposed Fix:

Convert np.float32 values to Python float before saving JSON.
Patch TrainerState.save_to_json to handle this conversion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant