Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make fc_type a dict to pass fc kwargs through #1201

Merged
merged 22 commits into from
May 16, 2024
Merged

Conversation

snarayan21
Copy link
Contributor

@snarayan21 snarayan21 commented May 14, 2024

Some custom FC layers will need custom kwargs. This PR enables that by changing fc_type from str to Union[str, Dict], and converting it to dict thereafter.

Default configs have also been moved to their own file, keeping those defaults easily accessible.

We will likely seek to deprecate fc_type down the line and replace it with fc_config instead.

llmfoundry/models/mpt/configuration_mpt.py Outdated Show resolved Hide resolved
llmfoundry/models/mpt/configuration_mpt.py Outdated Show resolved Hide resolved
llmfoundry/models/mpt/configuration_mpt.py Outdated Show resolved Hide resolved
llmfoundry/models/layers/ffn.py Outdated Show resolved Hide resolved
llmfoundry/models/layers/ffn.py Show resolved Hide resolved
@snarayan21
Copy link
Contributor Author

finally fixed the ci errors lfg

@snarayan21 snarayan21 requested a review from dakinggg May 16, 2024 04:51
@snarayan21 snarayan21 merged commit 38ae65b into main May 16, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants