Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a better incentive system to discourage model copies #71

Open
donaldknoller opened this issue Oct 8, 2024 · 4 comments
Open
Labels
discussion In Discussion

Comments

@donaldknoller
Copy link
Contributor

Although current systems in place (3% time penalty, model safetensors hash checking) help guard against blatant model copying, there are still potential avenues of exploitation.
The requirements for a solution that can accurately detect model copying are the following:

  1. The solution can distinguish between "copying" and fine tunes
  2. The solution is a relatively "fast" computation (ie. does not cost more execution time than the current scope of evaluation)

Further notes on tuning other subnet properties:

  1. It is desirable to keep the current temperature at roughly the same range. Although the current number of participants in the subnet may be small, this value can change in the future. Furthermore, there are existing discussions with validators regarding the emissions curve and the current level

  2. It is desirable to keep a relatively low registration cost. A low registration cost can encourage newer participants, and there is already an implied cost for miners who are training and evaluating their models offline.

Additional discussion points are welcome and it is expected that this may not result in a concrete solution, but at the very least provide discourse to find a way forward.

@CharewiczH
Copy link

In my opinion, the biggest problem is that every model gets emission, which encourages cloning... And the best miners are not properly rewarded. Therefore, I propose that only about 10% of the miners should earn, rather than everyone.

@donaldknoller
Copy link
Contributor Author

Thanks for your input.
Curious as to why it would be 10% specifically?
Given that the subnet is limited to 256 entrants, 10% would be roughly 25 entries. Implicitly, this means that 231 participants will never receive any emission at all. It also means that the worst performing entrant may immediately lose emissions with a slight change in scoring. Keep in mind that there may be future changes in the landscape where many more miners will want to participate, and each entry in itself may be a valuable submission in and of itself. Would this 10% be optimal in that scenario as well?

@CharewiczH
Copy link

I'm not convinced that 10% is the optimal solution, but I haven't come across any other subnet where every miner gets emission. For example, in SN37, only the top 1 and 2 receive emissions, which isn't necessarily ideal, but... On the other hand, if every miner earns something, it could encourage cloning and discourage the top miners, for whom the emissions are relatively small.

@donaldknoller
Copy link
Contributor Author

I think in the case of managing emissions, this may be better addressed by adjusting temperature, which other miners have requested as well.
I don't think that there is a worthwhile attempt for a "copying" miner to pay a registration fee to receive 0.001 tao in emissions vs no emissions, as long as the "top" miners are happy.
We will consider adjusting the temperature in a future update, and go from there in this case. Will keep this open for continued discussion

@donaldknoller donaldknoller added question Further information is requested discussion In Discussion and removed question Further information is requested labels Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
discussion In Discussion
Projects
None yet
Development

No branches or pull requests

2 participants