add federated contrastive learning baseline SimCLR and its linear prob evaluation #278
Conversation
|
|
|
|
||
|
|
||
| class SimCLRTransform(): | ||
| def __init__(self, is_sup, image_size=32): |
There was a problem hiding this comment.
please provide python-style docstring for the newly added classes and functions
| transform_train = SimCLRTransform(is_sup=False, image_size=32) | ||
| transform_test = T.Compose([ | ||
| T.ToTensor(), | ||
| T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))] |
There was a problem hiding this comment.
is it conventional to use 0.5 rather than the sample mean?
There was a problem hiding this comment.
Setting both parameters to 0.5 and using with T.totensor() can force the data to be scaled to the [-1,1] interval
There was a problem hiding this comment.
here the first arg is mean of the signals, to my knowledge, it is usually calculated from the available examples.
There was a problem hiding this comment.
T.totensor() can take the value of image from 0-255 to 0-1, and T.Normalize(0.5,0.5) can take 0-1 to -1-1 using function (x-mean)/std
federatedscope/cl/model/SimCLR.py
Outdated
|
|
||
|
|
||
| class Bottleneck(nn.Module): | ||
| expansion = 4 |
There was a problem hiding this comment.
is it necessary to use class attribute?
| representations = torch.cat([z1, z2], dim=0) | ||
| similarity_matrix = F.cosine_similarity(representations.unsqueeze(1), representations.unsqueeze(0), dim=-1) | ||
|
|
||
| l_pos = torch.diag(similarity_matrix, N) |
There was a problem hiding this comment.
I think, up to now, similarity_matrix is a 2N-by-2N matrix. Am I wrong? Why do we need to take the the above and below main diagonal?
federatedscope/cl/trainer/trainer.py
Outdated
| # print(len(x), x[0].size(), x[1].size(), label.size()) | ||
| x1, x2 = x[0], x[1] | ||
| z1, z2 = ctx.model(x1, x2) | ||
| if len(label.size()) == 0: |
There was a problem hiding this comment.
when will we enter such a branch?
There was a problem hiding this comment.
we enter this branch in contrastive learning with two augment data
There was a problem hiding this comment.
I mean when does the length of the size of label become zero
There was a problem hiding this comment.
I follow the torch_trainer and add this branch. Should I remove it?
| @@ -10,7 +10,7 @@ def get_optimizer(model, type, lr, **kwargs): | |||
| if isinstance(type, str): | |||
| if hasattr(torch.optim, type): | |||
| if isinstance(model, torch.nn.Module): | |||
| return getattr(torch.optim, type)(model.parameters(), lr, | |||
| return getattr(torch.optim, type)(filter(lambda p: p.requires_grad, model.parameters()), lr, | |||
There was a problem hiding this comment.
is any pfl algo affected by such a change? @yxdyc
|
This pr includes a new trainer, which is designed for conducting contrastive learning. @DavdGao could you have a look at that part for us? Thanks! |
| config = config | ||
| return data_dict, config | ||
|
|
||
| def Cifar4LP(config): |
There was a problem hiding this comment.
Duplicated code from line 118 to line 156.
federatedscope/cl/model/SimCLR.py
Outdated
|
|
||
|
|
||
| # Model class | ||
| class ResNet(nn.Module): |
There was a problem hiding this comment.
Please import ResNet from https://github.com/alibaba/FederatedScope/blob/master/federatedscope/contrib/model/resnet.py if there is no other concern.
There was a problem hiding this comment.
In this PR #267 , a ResNet model is already added in federatedscope/contrib/model/resnet.py. Please check if we still need to add a new resnet.
federatedscope/core/aggregator.py
Outdated
| @@ -104,12 +104,37 @@ def _para_weighted_avg(self, models, recover_fun=None): | |||
| return avg_model | |||
|
|
|||
|
|
|||
| class NoCommunicationAggregator(Aggregator): | |||
| class NoCommunicationAggregator(ClientsAvgAggregator): | |||
There was a problem hiding this comment.
@yxdyc Please have a look at this change to local mode, thanks.
| # Split data into dict | ||
| data_dict = dict() | ||
|
|
||
| # Splitter |
There was a problem hiding this comment.
Please refer to splitter
federatedscope/cl/model/SimCLR.py
Outdated
| @@ -0,0 +1,235 @@ | |||
| import torch | |||
There was a problem hiding this comment.
This file looks like a copy from https://github.com/akhilmathurs/orchestra/blob/main/models.py.
Please consider the copyright issues.
| @@ -0,0 +1,41 @@ | |||
| import torch | |||
There was a problem hiding this comment.
| @@ -0,0 +1,190 @@ | |||
| import math | |||
There was a problem hiding this comment.
To reproduce baselines, the sample_splitter is needed.
https://github.com/akhilmathurs/orchestra/blob/228d7a6379b6788e7dc288d3a9557d62b940c47a/utils.py#L44
|
| T.RandomResizedCrop(32, scale=(0.5, 1.0), interpolation=T.InterpolationMode.BICUBIC), | ||
| T.RandomHorizontalFlip(p=0.5), | ||
| T.ToTensor(), | ||
| T.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)) |
There was a problem hiding this comment.
I am still very curious about why we have to use such a mean and std. If it is a conventional usage in CL, please explain for us. The ultimate image classification task does not use this transformation, right?
There was a problem hiding this comment.
T.totensor() can take the value of image from 0-255 to 0-1, and T.Normalize(0.5,0.5) can take 0-1 to -1-1 using function (x-mean)/std. the data augement is time-costing and use sample mean and std wil take more time.
| splitter = get_splitter(config) | ||
| data_train = splitter(data_train) | ||
| data_val = data_train | ||
| data_test = splitter(data_test) |
There was a problem hiding this comment.
Although the original train and test data of CIFAR10 are iid, how to ensure that splitting them by our splitter respectively can keep the train and test data of a specific client iid?
No description provided.