Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ISSUE#12994]Optimize config operation. #13002

Open
wants to merge 7 commits into
base: develop
Choose a base branch
from
Open

[ISSUE#12994]Optimize config operation. #13002

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
4ac2eba
optimize config operation.
MatthewAden Dec 30, 2024
046940e
Update the unit tests.
MatthewAden Jan 1, 2025
745949c
Update Aspect.
MatthewAden Jan 1, 2025
08564eb
Refactor aspect.
MatthewAden Jan 10, 2025
e854f8f
Fix pmd.
MatthewAden Jan 10, 2025
dfd421e
Fix unit test.
MatthewAden Jan 10, 2025
83b5957
Fix capacityManagementAspect.
MatthewAden Jan 10, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
231 changes: 104 additions & 127 deletions config/src/main/java/com/alibaba/nacos/config/server/aspect/CapacityManagementAspect.java
View file
Open in desktop

Large diffs are not rendered by default.

212 changes: 86 additions & 126 deletions config/src/main/java/com/alibaba/nacos/config/server/aspect/ConfigChangeAspect.java
View file
Open in desktop

Large diffs are not rendered by default.

246 changes: 90 additions & 156 deletions config/src/main/java/com/alibaba/nacos/config/server/aspect/RequestLogAspect.java
View file
Open in desktop

Large diffs are not rendered by default.

32 changes: 7 additions & 25 deletions config/src/main/java/com/alibaba/nacos/config/server/controller/ConfigController.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,6 +214,7 @@ public Boolean publishConfig(HttpServletRequest request, HttpServletResponse res

ConfigRequestInfo configRequestInfo = new ConfigRequestInfo();
configRequestInfo.setSrcIp(RequestUtil.getRemoteIp(request));
configRequestInfo.setSrcType("http");
configRequestInfo.setRequestIpApp(RequestUtil.getAppName(request));
configRequestInfo.setBetaIps(request.getHeader("betaIps"));
configRequestInfo.setCasMd5(request.getHeader("casMd5"));
Expand Down Expand Up @@ -299,38 +300,19 @@ public Boolean deleteConfig(HttpServletRequest request, HttpServletResponse resp
String clientIp = RequestUtil.getRemoteIp(request);
String srcUser = RequestUtil.getSrcUserName(request);

return configOperationService.deleteConfig(dataId, group, tenant, tag, clientIp, srcUser);
return configOperationService.deleteConfig(dataId, group, tenant, tag, clientIp, srcUser, "http");
}

/**
* Execute delete config operation.
*
* @return java.lang.Boolean
* @author klw
* @Description: delete configuration based on multiple config ids
* @Date 2019/7/5 10:26
* @Param [request, response, dataId, group, tenant, tag]
* delete configs based on the IDs list.
*/
@DeleteMapping(params = "delType=ids")
@Secured(action = ActionTypes.WRITE, signType = SignType.CONFIG)
public RestResult<Boolean> deleteConfigs(HttpServletRequest request, @RequestParam(value = "ids") List<Long> ids) {
String clientIp = RequestUtil.getRemoteIp(request);
String srcUser = RequestUtil.getSrcUserName(request);
final Timestamp time = TimeUtils.getCurrentTime();
List<ConfigAllInfo> configInfoList = configInfoPersistService.removeConfigInfoByIds(ids, clientIp, srcUser);
if (CollectionUtils.isEmpty(configInfoList)) {
return RestResultUtils.success(true);
}
for (ConfigAllInfo configInfo : configInfoList) {
ConfigChangePublisher.notifyConfigChange(
new ConfigDataChangeEvent(configInfo.getDataId(), configInfo.getGroup(), configInfo.getTenant(),
time.getTime()));

ConfigTraceService.logPersistenceEvent(configInfo.getDataId(), configInfo.getGroup(),
configInfo.getTenant(), null, time.getTime(), clientIp, ConfigTraceService.PERSISTENCE_EVENT,
ConfigTraceService.PERSISTENCE_TYPE_REMOVE, null);
}
return RestResultUtils.success(true);

return RestResultUtils.success(configOperationService.deleteConfigs(ids, clientIp, srcUser));
}

@GetMapping("/catalog")
Expand Down Expand Up @@ -683,7 +665,7 @@ public RestResult<Map<String, Object>> importAndPublishConfig(HttpServletRequest
final String srcIp = RequestUtil.getRemoteIp(request);
String requestIpApp = RequestUtil.getAppName(request);
final Timestamp time = TimeUtils.getCurrentTime();
Map<String, Object> saveResult = configInfoPersistService.batchInsertOrUpdate(configInfoList, srcUser, srcIp,
Map<String, Object> saveResult = configOperationService.batchInsertOrUpdate(configInfoList, srcUser, srcIp,
null, policy);
for (ConfigInfo configInfo : configInfoList) {
ConfigChangePublisher.notifyConfigChange(
Expand Down Expand Up @@ -935,7 +917,7 @@ public RestResult<Map<String, Object>> cloneConfig(HttpServletRequest request,
final String srcIp = RequestUtil.getRemoteIp(request);
String requestIpApp = RequestUtil.getAppName(request);
final Timestamp time = TimeUtils.getCurrentTime();
Map<String, Object> saveResult = configInfoPersistService.batchInsertOrUpdate(configInfoList4Clone, srcUser,
Map<String, Object> saveResult = configOperationService.batchInsertOrUpdate(configInfoList4Clone, srcUser,
srcIp, null, policy);
for (ConfigInfo configInfo : configInfoList4Clone) {
ConfigChangePublisher.notifyConfigChange(
Expand Down
3 changes: 2 additions & 1 deletion config/src/main/java/com/alibaba/nacos/config/server/controller/v2/ConfigControllerV2.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ public Result<Boolean> publishConfig(ConfigForm configForm, HttpServletRequest r

ConfigRequestInfo configRequestInfo = new ConfigRequestInfo();
configRequestInfo.setSrcIp(RequestUtil.getRemoteIp(request));
configRequestInfo.setSrcType("http");
configRequestInfo.setRequestIpApp(RequestUtil.getAppName(request));
configRequestInfo.setBetaIps(request.getHeader("betaIps"));
configRequestInfo.setCasMd5(request.getHeader("casMd5"));
Expand Down Expand Up @@ -169,7 +170,7 @@ public Result<Boolean> deleteConfig(HttpServletRequest request, @RequestParam("d

String clientIp = RequestUtil.getRemoteIp(request);
String srcUser = RequestUtil.getSrcUserName(request);
return Result.success(configOperationService.deleteConfig(dataId, group, namespaceId, tag, clientIp, srcUser));
return Result.success(configOperationService.deleteConfig(dataId, group, namespaceId, tag, clientIp, srcUser, "http"));
}

/**
Expand Down
13 changes: 12 additions & 1 deletion config/src/main/java/com/alibaba/nacos/config/server/model/ConfigRequestInfo.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,14 +30,17 @@ public class ConfigRequestInfo implements Serializable {

private String srcIp;

private String srcType;

private String requestIpApp;

private String betaIps;

private String casMd5;

public ConfigRequestInfo(String srcIp, String requestIpApp, String betaIps, String casMd5) {
public ConfigRequestInfo(String srcIp, String srcType, String requestIpApp, String betaIps, String casMd5) {
this.srcIp = srcIp;
this.srcType = srcType;
this.requestIpApp = requestIpApp;
this.betaIps = betaIps;
this.casMd5 = casMd5;
Expand All @@ -54,6 +57,14 @@ public void setSrcIp(String srcIp) {
this.srcIp = srcIp;
}

public String getSrcType() {
return srcType;
}

public void setSrcType(String srcType) {
this.srcType = srcType;
}

public String getRequestIpApp() {
return requestIpApp;
}
Expand Down
10 changes: 0 additions & 10 deletions config/src/main/java/com/alibaba/nacos/config/server/monitor/MetricsMonitor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -175,21 +175,11 @@ public static Timer getReadConfigRtTimer() {
.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "readConfigRt");
}

public static Timer getReadConfigRpcRtTimer() {
return NacosMeterRegistryCenter
.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "readConfigRpcRt");
}

public static Timer getWriteConfigRtTimer() {
return NacosMeterRegistryCenter
.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "writeConfigRt");
}

public static Timer getWriteConfigRpcRtTimer() {
return NacosMeterRegistryCenter
.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "writeConfigRpcRt");
}

public static Timer getNotifyRtTimer() {
return NacosMeterRegistryCenter.timer(METER_REGISTRY, "nacos_timer", "module", "config", "name", "notifyRt");
}
Expand Down
1 change: 1 addition & 0 deletions config/src/main/java/com/alibaba/nacos/config/server/remote/ConfigPublishRequestHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ public ConfigPublishResponse handle(ConfigPublishRequest request, RequestMeta me

ConfigRequestInfo configRequestInfo = new ConfigRequestInfo();
configRequestInfo.setSrcIp(srcIp);
configRequestInfo.setSrcType("rpc");
configRequestInfo.setRequestIpApp(meta.getLabels().get(Constants.APPNAME));
configRequestInfo.setBetaIps(request.getAdditionParam("betaIps"));
configRequestInfo.setCasMd5(request.getCasMd5());
Expand Down
51 changes: 10 additions & 41 deletions config/src/main/java/com/alibaba/nacos/config/server/remote/ConfigRemoveRequestHandler.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,8 @@
import com.alibaba.nacos.api.exception.NacosException;
import com.alibaba.nacos.api.remote.request.RequestMeta;
import com.alibaba.nacos.auth.annotation.Secured;
import com.alibaba.nacos.common.utils.StringUtils;
import com.alibaba.nacos.config.server.model.event.ConfigDataChangeEvent;
import com.alibaba.nacos.config.server.model.gray.TagGrayRule;
import com.alibaba.nacos.config.server.service.ConfigChangePublisher;
import com.alibaba.nacos.config.server.service.repository.ConfigInfoGrayPersistService;
import com.alibaba.nacos.config.server.service.repository.ConfigInfoPersistService;
import com.alibaba.nacos.config.server.service.trace.ConfigTraceService;
import com.alibaba.nacos.config.server.service.ConfigOperationService;
import com.alibaba.nacos.config.server.utils.ParamUtils;
import com.alibaba.nacos.config.server.utils.TimeUtils;
import com.alibaba.nacos.core.control.TpsControl;
import com.alibaba.nacos.core.paramcheck.ExtractorManager;
import com.alibaba.nacos.core.paramcheck.impl.ConfigRequestParamExtractor;
Expand All @@ -39,8 +32,6 @@
import com.alibaba.nacos.plugin.auth.constant.SignType;
import org.springframework.stereotype.Component;

import java.sql.Timestamp;

/**
* handler to remove config.
*
Expand All @@ -50,14 +41,10 @@
@Component
public class ConfigRemoveRequestHandler extends RequestHandler<ConfigRemoveRequest, ConfigRemoveResponse> {

private final ConfigInfoPersistService configInfoPersistService;

private final ConfigInfoGrayPersistService configInfoGrayPersistService;
private final ConfigOperationService configOperationService;

public ConfigRemoveRequestHandler(ConfigInfoPersistService configInfoPersistService,
ConfigInfoGrayPersistService configInfoGrayPersistService) {
this.configInfoPersistService = configInfoPersistService;
this.configInfoGrayPersistService = configInfoGrayPersistService;
public ConfigRemoveRequestHandler(ConfigOperationService configOperationService) {
this.configOperationService = configOperationService;
}

@Override
Expand All @@ -66,36 +53,18 @@ public ConfigRemoveRequestHandler(ConfigInfoPersistService configInfoPersistServ
@ExtractorManager.Extractor(rpcExtractor = ConfigRequestParamExtractor.class)
public ConfigRemoveResponse handle(ConfigRemoveRequest configRemoveRequest, RequestMeta meta)
throws NacosException {
// check tenant
String tenant = configRemoveRequest.getTenant();
String dataId = configRemoveRequest.getDataId();
String group = configRemoveRequest.getGroup();
String tag = configRemoveRequest.getTag();
String underLine = "_";
String clientIp = meta.getClientIp();
ParamUtils.checkTenant(tenant);
ParamUtils.checkParam(dataId, group, "datumId", "rm");
ParamUtils.checkParam(tag);

try {
ParamUtils.checkTenant(tenant);
ParamUtils.checkParam(dataId, group, "datumId", "rm");
ParamUtils.checkParam(tag);
String persistEvent = ConfigTraceService.PERSISTENCE_EVENT;

String clientIp = meta.getClientIp();
String grayName = null;
if (StringUtils.isBlank(tag)) {

configInfoPersistService.removeConfigInfo(dataId, group, tenant, clientIp, null);
} else {
persistEvent = ConfigTraceService.PERSISTENCE_EVENT_TAG + underLine + tag;

grayName = TagGrayRule.TYPE_TAG + underLine + tag;
configInfoGrayPersistService.removeConfigInfoGray(dataId, group, tenant, grayName, clientIp, null);
}
final Timestamp time = TimeUtils.getCurrentTime();
ConfigTraceService.logPersistenceEvent(dataId, group, tenant, null, time.getTime(), clientIp, persistEvent,
ConfigTraceService.PERSISTENCE_TYPE_REMOVE, null);
ConfigChangePublisher.notifyConfigChange(
new ConfigDataChangeEvent(dataId, group, tenant, grayName, time.getTime()));
configOperationService.deleteConfig(dataId, group, tenant, tag, clientIp, null, "rpc");
return ConfigRemoveResponse.buildSuccessResponse();

} catch (Exception e) {
Loggers.REMOTE_DIGEST.error("remove config error,error msg is {}", e.getMessage(), e);
return ConfigRemoveResponse.buildFailResponse(e.getMessage());
Expand Down
36 changes: 34 additions & 2 deletions config/src/main/java/com/alibaba/nacos/config/server/service/ConfigOperationService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,11 @@
import com.alibaba.nacos.common.utils.MapUtil;
import com.alibaba.nacos.common.utils.NumberUtils;
import com.alibaba.nacos.common.utils.StringUtils;
import com.alibaba.nacos.config.server.model.ConfigAllInfo;
import com.alibaba.nacos.config.server.model.ConfigInfo;
import com.alibaba.nacos.config.server.model.ConfigOperateResult;
import com.alibaba.nacos.config.server.model.ConfigRequestInfo;
import com.alibaba.nacos.config.server.model.SameConfigPolicy;
import com.alibaba.nacos.config.server.model.event.ConfigDataChangeEvent;
import com.alibaba.nacos.config.server.model.form.ConfigForm;
import com.alibaba.nacos.config.server.model.gray.BetaGrayRule;
Expand Down Expand Up @@ -293,13 +295,13 @@ private int getMaxGrayVersionCount() {
* Synchronously delete all pre-aggregation data under a dataId.
*/
public Boolean deleteConfig(String dataId, String group, String namespaceId, String tag, String clientIp,
String srcUser) {
String srcUser, String srcType) {
String persistEvent = ConfigTraceService.PERSISTENCE_EVENT;
String grayName = "";
if (StringUtils.isBlank(tag)) {
configInfoPersistService.removeConfigInfo(dataId, group, namespaceId, clientIp, srcUser);
} else {
persistEvent = ConfigTraceService.PERSISTENCE_EVENT_TAG + "-" + tag;
persistEvent = ConfigTraceService.PERSISTENCE_EVENT_TAG + "_" + tag;
grayName = TagGrayRule.TYPE_TAG + "_" + tag;
configInfoGrayPersistService.removeConfigInfoGray(dataId, group, namespaceId, grayName, clientIp, srcUser);
deleteConfigTagv1(dataId, group, namespaceId, tag, clientIp, srcUser);
Expand All @@ -319,6 +321,36 @@ private void deleteConfigTagv1(String dataId, String group, String namespaceId,
configInfoTagPersistService.removeConfigInfoTag(dataId, group, namespaceId, tag, clientIp, srcUser);
}
}

/**
* Deletes configuration information based on the IDs list.
*/
public Boolean deleteConfigs(List<Long> ids, String srcIp, String srcUser) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deleteConfigs should check capacity , or can we split deleteConfigs operation to serval single deleteConfig operation in ConfigController?

List<ConfigAllInfo> configInfoList = configInfoPersistService.removeConfigInfoByIds(ids, srcIp, srcUser);
if (configInfoList == null || configInfoList.isEmpty()) {
return true;
}

Timestamp time = TimeUtils.getCurrentTime();
for (ConfigAllInfo configInfo : configInfoList) {
ConfigChangePublisher.notifyConfigChange(
new ConfigDataChangeEvent(configInfo.getDataId(), configInfo.getGroup(), configInfo.getTenant(),
time.getTime()));
ConfigTraceService.logPersistenceEvent(configInfo.getDataId(), configInfo.getGroup(),
configInfo.getTenant(), null, time.getTime(), srcIp, ConfigTraceService.PERSISTENCE_EVENT,
ConfigTraceService.PERSISTENCE_TYPE_REMOVE, null);
}

return true;
}

/**
* Batch insert or update configuration information.
*/
public Map<String, Object> batchInsertOrUpdate(List<ConfigAllInfo> configInfoList, String srcUser, String srcIp,
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all batch operation invoked ConfigOperationService single operation interface, all aspects works at single config publish & delete interface of ConfigOperationService

Map<String, Object> configAdvanceInfo, SameConfigPolicy policy) throws NacosException {
return configInfoPersistService.batchInsertOrUpdate(configInfoList, srcUser, srcIp, configAdvanceInfo, policy);
}

public Map<String, Object> getConfigAdvanceInfo(ConfigForm configForm) {
Map<String, Object> configAdvanceInfo = new HashMap<>(10);
Expand Down
Loading
Loading