[BUG] AzureResourceManager throws NullPointerException in context of ContainerGroup #27083
Closed
3 tasks done
Assignees
Labels
ARM
customer-reported
Issues that are reported by GitHub users external to the Azure organization.
Mgmt
This issue is related to a management-plane library.
question
The issue doesn't require a change to the product in order to be resolved. Most issues start as that
Comments
ghost
added
needs-triage
6 tasks
|
Thanks for filing this github issue, @sparsick. @weidongxu-microsoft could you please help route this? |
ghost
removed
the
needs-triage
|
@sparsick AzureResourceManager azureResourceManager = AzureResourceManager
.configure()
.withLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS) // for enabling log
.authenticate(credential, profile)and provide log response from: ContainerGroup containerGroup =
azureResourceManager.containerGroups().getByResourceGroup(resourceGroup, agent.getNodeName());when the exception occurs. And redact any sensitive information, thanks! |
|
There's a default log size limit, so if the upper code snippet can't log the response in complete, you can use following code to bypass it: AzureResourceManager azureResourceManager = AzureResourceManager
.configure()
.withLogOptions(new HttpLogOptions()
.setLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS)
.setResponseLogger((logger, loggingOptions) -> {
final HttpResponse response = loggingOptions.getHttpResponse();
String contentLengthString = response.getHeaderValue("Content-Length");
String bodySize = (CoreUtils.isNullOrEmpty(contentLengthString))
? "unknown-length body"
: contentLengthString + "-byte body";
StringBuilder responseLogMessage = new StringBuilder();
responseLogMessage.append("<-- ")
.append(response.getStatusCode())
.append(" ")
.append(response.getRequest().getUrl())
.append(" (")
.append(loggingOptions.getResponseDuration().toMillis())
.append(" ms, ")
.append(bodySize)
.append(")")
.append(System.lineSeparator());
HttpResponse bufferedResponse = response.buffer();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
WritableByteChannel bodyContentChannel = Channels.newChannel(outputStream);
return bufferedResponse.getBody()
.flatMap(byteBuffer -> {
try {
bodyContentChannel.write(byteBuffer.duplicate());
return Mono.just(byteBuffer);
} catch (IOException ex) {
return Mono.error(ex);
}
})
.doFinally(ignored -> {
responseLogMessage.append(", Response body:")
.append(System.lineSeparator())
.append(outputStream.toString(StandardCharsets.UTF_8))
.append(System.lineSeparator())
.append(" <-- END HTTP");
logger.info(responseLogMessage.toString());
}).then(Mono.just(bufferedResponse));
})
)
.authenticate(credential, profile) |
|
@XiaofeiCao the log information with your requested log level: 2022-02-16 14:09:29.645+0000 [id=79] INFO c.m.j.c.a.AciCleanTask$DeploymentRegistrar#registerDeployment: AzureAciCleanUpTask: registerDeployment: Registering deployment test-private-d3zld1qx in jenkins-rg
2022-02-16 14:09:29.890+0000 [id=79] INFO c.m.j.c.aci.AciService#createDeployment: Waiting for deployment test-private-d3zld1qx
2022-02-16 14:09:29.892+0000 [id=79] INFO c.a.c.util.logging.ClientLogger#info: --> GET https://management.azure.com/subscriptions/subscriptionId/resourcegroups/jenkins-rg/providers/Microsoft.Resources/deployments/test-private-d3zld1qx?api-version=REDACTEDTry count: 1(empty body)--> END GET
2022-02-16 14:09:39.959+0000 [id=79] INFO c.a.c.util.logging.ClientLogger#info: --> GET https://management.azure.com/subscriptions/subscriptionId/resourcegroups/jenkins-rg/providers/Microsoft.Resources/deployments/test-private-d3zld1qx?api-version=REDACTEDTry count: 1(empty body)--> END GET
2022-02-16 14:09:50.012+0000 [id=79] INFO c.a.c.util.logging.ClientLogger#info: --> GET https://management.azure.com/subscriptions/subscriptionId/resourcegroups/jenkins-rg/providers/Microsoft.Resources/deployments/test-private-d3zld1qx?api-version=REDACTEDTry count: 1(empty body)--> END GET
2022-02-16 14:10:00.043+0000 [id=79] INFO c.a.c.util.logging.ClientLogger#info: --> GET https://management.azure.com/subscriptions/subscriptionId/resourcegroups/jenkins-rg/providers/Microsoft.Resources/deployments/test-private-d3zld1qx?api-version=REDACTEDTry count: 1(empty body)--> END GET
2022-02-16 14:10:10.069+0000 [id=79] INFO c.a.c.util.logging.ClientLogger#info: --> GET https://management.azure.com/subscriptions/subscriptionId/resourcegroups/jenkins-rg/providers/Microsoft.Resources/deployments/test-private-d3zld1qx?api-version=REDACTEDTry count: 1(empty body)--> END GET
2022-02-16 14:10:20.100+0000 [id=79] INFO c.a.c.util.logging.ClientLogger#info: --> GET https://management.azure.com/subscriptions/subscriptionId/resourcegroups/jenkins-rg/providers/Microsoft.Resources/deployments/test-private-d3zld1qx?api-version=REDACTEDTry count: 1(empty body)--> END GET
2022-02-16 14:10:20.132+0000 [id=79] INFO c.m.j.c.aci.AciService#createDeployment: Deployment test-private-d3zld1qx succeed
2022-02-16 14:10:20.132+0000 [id=79] INFO c.m.j.c.aci.AciCloud#waitToOnline: Waiting agent test-private-5s16p to online
2022-02-16 14:10:20.133+0000 [id=79] INFO c.a.c.util.logging.ClientLogger#info: --> GET https://management.azure.com/subscriptions/subscriptionId/resourceGroups/jenkins-rg/providers/Microsoft.ContainerInstance/containerGroups/test-private-5s16p?api-version=REDACTEDTry count: 1(empty body)--> END GET
2022-02-16 14:10:20.272+0000 [id=79] WARNING c.m.j.c.aci.AciCloud#waitToOnline: Waiting for Agent test-private-5s16p produces a NullPointerException, but it is ignored.
2022-02-16 14:10:35.519+0000 [id=79] INFO c.a.c.util.logging.ClientLogger#info: --> GET https://management.azure.com/subscriptions/subscriptionId/resourceGroups/jenkins-rg/providers/Microsoft.ContainerInstance/containerGroups/test-private-5s16p?api-version=REDACTEDTry count: 1(empty body)--> END GETI added a workaround. Therefore, the log message looks a bit different from the original post. |
|
Hi @sparsick , AzureResourceManager azureResourceManager = AzureResourceManager
.configure()
.withLogOptions(new HttpLogOptions()
.setLogLevel(HttpLogDetailLevel.BODY_AND_HEADERS)
.setResponseLogger((logger, loggingOptions) -> {
final HttpResponse response = loggingOptions.getHttpResponse();
String contentLengthString = response.getHeaderValue("Content-Length");
String bodySize = (CoreUtils.isNullOrEmpty(contentLengthString))
? "unknown-length body"
: contentLengthString + "-byte body";
StringBuilder responseLogMessage = new StringBuilder();
responseLogMessage.append("<-- ")
.append(response.getStatusCode())
.append(" ")
.append(response.getRequest().getUrl())
.append(" (")
.append(loggingOptions.getResponseDuration().toMillis())
.append(" ms, ")
.append(bodySize)
.append(")")
.append(System.lineSeparator());
HttpResponse bufferedResponse = response.buffer();
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
WritableByteChannel bodyContentChannel = Channels.newChannel(outputStream);
return bufferedResponse.getBody()
.flatMap(byteBuffer -> {
try {
bodyContentChannel.write(byteBuffer.duplicate());
return Mono.just(byteBuffer);
} catch (IOException ex) {
return Mono.error(ex);
}
})
.doFinally(ignored -> {
responseLogMessage.append(", Response body:")
.append(System.lineSeparator())
.append(outputStream.toString(StandardCharsets.UTF_8))
.append(System.lineSeparator())
.append(" <-- END HTTP");
logger.info(responseLogMessage.toString());
}).then(Mono.just(bufferedResponse));
})
)
.authenticate(credential, profile)The log should look like : 13:41:13.035 [reactor-http-nio-2] INFO com.azure.resourcemanager.containerinstance.implementation.ContainerGroupsClientImpl$ContainerGroupsService.getByResourceGroup - <-- 200 https://management.azure.com/subscriptions/subscriptionId/resourceGroups/jenkins-rg/providers/Microsoft.ContainerInstance/containerGroups/test-private-5s16p?api-version=REDACTED (4701 ms, 3003-byte body)Response body:XXXXXXThanks! |
|
@XiaofeiCao I'm sorry. I misinterpret The log output after I add your second suggestion: |
|
@sparsick It's fine, I can see now from your log that the I think we can safely assume that missing of |
|
@XiaofeiCao Thank you. I recheck our ARM Template that we use for creating the deployment. We set the protocol to A sample of our ARM Template for creating the deployment {
"$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
"contentVersion": "1.0.0.0",
"parameters": {},
"variables": {
"containerName": "fancyname",
"containerImage": "helloworld",
"cpu": "cpu",
"memory": "memory",
"osType": "linux",
"jenkinsInstance": "instanceId",
"ipType": "Private",
"vnetResourceGroupName": "vnetResourceGroup",
"vnetName": "vnet",
"subnetName": "subnet"
},
"resources": [
{
"name": "[variables('containerName')]",
"type": "Microsoft.ContainerInstance/containerGroups",
"apiVersion": "2021-09-01",
"tags": {
"JenkinsInstance": "[variables('jenkinsInstance')]"
},
"location": "[resourceGroup().location]",
"properties": {
"containers": [
{
"name": "[variables('containerName')]",
"properties": {
"command": [
"command"
],
"image": "[variables('containerImage')]",
"resources": {
"requests": {
"cpu": "[variables('cpu')]",
"memoryInGb": "[variables('memory')]"
}
},
"ports": [
{
"port": "80"
}
],
"environmentVariables": [],
"volumeMounts": []
}
}
],
"osType": "[variables('osType')]",
"volumes": [],
"ipAddress": {
"type": "[variables('ipType')]",
"ports": [
{
"protocol": "tcp",
"port": "80"
}
]
},
"imageRegistryCredentials": [],
"subnetIds": [
{
"id": "[resourceId(variables('vnetResourceGroupName'), 'Microsoft.Network/virtualNetworks/subnets', variables('vnetName'), variables('subnetName'))]"
}
]
}
}
],
"outputs": {
"containerIPv4Address": {
"type": "string",
"value": "[reference(resourceId('Microsoft.ContainerInstance/containerGroups/', variables('containerName'))).ipAddress.ip]"
}
}
} |
|
@sparsick Thanks for your template sample and yes it's actually setting the protocol... Since you mentioned that the NPE is from time to time and the templates are constructed dynamically, can it be that some of them are without To be sure whether this is the case, a log for the actual ARM template used here before the NPE will be of great help! |
|
@sparsick feel free to just add that to the plugin in general, it can be useful for debugging, see pattern in the virtual machines plugin: https://github.com/jenkinsci/azure-vm-agents-plugin/blob/master/src/main/java/com/microsoft/azure/vmagent/AzureVMManagementServiceDelegate.java#L595-L599 This is the template file: Seems the values are set here: I wonder if the port is empty in some cases |
|
@timja I hear you. I doubt that empty ports will result in NPE... However, I tried @sparsick 's armTemplate and to find That said, for quick fix, you can add a protocol here so that deployment service will not ignore it. Or you can wait for next released sdk, which will have NPE fixed in this situation. |
|
@timja 👍 I will prepare a PR with your suggestion. @XiaofeiCao I will prepare a new logout where the ARM template is also logged. |
6 tasks
|
@XiaofeiCao I generate a new log output where the deployment template is also logged. The whole log output when starting 4 agents at the same time based on the same deployment template. |
|
@sparsick Thanks, the log is great! I can tell from your log that the templates are actually the same. I'll add some protections for this situation. You can expect it to be fixed in the next sdk release. In the meanwhile, if you want, you can add a |
6 tasks
|
@sparsick Hi, new SDK has just been released, you can try it out! |
|
@XiaofeiCao Thank you for mentioning me. I opened a PR in https://github.com/jenkinsci/azure-sdk-plugin/. After releasing this plugin, I can test it in the original plugin. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Describe the bug
When using many Azure container instances with private IP address, then the AzureResourceManager throws time to time NullPointerException in context of ContainerGroup.
Exception or Stack Trace
To Reproduce
Steps to reproduce the behavior: see issue jenkinsci/azure-container-agents-plugin#101
Code Snippet
see
Expected behavior
No NullPointerException is thrown.
Setup (please complete the following information):
Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report
The text was updated successfully, but these errors were encountered: