In one api fails, all other edge functionality gets blocked

[ravindrasingh22]

Describe the bug
APIGEE edge executes management APIs to fetch API products, developers etc.
If the endpoint for API product fails with some particular product detail fetch somehow it is not allowing developers to do anything else on the page.

To Reproduce
1- There is a private product on the instance that has some issue on APIGEE cloud.
2- When I enable the module it fetches all of the products available on apigee orgs
3- When a particular product comes in a loop system start throwing en error and page doesn't get executed.

Expected behavior
If there is an error on a particular product should not be synced to the devportal rest of the product should work fine.

Version Info
8.x-1.6 Apigee Edge module.

[arunz6161]

@ravindrasingh22 thanks for logging the issue. To help us reproduce the issue, could you share details of the error thrown by edge when the product fetch fails? You can enable debug module to get the response from edge: https://www.drupal.org/docs/8/modules/apigee-edge/monitor-apigee-edge-debug-logs

[ravindrasingh22]

its 500 error on apigee for that particular api product with message.
API Product [Developer-XXXXXXXXX-Products] does not exist for tenant [0xxxx9c191] and id [null] Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->withController() (line 220 of /var/www/html/web/modules/contrib/apigee_edge/src/Entity/Storage/EdgeEntityStorageBase.ph
p).

The way simple way to simulate the apiproducts error is put the dummy reponse in api call
GET /v1/organizations/oooooxxxxx/apiproducts?expand=true&startKey=Developer-xxxxxx-Products and after that you would see drupal behavior.

[arunz6161]

Thanks @ravindrasingh22 for sharing the error details. - we will prioritize this issue in our current sprint and update soon.

[arlina-espinoza]

Thanks for the details @ravindrasingh22. I'd like to point out that the API products are not synced into Drupal, they are remote entities (read from Edge), and if configured within Drupal, only cached temporarily to improve performance.
In any case, the way the API products are loaded is:

1. Fetch a paginated listing from the management API (GET /v1/organizations/oooooxxxxx/apiproducts?expand=true).
2. If list isn't empty, get the last API product returned from the listing, as use its ID as the startKey to request the next page of results (GET /v1/organizations/oooooxxxxx/apiproducts?expand=true&startKey=Developer-xxxxxx-Products). Remove the first item from this result set as it was already included in the previous call.
3. Repeat step 2 until all API products are loaded.

From your comment, it looks like the error is getting thrown on step 2, when specifying the startKey. The issue is that this module cannot simply skip on an error on an single API product because it looks like a whole page of results is failing, so no way to selectively skip on a single failing one.
This, as you mentioned, seems to be a misconfiguration on Edge cloud - I've only been able to reproduce the same does not exist for tenant response when using a non-existing API product as the startKey parameter. Could you share the failing API product ID/name to see if it could be causing an issue when URL encoding it?

[ravindrasingh22]

@arlina-espinoza - Yes, you understood correctly, there is an issue on Edge only with one specific which I can't fix/delete that. [Request is already raised on APIGEE support for that], but my concern is one buggy product should not block the other functionalities. And being a coder I feel it is doable on this edge module. it should capture that product in logs like unable to fetch XXX product. but it should not block other functionality.

[arlina-espinoza]

I think we can update the code that does the listing so that if a call with expand=true fails, it would retry fetching that page with just it's IDs (expand=false), and then loading each individually. It is definitively not as performant but it could probably work, assuming the call with expand=false works.
@ravindrasingh22 Could you try the following two calls from CURL or postman and let me know what they return?:

• GET /v1/organizations/oooooxxxxx/apiproducts?expand=true&startKey=Developer-xxxxxx-Products
• GET /v1/organizations/oooooxxxxx/apiproducts?expand=false&startKey=Developer-xxxxxx-Products

And also what is the ID of the product that is failing, to see if it could be causing an issue when URL encoding it?
Thanks!

[arlina-espinoza]

@ravindrasingh22 I posted a PR for this, feel free to try and let us know your feedback: #340

[ravindrasingh22]

@arlina-espinoza - I have tested your PR I still get the same problem.

{
"code": "cps.kms.ApiProductDoesNotExist",
"message": "API Product [Developer-XXXXXXXX-Products] does not exist for tenant [XXXXXX] and id [null]",
"contexts": []
}

[ravindrasingh22]

From Drupal this is the log -
API Product [Developer-XXXXXXX-Products] does not exist for tenant [xxxxxxx] and id [null] Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->withController() (line 220 of /var/www/html/web/modules/contrib/apigee_edge/src/Entity/Storage/EdgeEntityStorageBase.php).#0 /var/www/html/web/modules/contrib/apigee_edge/src/Entity/Storage/EdgeEntityStorageBase.php(282): Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->withController(Object(Closure)) #1 /var/www/html/web/modules/contrib/apigee_edge/src/Entity/Storage/EdgeEntityStorageBase.php(114): Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->getFromStorage(NULL) #2 /var/www/html/web/core/lib/Drupal/Core/Entity/EntityStorageBase.php(300): Drupal\apigee_edge\Entity\Storage\EdgeEntityStorageBase->doLoadMultiple(NULL) #3 /var/www/html/web/modules/contrib/apigee_edge/src/Entity/Form/DeveloperAppFormTrait.php(132): Drupal\Core\Entity\EntityStorageBase->loadMultiple() #4 /var/www/html/web/modules/contrib/apigee_edge/src/Entity/Form/AppCreateForm.php(126): Drupal\apigee_edge\Entity\Form\DeveloperAppCreateFormBase->apiProductList(Array, Object(Drupal\Core\Form\FormState)) #5 /var/www/html/web/modules/contrib/apigee_edge/src/Entity/Form/AppCreateForm.php(77): Drupal\apigee_edge\Entity\Form\AppCreateForm->apiProductsFormElement(Array, Object(Drupal\Core\Form\FormState)) #6 /var/www/html/web/core/lib/Drupal/Core/Entity/EntityForm.php(149): Drupal\apigee_edge\Entity\Form\AppCreateForm->form(Array, Object(Drupal\Core\Form\FormState)) #7 /var/www/html/web/modules/contrib/apigee_edge/src/Entity/Form/DeveloperAppCreateFormForDeveloper.php(45): Drupal\Core\Entity\EntityForm->buildForm(Array, Object(Drupal\Core\Form\FormState)) #8 [internal function]: Drupal\apigee_edge\Entity\Form\DeveloperAppCreateFormForDeveloper->buildForm(Array, Object(Drupal\Core\Form\FormState), Object(Drupal\user\Entity\User)) #9 /var/www/html/web/core/lib/Drupal/Core/Form/FormBuilder.php(520): call_user_func_array(Array, Array) #10 /var/www/html/web/core/lib/Drupal/Core/Form/FormBuilder.php(277): Drupal\Core\Form\FormBuilder->retrieveForm('developer_app_a...', Object(Drupal\Core\Form\FormState)) #11 /var/www/html/web/core/lib/Drupal/Core/Controller/FormController.php(91): Drupal\Core\Form\FormBuilder->buildForm(Object(Drupal\apigee_edge\Entity\Form\DeveloperAppCreateFormForDeveloper), Object(Drupal\Core\Form\FormState)) #12 [internal function]: Drupal\Core\Controller\FormController->getContentResult(Object(Symfony\Component\HttpFoundation\Request), Object(Drupal\Core\Routing\RouteMatch)) #13 /var/www/html/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(123): call_user_func_array(Array, Array) #14 /var/www/html/web/core/lib/Drupal/Core/Render/Renderer.php(573): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber{closure}() #15 /var/www/html/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(124): Drupal\Core\Render\Renderer->executeInRenderContext(Object(Drupal\Core\Render\RenderContext), Object(Closure)) #16 /var/www/html/web/core/lib/Drupal/Core/EventSubscriber/EarlyRenderingControllerWrapperSubscriber.php(97): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->wrapControllerExecutionInRenderContext(Array, Array) #17 /var/www/html/vendor/symfony/http-kernel/HttpKernel.php(151): Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber->Drupal\Core\EventSubscriber{closure}() #18 /var/www/html/vendor/symfony/http-kernel/HttpKernel.php(68): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1) #19 /var/www/html/web/core/lib/Drupal/Core/StackMiddleware/Session.php(57): Symfony\Component\HttpKernel\HttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #20 /var/www/html/web/core/lib/Drupal/Core/StackMiddleware/KernelPreHandle.php(47): Drupal\Core\StackMiddleware\Session->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #21 /var/www/html/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(106): Drupal\Core\StackMiddleware\KernelPreHandle->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #22 /var/www/html/web/core/modules/page_cache/src/StackMiddleware/PageCache.php(85): Drupal\page_cache\StackMiddleware\PageCache->pass(Object(Symfony\Component\HttpFoundation\Request), 1, true) #23 /var/www/html/web/core/lib/Drupal/Core/StackMiddleware/ReverseProxyMiddleware.php(47): Drupal\page_cache\StackMiddleware\PageCache->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #24 /var/www/html/web/core/lib/Drupal/Core/StackMiddleware/NegotiationMiddleware.php(52): Drupal\Core\StackMiddleware\ReverseProxyMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #25 /var/www/html/vendor/stack/builder/src/Stack/StackedHttpKernel.php(23): Drupal\Core\StackMiddleware\NegotiationMiddleware->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #26 /var/www/html/web/core/lib/Drupal/Core/DrupalKernel.php(694): Stack\StackedHttpKernel->handle(Object(Symfony\Component\HttpFoundation\Request), 1, true) #27 /var/www/html/web/index.php(19): Drupal\Core\DrupalKernel->handle(Object(Symfony\Component\HttpFoundation\Request)) #28 {main}

[cnovak]

@ravindrasingh22 this issue seems to be a problem with the Edge API, and it seems like the fix should be on the API side instead of us putting in code changes to work around this bug for all paging API calls where this could happen. I would also like to get more details on the root cause of the issue that you are seeing. Would you be able to give me the case # you put into support for this issue? Feel free to email me directly if you do not want to share the case in this issue. Thanks!

[cnovak]

I am going to close this ticket since there is a ticket in to Apigee support to fix the Apigee API that we call, but let us know if you need more help @ravindrasingh22 by adding more comments.

It is hard to fix this issue since we cannot reproduce it without access to your org with this error, too bad @arlina-espinoza 's fix did not help the issue.