All transactions unknown & 100% error rate post PHP8.1 upgrade (laravel v8) #387
Comments
|
Hi, There are a couple of things that caught our attention, and we'll need a little more information from you.
This value is set: https://docs.newrelic.com/docs/apm/agents/php-agent/configuration/php-agent-configuration/#inivar-daemon-port
|
|
Hello @zsistla
I have not specifically killed the processes, but I have restarted the daemon and apache many times. |
|
Hi, Thanks for the quick response. From the ps command, we see However, your log snippet indicates the agent is expecting to automatically start the daemon, but the agent will not automatically start the daemon if it finds a newrelic.cfg file in the /etc/newrelic/ directory. Do you need to start the daemon manually for a specific reason? The standard installation for the PHP agent will automatically start the daemon if it detects that it is not running. The license key is configured for the agent in a PHP INI file, and it can be modified on a per-directory or per-virtual host basis. The daemon also is configured via the agent configuration (INI) file. Because this newrelic.cfg file exists, the agent isn't automatically starting the daemon. Can you please try to:
|
|
Hello @zsistla, Before I do this, since it's a production server. Can you please clarify why this would make a difference ? The New Relic Daemon is running, we can see this from my snippet where I checked the deamon status, and equally from ps aux. We also know that new relic is receiving a connection, there is throughput and transactions they are just confused in some way. This seems to me to be an issue with new relics interpretation of the application rather than its run status. Thanks, Frank. |
|
Hi Frank,
Also, is this a new installation of the PHP Agent? |
|
It was working with Laravel 8 prior to the fresh installation of the new latest agent. I can't tell you what version it was on before i'm afraid. New Relic had been entirely broken for a couple of months while we were waiting for the extremely late release of PHP8.1 support. I have followed the above steps. New result of ps aux is as follows : I am still seeing an average error rate of 100% and transactions are still all bundled into 'unknown' as name. Latest logs from newrelic-deamon.log : Latest lines from php_agent.log |
|
Just to be clear - things were working with a version of Laravel 8 (possibly different than is being used now) and a previous agent version and a previous PHP version? So at least two things have changed (agent version and PHP version)? |
|
To be abundantly clear
The Laravel version has not changed. |
|
The log you provided seems to indicate the PHP agent extension which loads into your application was unable to talk to the newrelic-daemon which handles communication with the backend. I think figuring out this problem is going to help us understand what is happening. Having verbose log output might help - here are some instructions on setting this up. At this point I would recommend opening a case with support so we can track the case better. |
|
How do i open a case with support - I haven't found a way to do this. |
|
Sure I can help - our main portal for support is support.newrelic.com. |
|
@mfulb - I see no way of raising a support case. Only how to view existing cases. If you click through to i need more support it just redirects you to forums etc. Unfortunately all too typical. |
|
@mfulb - We have the newrelic.ini config stored in our ansible role, did the ini options change in the php8.1 release ? |
|
Disregard ^ I found the stub on this repo and it's identical (apart from actual values for license key, and a proper application name) |
|
Any update on this? We are facing the same problem, also there is a related discussion with no answer |
|
We're really looking for a sollution on this as NewRelic now basically is unusable and it has been for a while. Can someone maybe try out my statement in #392? That this appears to be Opcache related by disabling opcache? I understand that that's not something you're able to do on a production environment, but maybe someone has this issue on a non-prod environment. This knowledge won't fix this bug, but maybe help someone with knowledge fixing things :) |
|
@mfulb - It's time this was taken by the reins. New Relic is a paid service - We waited patiently for 3 months for PHP8.1 release and now people are still having issues. That means, some people have been paying for a broken product for 4 months at this point. Have the team at least attempted to spin up NewRelic on PHP8.1 with Laravel installed ? Can you confirm that it works out of the box on a blank Laravel application ? |
|
I managed to get transaction names working for about 10 minutes, we had two versions of the new-relic.ini file (identical) which was clearly causing some issues, but even with that removed it went back to labelling all transactions as unknown. I have some verbose error logging. I find the line Even with this provided, i'd like confirmation that new relic have an instance of at least laravel 8.x running on php8.1 and can confirm that the agent works on a blank install. Here are the processes for good measure : |
|
I'd like to report that our logging looks identical to the logging above. What I did notice were the "Skipping instrumented function" lines. Whenver we have a request that IS working, these line's dont appear. |
|
@rjp-thijs - Are you by any chance using redis-sentinel ? |
|
Ok, so confession time, the error rate being 100% was actually the fault of an extension we had for Redis Sentinel that was firing a deprecation on every request, so although the site was fully functional New Relic was registering the error rate as 100%. This was by bad because these errors were not in php errors, or laravels errors so i disregarded them as bogus. The transactions labelled as 'unknown' however continues to be an issue. Every time i restart apache (and therefore new-relic) it works for a few minutes then transactions go back to being grouped under unknown. |
|
We're not using redis-sentinel. What you've described above matches our problem. I can't say it has to do with reboots. For us it almost looks "random", but around 90% of the requests ends up "unknown". But as I wrote in the other thread, it happens that the a transaction might end up in unknown, and if the same request happens again it might(!) end up as a named tansaction. More often it will still go to unknown. As you can see on the screenshot below the route /admin/shipments (aka admin.shipments.index) goes most often to "unknown" but has results as a named entry. |
|
Also one very intresting thing to add (which again makes me more think this is happening due to Opcache). We have two services, the one thats used in nginx and one that monitors all our console commands / terminal. The one that monitores nginx has the problem with naming almost all transactions as unknown and has opcache enabled. The one that monitors console commands doesn't name anything unknown. |
Exactly what I'm observing as well. Frankly, New Relic is testing my patience as a paying customer. Might be time to consider other options. |
|
Hey guys, here we have the same problem. All transactions are unknown. Laravel 8 with php 8.1. we use naming in routes api.php
|
|
Anyone know how to actually raise a support case ? According to the documentation here : https://docs.newrelic.com/docs/new-relic-solutions/solve-common-issues/find-help-use-support-portal/ You should be able to raise a support ticket, but I don't have a button. Can you see if you have a button and if so create a ticket directing them to this issue. Otherwise it's going to be weeks before we get a solution. |
|
Sadly the option doesn't seem to be there for us either. |
|
@zsistla could you please raise to the NR team? |
|
@chris-skilltech while the package does help it doesn't seem to fix error reporting |
|
@trothwell-carsguide We've been using the following in our public function register()
{
$this->reportable(function (Throwable $e) {
if (function_exists('newrelic_notice_error')) {
newrelic_notice_error($e);
}
});
} |
|
Using the middleware from above that contains WebTransaction/Custom/OurAccountName\SearchController@search instead of So just be aware of that. Upgrading to New Relic 10 alone didn't fix this for us FYI |
|
@ak-war, re:
Any updates on when in October the release will be? |
|
Come on now @ak-war we pay a lot of money for this service which has been useless for months. No updates at all from your end. We deserve a reply that's better than "We don't know, we hope soon". |
|
Unfortunately, this is even not working on the dev branch. I tried to compile the PHP agent from the source still it fails. I'm testing newrelic integration with Laravel, drop is the point when I restarted my docker.
|
|
If this service doesn't work, then why do we pay for it? |
Tbh, we are planning on moving to Sentry in the next few days (our |
|
Any update here? |
|
Thank you everyone for your constructive feedback. A release will be made around November 3 that will be fixing this issue. |
|
@ak-war Is that a final release or a dev release ? Can we test this ? |
|
@EvanRijn It will be an official release, we do not have a dev release. |
|
@ak-war perfect! is tomorrow still the release day ? |
|
New Relic PHP Agent v10.3.0.315 has been released with the fix: https://github.com/newrelic/newrelic-php-agent/releases. |
|
Docs referenced on the release notes are not working: |
|
@nsinisterra thanks for catching that - links should be correct now. |
|
Awesome team! I'm able to confirm that this update fix the problem! No other issues for now. NOTE: For other doing the update right now, we have 4 apps running the APM Agent. In 3 of them, make an |
|
Unfortunately we're still having the same issue on the new agent (10.3.0.315-1). Seeing a lot of messages in the php_agent logs with: |
|
I have confirmed that we've successfully updated the agent using Are there any other steps we need to take other than installing the new agent, or is this issue not really fixed? |
Can you share the installed module list ? Can you share the installed module list ? |
|
I'm a bit unclear. The supported operating systems lists CentOS 6 or higher, but then states CentOS 6 support was dropped. Is CentOS 7 supported? |
Our customers are also using Magento 2.4.4 (and up) are also affected, and also the only ones using PHP 8.1. Those using versions under 2.4.4 for Magento and < PHP 8.1 are having no issues. |
|
We're also receiving issues about all requests marked as What strikes to me is that prior to the Please let us know/keep us posted about the status of this problem, as many merchants and developers are preparing their applications to be ready for both black friday and the holiday season in general. |
|
@theloveofcode or @tdgroot - Would one of you mind opening a new github issue on the repo so we can track the Magento 2 impact? We'd like to increase visibility on this issue and separate it from this Laravel-specific issue. We believe this and #436 were resolved by the |
|
Resolved by |
|
I upgraded very pessimistically but i am happy to report that at least for the last hour, transactions are now correct labelled and not unknown. Let's hope that support for PHP8.2 takes less than a year once it's released. (Laravel V9, New Relic 10.3.0, PHP8.1) |
|
Hi @tdgroot, hope you doing well. Are you still experiencing an issue with NR Thanks in advance for your help! |
and others
Update on this issue
The error rate being 100% was actually legitimate and I have resolved it BUT all transactions are still being labelled as unknown.
Description
1 day after switching to the new release all of the transactions are labeled as unknown and the error rate is 100% despite no errors being thrown.
I have opened a forum discussion for this but I have had no response : https://discuss.newrelic.com/t/php-new-relic-completely-broken-after-php-8-1-upgrade/179337
Relevant Logs / Console output
DAEMON STATUS
PHP AGENT LOGS
Daemon logs from today are empty
Your Environment
We are running out of the box newrelic.ini settings and config template
Additional context
The text was updated successfully, but these errors were encountered: