-
-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Understand why conversions are sometimes attributed to "Direct entry" or the payment provider, instead of the current channel #18612
Comments
@mattab Any idea what this feature could be named and how we would configure this? Not sure if we want to have this per site and globally or whether it be enough only one way etc. It could get confusing maybe for people that don't need this feature. Maybe it could be even a separate plugin? |
In my point of view, if a user :
Then the referrer should at least be the 1st one! (would it be possible to accumulate referrers?) And if the other referrer is considered as "direct entry” (what is done in case of excluded referrer) it means the only real referrer is still the 1st one... Note also that in case of payment gateway, the returning back delay should not exceed short delay as in other ways, the payment token would probably expire... Then the scenario where the 1st referrer is a payment gateway should be very rare case... |
Thanks @tsteur for creating the issue here on my behalf. I largely agree with @heurteph-ei - the main reason for referral data is to monitor how traffic is being driven to the website, especially if we attribute value from sales to a referrer. I can't really see any benefit of having "direct entry" as the referrer as it is misleading. However I do see a legitmate scenario that could be obscured by always using the first referrer. Here are 2 examples
In the above scenario - Paypal (for example) should be set up as a referral exclusion and therefore the 1st referrer should be used (Email).
In the above scenario we may want to know both the former or the latter referrer. With this in mind I would suggest either...
or
|
What i don't understand yet, is that the scenarios above should already work meaning: @adsham @heurteph-ei Could you maybe confirm if the subsequent visits (the one with referrer "Paypal" or the direct entries) ones occured within 30 min of their last pageview? (the only other reason i can think of is that people use a different browser to pay and that Matomo cannot recognise them as the same user, but that should be very edge case) |
Hi @mattab Thanks for picking this up. You will see that several users have picked up on this issue here https://forum.matomo.org/t/referral-exclusions/33582/22 and possibly here too https://forum.matomo.org/t/what-can-explain-the-mysterious-too-many-direct-entries-phenomenon/31721/6 I have also just looked through our data and can confirm that there are no "delays" as you are suggesting. However I'm not really in a position to fully test this out at the moment (we do not have a test payment provider currently configured) perhaps @heurteph-ei or someone from the forum could do this? I will ask there too. Could the resolution in the article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ be causing the original referrer to be overwritten with "direct entry"? Many thanks once again, hopefully we can find a resolution to this soon! |
Hello, for me its very simple and there are different ways, how you can create the "correct" goal*1. *1Goal: as already written. We have to know, from where the customer is coming from. I am not a technician, so I have no idea, why some 30 mins. timeframe is here on the table. To make it lesser complicated: *2 this would have the advantage, that you can directly see a fast overview of the daily payment gatesways too, without needing your ERP. Exlucde paymentgateways and putting all referrers as direct entry or use the gateway as referrer is for nobody really helpful. |
Hey there, it is great that this issue is picket up now. I have been experiencing this problem for a long time now. Here is our setup:
This completely messes up all conversion attribution, as most customers use paypal. All customers return from paypal within the 30 minute timespan. I would opt for an "attribute to first/last referrer" configuration in matomo. Thanks for taking care of this and let me know if I can be of further help. Cheers Jens P.S.: If I would use the ignore referrer option, the new session would have a direct access referrer which even more messes up attribution, since I cannot identify anymore that paypal messed up the referrer. |
Coming from GA and having struggled with the same issue there years ago, I quickly ran into this again with Matomo. Specifically the 'payment service provider' scenario. With GA we started with exclusion list but could finally solve this for good by adding the 'ignore_referrer' parameter to the PSP return/error/pending page (e.g. /thank-you?ignore_referrer=true) so regardless of the payment method, the site in between is always ignored. This works better than referral exclusion lists because the payment urls tend to change over time and new ones are easily overlooked. A lot of e-commerce sites will already be using this parameter so would supporting this not solve the issue for many cases? |
We have the same problem. I think most have the same problem and it's important to find a solution for this. The conversions are thus virtually useless, since it distorts everything. Hope somebody will find a solution soon. |
Perhaps then this is a bug rather than an enhancement? - it seems that many of us are experiencing this issue. |
I can give you these details:
|
Any update on this? It seems to be pushed back again into a later milestone. The referral data is an important part of our service and we have recently seen a client leave to a competitor because of the inaccurate data in our reports (from Matomo). The initial issue was raised almost 3 years ago in the forum https://forum.matomo.org/t/referral-exclusions/33582/25 and this Github issue was raised 2 months ago and pushed back into milstone 4.10 (4.8 released today). Please could we have some committment as to when this will be addressed as it's getting harder to explain to clients that the issue is being looked at but cannot tell them when it will be looked into/fixed. Many thanks in advance! |
Hello. |
Please read this again and understand, that your solution is not a solution. This will not fix the bug. It's just a workaround which will result in more issues without fixing the main issue. |
@ts1985 That topic is not as easy as you might think. Updating the referrer is only done in some specific cases and only if the previously stored referrer was direct. So that can't overwrite any other referrers. But that actually isn't the problem the issue topic was about. The conversion attribution is not only defined by the visit referrer. If there is an attribution cookie, that one will overwrite the conversion attribution. As this one is handled in javascript, we cant really do anything on the server side. If you don't want a conversion being attributed to a certain service provider, you would need to unset the referrer for the javascript tracking or use |
I think it is that easy. Just don't update the referrer. It's a bug that the referrer is updated for the same visit. And another big problem, also described here by different people, is, that there are so many "direct" visits. Some compared to Google Analytics for example and that there are much less "direct" visits. |
Well, not updating the referrer anymore will bring you even more direct visits 🤷 |
There is a complete thread about this "too many direct entries" topic: https://forum.matomo.org/t/what-can-explain-the-mysterious-too-many-direct-entries-phenomenon/31721 It's a real problem. I know it's another problem but it seems the Referrer topic in general is problematic for Matomo. |
We're experiencing the same problems that conversions aren't attributed correctly to the campaigns, but to visits from Paypal, or to "deref-gmx.net". The latter is an example that hasn't been mentioned yet; it's from an email campaign that includes campaign parameters in the email, but the visit (and subsequent ecommerce purchase) is still attributed to a website, with no trace of the campaign to be found in the visitor's profile. (I've disable cookies, don't know if that has anything to do with it.) What we're seeing is that according to Matomo's statistics, none of our campaigns in the recent months have led to any conversions, even though this is very unlikely in general, we see a spike in purchases after campaigns, and at least for some conversions we have evidence that they definitely came from a campaign. @sgiehl From what you described, you intend to add a way to ignore certain referers, but that requires everyone to find out which URLs they should add, and also will exclude some legitimate referers. The example with deref-gmx.net shows how fragile and open ended that is, as you probably wouldn't have thought of this domain. (The other idea about the URL parameter wouldn't work at all, as we cannot influence it, obviously.) Could you describe how this would get us correct attributions both in the absence of a full knowledge of the referrers we have to exclude, and without requiring every single user to adjust their configuration? From what I understand, the underlying problem here really is that the source of a visit is ever changed after the fact at all. This seems fundamentally wrong: The source of a visit is fully determined by the referrer (or parameter) of the first visited page. If, as you described above, the events are sometimes not submitted in the right order, then that can be worked around e.g. by only assigning a source after a few seconds. But importantly, once a source has been decided on, it should never be changed anymore, because it is by then a historic fact that shouldn't be affected by any later events. It also seems wrong that direct visits are treated differently. A direct (= unknown) visit is a source like any other and shouldn't be changed after the fact either. In particular, if we don't know where a user came from at first, the fact that at a later time they came from an identifiable source still doesn't give us any knowledge about their first visit. Instead, there probably needs to be a distinct state for the source, namely "uninitialized", which would be different from a "direct visit"; this state would only exist as long as we don't have enough information to make a decision yet. Now, this doesn't take the more complicated cases into consideration, e.g. the campaign id changing in the middle of a visit. IMO these should probably be counted as a new visit, as such a change is evidence that the user left and then returned through a different channel. But whatever is decided here, the complicated cases should not affect the bevaviour of the simple cases. tl;dr: May I suggest to implement a mode where it only determines the source from the first page visit (or the first few visits in the first few seconds, if technically necessary), never changes it afterwards, and treats direct visits like any other? Then we can experiment with that, and if it works out, make it the default mode, leaving the previous mode for backwards compatibility or even removing it. I'm pretty sure this is what many of the participants (and lurkers) in this discussion want, even if it later might have to be adjusted to handle some edge cases. |
Not updating the referrer at all can be easily achieved by removing a couple of methods: matomo/plugins/Referrers/Columns/ReferrerType.php Lines 58 to 68 in 1155273
matomo/plugins/Referrers/Columns/ReferrerName.php Lines 40 to 50 in 1155273
matomo/plugins/Referrers/Columns/ReferrerUrl.php Lines 53 to 63 in c973567
Removing those would fully stop updating any referrer information after the first tracking request of a visit. Only setting/updating it for the first page view or within a certain time frame sound fine for me, but that's not a decision I can make. ping @mattab But: Conversions might still be attributed to another referrer. Those attributions are handled using an attribution cookie, which might be updated when returning from any external service (as by default it uses the last referrer). |
Can these be removed in hosted instances of Matomo? |
@adsham this can be only removed if you have access to the source code. For Matomo Cloud this can't be changed. |
@schuetzm if you disable cookies, then the "same-visit" conversions should be still attributed, but indeed any visit from a newsletter from days ago or weeks ago will not be attributed to the newsletter. Only visits generated from the newsletter and directly converting will be attributed. (ref = https://matomo.org/faq/general/faq_156/)
the knowledge of the problematic referrers is clear to people who have the problem, because most of the goal conversions / ecommerce conversions are attributed to these "problematic referrers" so they appear in many places in the reports and it looks buggy. So if people realise this is buggy, then hopefully they will find the feature (although I can see how that won't be easy for many people).
That's why the feature lets you enter the domain within the UI of Matomo so you can enter that domain name there and it will apply to all websites automatically (or you can only set it for one website if you want) If we get complaints from a few people like we did for
Fyi that's already the case, see: https://matomo.org/faq/how-to/faq_19616/ |
@adsham I believe your attribution issues will be fixed on Matomo Cloud once the PRs are merged and released and deployed on the Cloud. Not sure yet when this will be all done but should be by end of July. |
Thanks! I commented them out and invalidated the historical data, but there are still many conversions that are attributed to Paypal. Would this change only apply to future visits? |
@schuetzm Yes. That only applies to future visits. |
I disabled referrer overwrites in May in our instance of matomo. And I have analyzed 4 last months in our matomo db (direct sql queries). Referrer overwrites gave a large wrong statistics. 12 conversions in April for one referrer site but only 3 of them are correct. A common picture is that user bought a product (with direct entry) and then wanders back and forth to the blog or forum and then returns to the main site. I am not going to argue with matomo team anymore, I am used to make a series of changes in the matomo code after each applied update. |
@Demichev Do you know what the referrer type was set to? Was the initial value |
I can also confirm that with the above mentioned functions disabled we're now seeing more plausible statistics: many direct visits, many from particular search engines, and a few from campaigns. (Although there are still a few from Paypal, where it probably lost the connection because we're not using cookies.) So, can we please have a switch for this behaviour in the configuration? Do you want me to open a separate issue for this feature request? |
As I can see all refs to our site with null name have type = 1 and with filled name have types 2 (search engines I guess), 3 (other sites), 7 (YouTube). I don't know what it means. |
@Demichev See Lines 26 to 30 in 53c00a7
Not sure if storing the referrer url, if it is flagged as a direct entry, is correct. An url from the same site imho should never be stored as referrer url or at least I can't see any good reason for that. |
when this is happening and causing data issues, then it may be worth also tracking all sub-domains/sites in one new Matomo website using cross domain tracking https://matomo.org/faq/how-to/faq_23654/ (Alternatively another workaround (with downside) is you can also define your blog and forum as "alias URLs" in the main site in Matomo: "Alias URLs: domains and subdomains that are tracked in this website. This will ensure that tracked domains don’t appear in the Referrer report." see https://matomo.org/faq/how-to/create-and-manage-websites/ -- But downside is that then the blog and forum won't be attributed as a source directly anymore... (unless you cross link with the URLs campaign parameters) |
I know about multi domain visitors, we plan to test it in the future.
Don't know a good solution but current inconsistency when different refs are stored for url and name looks wrong for me. |
Thanks again for all the valuable inputs. I will close this issue here now. Based on the discussion in here, we have implemented some new features that allow excluding referrers:
I'm aware that besides those features there is still a discussion about updating referrer attributions in general. |
This issue has been mentioned on Matomo forums. There might be relevant details there: https://forum.matomo.org/t/multi-channel-conversion-attribution-models-comparison/45580/12 |
This issue has been mentioned on Matomo forums. There might be relevant details there: https://forum.matomo.org/t/does-matomo-attribute-more-traffic-to-the-direct-channel/47334/2 |
This issue has been mentioned on Matomo forums. There might be relevant details there: |
from https://forum.matomo.org/t/referral-exclusions/33582/25
The resolution suggested by Matomo in this article https://matomo.org/faq/how-to/how-do-i-add-a-referral-exclusion-in-matomo/ does not really address the underlying issue - it merely recategorizes the referrer from “website” to “direct entry”, which in reality isn’t the true referrer for the user’s visit.
Here is an example of what is happening.
As a result we are over attributing the channel “direct entry” greatly (40% vs 15% on GA).
I'm not sure if we actually should call this feature "Referrer exclusion" as it's more like an "ignore refrerrer" or "keep the original referrer"? We wouldn't exclude this tracking request. We would still track that referrer but re-use the original referrer from the same visit. If it's from a different visit then we would use again a direct visit?
The text was updated successfully, but these errors were encountered: