Include GeoIP in core after improvements

[mattab]

See doc: Geo Locate visitors countries cities and regions.

GeoIP plugin #45 is one of the most popular plugins. For a web analytics tool, getting user countries as accurately as possible is critical, and Piwik should help users in this direction.

When the plugin is released in trunk, we should update the FAQ, website pages and wiki pages mentionning GeoIP, and mark as closed the GeoIP ticket #45. For Goals compatibility of GeoIP plugin, see #1434.

Please let us know in the comments your feedback. If you would like to participate... well you know what to do!

[gka]

Just some input to clarify the terms "country" and "region". Refering to the list of administrative levels used in OpenStreetMaps a country would correspond to admin level 2 while regions would correspond to admin level 4.

[gka]

Do we record lat/long for each visitor, or do we assume that other systems will know where to plot a given City

I think it is NOT necessary to record lat/long for each visitor. It is sufficient to record the city id. The GeoIP db would resolve each visitors location within the same city to the same lat/long anyway. In fact for each city there is only one lat/long stored in the GeoLite City DB (more precisely in the cityByCountry table).

As the number of available cities (= pairs of lat/long) differs between the different GeoIP databases, it makes no sense to put these information into other systems like the world map.

[anonymous-matomo-user]

if I might add to this. The maxmind db gives a city lookup. This does not work how people think it will.
Blocks of IP numbers are sold to to service providers who resell to end users. However, the IP issuing authority assign the city of the ISP address to all the IP numbers. At least that is how it works in the UK. Things may vary in different countries and ISPs don't reallocate city when they sell dedicated IP numbers to end users.
The result is that city lookup generally only gives the city of the ISP and not where the visitor is visting from. The ISP can be anywhere in the country and hundreds of miles from where the visitor is based. In other words, city lookup is useless except for giving the location of ISPs. This also means that lat long is useless too since it seems to be based on city lookup.
When IPV6 is rolled out and if, and only if, ISPs allocate city to users when they purchase a fixed IP then city lookup may become useful. But many ISPs still use dynamically allocated IPs so it wouldn't work in that case either.
In short the concept of providing city and/or lat/long of vistors is fundamentlly flawed.

[oparoz]

+1 for this, especially the Apache module detection routine. I get a few fatal errors in my logs because the plugin insists on loading the local files instead of getting the data from Apache.

[anonymous-matomo-user]

I have the commercial db of Maxmind. You can use it if you want for developing the new plugin. Let me know how I can contact you.

[mattab]

See a bug report in the forum in the php script to update past visits: http://forum.piwik.org/read.php?2,71587,page=2#msg-71784

and fix: http://forum.piwik.org/read.php?2,70989,page=1#msg-92434

[robocoder]

I'll take this on, in conjunction with the ipv6 ticket.

[mattab]

• New report: Users companies? I believe that we can get the 'Enterprise/Company' that Visitors connect from, with the GeoIP data. It would be interesting to propose this new report in Users > Countries page.

[anonymous-matomo-user]

Great idea.

[robocoder]

In the existing GeoIP plugin, there's a misc/.htaccess file. We don't want this in the new plugin. Access to geoipUpdateRows.php (or equivalent) should be guarded via token_auth.

[anonymous-matomo-user]

Should I delete the .htaccess file in there?

[robocoder]

Yes, you can remove the .htaccess file. After you've run it once, you shouldn't have to run it again.

[anonymous-matomo-user]

$ php geoipUpdateRows.php

Fatal error: Call to undefined function _parse_ini_file() in /home/kiplingw/avaneya.com/piwik/core/Config.php on line 373

???

I also removed the .htaccess file.

[robocoder]

This should be fixed in the updated .zip that I attached to #45. Heres the patch so you don't have to redownload the .zip:

Index: geoipUpdateRows.php
===================================================================
--- geoipUpdateRows.php (revision 51)
+++ geoipUpdateRows.php (working copy)
@@ -20,8 +20,8 @@
        . PATH_SEPARATOR . PIWIK_INCLUDE_PATH . '/libs'
        . PATH_SEPARATOR . PIWIK_INCLUDE_PATH . '/plugins');

+require_once PIWIK_INCLUDE_PATH . '/libs/upgradephp/upgrade.php';
 require_once PIWIK_INCLUDE_PATH . '/core/testMinimumPhpVersion.php';
-
 require_once PIWIK_INCLUDE_PATH . '/core/Loader.php';

 $GLOBALS['PIWIK_TRACKER_DEBUG'] = false;

[anonymous-matomo-user]

Thanks. Applied. How can I test it?

[anonymous-matomo-user]

I ran

$ php geoipUpdateRows.php

It finished execution (no output), and I noticed the UserCountry_ thing is still there in the stats. Should I just ignore that for now and assume new stats will not have that?

[robocoder]

yes

[anonymous-matomo-user]

Thank you =)

[mattab]

To answer questions in the ticket:

• we should track regions in the log_visit table (new field, char(2) ?) as per specification in: http://www.maxmind.com/app/fips10_4
• we should also track cities, and add a new field for cities.
  • the API for cities, would return the 'label' being the city name, and also one column for latitude, one column for longitude, since this is required to draw the Cities Maps Open Source SVG Map to show cities and regions #1652
  • this means, that we don't store lat/long in log_visit

[robocoder]

Replying to matt:

• this means, that we don't store lat/long in log_visit

I'm thinking of keeping lat/long because:

• in Open Source SVG Map to show cities and regions #1652, greg says the map plots by lat/long; including it in log_visit avoids a city-to-latlong lookup
• the GeoIP plugin already stores lat/long in log_visit
• future support for HTML5 navigator.geolocation

[robocoder]

re: comment:13

I would like to propose:

• rolling the provider plugin into the geolocation plugin
• if the geolocation plugin can get the organization field, it populate location_provider
• otherwise, fallback to the gethostbyaddr() method

[mattab]

I would like to propose:

• rolling the provider plugin into the geolocation plugin
• if the geolocation plugin can get the organization field, it populate location_provider
• otherwise, fallback to the gethostbyaddr() method

great idea!

The only thing, is please make sure the few "Provider" special cases are still working. In particular, VisitorGenerator & proxy-piwik.php disable the Provider lookup because it is too slow

[robocoder]

Ok. There are a couple of third party plugins (e.g., KSVisitorImport and TrackerSecondaryDb) that also disable the Provider plugin.

[mattab]

as a note, these plugins will be obsolete once we implement #134

[anonymous-matomo-user]

+1 vote for adding regions onto this as well. They are available in GeoLiteCity, so might as well use them. It would be great to include this into a regional map as well that the country map can drill down into.

[anonymous-matomo-user]

Copy of my coment to #5465 (sorry, I used wrong ticket, apparantly, I knew there was one specifically for integration of GeoIP into core):

This new plugin sounds promising. But I hope you are going to also keep the old browser language/country detection, maybe named as such. I personally consider that language display equally important as the IP location display.

Following scenario: I'm on a travel around the world, and have a travel blog. People accessing that blog are often people I have met on the trip, often still traveling. Now, when I see my Piwik logs, the IP location (which I currently check manually) is surely interesting, but what tells me more about a visitor is actually his browser language. If you check the IP address I am writing this from, you will see that it is Malaysian. How much do I have to do with Malaysia? Nothing. My browser language is German of Germany, which tells more. And the combination of the two IP location and browser country (i.e. the current detection) actually provides one more detail: the visitor is most likely a traveler or an expat. I can imagine website who interested in that marketing information.

You would not believe how many travelers roam the world this days. And I would say most of them use the often free WiFi (at their place of stay, bars and restaurants all over Southeast Asia) with their own devices: Laptops, Phones, Tablets, etc. It seems to be the new way of travel, with people sticking their noses into displays half of their time, with most of that time on Facebook.

P.S.: Since there are countries with several languages (Belguim, etc.), but also countries with common language (UK, US, etc.), maybe both, the browser country and its language could be shown (if provided by browser). Additionally to the IP location provided by this plugin.

[robocoder]

jawsmith: #638

[anonymous-matomo-user]

+1 on jawsmith proposal on having a combined vision on location against visitor's preferred language.
As a belgian developer I can tell you that this kind of information can be of crucial interest in a country like Belgium, but in many others too. For example, usage of the spanish language in some regions of the US can be an important factor I think...

I imagine an ideal "Vistor countries" GeoIP plugin offering the current "Countries" split, clicking a country name would open a "Regions" list, clicking a region name would open (the currently available) "Cities" list. Then an additional button could be fit at the bottom, between the "Display simple table" and "Display a table with more metrics" that would "Display a table with languages". That table could have one additional column for each language that was detected...

[anonymous-matomo-user]

Replying to vipsoft:

jawsmith: #638

Thank you very much for the info on the browser language detection plugin! That just leaves the browser country detection, in case the GeoIP plugin replaces it in core. (E.g.: Is it a British or an American accessing my website from the Philippines?)

[gka]

Do we recording regions as well as Countries?
Do we record Cities?

As the new world map widget will be able to display data for regions and cities, it would be amazing if Piwik would be able to record the data for regions and cities :)

Do we record lat/long for each visitor, or do we assume that other systems (eg. the world map) will know where to plot a given City (and maintain their own database)?

Nope, the world map doesn't store locations for every city. Instead, it will be able to plot any given lat/long onto the map.

[diosmosis]

(In [7181]) Refs #1823, fade 'Done' in & out after successfully switched location providers.

[diosmosis]

(In [7186]) Refs #1823, many changes including:

• Add warning if old GeoIP plugin is used.
• Display links to installation instructions for different providers if they are not installed.
• Add report documentation for country, region, continent and city reports, round latitude/longitude.
• Increase 'Done' timeout when switching providers
• Display quick start instructions for GeoIP if no GeoIP provider is currently working.
• Add script to geolocate old data.

[diosmosis]

(In [7187]) Refs #1823, display informative note when Region + City reports have no location data, merge unknown rows in region & city reports and make sure latitude/longitude is rounded in API output.

[mattab]

(In [7203]) Refs #1823 Adding important-to-have target=_blank

[mattab]

(In [7204]) Minor text change & provider ordering Refs #1823

[mattab]

(In [7205]) remove test Refs #1823

[diosmosis]

This bug is fixed. :) I created a ticket for improvements here: #3442

[mattab]

See doc: Geo Locate visitors countries cities and regions.

[diosmosis]

(In [7234]) Refs #1823, add note to geoipUpdateRows.php that tells user to re-process their reports.

[diosmosis]

(In [7236]) Refs #1823, add note to IP anonymization about geolocation accuracy.

[diosmosis]

(In [7260]) Refs #1823, add alternative check for GEOIP_COUNTRY_CODE $_SERVER var to ServerBased GeoIP implementation.

[diosmosis]

(In [7281]) Refs #1823, do broken check w/ both GEOIP_ADDR & GEOIP_COUNTRY_CODE.

[diosmosis]

(In [7283]) Refs #1823, added redundant trusted hosts warning to general settings page & display help icon that links to faq in warning.

[diosmosis]

[7628] refs this ticket, not #1253. (typo in commit msg)

[mattab]

We need YOUR help! We are running a crowd funding campaign to raise funds to implement the detailed Visitors Maps of Countries, Regions and Cities (for all countries)!

These maps will be beautiful, usable, and built using open standards SVG+JS. They will show detailed visitor count, conversion rates, by Country but also (New!) by city and region.

Pledge now at: http://crowdfunding.piwik.org/analytics-maps-world-country-city-region/

Piwik needs you!