Beyond "Trust Me": How to Properly Evaluate IP Data Accuracy

When comparing IP geolocation providers, how do you know which one is right when they disagree? The natural tendency is to trust what's familiar — usually data from your existing provider. But what if your current "source of truth" is actually wrong?

Being an outlier doesn't mean data is inaccurate. In fact, when it comes to IP geolocation, it often means IPinfo is identifying the true location while others are relying on outdated or self-reported data.

This post will provide a framework for genuinely evaluating IP data accuracy beyond simply comparing providers against each other.

Why Do Providers Give Different Answers?

Multiple providers may place the same IP address in different locations. This happens because of:

• Dependence on outdated records: Many providers rely heavily on WHOIS data and geofeeds, which are often self-reported and unverified.
• Intentional misreporting: Hosting providers sometimes deliberately falsify server locations to attract customers. We found a case where servers physically located in Amsterdam were advertised as being in 27 different countries.

• Methodological differences: Some providers use empirical measurements, while others depend on self-reported or static datasets.

Why Consensus Isn't a Reliable Measure of Accuracy

It’s tempting to assume that the majority is correct, but in IP geolocation, that's often not true. Many providers pull from the same flawed sources, creating an illusion of agreement even when the data is wrong.

Outlier results aren't always mistakes — in fact, they may indicate more rigorous, reality-based measurement.

For example, in one case, the IP address 64.138.26.13 was placed in the U.S. by every other provider. But ping tests from multiple locations revealed the lowest RTTs came from Singapore — evidence that the server was actually in Singapore. Our findings were later confirmed via WHOIS records showing a local office of Haemonetics Corporation.

You can read the full analysis in our post on ping-based geolocation versus WHOIS records.

Measuring Accuracy Requires Trusted Ground Truth

The gold standard for evaluating IP geolocation accuracy is comparing against a reliable set of "ground truth" data — IP addresses with known, verified locations. Here's how to establish and use ground truth effectively:

Sources of Ground Truth Data

Organizations often have access to various sources of ground truth:

1. Device Location Data: Mobile apps or websites can collect GPS coordinates alongside IP addresses, creating a dataset of known locations.
2. Customer-Reported Locations: Information gathered during signup, verification, or from support tickets.
3. Corporate Network Data: For enterprise clients, the exact locations of office IP ranges are known.
4. Verification Systems: Multi-factor authentication or login verification systems that confirm user locations.

Avoiding Ground Truth Pitfalls

Not all ground truth is created equal. Watch out for these common issues:

• Circular References: Ensure your ground truth isn't derived from IP geolocation itself. Some mobile SDKs fall back to IP-based location when GPS is unavailable.
  • Perfect Match Suspicion: Be wary of ground truth data that aligns too perfectly with a specific provider's geolocation data. For example, if your device locations match exactly the coordinates that MaxMind gives for a city (rather than showing natural variation within the city), this could indicate the data source is using IP geolocation as a fallback rather than actual device location. Legitimate GPS-based ground truth should show natural distribution patterns within cities and regions.
• VPN and Proxy Usage: Users may connect through VPNs or proxies, causing their IP addresses to appear from different locations than their physical presence.
• Self-Reported Inaccuracies: Users may provide incorrect information about their location, either accidentally or deliberately.
• Outdated Information: IP assignments change frequently — ground truth must be recent to be valid. The importance of recency changes with the geographical resolution you’re interested in: IPs change cities much more often than they change countries.
• Mobile Carrier Challenges: Mobile carrier IPs represent a special case where traditional geolocation concepts can break down. Many carriers use Carrier-Grade NAT to share IPs among hundreds or thousands of devices, sometimes across entire regions or countries. In such cases, these IPs can't be reliably geolocated to a specific city or even region e.g., a device in Seattle and another in Miami might simultaneously share the same IP address through their mobile carrier's network. It's important to note that not all mobile carriers behave this way; some assign IPs in a more geographically-aware manner. Accuracy evaluation involving mobile carrier IPs should take these differences into account, as IP assignment practices can significantly affect geolocation reliability. 

Qualifying Your Ground Truth

To ensure your ground truth is reliable:

1. Verify the data collection methodology
2. Understand how location was determined 
3. Check for timestamps to ensure recency
4. Filter out known VPNs and proxies, and mobile carrier IPs
5. Cross-validate with other confirmation methods when possible

Defining Accuracy Metrics

Once you have reliable ground truth, you need consistent metrics to evaluate accuracy:

Distance-Based Metrics

• Median Distance Error: The median distance between predicted locations and actual locations across all IPs in your dataset. This metric is less affected by outliers than the mean.
• Mean Distance Error: The average distance between predicted locations and actual locations. This will be higher than the median if there are significant outliers.
• 90th Percentile Error: The maximum error for 90% of your dataset. This helps understand the worst-case scenarios while excluding extreme outliers.

Percentage-Based Metrics

• Accuracy Within X km: The percentage of IPs geolocated within a specific distance (e.g., 10km, 50km, 100km) of their true location.
• Correct City/Region/Country Rate: The percentage of IPs assigned to the correct administrative division. 

Coverage Metrics

• Percentage of IPs With Results: Some providers may return "unknown" for challenging IPs. This metric helps understand overall coverage.
• Confidence Radius Accuracy: For providers that offer a confidence radius, what percentage of true locations fall within the stated radius?

If Ground Truth Isn't Available: Physics Doesn't Lie

When you don't have access to verified ground truth, network physics provides an independent verification method:

Speed of Light: The Ultimate Authority

Network communications are bound by physics — specifically, the speed of light in fiber optic cables (approximately 200,000 km/second). This means a server truly located in Amsterdam cannot respond to a ping from Amsterdam in less than 1ms while taking 150ms to respond from Singapore.

Round-trip time (RTT) measurements from known locations provide empirical evidence that can't be falsified, unlike documentation that can say anything.

Evidence Through Triangulation

Multiple measurement points can be used to triangulate an IP's location. Similar to GPS, this approach creates a confidence radius around the predicted location. By comparing RTT from numerous global vantage points, you can determine not just the country but often the city where an IP is located with high confidence.

Conducting Independent Verification

You can perform this verification yourself:

1. Use tools like ping.sx, check-host.net to test from multiple global locations
2. Sort by lowest RTT to determine likely actual location
3. Consider physical constraints (a response from Amsterdam in <10ms means the server is at least somewhat close to Amsterdam)
4. Look for supporting evidence in hostnames, ASN registration, and other metadata

6 Step Framework for Proper IP Data Evaluation

Combining ground truth with network physics verification gives you a robust framework:

1	Identify test IPs	Select addresses from locations critical to your business Include IP addresses from various regions and network types Ensure diversity in ASNs and connection types (datacenter, residential, mobile)
2	Define your accuracy requirements	Determine acceptable error thresholds for your use cases Decide which metrics matter most for your application
3	Establish ground truth when possible	Collect and verify IP addresses with known locations Document the verification method for each ground truth IP
4	Compare provider results	Run the same test dataset through different providers Calculate consistent metrics across all providers
5	Conduct independent verification	Use network physics (RTT) to validate disputed results Document evidence when providers disagree
6	Evaluate provider methodology	Ask providers to explain their verification process Understand how they handle edge cases and conflicts

Common Evaluation Pitfalls to Avoid

• Assuming majority consensus equals accuracy: Most providers copy from the same sources
• Relying on visual spot-checking: A systematic approach is needed
• Testing only major cities: Edge cases reveal the most about data quality
• Focusing on country-level only: City-level accuracy reveals true data quality differences
• Using stale ground truth: IP assignments change frequently

How IPinfo's Approach Is Different

IPinfo's methodology uses multiple measurement points to triangulate an IP's location using our Probe Network. Similar to GPS, this approach creates a confidence radius around the predicted location. By comparing RTT from numerous global vantage points, we can determine not just the country but often the city where an IP is located with high confidence.

When we find discrepancies between our measurements and what WHOIS records claim, we rely on the physics-based evidence. You can read more about how we measure our accuracy against ground truth data here.

Unlike providers who primarily rely on WHOIS records and geofeeds, IPinfo's approach centers on empirical evidence:

• Global probe network: Our 900+ probe servers spread across the world provide actual measurement data
• Full internet-wide scanning: We perform billions of measurements weekly to verify IP locations
• Ground truth validation: We validate our predictions against known-location datasets
• Transparency: We show our evidence and explain when we disagree with other providers

When providers disagree about an IP's location, we encourage you to check for yourself. IP geolocation accuracy is too important to leave to trust or consensus. By using a combination of reliable ground truth data and physics-based verification, you can objectively determine which provider delivers the most accurate results for your specific needs.

When providers disagree, don't assume the outlier is wrong, investigate using independent verification and appropriate metrics. As our research has demonstrated, being different often means being right when everyone else is wrong.

For assistance with conducting a thorough evaluation of IP data providers, contact our data team. We're happy to help you set up a structured comparison based on evidence that meets your unique requirements.