[feature] Allow estimating geographic location based on WHOIS data #1034

- Estimated geographic location logic, closes #1034.
- Added notification and admin warning, closes #1035.
- Set location estimated flag to ``False`` on manual update,
  closes #1036.
- Added admin and API filters for estimated locations,
  closes #1028.

---------

Signed-off-by: DragnEmperor <[email protected]>

Author: DragnEmperor

docs/index.rst

@@ -49,6 +49,7 @@ the OpenWISP architecture.
     user/openvpn.rst
     user/subnet-division-rules.rst
     user/whois.rst
+    user/estimated-location.rst
     user/rest-api.rst
     user/settings.rst
 

docs/user/estimated-location.rst

@@ -0,0 +1,80 @@
+Estimated Location
+==================
+
+.. important::
+
+    The **Estimated Location** feature is **disabled by default**.
+
+    Before enabling it, the :doc:`WHOIS Lookup feature <whois>` must be
+    enabled. Then set
+    :ref:`OPENWISP_CONTROLLER_ESTIMATED_LOCATION_ENABLED` to ``True``
+
+.. contents:: **Table of contents**:
+    :depth: 1
+    :local:
+
+Overview
+--------
+
+The Estimated Location feature automatically creates or updates a device’s
+location based on latitude and longitude information retrieved from the
+WHOIS Lookup feature.
+
+Trigger Conditions
+------------------
+
+Estimated Location is triggered when:
+
+- A **fresh WHOIS lookup** is performed for a device.
+- Or when a WHOIS record already exists for the device’s IP **and**:
+
+  - The device’s last IP address is **public**.
+  - WHOIS lookup and Estimated Location is **enabled** for the device’s
+    organization.
+
+Behavior
+--------
+
+The system will **attach the already existing matching location** of
+another device with same ip to the current device if:
+
+- Only one device is found with that IP and it has a location.
+- The current device **has no location** or that location is
+  **estimated**.
+
+If there are multiple devices with location for the same IP, the system
+will **not attach any location** to the current device and a notification
+will be sent suggesting the user to manually assign/create a location for
+the device.
+
+If there is **no matching location**, a new estimated location is created
+or the existing one is updated using coordinates from the WHOIS record,
+but only if the existing location is estimated.
+
+If two devices share the same IP address and are assigned to the same
+location, and the last IP of one of the devices is updated, the system
+will create a new estimated location for that device.
+
+Visibility of Estimated Status
+------------------------------
+
+The estimated status of a location is visible on the location page if the
+feature is enabled for the organization. The location admin page also
+includes indicators for the estimated status.
+
+- The name of the location will have suffix **(Estimated Location :
+  <ip_address>)**.
+- A warning on top of the page.
+- **Is Estimated** field.
+
+Changes to the ``coordinates`` and ``geometry`` of the estimated location
+will set the ``is_estimated`` field to ``False`` and remove the
+"(Estimated Location)" suffix with IP from the location name.
+
+In REST API, the field will be visible in the :ref:`Device Location
+<device_location_estimated>`, :ref:`Location list
+<location_list_estimated>`, :ref:`Location Detail
+<location_detail_estimated>` and :ref:`Location list (GeoJson)
+<location_geojson_estimated>` if the feature is **enabled**. The field can
+also be used for filtering in the location list (including geojson)
+endpoints and in the :ref:`Device List <device_list_estimated_filters>`.

docs/user/rest-api.rst

@@ -76,6 +76,14 @@ If :doc:`WHOIS Lookup feature <whois>` is enabled, each device in the list
 response will also include a ``whois_info`` field with related brief WHOIS
 information.
 
+.. _device_list_estimated_filters:
+
+**Estimated Location Filters**
+
+if :doc:`Estimated Location feature <estimated-location>` is enabled,
+devices can be filtered based on the estimated nature of their location
+using the ``geo_is_estimated``.
+
 **Available filters**
 
 You can filter a list of devices based on their configuration status using
@@ -544,13 +552,26 @@ of certificate's organization as show in the example below:
 
     GET /api/v1/controller/cert/{common_name}/group/?org={org1_slug},{org2_slug}
 
+.. |est_loc| replace:: Estimated Location feature
+
+.. _est_loc: estimated-location.html
+
+.. |estimated_details| replace:: If |est_loc|_ is enabled, the location
+    response will also include ``is_estimated`` status field.
+
 Get Device Location
 ~~~~~~~~~~~~~~~~~~~
 
 .. code-block:: text
 
     GET /api/v1/controller/device/{id}/location/
 
+.. _device_location_estimated:
+
+**Estimated Status**
+
+|estimated_details|
+
 .. _create_device_location:
 
 Create Device Location
@@ -787,6 +808,14 @@ List Locations
 
     GET /api/v1/controller/location/
 
+.. _location_list_estimated:
+
+**Estimated Status**
+
+|estimated_details|
+
+Locations can also be filtered using the ``is_estimated``.
+
 **Available filters**
 
 You can filter using ``organization_id`` or ``organization_slug`` to get
@@ -868,6 +897,12 @@ Get Location Details
 
     GET /api/v1/controller/location/{pk}/
 
+.. _location_detail_estimated:
+
+**Estimated Status**
+
+|estimated_details|
+
 Change Location Details
 ~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -910,6 +945,14 @@ List Locations with Devices Deployed (in GeoJSON Format)
 
     GET /api/v1/controller/location/geojson/
 
+.. _location_geojson_estimated:
+
+**Estimated Status**
+
+|estimated_details|
+
+Locations can also be filtered using the ``is_estimated``.
+
 **Available filters**
 
 You can filter using ``organization_id`` or ``organization_slug`` to get

docs/user/settings.rst

@@ -795,3 +795,20 @@ Maxmind Account ID required for the :doc:`WHOIS Lookup feature <whois>`.
 ============ =======
 
 Maxmind License Key required for the :doc:`WHOIS Lookup feature <whois>`.
+
+.. _openwisp_controller_whois_estimated_location_enabled:
+
+``OPENWISP_CONTROLLER_WHOIS_ESTIMATED_LOCATION_ENABLED``
+--------------------------------------------------------
+
+============ =========
+**type**:    ``bool``
+**default**: ``False``
+============ =========
+
+Allows enabling the optional :doc:`Estimated Location feature
+<estimated-location>`.
+
+.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-location-setting.png
+    :target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-location-setting.png
+    :alt: Estimated Location setting

docs/user/whois.rst

@@ -28,6 +28,7 @@ associated with the device's public IP address and includes:
 - CIDR block assigned to the ASN
 - Physical address registered to the ASN
 - Timezone of the ASN's registered location
+- Coordinates (Latitude and Longitude)
 
 Trigger Conditions
 ------------------
@@ -40,25 +41,23 @@ A WHOIS lookup is triggered automatically when:
 However, the lookup will only run if **all** the following conditions are
 met:
 
+- The device is either **newly created** or has a **changed last IP**.
 - The device's last IP address is **public**.
 - There is **no existing WHOIS record** for that IP.
 - WHOIS lookup is **enabled** for the device's organization.
 
-Behavior with Shared IP Addresses
----------------------------------
+Managing WHOIS Records
+----------------------
 
-If multiple devices share the same public IP address and one of them
-switches to a different IP, the following occurs:
-
-- A lookup is triggered for the **new IP**.
-- The WHOIS record for the **old IP** is deleted.
-- The next time a device still using the old IP fetches its checksum, a
-  new lookup is triggered, ensuring up-to-date data.
+If a device updates its last IP address, lookup is triggered for the **new
+IP** and the **WHOIS record for the old IP** is deleted if no active
+devices are associated with that IP address.
 
 .. note::
 
     When a device with an associated WHOIS record is deleted, its WHOIS
-    record is automatically removed.
+    record is automatically removed only if no active devices are
+    associated with that IP address.
 
 .. _controller_setup_whois_lookup:
 
@@ -79,6 +78,26 @@ Setup Instructions
    - Set :ref:`OPENWISP_CONTROLLER_WHOIS_GEOIP_ACCOUNT` to **Account ID**.
    - Set :ref:`OPENWISP_CONTROLLER_WHOIS_GEOIP_KEY` to **License Key**.
 
+6. Restart the application/containers if using ansible-openwisp2 or
+   docker.
+7. Run the ``clear_last_ip`` management command to clear the last IP
+   address of **all active devices across organizations**.
+
+   - If using ansible-openwisp2 (default directory is /opt/openwisp2,
+     unless changed in Ansible playbook configuration):
+
+     .. code-block:: bash
+
+         source /opt/openwisp2/env/bin/activate
+         python /opt/openwisp2/src/manage.py clear_last_ip
+
+   - If using docker:
+
+     .. code-block:: bash
+
+         docker exec -it <openwisp_container_name> sh
+         python manage.py clear_last_ip
+
 Viewing WHOIS Lookup Data
 -------------------------
 

openwisp_controller/config/admin.py

@@ -1384,7 +1384,7 @@ def get_fields(self, request, obj=None):
         if app_settings.REGISTRATION_ENABLED:
             fields += ["registration_enabled", "shared_secret"]
         if app_settings.WHOIS_CONFIGURED:
-            fields += ["whois_enabled"]
+            fields += ["whois_enabled", "estimated_location_enabled"]
         fields += ["context"]
         return fields
 

openwisp_controller/config/base/device.py

@@ -287,7 +287,7 @@ def save(self, *args, **kwargs):
         state_adding = self._state.adding
         super().save(*args, **kwargs)
         if app_settings.WHOIS_CONFIGURED:
-            self._check_last_ip()
+            self._check_last_ip(creating=state_adding)
         if state_adding and self.group and self.group.templates.exists():
             self.create_default_config()
         # The value of "self._state.adding" will always be "False"
@@ -510,9 +510,13 @@ def whois_service(self):
         """
         return WHOISService(self)
 
-    def _check_last_ip(self):
-        """Trigger WHOIS lookup if last_ip is not deferred."""
+    def _check_last_ip(self, creating=False):
+        """
+        Process details and location related to last_ip if last_ip has
+        changed or is being set for the first time.
+        """
         if self._initial_last_ip == models.DEFERRED:
             return
-        self.whois_service.trigger_whois_lookup()
+        if creating or self.last_ip != self._initial_last_ip:
+            self.whois_service.process_ip_data_and_location()
         self._initial_last_ip = self.last_ip

openwisp_controller/config/base/multitenancy.py

@@ -39,6 +39,11 @@ class AbstractOrganizationConfigSettings(UUIDModel):
         fallback=app_settings.WHOIS_ENABLED,
         verbose_name=_("WHOIS Enabled"),
     )
+    estimated_location_enabled = FallbackBooleanChoiceField(
+        help_text=_("Whether the estimated location feature is enabled"),
+        fallback=app_settings.ESTIMATED_LOCATION_ENABLED,
+        verbose_name=_("Estimated Location Enabled"),
+    )
     context = JSONField(
         blank=True,
         default=dict,
@@ -71,6 +76,15 @@ def clean(self):
                     )
                 }
             )
+        if not self.whois_enabled and self.estimated_location_enabled:
+            raise ValidationError(
+                {
+                    "estimated_location_enabled": _(
+                        "Estimated Location feature requires "
+                        "WHOIS Lookup feature to be enabled."
+                    )
+                }
+            )
         return super().clean()
 
     def save(