remove broken blueliv bots

collector is broken because of it's dependency
parser is obsolete with the removed collector

fixes #2593

Author: sebix

CHANGELOG.md

@@ -25,9 +25,11 @@ Please refer to the [NEWS](NEWS.md) for a list of changes which have an affect o
 
 ### Bots
 #### Collectors
+- Remove `intelmq.bots.collectors.blueliv` as it uses an unmaintained library, does not work any more and breaks other CI tests (fixes #2593, PR#2632 by Sebastian Wagner).
 
 #### Parsers
 - `intelmq.bots.parsers.cymru.parser_cap_program`: Add mapping for TOR and ipv6-icmp protocol (PR#2621 by Mikk Margus Möll).
+- Remove `intelmq.bots.collectors.blueliv` as it is obsolete with the removed collector (PR#2632 by Sebastian Wagner).
 
 #### Experts
 - `intelmq.bots.experts.asn_lookup.expert`:

docs/user/bots.md

@@ -1045,34 +1045,6 @@ input. If you intend to link two IntelMQ instance via TCP, have a look at the TC
 
 ---
 
-### Blueliv Crimeserver <div id="intelmq.bots.collectors.blueliv.collector_crimeserver" />
-
-Collects report messages from Blueliv API.
-
-For more information visit <https://github.com/Blueliv/api-python-sdk>
-
-**Module:** `intelmq.bots.collectors.blueliv.collector_crimeserver`
-
-**Requirements**
-
-Install the required library:
-
-```bash
-pip3 install -r intelmq/bots/collectors/blueliv/REQUIREMENTS.txt
-```
-
-**Parameters (also expects [feed parameters](#feed-parameters)):**
-
-**`api_key`**
-
-(required, string) location of information resource, see <https://map.blueliv.com/?redirect=get-started#signup>
-
-**`api_url`**
-
-(optional, string) The optional API endpoint. Defaults to `https://freeapi.blueliv.com`.
-
----
-
 ### Calidog Certstream <div id="intelmq.bots.collectors.calidog.collector_certstream" />
 
 A Bot to collect data from the Certificate Transparency Log (CTL). This bot works based on certstream library
@@ -1388,16 +1360,6 @@ No additional parameters.
 
 ---
 
-### Blueliv Crimeserver <div id="intelmq.bots.parsers.blueliv.parser_crimeserver" />
-
-Parses data from Blueliv Crimeserver feed.
-
-**Module:** `intelmq.bots.parsers.blueliv.parser_crimeserver`
-
-No additional parameters.
-
----
-
 ### Calidog Certstream <div id="intelmq.bots.parsers.calidog.parser_certstream" />
 
 Parses data from Certificate Transparency Log.

intelmq/bots/collectors/blueliv/REQUIREMENTS.txt

@@ -188,39 +188,6 @@ providers:
       revision: 2020-07-07
       documentation: https://urlhaus.abuse.ch/feeds/
       public: true
-  Blueliv:
-    CrimeServer:
-      description: Blueliv Crimeserver Collector is the bot responsible to get the
-        report through the API.
-      additional_information:
-        The service uses a different API for free users and paying subscribers. In 'CrimeServer'
-        feed the difference lies in the data points present in the feed. The non-free API
-        available from Blueliv contains, for this specific feed, following extra fields not
-        present in the free API;
-        "_id" - Internal unique ID
-        "subType" - Subtype of the Crime Server
-        "countryName" - Country name where the Crime Server is located, in English
-        "city" - City where the Crime Server is located
-        "domain" - Domain of the Crime Server
-        "host" - Host of the Crime Server
-        "createdAt" - Date when the Crime Server was added to Blueliv CrimeServer database
-        "asnCidr" - Range of IPs that belong to an ISP (registered via Autonomous System Number (ASN))
-        "asnId" - Identifier of an ISP registered via ASN
-        "asnDesc" Description of the ISP registered via ASN
-      bots:
-        collector:
-          module: intelmq.bots.collectors.blueliv.collector_crimeserver
-          parameters:
-            rate_limit: 3600
-            api_key: __APIKEY__
-            name: __FEED__
-            provider: __PROVIDER__
-        parser:
-          module: intelmq.bots.parsers.blueliv.parser_crimeserver
-          parameters:
-      revision: 2018-01-20
-      documentation: https://www.blueliv.com/
-      public: false
   Team Cymru:
     CAP:
       description: Team Cymru provides daily lists of compromised or abused devices

intelmq/bots/collectors/blueliv/__init__.py

@@ -41,7 +41,8 @@
            'v320_update_turris_greylist_url',
            'v322_url_replacement',
            'v322_removed_feeds_and_bots',
-           'v340_deprecations'
+           'v340_deprecations',
+           'v341_blueliv_removal',
            ]
 
 
@@ -974,6 +975,29 @@ def v340_deprecations(configuration, harmonization, dry_run, **kwargs):
     return message or changed, configuration, harmonization
 
 
+def v341_blueliv_removal(configuration, harmonization, dry_run, **kwargs):
+    """
+    Remove blueliv collector and parser
+    """
+    message = ''
+    discontinued_bots = []
+    discontinued_bots_modules = (
+        "intelmq.bots.collectors.blueliv.collector_crimeserver",
+        "intelmq.bots.parsers.blueliv.parser_crimeserver",
+    )
+
+    for bot_id, bot in configuration.items():
+        if bot_id == 'global':
+            continue
+        if bot["module"] in discontinued_bots_modules:
+            discontinued_bots.append(bot_id)
+
+    if discontinued_bots:
+        message = f"Found discontinued bots: {', '.join(discontinued_bots)}. Remove the affected bots from the configuration."
+
+    return message, configuration, harmonization
+
+
 UPGRADES = OrderedDict([
     ((1, 0, 0, 'dev7'), (v100_dev7_modify_syntax,)),
     ((1, 1, 0), (v110_shadowserver_feednames, v110_deprecations)),
@@ -1004,7 +1028,7 @@ def v340_deprecations(configuration, harmonization, dry_run, **kwargs):
     ((3, 3, 0), ()),
     ((3, 3, 1), ()),
     ((3, 4, 0), (v340_deprecations, )),
-    ((3, 4, 1), ()),
+    ((3, 4, 1), (v341_blueliv_removal, )),
 ])
 
 ALWAYS = (harmonization,)