when i run python3 dagda.py vuln --product python --product_version=3.8-slim error 500

[tbdavid94]

Short description

I run python3 dagda.py vuln --product python --product_version=3.8-slim

Reproduction steps

1. Start Dagda server with the parameters '...'
2. Type the Dagda CLI command vuln --product python --product_version=3.8-slim
3. Return {
   "err": 500,
   "msg": "Internal Server Error"
   }
4. Show logs dagda server
   dagda | <2023-12-13 09:54:51,440> <sysdig_falco_monitor> <pre_check:83> <I'm running inside a docker container, so I can't check if the kernel headers are installed in the host operating system. Please, review it!!>
   dagda | <2023-12-13 09:55:29,060> <dagda_server> run:104 <Falcosecurity/falco output file not found.>
   dagda | <2023-12-13 09:55:29,060> <dagda_server> run:105
   dagda | <2023-12-13 10:08:22,749> <log_exception:1454> <Exception on /v1/vuln/products/python/3.8-slim [GET]>
   dagda | Traceback (most recent call last):
   dagda | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2070, in wsgi_app
   dagda | response = self.full_dispatch_request()
   dagda | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1515, in full_dispatch_request
   dagda | rv = self.handle_user_exception(e)
   dagda | File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
   dagda | return cors_after_request(app.make_response(f(*args, **kwargs)))
   dagda | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1513, in full_dispatch_request
   dagda | rv = self.dispatch_request()
   dagda | File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1499, in dispatch_request
   dagda | return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
   dagda | File "/opt/app/api/service/vuln.py", line 57, in get_vulns_by_product_and_version
   dagda | vulns = InternalServer.get_mongodb_driver().get_vulnerabilities(product, version)
   dagda | File "/opt/app/driver/mongodb_driver.py", line 322, in get_vulnerabilities
   dagda | for bid in bid_cursor:
   dagda | File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1238, in next
   dagda | if len(self.__data) or self._refresh():
   dagda | File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1155, in _refresh
   dagda | self.__send_message(q)
   dagda | File "/usr/local/lib/python3.8/site-packages/pymongo/cursor.py", line 1044, in __send_message
   dagda | response = client._run_operation(
   dagda | File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1424, in _run_operation
   dagda | return self._retryable_read(
   dagda | File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1525, in _retryable_read
   dagda | return func(session, server, sock_info, secondary_ok)
   dagda | File "/usr/local/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1420, in _cmd
   dagda | return server.run_operation(
   dagda | File "/usr/local/lib/python3.8/site-packages/pymongo/server.py", line 130, in run_operation
   dagda | _check_command_response(first, sock_info.max_wire_version)
   dagda | File "/usr/local/lib/python3.8/site-packages/pymongo/helpers.py", line 167, in _check_command_response
   dagda | raise OperationFailure(errmsg, code, response, max_wire_version)
   dagda | pymongo.errors.OperationFailure: text index required for $text query (no such collection 'vuln_database.bid'), full error: {'ok': 0.0, 'errmsg': "text index required for $text query (no such collection 'vuln_database.bid')", 'code': 27, 'codeName': 'IndexNotFound'}

Expected results

I expected that by analyzing the docker image and showing the report

Actual results

Instead of '...', what I saw was that '...' happened instead.

On which platforms did you notice this:

Please complete the following information:

• OS: Ubuntu
• OS Version: Jammy
• Python version: 3.8

[tbdavid94]

@eliasgranderubio help me check issue, many thanks

[6AlexMan]

@eliasgranderubio help me check issue, many thanks

I also encountered the same problem as you, but I have already solved it.

0x01 Solution

Based on the error log you provided, the root cause of the problem is that the parameter passed when calling insert_many() is not a non-empty list, which resulted in a TypeError: documents must be a non-empty list being thrown.
The specific analysis is as follows:

1. The error occurred in the bulk_insert_exploit_db_ids function in the file dagda/driver/mongodb_driver.py, where self.db.exploit_db.insert_many(products) was called.
2. According to the error, when products was passed to insert_many, it was not a non-empty list. It might be an empty list [], None, or some other non-list type.
3. From the stack trace, the parameter passed to bulk_insert_exploit_db_ids from the upstream is exploit_db_list, which might be empty.
   Solution suggestions:

• Check if the exploit_db_list passed when calling bulk_insert_exploit_db_ids contains valid content, and print or log to confirm that it is not an empty list.
• Before calling insert_many inside bulk_insert_exploit_db_ids, first check if products is a non-empty list. If it is empty, skip the insertion or handle it accordingly. For example:

if products and isinstance(products, list):
    self.db.exploit_db.insert_many(products)
else:
    pass

4. In addition, you need to review all the places in dagda/driver/mongodb_driver.py where the insert_many method is called, and make the necessary adjustments following the aforementioned approach.

0x02 Future issues

In dagda/dagda/vulnDB/db_composer.py, the following code also contains a bug:

def compose_vuln_db(self):
    ...
    ...
    ...
    # -- RHSA (Red Hat Security Advisory) and RHBA (Red Hat Bug Advisory)
    # Adding or updating rhsa and rhba collections
    if InternalServer.is_debug_logging_enabled():
        DagdaLogger.get_logger().debug('Updating RHSA & RHBA collections ...')

    self.mongoDbDriver.delete_rhba_collection()
    self.mongoDbDriver.delete_rhba_info_collection()
    self.mongoDbDriver.delete_rhsa_collection()
    self.mongoDbDriver.delete_rhsa_info_collection()
    bz2_file = get_http_resource_content('https://www.redhat.com/security/data/oval/rhsa.tar.bz2')
    rhsa_list, rhba_list, rhsa_info_list, rhba_info_list = get_rhsa_and_rhba_lists_from_file(bz2_file)
    self.mongoDbDriver.bulk_insert_rhsa(rhsa_list)
    self.mongoDbDriver.bulk_insert_rhba(rhba_list)
    self.mongoDbDriver.bulk_insert_rhsa_info(rhsa_info_list)
    self.mongoDbDriver.bulk_insert_rhba_info(rhba_info_list)
    ...
    ...
    ...

Because the resource pointed to by https://www.redhat.com/security/data/oval/rhsa.tar.bz2 no longer exists, it is impossible to download RHSA (Red Hat Security Advisory) and RHBA (Red Hat Bug Advisory). My approach was to comment out this part of the code first.