`web.search_urls()` is too lenient

[dgw]

The regex pattern here:

sopel/sopel/tools/web.py

Line 253 in bc5843b

re_url = r'((?:%s)(?::\/\/\S+))' % schemes_patterns

doesn't ignore all IRC formatting characters. Bold and monospace formatting, for example, will be included in the match if it's right up against the end of the URL.

Additionally, text before the protocol (with no whitespace or other word boundary) doesn't appear to be ignored either. For example, "bold of youhttps://github.com/sopel-irc/sopel" (note lack of space between you and https) resulted in url trying to fetch a page, when it should have been ignored because youhttps is not a valid protocol. Hypothetically there could be custom protocols that end with one of the known strings, and Sopel should make sure the whole protocol string matches what it's looking for.

This leniency affects core message dispatch because it's used to build PreTrigger.urls:

sopel/sopel/trigger.py

Lines 269 to 271 in bc5843b

	# Search URLs after CTCP parsing
	self.urls = tuple(
	web.search_urls(self.args[-1], schemes=url_schemes))

Of note: web.search_urls() has a clean parameter that causes it to run web.trim_urls() on the found matches, which the PreTrigger code doesn't make use of.

[dgw]

Bold and monospace formatting, for example, will be included in the match if it's right up against the end of the URL.

This is still the case after #2697, so I will reopen the issue. Test function for reference:

def test_search_urls_strict_formatting():
    urls = list(search_urls(
        'You are making a \x02bold mistake http://example.com\x02 '
        '(and an \x1Dhttps://example.com italic one\x1D too). '
        'What if URLs were colored?! \x0303https://red.link\x03'
    ))
    assert len(urls) == 3
    assert urls == [
        'http://example.com',
        'https://example.com',
        'https://red.link',
    ]

# results as of c4c59147450a6060a3d8bd114ec5c679d4a024f2:
#
# =================================== FAILURES ===================================
# ______________________ test_search_urls_strict_formatting ______________________
# test/tools/test_tools_web.py:86: in test_search_urls_strict_formatting
#     assert len(urls) == 3
# E   AssertionError: assert 2 == 3
# E    +  where 2 = len(['http://example.com\x02', 'https://example.com'])
#
# The first URL match includes an unwanted formatting byte,
# and the third link is missing entirely.

To allow for "fun" cases like, say, colored URLs (\x0303https://red.link\x03), might be necessary to just formatting.plain() the whole argument to search_urls() before searching for links in it.