"No cached copy or other error"

[homebysix]

For increased control and security, I'm running SavingThrow pointing at an internal server for definitions, rather than at GitHub. Like so:

NEFARIOUS_FILE_SOURCES = [ 'https://jss.pretendco.com:8443/SavingThrow/MacKeeper.adf',
                           'https://jss.pretendco.com:8443/SavingThrow/ZipCloud.adf',
                           'https://jss.pretendco.com:8443/SavingThrow/AskToolbar.adf',
                           'https://jss.pretendco.com:8443/SavingThrow/AwesomeScreenshot.adf' ]
HT203987_URL = "https://jss.pretendco.com:8443/SavingThrow/Apple-HT203987.adf"
NEFARIOUS_FILE_SOURCES.append(HT203987_URL)

However this doesn't seem to work. Running sudo python custom_SavingThrow.py -v reveals that the .adf files aren't found.

Attempting to update Adware list: https://jss.pretendco.com:8443/SavingThrow/MacKeeper.adf
Update failed: . Looking for cached copy
Error: No cached copy of https://jss.pretendco.com:8443/SavingThrow/MacKeeper.adf or other error 
Attempting to update Adware list: https://jss.pretendco.com:8443/SavingThrow/ZipCloud.adf
Update failed: . Looking for cached copy
Error: No cached copy of https://jss.pretendco.com:8443/SavingThrow/ZipCloud.adf or other error 
Attempting to update Adware list: https://jss.pretendco.com:8443/SavingThrow/AskToolbar.adf
Update failed: . Looking for cached copy
Error: No cached copy of https://jss.pretendco.com:8443/SavingThrow/AskToolbar.adf or other error 
Attempting to update Adware list: https://jss.pretendco.com:8443/SavingThrow/AwesomeScreenshot.adf
Update failed: . Looking for cached copy
Error: No cached copy of https://jss.pretendco.com:8443/SavingThrow/AwesomeScreenshot.adf or other error 
Attempting to update Adware list: https://jss.pretendco.com:8443/SavingThrow/Apple-HT203987.adf
Update failed: . Looking for cached copy
Searching for files and processes defined in: Downlite and VSearch Variants
Searching for files and processes defined in: Conduit, Trovi, MyBrand, and Search Protect Variants
Searching for files and processes defined in: Genieo and InstallMac
<result>False</result>

However the files do exist at those URLs, as evidenced by curl output:

curl https://jss.pretendco.com:8443/SavingThrow/MacKeeper.adf

<AdwareDefinition>
    <Version>1.0</Version>
    <DefinitionAuthor>Shea Craig</DefinitionAuthor>
    <DefinitionSource>Durham Academy</DefinitionSource>
    <Adware>
        <AdwareName>MacKeeper</AdwareName>
        <File>/Users/*/Library/LaunchAgents/com.*.MacKeeper.Helper.plist</File>
        <File>/Users/*/Documents/MacKeeper Backups</File>
        <File>/Applications/MacKeeper.app</File>
        <File>/Users/*/Library/Application Support/MacKeeper Helper</File>
        <File>/Users/*/Library/Caches/com.mackeeper.MacKeeper</File>
        <Process>MacKeeper</Process>
    </Adware>
</AdwareDefinition>

The server has a self-signed SSL cert (for now). Do I need to modify the add_adware_from_url function to ignore certificate warnings?

[sheagcraig]

I would fire up python and try the following real quick:

import requests
response = requests.get("https://jss.pretendco.com:8443/SavingThrow/MacKeeper.adf")

The keyword argument "verify=True" may be added to the get as well.

After the get, the response object will have the return code, message and headers available for perusal. If you do a dir(reponse) on it, you'll get a list of all of its methods and properties.

Let me know if that uncovers anything!

[sheagcraig]

You'll need requests for this of course...
easy_install -U requests

[homebysix]

Regardless of whether I use

>>> response = requests.get("https://jss.pretendco.com:8443/SavingThrow/MacKeeper.adf", verify=True)

or

>>> response = requests.get("https://jss.pretendco.com:8443/SavingThrow/MacKeeper.adf", verify=False)

the result is always

/Library/Python/2.7/site-packages/requests-2.7.0-py2.7.egg/requests/packages/urllib3/util/ssl_.py:90: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Python/2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 69, in get
    return request('get', url, params=params, **kwargs)
  File "/Library/Python/2.7/site-packages/requests-2.7.0-py2.7.egg/requests/api.py", line 50, in request
    response = session.request(method=method, url=url, **kwargs)
  File "/Library/Python/2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 465, in request
    resp = self.send(prep, **send_kwargs)
  File "/Library/Python/2.7/site-packages/requests-2.7.0-py2.7.egg/requests/sessions.py", line 573, in send
    r = adapter.send(request, **kwargs)
  File "/Library/Python/2.7/site-packages/requests-2.7.0-py2.7.egg/requests/adapters.py", line 431, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [Errno 1] _ssl.c:507: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

We're on version 9.72 of the JSS.

[sheagcraig]

What version of python-jss is it using?

Newer versions ( > 0.5.7) should force a TLS connection. JAMF forced the JSS to stop using SSLv23 a few updates ago, and that's the error that you're getting.

You could do a quick:
easy_install -U python-jss
to get the update.

[homebysix]

The computer I'm running this from doesn't have python-jss installed. The JSS is serving as the place where the .adf file lives, but that's where the JSS's involvement ends. No need to use python-jss to access anything.

To make sure it's not a JSS/Tomcat issue, I uploaded the .adf file to another website I control, and I got the same error:

python custom_SavingThrow.py -v

Attempting to update Adware list: http://awesomewebsite.com/MacKeeper.adf
Update failed: . Looking for cached copy
Error: No cached copy of http://awesomewebsite.com/MacKeeper.adf or other error 
Searching for files and processes defined in: Downlite and VSearch Variants
Searching for files and processes defined in: Conduit, Trovi, MyBrand, and Search Protect Variants
Searching for files and processes defined in: Genieo and InstallMac
<result>False</result>

And yet it still loads fine in a browser:

This time I didn't even use https, so I'm now convinced it's not an SSL issue.

[sheagcraig]

Oh yeah, what am I thinking!

Do you get the same exception when you try to do the get, even on the non-HTTPS server?

The reason I jumped to the TLS conclusion is because that's the same error you get from a JSS... I'm sure any updated Apache, nginx, etc, has also disabled sslv23. I'll take a look about adding a TLS adapter to SavingThrow soon to test.

Of course, that still doesn't explain why you'd be getting it on an unencrypted server.

[homebysix]

Interesting. Both of these work fine (or at least don't produce any error output):

import requests
response = requests.get("http://awesomewebsite.com/MacKeeper.adf", verify=True)
response = requests.get("http://awesomewebsite.com/MacKeeper.adf", verify=False)

So if that works in isolation, why doesn't it work in the context of SavingThrow.py?

[sheagcraig]

Where are we on this @homebysix? Still broken?

[sheagcraig]

Also,
SavingThrow (currently) just uses urllib2, not requests.

Obviously I have a lot more experience trying to get requests to work. Getting TLS working on urllib2 is supposedly doable, I think.

[homebysix]

TBH, not using SavingThrow regularly any more. The org I was helping is now investigating other options for cross-platform endpoint security, including osquery.

[sheagcraig]

So where we are with this is that we should probably shell out to curl to get SSL working.

[sheagcraig]

I haven't heard any further requests for this. Filing this away as a future enhancement.