Skip to content

Handle invalid domains more gracefully#53

Open
S4lt5 wants to merge 4 commits into
cisagov:developfrom
S4lt5:invalid-domain-handling
Open

Handle invalid domains more gracefully#53
S4lt5 wants to merge 4 commits into
cisagov:developfrom
S4lt5:invalid-domain-handling

Conversation

@S4lt5

@S4lt5 S4lt5 commented Nov 21, 2022

Copy link
Copy Markdown

🗣 Description

As per #42, invalid domains should be handled gracefully.

These are checked for in main() before running the executor, and put in a separate "invalid domains" bucket.

Also, as part of code cleanup, the domain class was moved to a separate python file.

There is a LOT to do and a lot of redundant call chains to be removed, but each journey starts with a single step!

💭 Motivation and context

🧪 Testing

Added unit tests for valid/invalid cdnCheck() and main() calls with valid/invalid domains

Sample output

findcdn list "*.domain.com"
0 Domains Validated.
1 domain(s) rejected, listed below:
	*.domain.com
0it [00:00, ?it/s]
{
    "date": "11/21/2022, 11:44:09",
    "cdn_count": "0",
    "domains": {},
    "invalid_domains": [
        "*.domain.com"
    ]
}
Domain processing completed.
0 domains had CDN's out of 1.
1 Domains Validated.
2 domain(s) rejected, listed below:
	*.foo.bar
	wildcard domain
[Pending: 0 jobs]==[Threads: 1]: 100%|████████████████████████████████████████████████████| 1/1 [00:01<00:00,  1.26s/it]
{
    "date": "11/21/2022, 11:44:53",
    "cdn_count": "1",
    "domains": {
        "cisa.gov": {
            "IP": "'104.117.51.217'",
            "cdns": "'.edgekey.net', '.akamaitechnologies.fr'",
            "cdns_by_names": "'Akamai', 'Akamai'"
        }
    },
    "invalid_domains": [
        "*.foo.bar",
        "wildcard domain"
    ]
}
Domain processing completed.
1 domains had CDN's out of 3.

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • All future TODOs are captured in issues, which are referenced
    in code comments.
  • All relevant type-of-change labels have been added.
  • I have read the CONTRIBUTING document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated
    to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.

✅ Pre-merge checklist

  • Revert dependencies to default branches.
  • Finalize version.

✅ Post-merge checklist

  • Add a tag or create a release.

@lgtm-com

lgtm-com Bot commented Nov 21, 2022

Copy link
Copy Markdown

This pull request introduces 1 alert when merging b3b320d into 7bbf50e - view on LGTM.com

new alerts:

  • 1 for Unused import

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. Please enable GitHub code scanning, which uses the same CodeQL engine ⚙️ that powers LGTM.com. For more information, please check out our post on the GitHub blog.

@Pascal-0x90 Pascal-0x90 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the changes. I think this is a great addition. Going to say this should be merged on the note that I am also going to be working more of the PRs too so this may also end up being rearranged. I want to get rid of the complexity you keep pointing out because I am also noticing that as well.

@Pascal-0x90 Pascal-0x90 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I forgot to mention, please make sure you run pre-commit hooks before merging so the tests pass. Looks like pre-commit hooks had some issues.

@S4lt5

S4lt5 commented Nov 23, 2022

Copy link
Copy Markdown
Author

I think those "test_list_broken/est_file_broken" tests that fail, were failing before. I actually don't even understand the tests to be honest.

I'll double check with the other branches.

@Pascal-0x90

Copy link
Copy Markdown
Collaborator

Good point. The tests should probably get a good review as well while I am on this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants