NB-Public/green-spider
14
0
Fork 1
mirror of https://github.com/netzbegruenung/green-spider.git synced 2024-05-02 00:54:52 +02:00
Code Issues 2 Projects Releases Wiki Activity

Mofivy url_reachabliity to remove URLs if status 404

Browse source
This commit is contained in:
Marian Steinbach 2018-10-02 22:45:18 +02:00
parent cf9136b481
commit 2cebb3ff28
1 changed files with 4 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

4
checks/url_reachability.py
View file

@ -60,6 +60,10 @@ class Checker(AbstractChecker):
# remove source URL, add target URL to config.urls
self.config.remove_url(url)
self.config.add_url(r.url)
# remove 404 etc
if r.status_code > 400:
self.config.remove_url(url)
except Exception as exc:
logging.info("Exception for URL %s: %s %s", url, str(type(exc)), exc)