Marian Steinbach
b3bb8f34c3
Problembehebung mit Exception-Daten im Ergbebnis, die nicht geschrieben werden können, und Spidern einzelner Sites ( #132 )
...
* WIP commit for single job execution
* Convert exception to string
* Pass more arguments
* Move python modules lsit into requirements.txt
* Document single site spidering
* Remove debugging
2019-11-22 23:13:57 +01:00
Marian Steinbach
7788166b0c
Fix logging output in dns_resolution
2019-11-22 08:38:26 +01:00
Marian Steinbach
68f2288617
Prüfe DNS auf IPv6 AAAA Record ( #124 )
...
* Add check for IPv6 AAAA record
* Adapt rating/resolvable
2019-07-15 22:59:33 +02:00
Marian Steinbach
04a1e98b79
Prüfe Existenz von /favicon.ico und werte dies ebenso wie ein Icon, das im HTML Head verlinkt ist ( #115 )
...
* Fix full JSON export
* Update ignore list
* Update README
* Check for /favicon.ico and rate it as icon available
* Remove broken cookies test
2019-05-05 22:26:41 +02:00
Marian Steinbach
7621b7ef75
Remove debugging output
2019-05-03 22:54:05 +02:00
Marian Steinbach
56f9f1ba86
Check third party cookies
2019-04-29 10:09:25 +02:00
Marian Steinbach
5e8347916c
Fehlerbehebung im url_reachability check ( #108 )
...
* Fix detection of redirects to bad domains
* Fix bad domain check
* Add --url flag to spider for faster debugging
* Pass args to make spider
* Add spidering of a single URL for debugging purposes
* Fix tests
* Fix test in CI
* Remove pip upgrade
2019-04-19 00:35:28 +02:00
Marian Steinbach
16a05b751b
Several fixes for edge cases
2018-12-17 23:54:09 +01:00
Marian Steinbach
3b8328d804
Fixing several bugs in spider code
2018-12-17 17:31:09 +01:00
Marian Steinbach
3b9ead330d
Load feeds and gather info ( #103 )
2018-12-07 16:32:42 +01:00
Marian Steinbach
3063a4488d
Detect frameset ( #102 )
...
* Add frameset checker
* Remove unused variable (unrelated)
2018-12-07 16:31:56 +01:00
Marian Steinbach
deff95306b
Extend CMS detection for Urwahl3000 theme ( #96 )
...
* Extend check for Urwahl3000 theme
* Remove unused import
2018-12-05 21:27:45 +01:00
Marian Steinbach
d0e3a4210f
Fix link raters (social media links, contact link) ( #95 )
...
* Fix rating for contact_link and social_media_link
* Skip checks when dependencies not met
2018-11-28 23:46:40 +01:00
Marian Steinbach
678f319e73
Detect two more specific generators
2018-11-28 22:02:30 +01:00
Marian Steinbach
3ba6940e94
Add criteria: social media links, contact link ( #90 )
...
* Add hyperlink checker
* Add rating for contact and social media links
* Update a comment
* Remove hyperlinks details from final payload
2018-11-20 22:47:34 +01:00
Marian Steinbach
325caee2bb
Detect generator jimdo ( #81 )
2018-11-05 23:00:01 +01:00
Marian Steinbach
df1f0bb452
Detect Drupal ( #80 )
2018-11-05 22:32:06 +01:00
Marian Steinbach
fd4a29da8e
Collect cookies in load_in_browser check ( #74 )
2018-10-04 21:21:30 +02:00
Marian Steinbach
c065da4957
More unittests for checks ( #73 )
...
* Add test for dns_resolution
* Add test for domain_variations
* Add test for duplicate_content
2018-10-03 22:43:22 +02:00
Marian Steinbach
57f8dea4e0
Improve certificate check to support SNI ( #71 )
...
* Fix the certificate check to support SNI
* Better tests for the certificate check
* Activate verbose output when running make test
* Add commenting on the spider test
2018-10-03 21:01:52 +02:00
Marian Steinbach
ae6a2e83e9
Refactor and modularize spider ( #70 )
...
See PR description for details
2018-10-03 11:05:42 +02:00
Marian Steinbach
