Commit Graph

15 Commits

Author SHA1 Message Date
Marian Steinbach 618e29d763
Job-Verwaltung mit RQ, und vieles mehr (#149)
* CLI: remove 'jobs' command, add 'manager'

* Add job definition

* Move jobs to manage folder

* Rename jobs to manager

* Add rq and redis dependencies

* Add docker-compose YAML

* Downgrade to alpine 3.8

* Adjust paths in Dockerfile, remove entrypoint

* Rename 'make spiderjobs' to 'make jobs'

* Fix docker exectution

* Adapt 'make jobs'

* Fix metadata scheme

* Add docker dependency

* Rendomize queue (a bit)

* Use latest image, remove debug output

* Make docker-compose file downwards-compatible

* Use latest instead of dev image tag

* Update docker-compose.yaml

* Adapt job start script

* Fix redis connection in manager

* Add support for increasing timeout via environment variable

* Adapt load_in_browser to cookies table schema change

* Fix execution

* Mitigate yaml warning

* Bump some dependency versions

* Report resource usage stats for each job

* checks/load_in_browser: Return DOM size, prevent multiple page loads

* Update .dockerignore

* Code update

* Script update

* Update README.md

* WIP

* WIP commit

* Update Dockerfile to alpine:edge and chromium v90

* Update TestCertificateChecker

* Set defaults for __init__ function

* Detect sunflower theme

* Update unit test for new datetime (zero-basing)

* Set logging prefs from Chromium in a new way

* Move datastore client instantiation

As it is not needed for all commands

* Change green-directory repository URL

* Add git settings for cloning green-directory

* Pin alpine version 3.14, fix py3-cryptography

* Use plain docker build progress output

* Add volumes to 'make test' docker run command

* Fix bug

* Update example command in README

* Update dependencies

* Add creation of Kubernetes jobs
2021-11-11 20:15:43 +01:00
Marian Steinbach 68f2288617
Prüfe DNS auf IPv6 AAAA Record (#124)
* Add check for IPv6 AAAA record

* Adapt rating/resolvable
2019-07-15 22:59:33 +02:00
Marian Steinbach 04a1e98b79
Prüfe Existenz von /favicon.ico und werte dies ebenso wie ein Icon, das im HTML Head verlinkt ist (#115)
* Fix full JSON export

* Update ignore list

* Update README

* Check for /favicon.ico and rate it as icon available

* Remove broken cookies test
2019-05-05 22:26:41 +02:00
Marian Steinbach cff4d55f17 Fix problem where pageload was not counted 2019-05-03 22:54:25 +02:00
Marian Steinbach 56f9f1ba86 Check third party cookies 2019-04-29 10:09:25 +02:00
Marian Steinbach 16a05b751b Several fixes for edge cases 2018-12-17 23:54:09 +01:00
Marian Steinbach d0e3a4210f
Fix link raters (social media links, contact link) (#95)
* Fix rating for contact_link and social_media_link

* Skip checks when dependencies not met
2018-11-28 23:46:40 +01:00
Marian Steinbach 39cba1595a Fix contact link rating 2018-11-23 22:16:26 +01:00
Marian Steinbach 3ba6940e94
Add criteria: social media links, contact link (#90)
* Add hyperlink checker

* Add rating for contact and social media links

* Update a comment

* Remove hyperlinks details from final payload
2018-11-20 22:47:34 +01:00
Marian Steinbach 4524cb5714
Consider site reachable only with status code < 400 (#89) 2018-11-20 20:14:52 +01:00
Marian Steinbach 924981659b
Allow Titillium together with Arvo (#78) 2018-11-05 23:18:11 +01:00
Marian Steinbach 0538e437ea Fix NoneType error in rater responsive_layout 2018-10-07 21:14:29 +02:00
Marian Steinbach 18be6e7adf Fix rater no_network_errors 2018-10-07 20:55:57 +02:00
Marian Steinbach 4251df6b06
Fixes for two problems found during spidering (#75) 2018-10-05 10:25:05 +02:00
Marian Steinbach ae6a2e83e9
Refactor and modularize spider (#70)
See PR description for details
2018-10-03 11:05:42 +02:00