Commit Graph

11 Commits

Author SHA1 Message Date
Marian Steinbach 48fcecf256 Ignore k8s-jobs 2021-11-11 21:33:00 +01:00
Marian Steinbach 618e29d763
Job-Verwaltung mit RQ, und vieles mehr (#149)
* CLI: remove 'jobs' command, add 'manager'

* Add job definition

* Move jobs to manage folder

* Rename jobs to manager

* Add rq and redis dependencies

* Add docker-compose YAML

* Downgrade to alpine 3.8

* Adjust paths in Dockerfile, remove entrypoint

* Rename 'make spiderjobs' to 'make jobs'

* Fix docker exectution

* Adapt 'make jobs'

* Fix metadata scheme

* Add docker dependency

* Rendomize queue (a bit)

* Use latest image, remove debug output

* Make docker-compose file downwards-compatible

* Use latest instead of dev image tag

* Update docker-compose.yaml

* Adapt job start script

* Fix redis connection in manager

* Add support for increasing timeout via environment variable

* Adapt load_in_browser to cookies table schema change

* Fix execution

* Mitigate yaml warning

* Bump some dependency versions

* Report resource usage stats for each job

* checks/load_in_browser: Return DOM size, prevent multiple page loads

* Update .dockerignore

* Code update

* Script update

* Update README.md

* WIP

* WIP commit

* Update Dockerfile to alpine:edge and chromium v90

* Update TestCertificateChecker

* Set defaults for __init__ function

* Detect sunflower theme

* Update unit test for new datetime (zero-basing)

* Set logging prefs from Chromium in a new way

* Move datastore client instantiation

As it is not needed for all commands

* Change green-directory repository URL

* Add git settings for cloning green-directory

* Pin alpine version 3.14, fix py3-cryptography

* Use plain docker build progress output

* Add volumes to 'make test' docker run command

* Fix bug

* Update example command in README

* Update dependencies

* Add creation of Kubernetes jobs
2021-11-11 20:15:43 +01:00
Marian Steinbach 620610b48e Update ignore list 2019-05-04 23:00:56 +02:00
Marian Steinbach ae6a2e83e9
Refactor and modularize spider (#70)
See PR description for details
2018-10-03 11:05:42 +02:00
Marian Steinbach e64383c899 Add dev-shm to ignored list 2018-09-12 09:21:51 +02:00
Marian Steinbach 2c9e8d59cd Remove and ignore generated file 2018-05-25 20:06:09 +02:00
Marian Steinbach eecf0bf74b Ignore VS Code settings file 2018-05-23 20:18:39 +02:00
Marian Steinbach 280d13605e Add __pycache__ to ignored list 2018-05-03 11:43:29 +02:00
Marian Steinbach 7348c7bec8 Add site screenshots 2018-04-17 20:45:51 +02:00
Marian Steinbach 3513eddfe6 Update results, spider logic and webapp 2018-04-05 18:11:40 +02:00
Marian Steinbach 220a6ba629 First working code and results 2018-04-03 23:15:28 +02:00