Commit graph

18 commits

Author SHA1 Message Date
Marian Steinbach 618e29d763
Job-Verwaltung mit RQ, und vieles mehr (#149)
* CLI: remove 'jobs' command, add 'manager'

* Add job definition

* Move jobs to manage folder

* Rename jobs to manager

* Add rq and redis dependencies

* Add docker-compose YAML

* Downgrade to alpine 3.8

* Adjust paths in Dockerfile, remove entrypoint

* Rename 'make spiderjobs' to 'make jobs'

* Fix docker exectution

* Adapt 'make jobs'

* Fix metadata scheme

* Add docker dependency

* Rendomize queue (a bit)

* Use latest image, remove debug output

* Make docker-compose file downwards-compatible

* Use latest instead of dev image tag

* Update docker-compose.yaml

* Adapt job start script

* Fix redis connection in manager

* Add support for increasing timeout via environment variable

* Adapt load_in_browser to cookies table schema change

* Fix execution

* Mitigate yaml warning

* Bump some dependency versions

* Report resource usage stats for each job

* checks/load_in_browser: Return DOM size, prevent multiple page loads

* Update .dockerignore

* Code update

* Script update

* Update README.md

* WIP

* WIP commit

* Update Dockerfile to alpine:edge and chromium v90

* Update TestCertificateChecker

* Set defaults for __init__ function

* Detect sunflower theme

* Update unit test for new datetime (zero-basing)

* Set logging prefs from Chromium in a new way

* Move datastore client instantiation

As it is not needed for all commands

* Change green-directory repository URL

* Add git settings for cloning green-directory

* Pin alpine version 3.14, fix py3-cryptography

* Use plain docker build progress output

* Add volumes to 'make test' docker run command

* Fix bug

* Update example command in README

* Update dependencies

* Add creation of Kubernetes jobs
2021-11-11 20:15:43 +01:00
dependabot-preview[bot] aedb4b3acc
Merge pull request #169 from netzbegruenung/dependabot/pip/urllib3-1.25.9 2020-06-12 07:26:37 +00:00
dependabot-preview[bot] cc59deb262
Bump responses from 0.10.12 to 0.10.15
Bumps [responses](https://github.com/getsentry/responses) from 0.10.12 to 0.10.15.
- [Release notes](https://github.com/getsentry/responses/releases)
- [Changelog](https://github.com/getsentry/responses/blob/master/CHANGES)
- [Commits](https://github.com/getsentry/responses/compare/0.10.12...0.10.15)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-12 06:51:49 +00:00
dependabot-preview[bot] 18793b6968
Bump urllib3 from 1.25.7 to 1.25.9
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.25.7 to 1.25.9.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/master/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.25.7...1.25.9)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-17 07:07:23 +00:00
dependabot-preview[bot] 02677ac1de
Bump responses from 0.10.7 to 0.10.12 (#161)
Bumps [responses](https://github.com/getsentry/responses) from 0.10.7 to 0.10.12.
- [Release notes](https://github.com/getsentry/responses/releases)
- [Changelog](https://github.com/getsentry/responses/blob/master/CHANGES)
- [Commits](https://github.com/getsentry/responses/compare/0.10.7...0.10.12)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-21 00:55:17 +01:00
dependabot-preview[bot] 47ecf4024b
Bump beautifulsoup4 from 4.6.0 to 4.8.1 (#145) 2019-12-03 18:06:33 +00:00
dependabot-preview[bot] 30c5aee565
Bump httpretty from 0.9.4 to 0.9.7 (#138) 2019-11-27 12:55:33 +00:00
dependabot-preview[bot] a5d37cacea
Bump smmap2 from 2.0.3 to 2.0.5 (#146) 2019-11-26 15:28:25 +00:00
dependabot-preview[bot] e3165f25dc
Bump requests from 2.20.0 to 2.22.0 (#143) 2019-11-26 15:27:55 +00:00
dependabot-preview[bot] e24c2507e6
Bump responses from 0.9.0 to 0.10.7 (#147) 2019-11-26 15:13:17 +00:00
dependabot-preview[bot] 9789af1688
Bump urllib3 from 1.24.2 to 1.25.7 (#140) 2019-11-25 11:06:34 +00:00
dependabot[bot] 7c75e1194c
Bump urllib3 from 1.22 to 1.24.2 (#133) 2019-11-22 22:55:15 +00:00
dependabot[bot] 01a0000316
Bump requests from 2.18.4 to 2.20.0 (#134) 2019-11-22 22:26:40 +00:00
Marian Steinbach b3bb8f34c3
Problembehebung mit Exception-Daten im Ergbebnis, die nicht geschrieben werden können, und Spidern einzelner Sites (#132)
* WIP commit for single job execution

* Convert exception to string

* Pass more arguments

* Move python modules lsit into requirements.txt

* Document single site spidering

* Remove debugging
2019-11-22 23:13:57 +01:00
Marian Steinbach 9bea186008 Clean up dependencies 2018-05-04 00:38:28 +02:00
Marian Steinbach 914fb1f35e Detect and report responsive layout details 2018-04-23 09:11:50 +02:00
Marian Steinbach 0ecf7e306e Add beautifulsoup4 and update requirements.txt 2018-04-09 22:39:26 +02:00
Marian Steinbach 220a6ba629 First working code and results 2018-04-03 23:15:28 +02:00