Marian Steinbach
5e183dc34a
Update dependencies ( #356 )
...
* Update Google modules
* Update pyasn1 modules
* Update cffi
* Update gitdb and GitPython
* Update various modules
* Update feedparser
* Update w3lib
2024-04-03 22:29:33 +02:00
dependabot[bot]
99d0980686
Bump google-auth from 2.28.1 to 2.29.0 ( #351 )
2024-04-03 14:54:53 +00:00
dependabot[bot]
ee167db02f
Bump certifi from 2023.7.22 to 2024.2.2 ( #352 )
2024-04-03 14:31:46 +00:00
dependabot[bot]
a84c2244bd
Bump pyopenssl from 23.0.0 to 24.1.0 ( #349 )
2024-04-03 14:27:10 +00:00
dependabot[bot]
e2bc62e51a
Bump redis from 4.1.0 to 5.0.3 ( #350 )
2024-04-03 14:18:30 +00:00
dependabot[bot]
f2a4be1b9d
Bump urllib3 from 1.26.7 to 1.26.18 ( #355 )
2024-04-03 14:17:35 +00:00
dependabot[bot]
1136637ed8
Bump rq from 1.16.0 to 1.16.1 ( #344 )
2024-04-03 13:58:04 +00:00
Marian Steinbach
0c0bcbf54e
Mehrere Fixes und Verbesserungen ( #343 )
...
* Use UTC for feed item age calculation
* Improvements in run-job.sh script
* Prevent output buffering in job creation
* Remove unused environment variable references
* Print more detailed results count
* Bring back function to execute a single spider job
* Fix 'make spider' command
* Upgrade docker to 5.0.3
2024-03-07 11:31:16 +01:00
Marian Steinbach
c59db691a0
Reparatur und Aufräumen an der job execution ( #340 )
...
* Update jq URL
* Improve docker compose setup
* Script makeover: only one spider job, debian 11, add git clone
* Update image name
* Add some docs
* Pin click to v7 due to problems with rq
* Newline
* Improve manager code
* Add make tarket venv
* Remove obsolete 'spider' command from cli
* Remove git clone from manager code
* Remove worker functions from spider code
* Let 'make jobs' execute git clone and use docker compose
* Add 'spider' make target
* Update .dockerignore
* Add dryrun target to spider a URL without storing results
* Remove unused config entry
2024-03-04 17:18:37 +01:00
dependabot[bot]
2927acde84
Bump kubernetes from 20.13.0 to 29.0.0 ( #334 )
2024-03-01 17:42:22 +00:00
dependabot[bot]
27705c83cb
Bump idna from 2.10 to 3.6 ( #331 )
2024-03-01 17:29:18 +00:00
dependabot[bot]
b17896cdb5
Bump cssselect from 1.1.0 to 1.2.0 ( #335 )
2024-03-01 17:19:48 +00:00
dependabot[bot]
a0e0dae2b9
Bump google-auth from 2.13.0 to 2.28.1 ( #330 )
2024-03-01 17:14:42 +00:00
dependabot[bot]
e48f80cd50
Bump rq from 1.8.0 to 1.16.0 ( #333 )
2024-03-01 17:10:31 +00:00
dependabot[bot]
bfd5ebc25e
Bump smmap2 from 2.0.5 to 3.0.1 ( #338 )
2024-03-01 17:05:04 +00:00
dependabot[bot]
724708217d
Bump cachetools from 5.3.2 to 5.3.3 ( #332 )
2024-03-01 16:59:50 +00:00
dependabot[bot]
42bc89f251
Bump chardet from 3.0.4 to 5.2.0 ( #324 )
2024-02-21 11:55:34 +00:00
dependabot[bot]
fceda4f642
Bump beautifulsoup4 from 4.10.0 to 4.12.3 ( #314 )
2024-02-21 10:09:44 +00:00
dependabot[bot]
fd8947cbf1
Bump rsa from 4.7.2 to 4.9 ( #291 )
2024-02-21 10:00:11 +00:00
dependabot[bot]
e7a8511650
Bump smmap from 3.0.4 to 5.0.1 ( #323 )
2024-02-21 09:59:59 +00:00
dependabot[bot]
a5b67c31ff
Bump dnspython from 2.1.0 to 2.6.1 ( #318 )
2024-02-21 09:55:26 +00:00
dependabot[bot]
fe747e5e14
Bump cachetools from 4.2.4 to 5.3.2 ( #316 )
...
Bumps [cachetools](https://github.com/tkem/cachetools ) from 4.2.4 to 5.3.2.
- [Changelog](https://github.com/tkem/cachetools/blob/master/CHANGELOG.rst )
- [Commits](https://github.com/tkem/cachetools/compare/v4.2.4...v5.3.2 )
---
updated-dependencies:
- dependency-name: cachetools
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:54:46 +01:00
dependabot[bot]
335e4d3af4
Bump certifi from 2021.10.8 to 2023.7.22 ( #321 )
...
Bumps [certifi](https://github.com/certifi/python-certifi ) from 2021.10.8 to 2023.7.22.
- [Commits](https://github.com/certifi/python-certifi/compare/2021.10.08...2023.07.22 )
---
updated-dependencies:
- dependency-name: certifi
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:51:00 +01:00
dependabot[bot]
e8fae03f49
Bump gitpython from 3.1.24 to 3.1.41 ( #320 )
...
Bumps [gitpython](https://github.com/gitpython-developers/GitPython ) from 3.1.24 to 3.1.41.
- [Release notes](https://github.com/gitpython-developers/GitPython/releases )
- [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES )
- [Commits](https://github.com/gitpython-developers/GitPython/compare/3.1.24...3.1.41 )
---
updated-dependencies:
- dependency-name: gitpython
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:50:47 +01:00
dependabot[bot]
a839597168
Bump requests from 2.26.0 to 2.31.0 ( #322 )
...
Bumps [requests](https://github.com/psf/requests ) from 2.26.0 to 2.31.0.
- [Release notes](https://github.com/psf/requests/releases )
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md )
- [Commits](https://github.com/psf/requests/compare/v2.26.0...v2.31.0 )
---
updated-dependencies:
- dependency-name: requests
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:50:30 +01:00
dependabot[bot]
a4c822ebfb
Bump websocket-client from 0.59.0 to 1.7.0 ( #313 )
...
Bumps [websocket-client](https://github.com/websocket-client/websocket-client ) from 0.59.0 to 1.7.0.
- [Release notes](https://github.com/websocket-client/websocket-client/releases )
- [Changelog](https://github.com/websocket-client/websocket-client/blob/master/ChangeLog )
- [Commits](https://github.com/websocket-client/websocket-client/compare/v0.59.0...v1.7.0 )
---
updated-dependencies:
- dependency-name: websocket-client
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 09:27:17 +01:00
dependabot[bot]
ced7fbd9b9
Bump protobuf from 4.21.8 to 4.25.3 ( #311 )
...
Bumps [protobuf](https://github.com/protocolbuffers/protobuf ) from 4.21.8 to 4.25.3.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases )
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl )
- [Commits](https://github.com/protocolbuffers/protobuf/commits/v4.25.3 )
---
updated-dependencies:
- dependency-name: protobuf
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 09:26:42 +01:00
Marian Steinbach
0efab3295c
Chromium und Alpine update ( #319 )
...
* Update base image to alpine v3.19
* Update Chromium to 121
* Remove libssl1.1 (no longer available in alpine 3.19)
* Add --break-system-packages to pip install commands
* Print debugging info
* Upgrade actions/checkout to v4
* Fix entrypoints and commands
* Upgrade pyOpenSSL to v24.0.0
* Upgrade tenacity to v8
2024-02-21 09:19:33 +01:00
Marian Steinbach
5e723c94db
Make and use a versioned docker image ( #279 )
...
* Revert redis module to 4.1.0
* Revert dnspython to 2.1.0
* Revert click to 8.0.3
* Specify alpine 3.16.2, reorganize into multiple steps
* Replace 'latest' with 'main' everywhere
* Fix deprecation warnings
* Add Google root certificates
* Re-order APK packages, write list after installing
* Create VERSION file during docker image build
* Pin chromium version
2022-10-24 21:35:15 +02:00
dependabot[bot]
024ef118dd
Bump dnspython from 1.16.0 to 2.2.1 ( #253 )
...
Bumps [dnspython](https://github.com/rthalley/dnspython ) from 1.16.0 to 2.2.1.
- [Release notes](https://github.com/rthalley/dnspython/releases )
- [Changelog](https://github.com/rthalley/dnspython/blob/master/doc/whatsnew.rst )
- [Commits](https://github.com/rthalley/dnspython/compare/v1.16.0...v2.2.1 )
---
updated-dependencies:
- dependency-name: dnspython
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 14:36:31 +02:00
dependabot[bot]
47f327290d
Bump cffi from 1.14.5 to 1.15.1 ( #263 )
...
Bumps [cffi](http://cffi.readthedocs.org ) from 1.14.5 to 1.15.1.
---
updated-dependencies:
- dependency-name: cffi
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 14:35:58 +02:00
dependabot[bot]
f3f08f68df
Bump httpretty from 0.9.7 to 1.1.4 ( #222 )
...
Bumps [httpretty](https://github.com/gabrielfalcao/HTTPretty ) from 0.9.7 to 1.1.4.
- [Release notes](https://github.com/gabrielfalcao/HTTPretty/releases )
- [Changelog](https://github.com/gabrielfalcao/HTTPretty/blob/master/.release )
- [Commits](https://github.com/gabrielfalcao/HTTPretty/compare/0.9.7...1.1.4 )
---
updated-dependencies:
- dependency-name: httpretty
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 14:35:26 +02:00
dependabot[bot]
643dd4c6c9
Bump click from 7.1.2 to 8.1.3 ( #257 )
...
Bumps [click](https://github.com/pallets/click ) from 7.1.2 to 8.1.3.
- [Release notes](https://github.com/pallets/click/releases )
- [Changelog](https://github.com/pallets/click/blob/main/CHANGES.rst )
- [Commits](https://github.com/pallets/click/compare/7.1.2...8.1.3 )
---
updated-dependencies:
- dependency-name: click
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 14:19:36 +02:00
dependabot[bot]
2a8921b1fa
Bump protobuf from 3.19.1 to 4.21.8 ( #276 )
...
Bumps [protobuf](https://github.com/protocolbuffers/protobuf ) from 3.19.1 to 4.21.8.
- [Release notes](https://github.com/protocolbuffers/protobuf/releases )
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/generate_changelog.py )
- [Commits](https://github.com/protocolbuffers/protobuf/commits )
---
updated-dependencies:
- dependency-name: protobuf
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 14:03:31 +02:00
dependabot[bot]
c5a1feac5a
Bump redis from 3.5.3 to 4.3.4 ( #264 )
...
Bumps [redis](https://github.com/redis/redis-py ) from 3.5.3 to 4.3.4.
- [Release notes](https://github.com/redis/redis-py/releases )
- [Changelog](https://github.com/redis/redis-py/blob/master/CHANGES )
- [Commits](https://github.com/redis/redis-py/compare/3.5.3...v4.3.4 )
---
updated-dependencies:
- dependency-name: redis
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 16:57:00 +02:00
dependabot[bot]
8310e9760d
Bump responses from 0.13.3 to 0.22.0 ( #275 )
...
Bumps [responses](https://github.com/getsentry/responses ) from 0.13.3 to 0.22.0.
- [Release notes](https://github.com/getsentry/responses/releases )
- [Changelog](https://github.com/getsentry/responses/blob/master/CHANGES )
- [Commits](https://github.com/getsentry/responses/compare/0.13.3...0.22.0 )
---
updated-dependencies:
- dependency-name: responses
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-21 16:46:50 +02:00
Marian Steinbach
51052ebac1
Add tests action ( #273 )
...
* Add tests action
* Change test execution command
* Change action towards building a docker image
* Use pyyaml from alpine package
* Add test execution
* Enable running without terminal in CI
* Add creation of empty secret file
* Fill service account file
* Fix JSON
* Fix quiting
* Add more fields to fake secret
* Wrap execution in try/except
* Fix: local variable 'result' referenced before assignment
2022-10-21 16:27:15 +02:00
Marian Steinbach
a864675d85
Upgrade PyOpenSSL, remove unit tests using TLS 1.0 and 1.1 ( #274 )
...
* Upgrade pyOpenSSL to v22.1.0
* Remove unit test using TLS 1.0 and 1.1
* Upgrade base image to alpine 3.15.6
2022-10-18 09:48:25 +02:00
Marian Steinbach
ba40bcf5b4
Update kubernetes ( #242 )
...
* Add kubernetes module
* Update context name
* Remove pod anti-affinity rule
* Update image tag
2021-12-06 22:40:43 +01:00
Marian Steinbach
357ecb6fd3
Update some dependencies ( #237 )
2021-12-06 22:38:38 +01:00
Marian Steinbach
618e29d763
Job-Verwaltung mit RQ, und vieles mehr ( #149 )
...
* CLI: remove 'jobs' command, add 'manager'
* Add job definition
* Move jobs to manage folder
* Rename jobs to manager
* Add rq and redis dependencies
* Add docker-compose YAML
* Downgrade to alpine 3.8
* Adjust paths in Dockerfile, remove entrypoint
* Rename 'make spiderjobs' to 'make jobs'
* Fix docker exectution
* Adapt 'make jobs'
* Fix metadata scheme
* Add docker dependency
* Rendomize queue (a bit)
* Use latest image, remove debug output
* Make docker-compose file downwards-compatible
* Use latest instead of dev image tag
* Update docker-compose.yaml
* Adapt job start script
* Fix redis connection in manager
* Add support for increasing timeout via environment variable
* Adapt load_in_browser to cookies table schema change
* Fix execution
* Mitigate yaml warning
* Bump some dependency versions
* Report resource usage stats for each job
* checks/load_in_browser: Return DOM size, prevent multiple page loads
* Update .dockerignore
* Code update
* Script update
* Update README.md
* WIP
* WIP commit
* Update Dockerfile to alpine:edge and chromium v90
* Update TestCertificateChecker
* Set defaults for __init__ function
* Detect sunflower theme
* Update unit test for new datetime (zero-basing)
* Set logging prefs from Chromium in a new way
* Move datastore client instantiation
As it is not needed for all commands
* Change green-directory repository URL
* Add git settings for cloning green-directory
* Pin alpine version 3.14, fix py3-cryptography
* Use plain docker build progress output
* Add volumes to 'make test' docker run command
* Fix bug
* Update example command in README
* Update dependencies
* Add creation of Kubernetes jobs
2021-11-11 20:15:43 +01:00
dependabot-preview[bot]
aedb4b3acc
Merge pull request #169 from netzbegruenung/dependabot/pip/urllib3-1.25.9
2020-06-12 07:26:37 +00:00
dependabot-preview[bot]
cc59deb262
Bump responses from 0.10.12 to 0.10.15
...
Bumps [responses](https://github.com/getsentry/responses ) from 0.10.12 to 0.10.15.
- [Release notes](https://github.com/getsentry/responses/releases )
- [Changelog](https://github.com/getsentry/responses/blob/master/CHANGES )
- [Commits](https://github.com/getsentry/responses/compare/0.10.12...0.10.15 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-12 06:51:49 +00:00
dependabot-preview[bot]
18793b6968
Bump urllib3 from 1.25.7 to 1.25.9
...
Bumps [urllib3](https://github.com/urllib3/urllib3 ) from 1.25.7 to 1.25.9.
- [Release notes](https://github.com/urllib3/urllib3/releases )
- [Changelog](https://github.com/urllib3/urllib3/blob/master/CHANGES.rst )
- [Commits](https://github.com/urllib3/urllib3/compare/1.25.7...1.25.9 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-17 07:07:23 +00:00
dependabot-preview[bot]
02677ac1de
Bump responses from 0.10.7 to 0.10.12 ( #161 )
...
Bumps [responses](https://github.com/getsentry/responses ) from 0.10.7 to 0.10.12.
- [Release notes](https://github.com/getsentry/responses/releases )
- [Changelog](https://github.com/getsentry/responses/blob/master/CHANGES )
- [Commits](https://github.com/getsentry/responses/compare/0.10.7...0.10.12 )
Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-21 00:55:17 +01:00
dependabot-preview[bot]
47ecf4024b
Bump beautifulsoup4 from 4.6.0 to 4.8.1 ( #145 )
2019-12-03 18:06:33 +00:00
dependabot-preview[bot]
30c5aee565
Bump httpretty from 0.9.4 to 0.9.7 ( #138 )
2019-11-27 12:55:33 +00:00
dependabot-preview[bot]
a5d37cacea
Bump smmap2 from 2.0.3 to 2.0.5 ( #146 )
2019-11-26 15:28:25 +00:00
dependabot-preview[bot]
e3165f25dc
Bump requests from 2.20.0 to 2.22.0 ( #143 )
2019-11-26 15:27:55 +00:00
dependabot-preview[bot]
e24c2507e6
Bump responses from 0.9.0 to 0.10.7 ( #147 )
2019-11-26 15:13:17 +00:00