NB-Public/green-spider
14
0
Fork 1
mirror of https://github.com/netzbegruenung/green-spider.git synced 2024-05-05 18:33:40 +02:00
Code Issues 2 Projects Releases Wiki Activity
green-spider/requirements.txt

43 lines
882 B
Plaintext
Raw Normal View History

Bump beautifulsoup4 from 4.10.0 to 4.12.3 (#314)
2024-02-21 11:09:44 +01:00
beautifulsoup4==4.12.3
Bump cachetools from 5.3.2 to 5.3.3 (#332)
2024-03-01 17:59:50 +01:00
cachetools==5.3.3
Bump certifi from 2023.7.22 to 2024.2.2 (#352)
2024-04-03 16:31:46 +02:00
certifi==2024.2.2
Update dependencies (#356) * Update Google modules * Update pyasn1 modules * Update cffi * Update gitdb and GitPython * Update various modules * Update feedparser * Update w3lib
2024-04-03 22:29:33 +02:00
cffi==1.16.0
Bump chardet from 3.0.4 to 5.2.0 (#324)
2024-02-21 12:55:34 +01:00
chardet==5.2.0
Reparatur und Aufräumen an der job execution (#340) * Update jq URL * Improve docker compose setup * Script makeover: only one spider job, debian 11, add git clone * Update image name * Add some docs * Pin click to v7 due to problems with rq * Newline * Improve manager code * Add make tarket venv * Remove obsolete 'spider' command from cli * Remove git clone from manager code * Remove worker functions from spider code * Let 'make jobs' execute git clone and use docker compose * Add 'spider' make target * Update .dockerignore * Add dryrun target to spider a URL without storing results * Remove unused config entry
2024-03-04 17:18:37 +01:00
click>=7,<8
Bump cssselect from 1.1.0 to 1.2.0 (#335)
2024-03-01 18:19:48 +01:00
cssselect==1.2.0
Bump dnspython from 2.1.0 to 2.6.1 (#318)
2024-02-21 10:55:26 +01:00
dnspython==2.6.1
Mehrere Fixes und Verbesserungen (#343) * Use UTC for feed item age calculation * Improvements in run-job.sh script * Prevent output buffering in job creation * Remove unused environment variable references * Print more detailed results count * Bring back function to execute a single spider job * Fix 'make spider' command * Upgrade docker to 5.0.3
2024-03-07 11:31:16 +01:00
docker==5.0.3
Update dependencies (#356) * Update Google modules * Update pyasn1 modules * Update cffi * Update gitdb and GitPython * Update various modules * Update feedparser * Update w3lib
2024-04-03 22:29:33 +02:00
feedparser==6.0.11
gitdb==4.0.11
GitPython==3.1.43
google-api-core==2.18.0
Bump google-auth from 2.28.1 to 2.29.0 (#351)
2024-04-03 16:54:53 +02:00
google-auth==2.29.0
Update dependencies (#356) * Update Google modules * Update pyasn1 modules * Update cffi * Update gitdb and GitPython * Update various modules * Update feedparser * Update w3lib
2024-04-03 22:29:33 +02:00
google-cloud-core==2.4.1
google-cloud-datastore==2.19.0
google-cloud-storage==2.16.0
googleapis-common-protos==1.63.0
Job-Verwaltung mit RQ, und vieles mehr (#149) * CLI: remove 'jobs' command, add 'manager' * Add job definition * Move jobs to manage folder * Rename jobs to manager * Add rq and redis dependencies * Add docker-compose YAML * Downgrade to alpine 3.8 * Adjust paths in Dockerfile, remove entrypoint * Rename 'make spiderjobs' to 'make jobs' * Fix docker exectution * Adapt 'make jobs' * Fix metadata scheme * Add docker dependency * Rendomize queue (a bit) * Use latest image, remove debug output * Make docker-compose file downwards-compatible * Use latest instead of dev image tag * Update docker-compose.yaml * Adapt job start script * Fix redis connection in manager * Add support for increasing timeout via environment variable * Adapt load_in_browser to cookies table schema change * Fix execution * Mitigate yaml warning * Bump some dependency versions * Report resource usage stats for each job * checks/load_in_browser: Return DOM size, prevent multiple page loads * Update .dockerignore * Code update * Script update * Update README.md * WIP * WIP commit * Update Dockerfile to alpine:edge and chromium v90 * Update TestCertificateChecker * Set defaults for __init__ function * Detect sunflower theme * Update unit test for new datetime (zero-basing) * Set logging prefs from Chromium in a new way * Move datastore client instantiation As it is not needed for all commands * Change green-directory repository URL * Add git settings for cloning green-directory * Pin alpine version 3.14, fix py3-cryptography * Use plain docker build progress output * Add volumes to 'make test' docker run command * Fix bug * Update example command in README * Update dependencies * Add creation of Kubernetes jobs
2021-11-11 20:15:43 +01:00
html-similarity==0.3.3
Bump httpretty from 0.9.7 to 1.1.4 (#222) Bumps [httpretty](https://github.com/gabrielfalcao/HTTPretty) from 0.9.7 to 1.1.4. - [Release notes](https://github.com/gabrielfalcao/HTTPretty/releases) - [Changelog](https://github.com/gabrielfalcao/HTTPretty/blob/master/.release) - [Commits](https://github.com/gabrielfalcao/HTTPretty/compare/0.9.7...1.1.4) --- updated-dependencies: - dependency-name: httpretty dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-10-24 14:35:26 +02:00
httpretty==1.1.4
Bump idna from 2.10 to 3.6 (#331)
2024-03-01 18:29:18 +01:00
idna==3.6
Bump kubernetes from 20.13.0 to 29.0.0 (#334)
2024-03-01 18:42:22 +01:00
kubernetes==29.0.0
Update dependencies (#356) * Update Google modules * Update pyasn1 modules * Update cffi * Update gitdb and GitPython * Update various modules * Update feedparser * Update w3lib
2024-04-03 22:29:33 +02:00
parsel==1.6.0 # html-similarity 0.3.3 depends on parsel==1.6.0
protobuf==4.25.3 # <5.0.0 requested by google-api-core 2.18.0
pyasn1==0.6.0
pyasn1-modules==0.4.0
pycparser==2.22
Bump pyopenssl from 23.0.0 to 24.1.0 (#349)
2024-04-03 16:27:10 +02:00
pyOpenSSL==24.1.0
Update dependencies (#356) * Update Google modules * Update pyasn1 modules * Update cffi * Update gitdb and GitPython * Update various modules * Update feedparser * Update w3lib
2024-04-03 22:29:33 +02:00
pytz==2024.1
Bump redis from 4.1.0 to 5.0.3 (#350)
2024-04-03 16:18:30 +02:00
redis==5.0.3
Bump requests from 2.26.0 to 2.31.0 (#322) Bumps [requests](https://github.com/psf/requests) from 2.26.0 to 2.31.0. - [Release notes](https://github.com/psf/requests/releases) - [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md) - [Commits](https://github.com/psf/requests/compare/v2.26.0...v2.31.0) --- updated-dependencies: - dependency-name: requests dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 10:50:30 +01:00
requests==2.31.0
Update dependencies (#356) * Update Google modules * Update pyasn1 modules * Update cffi * Update gitdb and GitPython * Update various modules * Update feedparser * Update w3lib
2024-04-03 22:29:33 +02:00
responses==0.25.0
Bump rq from 1.16.0 to 1.16.1 (#344)
2024-04-03 15:58:04 +02:00
rq==1.16.1
Bump rsa from 4.7.2 to 4.9 (#291)
2024-02-21 11:00:11 +01:00
rsa==4.9
Job-Verwaltung mit RQ, und vieles mehr (#149) * CLI: remove 'jobs' command, add 'manager' * Add job definition * Move jobs to manage folder * Rename jobs to manager * Add rq and redis dependencies * Add docker-compose YAML * Downgrade to alpine 3.8 * Adjust paths in Dockerfile, remove entrypoint * Rename 'make spiderjobs' to 'make jobs' * Fix docker exectution * Adapt 'make jobs' * Fix metadata scheme * Add docker dependency * Rendomize queue (a bit) * Use latest image, remove debug output * Make docker-compose file downwards-compatible * Use latest instead of dev image tag * Update docker-compose.yaml * Adapt job start script * Fix redis connection in manager * Add support for increasing timeout via environment variable * Adapt load_in_browser to cookies table schema change * Fix execution * Mitigate yaml warning * Bump some dependency versions * Report resource usage stats for each job * checks/load_in_browser: Return DOM size, prevent multiple page loads * Update .dockerignore * Code update * Script update * Update README.md * WIP * WIP commit * Update Dockerfile to alpine:edge and chromium v90 * Update TestCertificateChecker * Set defaults for __init__ function * Detect sunflower theme * Update unit test for new datetime (zero-basing) * Set logging prefs from Chromium in a new way * Move datastore client instantiation As it is not needed for all commands * Change green-directory repository URL * Add git settings for cloning green-directory * Pin alpine version 3.14, fix py3-cryptography * Use plain docker build progress output * Add volumes to 'make test' docker run command * Fix bug * Update example command in README * Update dependencies * Add creation of Kubernetes jobs
2021-11-11 20:15:43 +01:00
selenium==3.141.0
Bump smmap from 3.0.4 to 5.0.1 (#323)
2024-02-21 10:59:59 +01:00
smmap==5.0.1
Bump smmap2 from 2.0.5 to 3.0.1 (#338)
2024-03-01 18:05:04 +01:00
smmap2==3.0.1
Update dependencies (#356) * Update Google modules * Update pyasn1 modules * Update cffi * Update gitdb and GitPython * Update various modules * Update feedparser * Update w3lib
2024-04-03 22:29:33 +02:00
soupsieve==2.5
Chromium und Alpine update (#319) * Update base image to alpine v3.19 * Update Chromium to 121 * Remove libssl1.1 (no longer available in alpine 3.19) * Add --break-system-packages to pip install commands * Print debugging info * Upgrade actions/checkout to v4 * Fix entrypoints and commands * Upgrade pyOpenSSL to v24.0.0 * Upgrade tenacity to v8
2024-02-21 09:19:33 +01:00
tenacity==8.2.3
Update dependencies (#356) * Update Google modules * Update pyasn1 modules * Update cffi * Update gitdb and GitPython * Update various modules * Update feedparser * Update w3lib
2024-04-03 22:29:33 +02:00
urllib3==1.26.18 # v2 requires code changes in timeout handling
w3lib==2.1.2
Bump websocket-client from 0.59.0 to 1.7.0 (#313) Bumps [websocket-client](https://github.com/websocket-client/websocket-client) from 0.59.0 to 1.7.0. - [Release notes](https://github.com/websocket-client/websocket-client/releases) - [Changelog](https://github.com/websocket-client/websocket-client/blob/master/ChangeLog) - [Commits](https://github.com/websocket-client/websocket-client/compare/v0.59.0...v1.7.0) --- updated-dependencies: - dependency-name: websocket-client dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-02-21 09:27:17 +01:00
websocket-client==1.7.0
Reference in a new issue Copy permalink