NB-Public/green-spider
14
0
Fork 1
mirror of https://github.com/netzbegruenung/green-spider.git synced 2024-05-06 10:53:41 +02:00
Code Issues 2 Projects Releases Wiki Activity
green-spider/Dockerfile

37 lines
1.2 KiB
Docker
Raw Normal View History

Update auf Chromium v122; Verbesserung am container image build (#339) * Use Chromium v122 * Pin base image to checksum * Change package sources, do not use edge/main
2024-03-01 17:51:56 +01:00
FROM alpine:3.19@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
Make and use a versioned docker image (#279) * Revert redis module to 4.1.0 * Revert dnspython to 2.1.0 * Revert click to 8.0.3 * Specify alpine 3.16.2, reorganize into multiple steps * Replace 'latest' with 'main' everywhere * Fix deprecation warnings * Add Google root certificates * Re-order APK packages, write list after installing * Create VERSION file during docker image build * Pin chromium version
2022-10-24 21:35:15 +02:00
Set chromium version to 123.0.6312.86-r0 (#354)
2024-04-03 15:45:02 +02:00
ENV CHROMIUM_VERSION=123.0.6312.86-r0
Make and use a versioned docker image (#279) * Revert redis module to 4.1.0 * Revert dnspython to 2.1.0 * Revert click to 8.0.3 * Specify alpine 3.16.2, reorganize into multiple steps * Replace 'latest' with 'main' everywhere * Fix deprecation warnings * Add Google root certificates * Re-order APK packages, write list after installing * Create VERSION file during docker image build * Pin chromium version
2022-10-24 21:35:15 +02:00
Update auf Chromium v122; Verbesserung am container image build (#339) * Use Chromium v122 * Pin base image to checksum * Change package sources, do not use edge/main
2024-03-01 17:51:56 +01:00
RUN echo "http://dl-cdn.alpinelinux.org/alpine/v3.19/community" >> /etc/apk/repositories && \
Make and use a versioned docker image (#279) * Revert redis module to 4.1.0 * Revert dnspython to 2.1.0 * Revert click to 8.0.3 * Specify alpine 3.16.2, reorganize into multiple steps * Replace 'latest' with 'main' everywhere * Fix deprecation warnings * Add Google root certificates * Re-order APK packages, write list after installing * Create VERSION file during docker image build * Pin chromium version
2022-10-24 21:35:15 +02:00
apk --update --no-cache add ca-certificates \
chromium=$CHROMIUM_VERSION \
chromium-chromedriver=$CHROMIUM_VERSION \
py3-cryptography python3-dev py3-grpcio py3-wheel py3-pip py3-lxml py3-yaml \
Chromium und Alpine update (#319) * Update base image to alpine v3.19 * Update Chromium to 121 * Remove libssl1.1 (no longer available in alpine 3.19) * Add --break-system-packages to pip install commands * Print debugging info * Upgrade actions/checkout to v4 * Fix entrypoints and commands * Upgrade pyOpenSSL to v24.0.0 * Upgrade tenacity to v8
2024-02-21 09:19:33 +01:00
build-base git icu-libs libssl3 libxml2 libxml2-dev libxslt libxslt-dev \
Make and use a versioned docker image (#279) * Revert redis module to 4.1.0 * Revert dnspython to 2.1.0 * Revert click to 8.0.3 * Specify alpine 3.16.2, reorganize into multiple steps * Replace 'latest' with 'main' everywhere * Fix deprecation warnings * Add Google root certificates * Re-order APK packages, write list after installing * Create VERSION file during docker image build * Pin chromium version
2022-10-24 21:35:15 +02:00
libffi-dev openssl-dev cargo
RUN apk info -v | sort
Change to run spider in Docker container
2018-05-03 10:22:10 +02:00
Problembehebung mit Exception-Daten im Ergbebnis, die nicht geschrieben werden können, und Spidern einzelner Sites (#132) * WIP commit for single job execution * Convert exception to string * Pass more arguments * Move python modules lsit into requirements.txt * Document single site spidering * Remove debugging
2019-11-22 23:13:57 +01:00
WORKDIR /workdir
Make and use a versioned docker image (#279) * Revert redis module to 4.1.0 * Revert dnspython to 2.1.0 * Revert click to 8.0.3 * Specify alpine 3.16.2, reorganize into multiple steps * Replace 'latest' with 'main' everywhere * Fix deprecation warnings * Add Google root certificates * Re-order APK packages, write list after installing * Create VERSION file during docker image build * Pin chromium version
2022-10-24 21:35:15 +02:00
# Execute time consuming compilations in a separate step
Chromium und Alpine update (#319) * Update base image to alpine v3.19 * Update Chromium to 121 * Remove libssl1.1 (no longer available in alpine 3.19) * Add --break-system-packages to pip install commands * Print debugging info * Upgrade actions/checkout to v4 * Fix entrypoints and commands * Upgrade pyOpenSSL to v24.0.0 * Upgrade tenacity to v8
2024-02-21 09:19:33 +01:00
RUN python3 -m pip install libcst==0.4.7 sgmllib3k==1.0.0 --break-system-packages
Make and use a versioned docker image (#279) * Revert redis module to 4.1.0 * Revert dnspython to 2.1.0 * Revert click to 8.0.3 * Specify alpine 3.16.2, reorganize into multiple steps * Replace 'latest' with 'main' everywhere * Fix deprecation warnings * Add Google root certificates * Re-order APK packages, write list after installing * Create VERSION file during docker image build * Pin chromium version
2022-10-24 21:35:15 +02:00
ADD https://pki.google.com/roots.pem /google_roots.pem
ENV GRPC_DEFAULT_SSL_ROOTS_FILE_PATH=/google_roots.pem
Problembehebung mit Exception-Daten im Ergbebnis, die nicht geschrieben werden können, und Spidern einzelner Sites (#132) * WIP commit for single job execution * Convert exception to string * Pass more arguments * Move python modules lsit into requirements.txt * Document single site spidering * Remove debugging
2019-11-22 23:13:57 +01:00
ADD requirements.txt /workdir/
Chromium und Alpine update (#319) * Update base image to alpine v3.19 * Update Chromium to 121 * Remove libssl1.1 (no longer available in alpine 3.19) * Add --break-system-packages to pip install commands * Print debugging info * Upgrade actions/checkout to v4 * Fix entrypoints and commands * Upgrade pyOpenSSL to v24.0.0 * Upgrade tenacity to v8
2024-02-21 09:19:33 +01:00
RUN pip install -r requirements.txt --break-system-packages
Problembehebung mit Exception-Daten im Ergbebnis, die nicht geschrieben werden können, und Spidern einzelner Sites (#132) * WIP commit for single job execution * Convert exception to string * Pass more arguments * Move python modules lsit into requirements.txt * Document single site spidering * Remove debugging
2019-11-22 23:13:57 +01:00
Make and use a versioned docker image (#279) * Revert redis module to 4.1.0 * Revert dnspython to 2.1.0 * Revert click to 8.0.3 * Specify alpine 3.16.2, reorganize into multiple steps * Replace 'latest' with 'main' everywhere * Fix deprecation warnings * Add Google root certificates * Re-order APK packages, write list after installing * Create VERSION file during docker image build * Pin chromium version
2022-10-24 21:35:15 +02:00
RUN python3 -m pip freeze
Change to run spider in Docker container
2018-05-03 10:22:10 +02:00
Job-Verwaltung mit RQ, und vieles mehr (#149) * CLI: remove 'jobs' command, add 'manager' * Add job definition * Move jobs to manage folder * Rename jobs to manager * Add rq and redis dependencies * Add docker-compose YAML * Downgrade to alpine 3.8 * Adjust paths in Dockerfile, remove entrypoint * Rename 'make spiderjobs' to 'make jobs' * Fix docker exectution * Adapt 'make jobs' * Fix metadata scheme * Add docker dependency * Rendomize queue (a bit) * Use latest image, remove debug output * Make docker-compose file downwards-compatible * Use latest instead of dev image tag * Update docker-compose.yaml * Adapt job start script * Fix redis connection in manager * Add support for increasing timeout via environment variable * Adapt load_in_browser to cookies table schema change * Fix execution * Mitigate yaml warning * Bump some dependency versions * Report resource usage stats for each job * checks/load_in_browser: Return DOM size, prevent multiple page loads * Update .dockerignore * Code update * Script update * Update README.md * WIP * WIP commit * Update Dockerfile to alpine:edge and chromium v90 * Update TestCertificateChecker * Set defaults for __init__ function * Detect sunflower theme * Update unit test for new datetime (zero-basing) * Set logging prefs from Chromium in a new way * Move datastore client instantiation As it is not needed for all commands * Change green-directory repository URL * Add git settings for cloning green-directory * Pin alpine version 3.14, fix py3-cryptography * Use plain docker build progress output * Add volumes to 'make test' docker run command * Fix bug * Update example command in README * Update dependencies * Add creation of Kubernetes jobs
2021-11-11 20:15:43 +01:00
ADD cli.py /workdir/
ADD manager /workdir/manager
ADD config /workdir/config
ADD checks /workdir/checks
ADD rating /workdir/rating
ADD spider /workdir/spider
ADD export /workdir/export
ADD job.py /workdir/
Make and use a versioned docker image (#279) * Revert redis module to 4.1.0 * Revert dnspython to 2.1.0 * Revert click to 8.0.3 * Specify alpine 3.16.2, reorganize into multiple steps * Replace 'latest' with 'main' everywhere * Fix deprecation warnings * Add Google root certificates * Re-order APK packages, write list after installing * Create VERSION file during docker image build * Pin chromium version
2022-10-24 21:35:15 +02:00
ADD VERSION /workdir/VERSION
Reference in a new issue Copy permalink