mirror of
https://github.com/netzbegruenung/green-spider.git
synced 2024-04-26 06:20:06 +02:00
Marian Steinbach
618e29d763
* CLI: remove 'jobs' command, add 'manager' * Add job definition * Move jobs to manage folder * Rename jobs to manager * Add rq and redis dependencies * Add docker-compose YAML * Downgrade to alpine 3.8 * Adjust paths in Dockerfile, remove entrypoint * Rename 'make spiderjobs' to 'make jobs' * Fix docker exectution * Adapt 'make jobs' * Fix metadata scheme * Add docker dependency * Rendomize queue (a bit) * Use latest image, remove debug output * Make docker-compose file downwards-compatible * Use latest instead of dev image tag * Update docker-compose.yaml * Adapt job start script * Fix redis connection in manager * Add support for increasing timeout via environment variable * Adapt load_in_browser to cookies table schema change * Fix execution * Mitigate yaml warning * Bump some dependency versions * Report resource usage stats for each job * checks/load_in_browser: Return DOM size, prevent multiple page loads * Update .dockerignore * Code update * Script update * Update README.md * WIP * WIP commit * Update Dockerfile to alpine:edge and chromium v90 * Update TestCertificateChecker * Set defaults for __init__ function * Detect sunflower theme * Update unit test for new datetime (zero-basing) * Set logging prefs from Chromium in a new way * Move datastore client instantiation As it is not needed for all commands * Change green-directory repository URL * Add git settings for cloning green-directory * Pin alpine version 3.14, fix py3-cryptography * Use plain docker build progress output * Add volumes to 'make test' docker run command * Fix bug * Update example command in README * Update dependencies * Add creation of Kubernetes jobs
27 lines
1 KiB
Docker
27 lines
1 KiB
Docker
FROM alpine:3.14
|
|
|
|
WORKDIR /workdir
|
|
|
|
ADD requirements.txt /workdir/
|
|
|
|
RUN echo "http://dl-4.alpinelinux.org/alpine/edge/main" >> /etc/apk/repositories && \
|
|
echo "http://dl-4.alpinelinux.org/alpine/edge/community" >> /etc/apk/repositories && \
|
|
apk --update --no-cache add ca-certificates chromium chromium-chromedriver \
|
|
python3-dev py3-grpcio py3-wheel py3-pip py3-lxml \
|
|
build-base git libxml2 libxml2-dev libxslt libxslt-dev libffi-dev openssl-dev cargo && \
|
|
pip install -r requirements.txt && \
|
|
apk del build-base
|
|
|
|
# As alpine's py3-cryptography did not work as of alpine v3.14, we use this hack from
|
|
# https://github.com/pyca/cryptography/issues/3344#issuecomment-650845512
|
|
RUN LDFLAGS="-L/opt/openssl/lib -Wl,-rpath,/opt/openssl/lib" CFLAGS="-I/opt/openssl/include" pip3 install -U cryptography
|
|
|
|
ADD cli.py /workdir/
|
|
ADD manager /workdir/manager
|
|
ADD config /workdir/config
|
|
ADD checks /workdir/checks
|
|
ADD rating /workdir/rating
|
|
ADD spider /workdir/spider
|
|
ADD export /workdir/export
|
|
ADD job.py /workdir/
|