Commit Graph

33 Commits

Author SHA1 Message Date
Marian Steinbach 618e29d763
Job-Verwaltung mit RQ, und vieles mehr (#149)
* CLI: remove 'jobs' command, add 'manager'

* Add job definition

* Move jobs to manage folder

* Rename jobs to manager

* Add rq and redis dependencies

* Add docker-compose YAML

* Downgrade to alpine 3.8

* Adjust paths in Dockerfile, remove entrypoint

* Rename 'make spiderjobs' to 'make jobs'

* Fix docker exectution

* Adapt 'make jobs'

* Fix metadata scheme

* Add docker dependency

* Rendomize queue (a bit)

* Use latest image, remove debug output

* Make docker-compose file downwards-compatible

* Use latest instead of dev image tag

* Update docker-compose.yaml

* Adapt job start script

* Fix redis connection in manager

* Add support for increasing timeout via environment variable

* Adapt load_in_browser to cookies table schema change

* Fix execution

* Mitigate yaml warning

* Bump some dependency versions

* Report resource usage stats for each job

* checks/load_in_browser: Return DOM size, prevent multiple page loads

* Update .dockerignore

* Code update

* Script update

* Update README.md

* WIP

* WIP commit

* Update Dockerfile to alpine:edge and chromium v90

* Update TestCertificateChecker

* Set defaults for __init__ function

* Detect sunflower theme

* Update unit test for new datetime (zero-basing)

* Set logging prefs from Chromium in a new way

* Move datastore client instantiation

As it is not needed for all commands

* Change green-directory repository URL

* Add git settings for cloning green-directory

* Pin alpine version 3.14, fix py3-cryptography

* Use plain docker build progress output

* Add volumes to 'make test' docker run command

* Fix bug

* Update example command in README

* Update dependencies

* Add creation of Kubernetes jobs
2021-11-11 20:15:43 +01:00
Marian Steinbach 725ed5439d Set share memory size 2019-11-22 08:39:56 +01:00
Marian Steinbach aac5fb3192 Some Makefile updates 2019-06-03 08:08:55 +02:00
Marian Steinbach 5e8347916c
Fehlerbehebung im url_reachability check (#108)
* Fix detection of redirects to bad domains

* Fix bad domain check

* Add --url flag to spider for faster debugging

* Pass args to make spider

* Add spidering of a single URL for debugging purposes

* Fix tests

* Fix test in CI

* Remove pip upgrade
2019-04-19 00:35:28 +02:00
Marian Steinbach c03ff21a9c
Simplify export (#88)
* Simplify exports

* Create file output in current working directory
2018-11-20 20:00:47 +01:00
Marian Steinbach 38481236ca
Add webapp deployment (#87)
* Add webapp deployment script

* Add some docs for webapp

* Some fixes in run-job.sh

* Update webapp deployment script

* Add some kubernetes job manifests

* Create index.yaml

* Remove local creation of the docker image from targets

* Update README.md
2018-11-20 19:54:23 +01:00
Marian Steinbach 2945372aaf
Fix README and Makefile (#72) 2018-10-04 21:20:53 +02:00
Marian Steinbach 57f8dea4e0
Improve certificate check to support SNI (#71)
* Fix the certificate check to support SNI
* Better tests for the certificate check
* Activate verbose output when running make test
* Add commenting on the spider test
2018-10-03 21:01:52 +02:00
Marian Steinbach ae6a2e83e9
Refactor and modularize spider (#70)
See PR description for details
2018-10-03 11:05:42 +02:00
Marian Steinbach 22d158c8b8 Fix docker image name in Makefile 2018-09-12 09:23:10 +02:00
Marian Steinbach 8580747e2a Docker file change from Debian stretch to Alpine 2018-09-12 00:42:40 +02:00
Marian Steinbach fda97574cb Remove webapp code 2018-08-28 22:28:34 +02:00
Marian Steinbach a463087b0a Add icon fetching to data export 2018-08-27 23:39:00 +02:00
Marian Steinbach 77a736c61f Comment makefile targets 2018-08-27 21:17:04 +02:00
Marian Steinbach ea47673ed2 More Makefile adaptation 2018-08-23 10:02:34 +02:00
Marian Steinbach 35f21f298d Replace the screenshots target with an export target 2018-08-23 10:02:07 +02:00
Marian Steinbach 2472af988d Add target to create spider jobs 2018-08-23 09:38:30 +02:00
Marian Steinbach 785fbb2951 Fix makefile for Screenshots 2018-08-17 07:09:21 +02:00
Marian Steinbach 989ed53acf Fetch screenshot data from database 2018-08-15 22:02:20 +02:00
Marian Steinbach 2034345d97 Add and apply tooltip 2018-06-06 23:35:07 +02:00
Marian Steinbach 2c9e8d59cd Remove and ignore generated file 2018-05-25 20:06:09 +02:00
Marian Steinbach ecd91588d2 Do not swipe /docs folder when publishing webapp 2018-05-25 19:21:09 +02:00
Marian Steinbach 8a7cf17701 Add volume for icons 2018-05-25 19:10:54 +02:00
Marian Steinbach 997519df35 More tests 2018-05-04 10:02:01 +02:00
Marian Steinbach 1e4cb2bce8 Run python unit tests in docker container 2018-05-03 12:01:30 +02:00
Marian Steinbach cf656ad2d1
Merge pull request #33 from netzbegruenung/unittests
Unit tests und CI
2018-05-03 11:55:06 +02:00
Marian Steinbach ab801e646b Add test target 2018-05-03 11:43:42 +02:00
Marian Steinbach 7291b09e83 Change pull command 2018-05-03 10:56:17 +02:00
Marian Steinbach 317624c2b4 Fix output path 2018-05-03 10:35:52 +02:00
Marian Steinbach 4de5890c60 Change to run spider in Docker container 2018-05-03 10:22:10 +02:00
Marian Steinbach 7348c7bec8 Add site screenshots 2018-04-17 20:45:51 +02:00
Marian Steinbach 48ac7f192d Fix "make webapp" target 2018-04-05 20:24:40 +02:00
Marian Steinbach f24276d891 Makefile and documentation 2018-04-05 19:39:09 +02:00