NB-Public/green-spider
14
0
Fork 1
mirror of https://github.com/netzbegruenung/green-spider.git synced 2024-04-27 14:54:52 +02:00
Code Issues 2 Projects Releases Wiki Activity
311 commits 7 branches 7 tags 4.1 MiB
Commit graph

5 commits

Author SHA1 Message Date
Marian Steinbach c59db691a0
Reparatur und Aufräumen an der job execution (#340) 
* Update jq URL

* Improve docker compose setup

* Script makeover: only one spider job, debian 11, add git clone

* Update image name

* Add some docs

* Pin click to v7 due to problems with rq

* Newline

* Improve manager code

* Add make tarket venv

* Remove obsolete 'spider' command from cli

* Remove git clone from manager code

* Remove worker functions from spider code

* Let 'make jobs' execute git clone and use docker compose

* Add 'spider' make target

* Update .dockerignore

* Add dryrun target to spider a URL without storing results

* Remove unused config entry
 2024-03-04 17:18:37 +01:00
Marian Steinbach ab942ca91d Fix full JSON export 2019-05-04 23:00:00 +02:00
Marian Steinbach c03ff21a9c
Simplify export (#88) 
* Simplify exports

* Create file output in current working directory
 2018-11-20 20:00:47 +01:00
Marian Steinbach 8ce8768465 Improve error messages in export 2018-10-08 08:42:29 +02:00
Marian Steinbach ae6a2e83e9
Refactor and modularize spider (#70) 
See PR description for details
 2018-10-03 11:05:42 +02:00