NB-Public/green-spider
14
0
Fork 1
mirror of https://github.com/netzbegruenung/green-spider.git synced 2024-05-03 17:43:40 +02:00
Code Issues 2 Projects Releases Wiki Activity
238 commits 11 branches 7 tags 4.1 MiB
Commit graph

19 commits

Author SHA1 Message Date
Marian Steinbach b3bb8f34c3
Problembehebung mit Exception-Daten im Ergbebnis, die nicht geschrieben werden können, und Spidern einzelner Sites (#132) 
* WIP commit for single job execution

* Convert exception to string

* Pass more arguments

* Move python modules lsit into requirements.txt

* Document single site spidering

* Remove debugging
 2019-11-22 23:13:57 +01:00
Marian Steinbach 68f2288617
Prüfe DNS auf IPv6 AAAA Record (#124) 
* Add check for IPv6 AAAA record

* Adapt rating/resolvable
 2019-07-15 22:59:33 +02:00
Marian Steinbach 0c59111044
Alpine downgrade to 3.8 (#118) 2019-06-04 08:08:53 +02:00
Marian Steinbach 576050d3cd Update alpine repository URLs 2019-06-03 08:12:30 +02:00
Marian Steinbach 9e5426ccde Use alpine 3.9 base image 2019-05-03 23:20:08 +02:00
Marian Steinbach 3b9ead330d
Load feeds and gather info (#103) 2018-12-07 16:32:42 +01:00
Marian Steinbach ae6a2e83e9
Refactor and modularize spider (#70) 
See PR description for details
 2018-10-03 11:05:42 +02:00
Marian Steinbach 8580747e2a Docker file change from Debian stretch to Alpine 2018-09-12 00:42:40 +02:00
Marian Steinbach e83fd5ecc0 Update selenium to 3.14.0 2018-09-11 23:57:11 +02:00
Marian Steinbach 25e5fc936c Replace PhantomJS with Chromedriver 2018-09-11 23:39:30 +02:00
Marian Steinbach 545ea671d8 Add retry for get_job_from_queue 2018-08-27 22:40:31 +02:00
Marian Steinbach 0d9b44b384 Dockerfile update 
- remove unused certifi module
- add google-cloud-datastore module
- add export script
 2018-08-23 09:37:53 +02:00
Marian Steinbach 997519df35 More tests 2018-05-04 10:02:01 +02:00
Marian Steinbach 9bea186008 Clean up dependencies 2018-05-04 00:38:28 +02:00
Marian Steinbach 1e4cb2bce8 Run python unit tests in docker container 2018-05-03 12:01:30 +02:00
Marian Steinbach c21511cd62 Add phantomjs 2018-05-03 11:29:21 +02:00
Marian Steinbach 15697016e7 Change docker image to contain chrome 2018-05-03 10:56:04 +02:00
Marian Steinbach 4de5890c60 Change to run spider in Docker container 2018-05-03 10:22:10 +02:00
Marian Steinbach 7ce779e52d Add Dockerfile 2018-05-03 08:54:05 +02:00