6 commits

Author	SHA1	Message	Date
Marian Steinbach	b3bb8f34c3	Problembehebung mit Exception-Daten im Ergbebnis, die nicht geschrieben werden können, und Spidern einzelner Sites (#132) ... * WIP commit for single job execution * Convert exception to string * Pass more arguments * Move python modules lsit into requirements.txt * Document single site spidering * Remove debugging	2019-11-22 23:13:57 +01:00
Marian Steinbach	5e8347916c	Fehlerbehebung im url_reachability check (#108) ... * Fix detection of redirects to bad domains * Fix bad domain check * Add --url flag to spider for faster debugging * Pass args to make spider * Add spidering of a single URL for debugging purposes * Fix tests * Fix test in CI * Remove pip upgrade	2019-04-19 00:35:28 +02:00
Marian Steinbach	3b8328d804	Fixing several bugs in spider code	2018-12-17 17:31:09 +01:00
Marian Steinbach	3ba6940e94	Add criteria: social media links, contact link (#90) ... * Add hyperlink checker * Add rating for contact and social media links * Update a comment * Remove hyperlinks details from final payload	2018-11-20 22:47:34 +01:00
Marian Steinbach	38481236ca	Add webapp deployment (#87) ... * Add webapp deployment script * Add some docs for webapp * Some fixes in run-job.sh * Update webapp deployment script * Add some kubernetes job manifests * Create index.yaml * Remove local creation of the docker image from targets * Update README.md	2018-11-20 19:54:23 +01:00
Marian Steinbach	ae6a2e83e9	Refactor and modularize spider (#70) ... See PR description for details	2018-10-03 11:05:42 +02:00