Commit Graph

32 Commits

Author SHA1 Message Date
Marian Steinbach a001301121 Cleanup 2018-08-27 23:23:48 +02:00
Marian Steinbach 545ea671d8 Add retry for get_job_from_queue 2018-08-27 22:40:31 +02:00
Marian Steinbach cb58b32c39 Remove icon downloading from spider 2018-08-27 21:17:14 +02:00
Marian Steinbach 3a7a553534 Add creating of a job for a particular URL 2018-08-23 10:01:22 +02:00
Marian Steinbach 88ec1f63f7 Big spider.py overhaul 2018-08-23 09:36:33 +02:00
Marian Steinbach 8fefc6c0c1 Add detection for more strings 2018-08-10 16:10:04 +02:00
Marian Steinbach c7523ac47f Add CMS detection for josephknowsbest 2018-08-10 11:26:02 +02:00
Marian Steinbach ccdf954524 Data update, screenshot update 2018-05-30 14:54:45 +02:00
Marian Steinbach d8cfbddf3a Set data path back to whole Germany 2018-05-25 19:27:44 +02:00
Marian Steinbach 45c4d72326 Add icon pull logic 2018-05-25 19:11:09 +02:00
Marian Steinbach 8a7cf17701 Add volume for icons 2018-05-25 19:10:54 +02:00
Christian Tramnitz 8df5c66813 Move append into for-loop. This allows mutliple web sites per entry to be added. 2018-05-10 22:47:49 +02:00
Marian Steinbach 997519df35 More tests 2018-05-04 10:02:01 +02:00
Marian Steinbach 16ca2699d0 Ensure stable result sorting 2018-05-03 13:55:17 +02:00
Marian Steinbach cf656ad2d1
Merge pull request #33 from netzbegruenung/unittests
Unit tests und CI
2018-05-03 11:55:06 +02:00
Marian Steinbach 0450cfaac2 Make function output deterministic 2018-05-03 11:44:00 +02:00
Marian Steinbach e15051952e Fix YAML file encoding problem 2018-05-03 11:29:11 +02:00
Marian Steinbach 4de5890c60 Change to run spider in Docker container 2018-05-03 10:22:10 +02:00
Marian Steinbach 914fb1f35e Detect and report responsive layout details 2018-04-23 09:11:50 +02:00
Marian Steinbach 7dd451c3b6 Increase score for HTTPS 2018-04-21 20:37:53 +02:00
Marian Steinbach 94241ba380 Add detection for most common CMSes
and write info to report
2018-04-20 23:36:00 +02:00
Marian Steinbach 278261db26 Some stability improvements 2018-04-19 11:36:14 +02:00
Marian Steinbach ea05bf5192 Move test criteria logic into spider, add scoring 2018-04-19 00:51:59 +02:00
Marian Steinbach 9672e41dba Add icon retrieval 2018-04-09 23:02:12 +02:00
Marian Steinbach 313f56f39e Set JSON output to UTF-8 encoding 2018-04-09 22:40:17 +02:00
Marian Steinbach 48ee1ab8a1 Add content checks 2018-04-09 22:39:53 +02:00
Marian Steinbach 5438818043 Set stable sorting for all arrays
to minimize the diffs between results submissions
2018-04-06 00:06:54 +02:00
Marian Steinbach 3513eddfe6 Update results, spider logic and webapp 2018-04-05 18:11:40 +02:00
Marian Steinbach 56995067ec New test logic, new results 2018-04-04 21:14:16 +02:00
Marian Steinbach fb634fc1f3 Reduce concurrency to reduce variance in timing results 2018-04-03 23:25:14 +02:00
Marian Steinbach b3da3ccc12 Round duration time in milliseconds 2018-04-03 23:24:53 +02:00
Marian Steinbach 220a6ba629 First working code and results 2018-04-03 23:15:28 +02:00