Marian Steinbach
eac5feb4f5
Kubernetes manifests: replace jobs with cronjobs
2018-11-28 22:19:03 +01:00
Marian Steinbach
678f319e73
Detect two more specific generators
2018-11-28 22:02:30 +01:00
Marian Steinbach
39cba1595a
Fix contact link rating
2018-11-23 22:16:26 +01:00
Marian Steinbach
3ba6940e94
Add criteria: social media links, contact link ( #90 )
...
* Add hyperlink checker
* Add rating for contact and social media links
* Update a comment
* Remove hyperlinks details from final payload
2018-11-20 22:47:34 +01:00
Marian Steinbach
4524cb5714
Consider site reachable only with status code < 400 ( #89 )
2018-11-20 20:14:52 +01:00
Marian Steinbach
c03ff21a9c
Simplify export ( #88 )
...
* Simplify exports
* Create file output in current working directory
2018-11-20 20:00:47 +01:00
Marian Steinbach
38481236ca
Add webapp deployment ( #87 )
...
* Add webapp deployment script
* Add some docs for webapp
* Some fixes in run-job.sh
* Update webapp deployment script
* Add some kubernetes job manifests
* Create index.yaml
* Remove local creation of the docker image from targets
* Update README.md
2018-11-20 19:54:23 +01:00
Marian Steinbach
924981659b
Allow Titillium together with Arvo ( #78 )
2018-11-05 23:18:11 +01:00
Marian Steinbach
325caee2bb
Detect generator jimdo ( #81 )
2018-11-05 23:00:01 +01:00
Marian Steinbach
df1f0bb452
Detect Drupal ( #80 )
2018-11-05 22:32:06 +01:00
Marian Steinbach
8ce8768465
Improve error messages in export
2018-10-08 08:42:29 +02:00
Marian Steinbach
0538e437ea
Fix NoneType error in rater responsive_layout
2018-10-07 21:14:29 +02:00
Marian Steinbach
18be6e7adf
Fix rater no_network_errors
2018-10-07 20:55:57 +02:00
Marian Steinbach
4251df6b06
Fixes for two problems found during spidering ( #75 )
2018-10-05 10:25:05 +02:00
Marian Steinbach
fd4a29da8e
Collect cookies in load_in_browser check ( #74 )
2018-10-04 21:21:30 +02:00
Marian Steinbach
2945372aaf
Fix README and Makefile ( #72 )
2018-10-04 21:20:53 +02:00
Marian Steinbach
c065da4957
More unittests for checks ( #73 )
...
* Add test for dns_resolution
* Add test for domain_variations
* Add test for duplicate_content
2018-10-03 22:43:22 +02:00
Marian Steinbach
57f8dea4e0
Improve certificate check to support SNI ( #71 )
...
* Fix the certificate check to support SNI
* Better tests for the certificate check
* Activate verbose output when running make test
* Add commenting on the spider test
2018-10-03 21:01:52 +02:00
Marian Steinbach
ae6a2e83e9
Refactor and modularize spider ( #70 )
...
See PR description for details
2018-10-03 11:05:42 +02:00
Marian Steinbach
7514aeb542
Add date and time to spider result export
2018-09-19 23:04:09 +02:00
Marian Steinbach
220b06fe79
Fix data export logging bug
2018-09-17 17:35:21 +02:00
Marian Steinbach
22d158c8b8
Fix docker image name in Makefile
2018-09-12 09:23:10 +02:00
Marian Steinbach
e64383c899
Add dev-shm to ignored list
2018-09-12 09:21:51 +02:00
Marian Steinbach
634ffa4a23
Merge pull request #67 from netzbegruenung/remove-phantomjs
...
Replace PhantomJS with Chromedriver
2018-09-12 09:02:52 +02:00
Marian Steinbach
8580747e2a
Docker file change from Debian stretch to Alpine
2018-09-12 00:42:40 +02:00
Marian Steinbach
eb9a29ac1c
Tweak chromedriver usage
2018-09-11 23:58:53 +02:00
Marian Steinbach
e83fd5ecc0
Update selenium to 3.14.0
2018-09-11 23:57:11 +02:00
Marian Steinbach
25e5fc936c
Replace PhantomJS with Chromedriver
2018-09-11 23:39:30 +02:00
Marian Steinbach
54b6d24b61
Update README.md
2018-08-28 22:39:19 +02:00
Marian Steinbach
d5e8f453aa
Merge pull request #65 from netzbegruenung/remove-webapp-code
...
Remove webapp code
2018-08-28 22:38:25 +02:00
Marian Steinbach
c642fd68fb
Update README
2018-08-28 22:28:40 +02:00
Marian Steinbach
fda97574cb
Remove webapp code
2018-08-28 22:28:34 +02:00
Marian Steinbach
fea2ce15b5
Merge pull request #64 from netzbegruenung/enable-screenshot-export
...
Enable screenshot export
2018-08-28 21:36:03 +02:00
Marian Steinbach
199d6da324
Uncomment screenshot export
2018-08-28 21:35:38 +02:00
Marian Steinbach
f2025a9476
Merge pull request #63 from netzbegruenung/add-license
...
Create LICENSE
2018-08-28 21:04:50 +02:00
Marian Steinbach
627384d1af
Merge pull request #62 from netzbegruenung/webapp-entry-types
...
Add support for RV and BV entry type display
2018-08-28 21:04:37 +02:00
Marian Steinbach
790faa20e3
Create LICENSE
2018-08-28 21:04:20 +02:00
Marian Steinbach
11dc9b6261
Add support for RV and BV
2018-08-28 20:59:39 +02:00
Marian Steinbach
0210e03d5d
Merge pull request #60 from netzbegruenung/download-icons
...
Remove downloading of icons
2018-08-28 20:51:05 +02:00
Marian Steinbach
e2077fe12b
Adapt display code for the favicon
2018-08-27 23:48:54 +02:00
Marian Steinbach
aa3b75add4
Data update with new icon data format
2018-08-27 23:39:28 +02:00
Marian Steinbach
a463087b0a
Add icon fetching to data export
2018-08-27 23:39:00 +02:00
Marian Steinbach
4abeb48cec
Add a site icon
2018-08-27 23:38:41 +02:00
Marian Steinbach
a001301121
Cleanup
2018-08-27 23:23:48 +02:00
Marian Steinbach
545ea671d8
Add retry for get_job_from_queue
2018-08-27 22:40:31 +02:00
Marian Steinbach
cb58b32c39
Remove icon downloading from spider
2018-08-27 21:17:14 +02:00
Marian Steinbach
77a736c61f
Comment makefile targets
2018-08-27 21:17:04 +02:00
Marian Steinbach
89f5f4fbe5
Merge pull request #59 from netzbegruenung/add-admin-scripts
...
Add devops scripts
2018-08-27 20:56:21 +02:00
Marian Steinbach
84e62b218a
Consolidate scripts
2018-08-27 20:50:18 +02:00
Marian Steinbach
aec2ce47f0
Adding devops scripts
2018-08-27 20:17:46 +02:00