Commit Graph

257 Commits

Author SHA1 Message Date
Markus Günther e59b05fc6c
Adds sunflower theme detection (#215) 2021-06-29 21:17:11 +02:00
dependabot-preview[bot] abb6fa5460
Upgrade to GitHub-native Dependabot (#203)
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2021-05-04 21:53:11 +02:00
b90g d55330b678
changed gcms ip (#199) 2021-01-18 11:25:45 +01:00
Marian Steinbach bc934b7479
Add dependabot configuration 2020-06-12 09:33:36 +02:00
dependabot-preview[bot] aedb4b3acc
Merge pull request #169 from netzbegruenung/dependabot/pip/urllib3-1.25.9 2020-06-12 07:26:37 +00:00
dependabot-preview[bot] 40b474cbe8
Merge pull request #175 from netzbegruenung/dependabot/pip/responses-0.10.15 2020-06-12 07:26:11 +00:00
dependabot-preview[bot] cc59deb262
Bump responses from 0.10.12 to 0.10.15
Bumps [responses](https://github.com/getsentry/responses) from 0.10.12 to 0.10.15.
- [Release notes](https://github.com/getsentry/responses/releases)
- [Changelog](https://github.com/getsentry/responses/blob/master/CHANGES)
- [Commits](https://github.com/getsentry/responses/compare/0.10.12...0.10.15)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-06-12 06:51:49 +00:00
dependabot-preview[bot] 18793b6968
Bump urllib3 from 1.25.7 to 1.25.9
Bumps [urllib3](https://github.com/urllib3/urllib3) from 1.25.7 to 1.25.9.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/master/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/1.25.7...1.25.9)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>
2020-04-17 07:07:23 +00:00
Norbert Tretkowski 7a8547b34e
Liste der Qualitätskriterien aktualisiert (#155) 2020-03-24 23:26:57 +01:00
dependabot-preview[bot] 02677ac1de
Bump responses from 0.10.7 to 0.10.12 (#161)
Bumps [responses](https://github.com/getsentry/responses) from 0.10.7 to 0.10.12.
- [Release notes](https://github.com/getsentry/responses/releases)
- [Changelog](https://github.com/getsentry/responses/blob/master/CHANGES)
- [Commits](https://github.com/getsentry/responses/compare/0.10.7...0.10.12)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
2020-03-21 00:55:17 +01:00
dependabot-preview[bot] 47ecf4024b
Bump beautifulsoup4 from 4.6.0 to 4.8.1 (#145) 2019-12-03 18:06:33 +00:00
dependabot-preview[bot] 30c5aee565
Bump httpretty from 0.9.4 to 0.9.7 (#138) 2019-11-27 12:55:33 +00:00
dependabot-preview[bot] a5d37cacea
Bump smmap2 from 2.0.3 to 2.0.5 (#146) 2019-11-26 15:28:25 +00:00
dependabot-preview[bot] e3165f25dc
Bump requests from 2.20.0 to 2.22.0 (#143) 2019-11-26 15:27:55 +00:00
dependabot-preview[bot] e24c2507e6
Bump responses from 0.9.0 to 0.10.7 (#147) 2019-11-26 15:13:17 +00:00
dependabot-preview[bot] 9789af1688
Bump urllib3 from 1.24.2 to 1.25.7 (#140) 2019-11-25 11:06:34 +00:00
Marian Steinbach ff6eae3955
Upgrade to alpine 3.9 (#135)
* Upgrade to alpine 3.9

* Set Python version to 3.7 (#136)
2019-11-24 23:54:26 +01:00
dependabot[bot] 7c75e1194c
Bump urllib3 from 1.22 to 1.24.2 (#133) 2019-11-22 22:55:15 +00:00
dependabot[bot] 01a0000316
Bump requests from 2.18.4 to 2.20.0 (#134) 2019-11-22 22:26:40 +00:00
Marian Steinbach b3bb8f34c3
Problembehebung mit Exception-Daten im Ergbebnis, die nicht geschrieben werden können, und Spidern einzelner Sites (#132)
* WIP commit for single job execution

* Convert exception to string

* Pass more arguments

* Move python modules lsit into requirements.txt

* Document single site spidering

* Remove debugging
2019-11-22 23:13:57 +01:00
Marian Steinbach 725ed5439d Set share memory size 2019-11-22 08:39:56 +01:00
Marian Steinbach 802a977fe5 Add required secrets for screenshots job 2019-11-22 08:38:49 +01:00
Marian Steinbach 7788166b0c Fix logging output in dns_resolution 2019-11-22 08:38:26 +01:00
Marian Steinbach 68f2288617
Prüfe DNS auf IPv6 AAAA Record (#124)
* Add check for IPv6 AAAA record

* Adapt rating/resolvable
2019-07-15 22:59:33 +02:00
Marian Steinbach 4401683133 Add shared memory to docker container 2019-07-14 22:51:05 +02:00
Marian Steinbach 1087cc25ec Little devops/README update 2019-07-14 22:50:44 +02:00
Marian Steinbach 0c59111044
Alpine downgrade to 3.8 (#118) 2019-06-04 08:08:53 +02:00
Marian Steinbach 576050d3cd Update alpine repository URLs 2019-06-03 08:12:30 +02:00
Marian Steinbach aac5fb3192 Some Makefile updates 2019-06-03 08:08:55 +02:00
Marian Steinbach 2f6cd9a304 Add extra docker run flags 2019-06-03 08:08:37 +02:00
Marian Steinbach c41e717d3d Update devops stuff 2019-05-05 22:42:48 +02:00
Marian Steinbach 9f7efddc61 Merge branch 'master' of https://github.com/netzbegruenung/green-spider 2019-05-05 22:41:34 +02:00
Marian Steinbach 04a1e98b79
Prüfe Existenz von /favicon.ico und werte dies ebenso wie ein Icon, das im HTML Head verlinkt ist (#115)
* Fix full JSON export

* Update ignore list

* Update README

* Check for /favicon.ico and rate it as icon available

* Remove broken cookies test
2019-05-05 22:26:41 +02:00
Marian Steinbach cc6a52bf45 Update README 2019-05-04 23:01:04 +02:00
Marian Steinbach 620610b48e Update ignore list 2019-05-04 23:00:56 +02:00
Marian Steinbach ab942ca91d Fix full JSON export 2019-05-04 23:00:00 +02:00
Marian Steinbach 9e5426ccde Use alpine 3.9 base image 2019-05-03 23:20:08 +02:00
Marian Steinbach cff4d55f17 Fix problem where pageload was not counted 2019-05-03 22:54:25 +02:00
Marian Steinbach 7621b7ef75 Remove debugging output 2019-05-03 22:54:05 +02:00
Marian Steinbach 56f9f1ba86 Check third party cookies 2019-04-29 10:09:25 +02:00
Marian Steinbach 5e8347916c
Fehlerbehebung im url_reachability check (#108)
* Fix detection of redirects to bad domains

* Fix bad domain check

* Add --url flag to spider for faster debugging

* Pass args to make spider

* Add spidering of a single URL for debugging purposes

* Fix tests

* Fix test in CI

* Remove pip upgrade
2019-04-19 00:35:28 +02:00
Marian Steinbach 2dfcf61cc0
Add netzbegruenung/green-spider-indexer to README 2019-04-11 22:43:16 +02:00
Marian Steinbach 16a05b751b Several fixes for edge cases 2018-12-17 23:54:09 +01:00
Marian Steinbach 3b8328d804 Fixing several bugs in spider code 2018-12-17 17:31:09 +01:00
Marian Steinbach 3b9ead330d
Load feeds and gather info (#103) 2018-12-07 16:32:42 +01:00
Marian Steinbach 3063a4488d
Detect frameset (#102)
* Add frameset checker

* Remove unused variable (unrelated)
2018-12-07 16:31:56 +01:00
Marian Steinbach deff95306b
Extend CMS detection for Urwahl3000 theme (#96)
* Extend check for Urwahl3000 theme
* Remove unused import
2018-12-05 21:27:45 +01:00
Marian Steinbach d0e3a4210f
Fix link raters (social media links, contact link) (#95)
* Fix rating for contact_link and social_media_link

* Skip checks when dependencies not met
2018-11-28 23:46:40 +01:00
Marian Steinbach eac5feb4f5 Kubernetes manifests: replace jobs with cronjobs 2018-11-28 22:19:03 +01:00
Marian Steinbach 678f319e73 Detect two more specific generators 2018-11-28 22:02:30 +01:00