Hilft Dir dabei, Deine BÜNDNIS 90/DIE GRÜNEN Website zu optimieren https://green-spider.netzbegruenung.de/
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

58 lines
1.8 KiB

Job-Verwaltung mit RQ, und vieles mehr (#149) * CLI: remove 'jobs' command, add 'manager' * Add job definition * Move jobs to manage folder * Rename jobs to manager * Add rq and redis dependencies * Add docker-compose YAML * Downgrade to alpine 3.8 * Adjust paths in Dockerfile, remove entrypoint * Rename 'make spiderjobs' to 'make jobs' * Fix docker exectution * Adapt 'make jobs' * Fix metadata scheme * Add docker dependency * Rendomize queue (a bit) * Use latest image, remove debug output * Make docker-compose file downwards-compatible * Use latest instead of dev image tag * Update docker-compose.yaml * Adapt job start script * Fix redis connection in manager * Add support for increasing timeout via environment variable * Adapt load_in_browser to cookies table schema change * Fix execution * Mitigate yaml warning * Bump some dependency versions * Report resource usage stats for each job * checks/load_in_browser: Return DOM size, prevent multiple page loads * Update .dockerignore * Code update * Script update * Update README.md * WIP * WIP commit * Update Dockerfile to alpine:edge and chromium v90 * Update TestCertificateChecker * Set defaults for __init__ function * Detect sunflower theme * Update unit test for new datetime (zero-basing) * Set logging prefs from Chromium in a new way * Move datastore client instantiation As it is not needed for all commands * Change green-directory repository URL * Add git settings for cloning green-directory * Pin alpine version 3.14, fix py3-cryptography * Use plain docker build progress output * Add volumes to 'make test' docker run command * Fix bug * Update example command in README * Update dependencies * Add creation of Kubernetes jobs
8 months ago
This rater evaluates the amount of data transferred for a page load.
Currently no score is given. The plan is however to reward site that
cause smaller transfers.
The rater uses Chrome performance log messages of type
from rating.abstract_rater import AbstractRater
class Rater(AbstractRater):
rating_type = 'number'
default_value = 0
depends_on_checks = ['load_in_browser']
max_score = 1.0
def __init__(self, check_results):
def rate(self):
value = self.default_value
score = 0
payloads_for_urls = []
for url in self.check_results['load_in_browser']:
payload = 0
if (self.check_results['load_in_browser'][url]['performance_log'] == [] or
self.check_results['load_in_browser'][url]['performance_log'] is None):
for lentry in self.check_results['load_in_browser'][url]['performance_log']:
if lentry['message']['method'] == 'Network.loadingFinished':
payload += lentry['message']['params']['encodedDataLength']
# Calculate score based on the largest value found for a URL.
# See https://github.com/netzbegruenung/green-spider/issues/11#issuecomment-600307544
# for details.
if len(payloads_for_urls) > 0:
value = max(payloads_for_urls)
if value < 994000:
score = 1
elif value < 1496000:
score = .5
return {
'type': self.rating_type,
'value': value,
'score': score,
'max_score': self.max_score,