Commit Graph

404 Commits

Author SHA1 Message Date
Jelle van der Waa
c6d91ecbd4
Fix readlinks import and rename inotify read links db 2021-04-28 23:16:57 +02:00
Jelle van der Waa
fdf5e3b6fc
Add links database reading and API
The $repo.links.tar.gz databases provided in the repositories are now
parsed by archweb, to be able to expose them via a simple API and to be
able to list the linked sonames in the package details overview.
2021-04-28 22:47:05 +02:00
Jelle van der Waa
7a109722aa
Add orphan not reproducible developer report
Fixes: #289
2021-04-28 22:34:55 +02:00
Jelle van der Waa
d7d3a6ede4
Add diffoscope/log link to reproducible devel report
For these reports to be useful, show a direct link to diffoscope and the
build logs. So our developers can more easily debug non reproducible
packages.
2021-04-28 22:34:35 +02:00
Jelle van der Waa
71e7724b4b
Add new report for orphan required packages
This dashboard lists packages which are orphan but required by other
packages and should get a maintainer.
2021-04-28 22:34:31 +02:00
Jelle van der Waa
9bc48164c5
Implement Last-Modifed header support for rebuilderd
Rebuilderd now supports the Last-Modified header and If-Modified-Since
when requesting the rebuilderd package list. This saves us from
iterating over all rebuilderd results and fetching the status form the
database.
2021-04-18 23:36:41 +02:00
Jelle van der Waa
6bb8a45d0b
Add retry mechanism for reporead updates
When reporead_inotify is unable to reach the postgresql server an
exception is thrown and nothing is retried which means db updates are
missed. This change adds a simple retry mechanism which retries up till
5 times * 5 seconds and after that the change is ignored.

Closes: #336
2021-03-10 22:13:37 +01:00
Jelle van der Waa
adb5eb8b1a
Fix trailing spaces in read_rebuilderd_status 2021-03-09 23:58:18 +01:00
Jelle van der Waa
e2466a3456
Improve importing performance by not saving for every updated
The Django object model does not check if the model was modified when
calling .save() so will always generate SQL queries for statuses which
are not changed.

Closes: #334
2021-03-09 23:54:25 +01:00
Jelle van der Waa
a1de7b9585
Use a lookup table for arches/repos for rebuilderd import
To reduce the amount of SQL queries for looking up which repo/arch the
package is in, prefetch them.
2021-03-09 23:54:24 +01:00
Jelle van der Waa
5c4c7fa709
Fix typo statusses 2021-03-09 22:58:12 +01:00
Jelle van der Waa
acc14afd5d
Import the build_id for the rebuilderd status
To add links to logs and diffoscope output a build_id has to be recorded
in our Rebuilderd model as links contain the build_id in their url.
2021-01-31 17:14:49 +01:00
Jelle van der Waa
3aeffc4ec2
Update rebuilderd import command
Rename read_reproducible_status to read_rebuilderd_status as we are
importing the status from rebuilderd.
2021-01-31 16:58:27 +01:00
Justin !
87977c599b Fix pep8 error 2020-11-11 22:23:29 +01:00
Justin !
c260e57e3c Do not lint migrations 2020-11-11 22:23:29 +01:00
Justin !
c6cf91a40d Fix pep8 errors 2020-11-11 22:23:29 +01:00
Jelle van der Waa
17fd534a4f Add core.db.tar.zst support for reporead
As Python does not support zstd compression yet, xtarfile a wrapper
around tarfile with zstd support is required.
2020-10-26 21:28:21 +01:00
Jelle van der Waa
c358541801
Only show bad packages in reports
To avoid confusion with reproducible.archlinux.org output, only show BAD
statusses. Unknown statuses are still queue'd up in rebuilderd so aren't
interesting.
2020-07-09 21:50:48 +02:00
Jelle van der Waa
21d1e50da8
Expand debug output for rebuilderd importer 2020-07-09 21:50:31 +02:00
Jelle van der Waa
1a15dea670 Add rebuilderd status import and reporting functionality
Import the rebuilderd status periodically with a django management
command into RebuilderdStats which holds one record per pkgname with
it's pkgver/pkgrel/epoch all recorded. Shown as a developer dashboard
and with opt in mail notifications for when a package becomes not
reproducible.
2020-06-29 16:28:27 +02:00
Jelle van der Waa
d0bafe48c4 Create migration for planet_rss help message. 2020-06-29 01:54:24 +02:00
Jelle van der Waa
17a12f07fb Introduce planet functionality in archweb
This change introduces a replacment for planet.archlinux.org which uses
a python 2 project to generate static html from multiple RSS feed
sources. For archweb a set of 'static' feeds can be created in the
django admin view for the Arch forums and other static feeds, archweb
users can add their own blog rss feed in their profile which will create
a Feed model.

When running the update_planet command, all Feed models are iterated
over and the rss feed is parsed. The latest FeedItem is queried matching
the current Feed model and every newer entry in the RSS feed is added as
new FeedItem. Since the body is also stored in the FeedItem there is a
limit to the amount of FeedItems per Feed configured in settings.py of
which the default is 25.

When a user is marked as inactive his Feed model and items are
removed automatically to avoid keeping stale data around.

Closes: #261
2020-02-03 21:25:31 +01:00
Felix Yan
4443a1c159 Fix a typo in commands/retire_user.py 2019-11-07 14:12:05 +01:00
Johannes Löthberg
2685a08098 devel/views: Invalidate template fragment cache on profile update
Closes #166.

Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2019-10-04 01:57:30 +02:00
Jelle van der Waa
40a955670a devel: Add sorting on groups to the admin view 2019-09-11 21:14:23 +02:00
Rowan Ruseler
850263374e trailing spaces 2019-06-28 11:56:13 +02:00
Evangelos Foutras
89af0c7799
reporead: skip sanity check for public testing repos 2019-06-22 13:40:41 +03:00
Jelle van der Waa
e3584454a3 devel: add non existing dependencies report
Add a new devel report which shows packages which have non-existing
make, opt or normal dependencies.

Closes: #111
2019-05-17 22:46:50 +02:00
Jelle van der Waa
8ef0c1a1b9 reporead: remove added list()
2to3 added an unrequired list() call around the django ORM query.
2019-04-27 21:35:49 +02:00
Johannes Löthberg
7d162a0953 reporead: Delete old FlagRequests when deleting package from repo
Signed-off-by: Johannes Löthberg <johannes@kyriasis.com>
2019-03-19 20:57:22 +01:00
Jelle van der Waa
b737f2b4c3 Revert "rewrite logic to put some key at the beginning of the OrderedDict"
This reverts commit adaa6d64c4.

old is undefined and therefore the code is broken.
2019-02-24 16:50:15 +01:00
Jelle van der Waa
e58c11fd52 tests: add test for pgp_import keys
Add a test to check the pgp_import functionality with one key, by
patching call_pgp so there is no need of a local pacman keyring.
2019-02-09 16:13:14 +01:00
Felix Yan
bdcfe16287 Fix also call_gpg's output 2019-02-09 14:58:48 +01:00
Aleksandar Trifunovic
203506c9cc
remove unecessary list calls 2019-02-01 21:27:57 +01:00
Aleksandar Trifunovic
adaa6d64c4
rewrite logic to put some key at the beginning of the OrderedDict 2019-02-01 21:27:57 +01:00
Jelle van der Waa
da18435177
switch to python3's buildin mock
Instead of using a python module use the build-in mock and remove the
module from requirements.txt
2019-01-28 21:02:39 +01:00
Jelle van der Waa
b8b85e7f21 Update migrations for Python 3 2019-01-21 21:19:58 +01:00
Jelle van der Waa
fcc810bfdf Update django reverse import 2019-01-21 21:19:58 +01:00
Jelle van der Waa
86f5b0ad27 In Python 3.2 assertItemsEqual is replaced by assertCountEqual 2019-01-21 21:19:58 +01:00
Frank Vanderham
0ca04132d0 Refactor Python 3 __unicode__ to __str__
In Python 3 for Django, class method __unicode__ must be changed to
__str__. This commit replaces all models that use __unicode__ to
__str__.
2019-01-21 21:19:58 +01:00
Jelle van der Waa
c612b19314 main: run 2to3 2019-01-21 21:19:58 +01:00
Jelle van der Waa
4bf7406cbf devel: switch to force_text 2019-01-21 21:19:58 +01:00
Jelle van der Waa
b911b3308e Remove all squashed migrations
this makes it easier to upgrade to Python 3 since less on_delete lines
have to be fixed.
2019-01-21 21:19:58 +01:00
jelle van der Waa
21211d3cb4
Merge pull request #154 from jelly/squashmigrations
squash all migrations
2018-12-16 15:29:02 +01:00
Jelle van der Waa
4572b1f3a7 devel: fix class name in test 2018-11-30 21:03:03 +01:00
Jelle van der Waa
890937ac9f devel: add pgp_import basic test 2018-11-30 21:02:53 +01:00
Jelle van der Waa
cdce8a4f88 deve: test listing actual developers 2018-11-23 20:46:40 +01:00
Jelle van der Waa
68e96cb4a1 devel: Add test for unsupported architecture 2018-11-20 22:34:04 +01:00
Jelle van der Waa
3fc26cdb27 Squash all migrations 2018-11-17 21:58:50 +01:00
Jelle van der Waa
9354c4407d Fix Foreignkey implicit on_delete on django < 2.0
Foreignkey used to implicitly be on_delete=models.CASCADE and has to be
explicit now.
2018-11-17 15:38:12 +01:00