This was around from the time when we handled timezones sanely and
Django did not; now that we are on 1.4 we no longer need our own code to
handle this.
Signed-off-by: Dan McGee <dan@archlinux.org>
It isn't worth it, as we run into the 999 max SQL statement variables
issue when using it on any significant amount of mirrors. Since this is
just a development database setup, and it isn't a command we need to run
especially fast, we can ditch it.
Signed-off-by: Dan McGee <dan@archlinux.org>
Managed to see this bubble up today when running the mirrorcheck command
on a less than ideal connection that was experiencing timeouts at the
wrong time.
Signed-off-by: Dan McGee <dan@archlinux.org>
The main changes in this patch implement rsync:// protocol checking
support by calling the rsync binary, requested in FS#29878. We track and
log much of the same things as we already do for FTP and HTTP URLs-
check time, last sync, total check duration, etc.
Also added in this patch is a configurable timeout value which defaults
to the previous hardcoded value of 10 seconds; this can be passed as an
option to the mirrorcheck command.
Signed-off-by: Dan McGee <dan@archlinux.org>
98% of the time, we won't need to update the existing values as it will
be the same as the prior run of this command. Do a quick check of the
old and new values and don't send anything to the database if there is
no need for an update.
Signed-off-by: Dan McGee <dan@archlinux.org>
This is most of the transition to Django 1.4 `USE_TZ = True`. We need to
ensure we don't mix aware and non-aware datetime objects when dealing
with datetimes in the code. Add a utc_now() helper method that we can
use most places, and ensure there is always a timezone attached when
necessary.
Signed-off-by: Dan McGee <dan@archlinux.org>
Now that we aren't seeing odd segfaults and hung tasks, we can remove
the traceback stuff from the scripts. Also use the 'io' module only, it
has been long enough.
Signed-off-by: Dan McGee <dan@archlinux.org>
We should be encouraging our mirrors to serve us the original file, not
something they create and come up with.
Signed-off-by: Dan McGee <dan@archlinux.org>
This prevents people from having to mess with these checkboxes at all in
the admin, and we incur no delay on their initial values being correct
waiting for the cron job to run.
Signed-off-by: Dan McGee <dan@archlinux.org>
We know we are doing updates when setting IPv4/IPv6 information, so set
force_update to True to save the useless select query on each save(). For
mirror checks, use a less cumbersome deque for the results since it is also
thread-safe, and have all the log entries committed in one go.
Signed-off-by: Dan McGee <dan@archlinux.org>
Don't use 'fmtstr % (arg1, arg2)' type format; logger can be passed a format
string and the arguments to populate it. Saves a bit of work for strings
that never end up getting displayed anyway.
Signed-off-by: Dan McGee <dan@archlinux.org>
We were seeing a lot of hangs and long-running never-ending processes. This
might be due to some multithreading issues within Django, so move the save()
calls to a loop after the join() on the threads doing the mirror polling.
Signed-off-by: Dan McGee <dan@archlinux.org>
Don't import all the constants from logging, just use logging.* instead.
Also, fix some typos that somehow snuck into one of my commits.
Signed-off-by: Dan McGee <dan@archlinux.org>
Dan:
* Fix up some style issues such as spacing between operators
* Ensure one failed lookup doesn't crash the whole script
* Be silent out of the box if there are no errors, just like mirrorcheck
Signed-off-by: PyroPeter <abi1789@googlemail.com>
Signed-off-by: Dan McGee <dan@archlinux.org>
We were seeing processes hang on the Arch server. It looks like there are
ways for socket.timeout to come out of the main check code, so add another
except block to catch this case. In addition, make sure we always call
task_done() even on failures so processes eventually die.
Signed-off-by: Dan McGee <dan@archlinux.org>
On an HTTP 404, FTP 550, or inability to parse the lastsync file, record the
duration of the check even though we couldn't get a time from the mirror.
This allows for these checks to show up as completed but in error, which is
more what.
Previously, inability to parse the date was also recorded as a success, so
change that to be a failure and record an error message with it.
Signed-off-by: Dan McGee <dan@archlinux.org>
This does the actual work of going out and checking the mirror status. In
short, it polls every active mirror URL for the 'lastsync' file and then
records the appropriate details. These include the contents of that file,
how long the total time to retrieve took, and any errors encountered.
In order to finish up a bit faster, we spawn several threads to do the
actual work. This parallelization allows the whole check process to take
around 30 seconds rather than several minutes.
Signed-off-by: Dan McGee <dan@archlinux.org>
