support/scripts/pkg-stats: check_url_status(): use HEAD requests to limit server load
The gnu.org admins have been blocking the IP address of machines running pkg-stats as the GET requests for the (many) packages with gnu.org URLs are seen as abusive. The resource body is not used, so use a HTTP HEAD request instead of a GET to limit server load and bandwidth use. Signed-off-by: Peter Korsgaard <peter@korsgaard.com> Signed-off-by: Julien Olivain <ju.o@free.fr>
This commit is contained in:
committed by
Julien Olivain
parent
2b4ee4e72f
commit
f9f3e6ccc6
@@ -469,7 +469,7 @@ async def check_url_status(session, pkg, npkgs, retry=True, verbose=False):
|
||||
global check_url_count
|
||||
|
||||
try:
|
||||
async with session.get(pkg.url) as resp:
|
||||
async with session.head(pkg.url) as resp:
|
||||
if resp.status >= 400:
|
||||
pkg.status['url'] = ("error", "invalid {}".format(resp.status))
|
||||
check_url_count += 1
|
||||
|
||||
Reference in New Issue
Block a user