Debug howto: strange 403 error status

[fauust]

Hi!
I am facing a problem with one of our mirrors (at MariaDB Foundation) and I am not sure how to debug it.

The URL is https://mariadb.in.ssimn.org and the mirror is down because of the following:

Nov 08 12:08:46 hostname mirrorbits[1750095]: 2024/11/08 12:08:46.486 CET mariadb.in.ssimn.org Down! Status: 403

But the mirror looks OK and I did a quick go program to try to simulate HEAD requests on hardcoded URL but I am not a go programmer so, I might not be simulating what mirrorbits does (and I am not sure that I understand exactly how the random URL check is done).

package main

import (
	"fmt"
	"net/http"
	"os"
)

var (
	userAgent = "Mirrorbits/v0.5.1" + " PING CHECK"
)

func main() {
	// requestURL := fmt.Sprintf("https://mariadb.in.ssimn.org/TIME")
	requestURL := fmt.Sprintf("https://mariadb.in.ssimn.org//mariadb-11.2.6/galera-26.4.20/deb/galera-4-dbg_26.4.20-deb10_amd64.deb.asc")
	req, err := http.NewRequest(http.MethodHead, requestURL, nil)
	if err != nil {
		fmt.Printf("client: could not create request: %s\n", err)
		os.Exit(1)
	}

	res, err := http.DefaultClient.Do(req)
	if err != nil {
		fmt.Printf("client: error making http request: %s\n", err)
		os.Exit(1)
	}

	fmt.Printf("client: got response!\n")
	fmt.Printf("client: status code: %d\n", res.StatusCode)
}

❯ go run ./head.go 
client: got response!
client: status code: 200

I asked the mirror maintainer to check for errors in the webserver (apache) logs but he could not find anything relevant.

Can someone kindly point me to a documentation or a debug option to try to investigate this?

[elboulangero]

I am not sure that I understand exactly how the random URL check is done

Mirrorbits picks a file that is present on the mirror (as it scans the mirror on a regular basis), and also present on the source (ie. it doesn't pick extra files on the mirror, if any).

Did you run your test from the same machines as the mirrorbits instance? Maybe something on the mirror didn't like that there are too many requests (one per minute) coming from one particular IP, so this IP was blocked?

[fauust]

Thanks @elboulangero!

Did you run your test from the same machines as the mirrorbits instance? Maybe something on the mirror didn't like that there are too many requests (one per minute) coming from one particular IP, so this IP was blocked?

Yes that was my first thought (rate limit or something network related) so I did all the tests on the server where mirrorbits run (mirror.mariadb.org). And I asked the mirror maintainer if the IP was banned or rate limited. This is not the case and also confirmed by running some benchmarks with the hey tool.

[elboulangero]

You could also check mirrorbits logs mariadb.in.ssimn.org to see if there's something special there.

FWIW, connecting to https://mariadb.in.ssimn.org (either from web browser or wget) times out from where I am.

[fauust]

hm:

faust@hz-mirror:~$ mirrorbits logs mariadb.in.ssimn.org
Error: Command not found: logs
Usage: mirrorbits [OPTIONS] COMMAND [arg...]

A smart download redirector.

Server commands:
    daemon    Start the server

CLI commands:
    add       Add a new mirror
    disable   Disable a mirror
    edit      Edit a mirror
    enable    Enable a mirror
    export    Export the mirror database
    list      List all mirrors
    refresh   Refresh the local repository
    reload    Reload configuration
    remove    Remove a mirror
    scan      (Re-)Scan a mirror
    show      Print a mirror configuration
    stats     Show download stats
    upgrade   Seamless binary upgrade
    version   Print version information
faust@hz-mirror:~$ mirrorbits list | grep ssimn
mariadb.in.ssimn.org 		down 		(Wed, 21 Aug 2024 09:43:43 UTC)

Are we on the same version?

faust@hz-mirror:~$ mirrorbits version
Client:
 Version:          v0.5.1
 Build:            e83e56a-master
 GoVersion:        go1.11.2
 Operating System: linux
 Architecture:     amd64
 Gomaxprocs:       2

Server:
 Version:          v0.5.1
 Build:            e83e56a-master
 GoVersion:        go1.11.2
 Operating System: linux
 Architecture:     amd64
 Gomaxprocs:       2

FWIW, connecting to https://mariadb.in.ssimn.org/ (either from web browser or wget) times out from where I am.

Weird, I can't find any POP that times out on it an surely not the mirrorbits instance...

faust@hz-mirror:~$ wget https://mariadb.in.ssimn.org
--2024-11-08 16:24:27--  https://mariadb.in.ssimn.org/
Resolving mariadb.in.ssimn.org (mariadb.in.ssimn.org)... 94.136.185.162
Connecting to mariadb.in.ssimn.org (mariadb.in.ssimn.org)|94.136.185.162|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 4380 (4.3K) [text/html]
Saving to: ‘index.html’

index.html                 100%[======================================>]   4.28K  --.-KB/s    in 0s      

2024-11-08 16:24:27 (205 MB/s) - ‘index.html’ saved [4380/4380]

[elboulangero]

logs was added in commit f4c686c, it's not been released. I run mirrorbits from latest git commit.

I can reach the mirror now:

$ wget --spider -S https://mariadb.in.ssimn.org//mariadb-11.2.6/galera-26.4.20/deb/galera-4-dbg_26.4.20-deb10_amd64.deb.asc
Spider mode enabled. Check if remote file exists.
--2024-11-09 10:28:10--  https://mariadb.in.ssimn.org//mariadb-11.2.6/galera-26.4.20/deb/galera-4-dbg_26.4.20-deb10_amd64.deb.asc
Resolving mariadb.in.ssimn.org (mariadb.in.ssimn.org)... 94.136.185.162
Connecting to mariadb.in.ssimn.org (mariadb.in.ssimn.org)|94.136.185.162|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Date: Sat, 09 Nov 2024 03:28:10 GMT
  Server: Apache/2.4.62 (Unix) OpenSSL/3.0.7
  Last-Modified: Tue, 22 Oct 2024 17:32:41 GMT
  ETag: "341-6251428012040"
  Accept-Ranges: bytes
  Content-Length: 833
  Vary: User-Agent
  Keep-Alive: timeout=5, max=100
  Connection: Keep-Alive
  Content-Type: application/pgp-signature
Length: 833 [application/pgp-signature]
Remote file exists.

[jbkempf]

logs was added in commit f4c686c, it's not been released. I run mirrorbits from latest git commit.

We need a release :D