Fix regression in --fail-http-to-httpsuri

[mzpqnxow]

Unless I missed a discussion, I think there was an unintentional regression in the code that supported the --fail-http-to-https flag. The flag is still implemented, and 99% of the logic is still there, it's just one small piece means the condition gets logged and a short-circuit occurs rather than a retry.

Refresher

The purpose of this was to provide some intelligent ways to deal with the small number of widely used implementations which, in certain configurations, responded to HTTP requests made on HTTPS ports. They provided HTTP 400 with a helpful error message - we identified a few in specific and supported those explicitly:

zgrab2/modules/http/scanner.go

Lines 569 to 576 in e91fc98

	sliceBuf := bodyText[:sliceLen]
	if strings.Contains(sliceBuf, "The plain HTTP request was sent to HTTPS port") ||
	strings.Contains(sliceBuf, "You're speaking plain HTTP") ||
	strings.Contains(sliceBuf, "combination of host and port requires TLS") ||
	strings.Contains(sliceBuf, "Client sent an HTTP request to an HTTPS server") {
	return zgrab2.NewScanError(zgrab2.SCAN_PROTOCOL_ERROR, errors.New("NGINX or Apache HTTP over HTTPS failure"))
	}
	}

Rather than treat it as a handshake failure, or any other sort of failure, it's treated as a special failure that just means "even though a valid HTTP response came back, retry using HTTPS"

Without this, HTTPS endpoints were just returning blank 400 pages to zgrab2, a very quiet data loss behavior.

How to Test (Server Side)

You can either set up nginx, apache, etc. with the configuration that emits one of the supported warnings, or you can simulate it with this hacky nonsense, which I have named friendly_server.py:

#!/usr/bin/env python3
# Will simulate servers that expect HTTPS, but will advise HTTP
# clients they should retry with HTTPS rather than just let the
# TLS handshake blow up
# For quick testing of --fail-http-to-https
# NOTE: zgrab2 will refuse to connect to RFC1918 addresses by
# default so you might want to create a temporary eth alias
import socket
import ssl
import threading

HOST = "0.0.0.0"
PORT = 8443

CERT_FILE = "pki.pem"
KEY_FILE = "pki.pem"

with open("pki.pem", mode="w") as fd:
    fd.write("""-----BEGIN CERTIFICATE-----
MIIDCTCCAfGgAwIBAgIUQ4qPGqiRya3PLzXEbPlPQdZTlBMwDQYJKoZIhvcNAQEL
BQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTI2MDYxMjAwNDY0NloXDTI3MDYx
MjAwNDY0NlowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEF
AAOCAQ8AMIIBCgKCAQEAp92H0VIpJFaWkI7f27Gb8QLKK9QWgGHVYNcUiUTrxdXH
f1g5az2BtDd/Ix5rSrjvW5sVFVdBLRT8iRbLRQGvCqCzktCuVc6t476DujGKH+Bd
quO0GJFB7X5U6eYhyUqpqyWI6YlewI48GM7CaGsE9q76nZ1mLgLCKV0sptSDoNA8
wtWMU3xndIX++6FQmrJFk15mOeF0zG6dgZh4ejdT9MrpFuFz2qjJ+U5jEQpxZKiH
junGUz/ivsLb+6ftf/nvLc+L7sPQAKErw5W42RKtiBQ0E/3HdDBaLOS2usu6jduG
eiiMZm8+38ef3MXMiIX7BTFpfODwVzDn9zzEBjkv8QIDAQABo1MwUTAdBgNVHQ4E
FgQUfRTTkEOPTZXW6j+Y96CxjaBvuT0wHwYDVR0jBBgwFoAUfRTTkEOPTZXW6j+Y
96CxjaBvuT0wDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAUSz8
NlyMv7to1MLULCZa3sdPHSHCxPd/nM0eJmyKP+NyE4pzxK/YfMcPkgiPdWL3zH3E
SYKcULlTYv2tUjQyt7nrzna9823/ERgVtg/eA4xqeFYJ5WdCXCBl94mWZouC0AyO
CxnpzSEzbrc7LLOjRMFJk952I/HMnA7l4mE+BA1eGGy4VCyKoWXJU/L3Crm4JLng
7hDvTJnQPwWKsJHSmPqgKikHlHuwAIrtumFaK8PbMYvZiMKabUxDGwHxtTFp+j+z
tm/Wy97jHmw05ROT8PQoRd/ccdbFJ/XG+pb/r/TthXB4Y5avS5y5vSjQAWly60K5
UWZZGOQM6MCPuPvI/Q==
-----END CERTIFICATE-----
-----BEGIN PRIVATE KEY-----
MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQCn3YfRUikkVpaQ
jt/bsZvxAsor1BaAYdVg1xSJROvF1cd/WDlrPYG0N38jHmtKuO9bmxUVV0EtFPyJ
FstFAa8KoLOS0K5Vzq3jvoO6MYof4F2q47QYkUHtflTp5iHJSqmrJYjpiV7AjjwY
zsJoawT2rvqdnWYuAsIpXSym1IOg0DzC1YxTfGd0hf77oVCaskWTXmY54XTMbp2B
mHh6N1P0yukW4XPaqMn5TmMRCnFkqIeO6cZTP+K+wtv7p+1/+e8tz4vuw9AAoSvD
lbjZEq2IFDQT/cd0MFos5La6y7qN24Z6KIxmbz7fx5/cxcyIhfsFMWl84PBXMOf3
PMQGOS/xAgMBAAECggEAA0hMKxYpssm0qP/6NaAL9hMBZAkCJaGEHCAoiSSdXaF4
BK/zc3rg6ea56DRkkbH7EDYAnAm4Pwrtgzq8X82QICuTR1goSIpBkTX8+muUkVoA
SuKn92EDwoKZY3d6CqcYmT2jJznl7er6FtwYJhWNo8s5IFLfGCA6rrdhM6PtLPxN
UAvi28lRZ8BS2b9yYcpttgW+tDY40ym7KVVjXT40dw2xW1d4SF8DfwX5+4veRhzk
pTNDhA5JAVAEtZNmWdBxy0NvEN3lRXIZDBgJGPYSAugGmeGLIoUrw6Q7aSfLFA8G
L/nK+7XtFyYVkWlLgF3QzQUFLOSaEUOJ59iSo80j1wKBgQDTS0seZn7aeH/BLBK0
eUAdHQ3Py+bHBrRoPh/7nkXCSUxS9htZwQBWEjQHIjx/f3SXPctC6t3XdtYSUEck
f/6SeS05F183tTrHs/6buPNXKd+pST5FDx9Ouj0zXwum6DvLSNq9+FZJoucGuKm/
BSLHzivGtoqIMJit4Q8Gb9ocDwKBgQDLYe0h+vBT7/4hbgMMJuCg1LaF25DLqQy/
FtAlPFEdjeXIj+DYgkaxn34Khg6a5qrymHQrcmk5NSRrD70XdxVPYS+xK7Oss83F
aJHIIDBSrU4cVw4DoW9HA/wagPdD64WDQhaUTCfHxq5QU8f4AZOCjg5JZZhv9+2X
zMTdxlvz/wKBgFRcPXO3zHIBplBD/nvb5UM6dGdAq7VO8fykwTQ+7u8xQK3m4Oim
7DFxNyCaBLmK88vmuh7Pkx8RrUrGjoD+MW/dvbaLpFKjowJU1RTrdQyaBVibi6ca
8I8IQKMgEZDkK+tv/RXX7uubhM3kMLG5pTdNE6FVBF53uJ3Du7IAOn71AoGAboZe
FsxHFGZkpYgicdUi12QheKb0jijlO2dLYCDiN/xn9lFyMFbD0j1JgrCNeVKHbeS5
pO69DhC6JNBdw0W8t/GpuE92ihEAbxo/txD/Eb8Kps4MWJ6/WUlqFIEnXqq8WLNu
j9K+/4sSKacOEdHRdu56BxsF9zs1y/vEtHzadcMCgYAj9+0oLm4/cPaULsnYDshE
leS9mQOmzGxUY/Ew2tOCDBw0xePhmDTcqQNlSTpjnFgc0w9cbT928KDYmnrsjlAp
HBTx+yb8CrgKsmT3TrotXPnTOtFNKNs4QdcWkQGDHtHXfMu5gyoOAKB8I9NoOQLr
3OvZ3WWzsRzxJ5jxxZ/3Gg==
-----END PRIVATE KEY-----
""")

BAD_REQUEST_BODY = "You're speaking plain HTTP"
BAD_REQUEST_RESPONSE = (
    "HTTP/1.1 400 Bad Request\r\n"
    "Content-Type: text/plain\r\n"
    f"Content-Length: {len(BAD_REQUEST_BODY)}\r\n"
    "Connection: close\r\n"
    "\r\n"
    f"{BAD_REQUEST_BODY}"
).encode("utf-8")

OK_BODY = "hello over tls\n"
OK_RESPONSE = (
    "HTTP/1.1 200 OK\r\n"
    "Content-Type: text/plain\r\n"
    f"Content-Length: {len(OK_BODY)}\r\n"
    "Connection: close\r\n"
    "\r\n"
    f"{OK_BODY}"
).encode("utf-8")


def is_plaintext_get(peeked: bytes) -> bool:
    return peeked.startswith(b"GET ")


def handle_client(raw_conn: socket.socket, client_addr, tls_context: ssl.SSLContext) -> None:
    raw_conn.settimeout(5)

    try:
        peeked = raw_conn.recv(8, socket.MSG_PEEK)
    except OSError:
        raw_conn.close()
        return

    if is_plaintext_get(peeked):
        try:
            raw_conn.sendall(BAD_REQUEST_RESPONSE)
        finally:
            raw_conn.close()
        return

    try:
        tls_conn = tls_context.wrap_socket(raw_conn, server_side=True)
    except ssl.SSLError:
        raw_conn.close()
        return

    with tls_conn:
        try:
            _ = tls_conn.recv(4096)
            tls_conn.sendall(OK_RESPONSE)
        except OSError:
            return


def main() -> None:
    tls_context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
    tls_context.load_cert_chain(certfile=CERT_FILE, keyfile=KEY_FILE)

    server_sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server_sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    server_sock.bind((HOST, PORT))
    server_sock.listen(128)

    print(f"Listening on {HOST}:{PORT} (TLS)")

    with server_sock:
        while True:
            conn, addr = server_sock.accept()
            thread = threading.Thread(target=handle_client, args=(conn, addr, tls_context), daemon=True)
            thread.start()


if __name__ == "__main__":
    main()

How To Test: zgrab Command

echo 2.2.2.1 | ./cmd/zgrab2/zgrab2 http -p 8443 --fail-http-to-https | jq '.'

Expected Output

{
  "ip": "2.2.2.1",
  "data": {
    "http": {
      "status": "success",
      "protocol": "http",
      "port": 8443,
      "result": {
        "response": {
          "status_line": "200 OK",
          "status_code": 200,
          "protocol": "HTTP/1.1",
          "protocol_major": 1,
          "protocol_minor": 1,
          "headers": {
            "content_length": [
              "15"
            ],
            "content_type": [
              "text/plain"
            ]
          },
          "body": "hello over tls\n",
          "body_sha256": "033aef276ffec11a316130c1434baea9657f0ab764e9adc999a860b9435168f2",
          "content_length": 15,
          "request": {
            "url": {
              "scheme": "https",
              "host": "2.2.2.1:8443",
              "path": "/"
            },
            "method": "GET",
            "protocol": "HTTP/1.1",
            "protocol_major": 1,
            "protocol_minor": 1,
            "headers": {
              "accept": [
                "*/*"
              ],
              "user_agent": [
                "Mozilla/5.0 zgrab/0.x"
              ]
            },
            "host": "2.2.2.1:8443",
            "tls_log": {
              "handshake_log": {
                "server_hello": {
                  "version": {
                    "name": "TLSv1.2",
                    "value": 771
                  },
                  "random": "ndhxIy7DkJ6VaLzVhsekF5uhT8PA7+EVl7/0hTY7Crg=",
                  "session_id": "HHHC6OMOb/kIVtLokHkP+UNpJ867Jns4goAUEysM9xY=",
                  "cipher_suite": {
                    "hex": "0x1302",
                    "name": "TLS_AES_256_GCM_SHA384",
                    "value": 4866
                  },
                  "compression_method": {
                    "hex": "0x00",
                    "name": "NULL",
                    "value": 0
                  },
                  "ocsp_stapling": false,
                  "ticket": false,
                  "secure_renegotiation": false,
                  "heartbeat": false,
                  "extended_master_secret": false,
                  "supported_versions": {
                    "selected_version": {
                      "name": "TLSv1.3",
                      "value": 772
                    }
                  },
                  "key_share": {
                    "hex": "0x001D",
                    "name": "x25519",
                    "value": 29
                  },
                  "extension_identifiers": [
                    43,
                    51
                  ]
                },
                "server_certificates": {
                  "certificate": {
                    "raw": "MIIDCTCCAfGgAwIBAgIUQ4qPGqiRya3PLzXEbPlPQdZTlBMwDQYJKoZIhvcNAQELBQAwFDESMBAGA1UEAwwJbG9jYWxob3N0MB4XDTI2MDYxMjAwNDY0NloXDTI3MDYxMjAwNDY0NlowFDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAp92H0VIpJFaWkI7f27Gb8QLKK9QWgGHVYNcUiUTrxdXHf1g5az2BtDd/Ix5rSrjvW5sVFVdBLRT8iRbLRQGvCqCzktCuVc6t476DujGKH+BdquO0GJFB7X5U6eYhyUqpqyWI6YlewI48GM7CaGsE9q76nZ1mLgLCKV0sptSDoNA8wtWMU3xndIX++6FQmrJFk15mOeF0zG6dgZh4ejdT9MrpFuFz2qjJ+U5jEQpxZKiHjunGUz/ivsLb+6ftf/nvLc+L7sPQAKErw5W42RKtiBQ0E/3HdDBaLOS2usu6jduGeiiMZm8+38ef3MXMiIX7BTFpfODwVzDn9zzEBjkv8QIDAQABo1MwUTAdBgNVHQ4EFgQUfRTTkEOPTZXW6j+Y96CxjaBvuT0wHwYDVR0jBBgwFoAUfRTTkEOPTZXW6j+Y96CxjaBvuT0wDwYDVR0TAQH/BAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAUSz8NlyMv7to1MLULCZa3sdPHSHCxPd/nM0eJmyKP+NyE4pzxK/YfMcPkgiPdWL3zH3ESYKcULlTYv2tUjQyt7nrzna9823/ERgVtg/eA4xqeFYJ5WdCXCBl94mWZouC0AyOCxnpzSEzbrc7LLOjRMFJk952I/HMnA7l4mE+BA1eGGy4VCyKoWXJU/L3Crm4JLng7hDvTJnQPwWKsJHSmPqgKikHlHuwAIrtumFaK8PbMYvZiMKabUxDGwHxtTFp+j+ztm/Wy97jHmw05ROT8PQoRd/ccdbFJ/XG+pb/r/TthXB4Y5avS5y5vSjQAWly60K5UWZZGOQM6MCPuPvI/Q==",
                    "parsed": {
                      "version": 3,
                      "serial_number": "385592350610891532648099422262299092915500323859",
                      "signature_algorithm": {
                        "name": "SHA256-RSA",
                        "oid": "1.2.840.113549.1.1.11"
                      },
                      "issuer": {
                        "common_name": [
                          "localhost"
                        ]
                      },
                      "issuer_dn": "CN=localhost",
                      "validity": {
                        "start": "2026-06-12T00:46:46Z",
                        "end": "2027-06-12T00:46:46Z",
                        "length": 31536000
                      },
                      "subject": {
                        "common_name": [
                          "localhost"
                        ]
                      },
                      "subject_dn": "CN=localhost",
                      "subject_key_info": {
                        "key_algorithm": {
                          "name": "RSA"
                        },
                        "rsa_public_key": {
                          "exponent": 65537,
                          "modulus": "p92H0VIpJFaWkI7f27Gb8QLKK9QWgGHVYNcUiUTrxdXHf1g5az2BtDd/Ix5rSrjvW5sVFVdBLRT8iRbLRQGvCqCzktCuVc6t476DujGKH+BdquO0GJFB7X5U6eYhyUqpqyWI6YlewI48GM7CaGsE9q76nZ1mLgLCKV0sptSDoNA8wtWMU3xndIX++6FQmrJFk15mOeF0zG6dgZh4ejdT9MrpFuFz2qjJ+U5jEQpxZKiHjunGUz/ivsLb+6ftf/nvLc+L7sPQAKErw5W42RKtiBQ0E/3HdDBaLOS2usu6jduGeiiMZm8+38ef3MXMiIX7BTFpfODwVzDn9zzEBjkv8Q==",
                          "length": 2048
                        },
                        "fingerprint_sha256": "50e2c65e7440bd6d5e1560abc2c461bf5f47a8b3a837db7fbafb1d4885f29888"
                      },
                      "extensions": {
                        "basic_constraints": {
                          "is_ca": true
                        },
                        "authority_key_id": "7d14d390438f4d95d6ea3f98f7a0b18da06fb93d",
                        "subject_key_id": "7d14d390438f4d95d6ea3f98f7a0b18da06fb93d"
                      },
                      "signature": {
                        "signature_algorithm": {
                          "name": "SHA256-RSA",
                          "oid": "1.2.840.113549.1.1.11"
                        },
                        "value": "USz8NlyMv7to1MLULCZa3sdPHSHCxPd/nM0eJmyKP+NyE4pzxK/YfMcPkgiPdWL3zH3ESYKcULlTYv2tUjQyt7nrzna9823/ERgVtg/eA4xqeFYJ5WdCXCBl94mWZouC0AyOCxnpzSEzbrc7LLOjRMFJk952I/HMnA7l4mE+BA1eGGy4VCyKoWXJU/L3Crm4JLng7hDvTJnQPwWKsJHSmPqgKikHlHuwAIrtumFaK8PbMYvZiMKabUxDGwHxtTFp+j+ztm/Wy97jHmw05ROT8PQoRd/ccdbFJ/XG+pb/r/TthXB4Y5avS5y5vSjQAWly60K5UWZZGOQM6MCPuPvI/Q==",
                        "valid": true,
                        "self_signed": true
                      },
                      "fingerprint_md5": "8928f54c4edf0ec3abd948f428ed70f4",
                      "fingerprint_sha1": "1f732f323862ca587e4cb10604d565c6281e7eb2",
                      "fingerprint_sha256": "4061fd5a466185b94329cc3a3bc503440878429393be058210b5ab9e049434dd",
                      "tbs_noct_fingerprint": "362c588e6ac9e80a72e63c5d5eb9af9a37dc37c720a30dab25bc493c547a7642",
                      "spki_subject_fingerprint": "8fa3bb1a409b639928481b58c1cd0f7990a73cda1396eb05e5feaa567e1f9d1a",
                      "tbs_fingerprint": "362c588e6ac9e80a72e63c5d5eb9af9a37dc37c720a30dab25bc493c547a7642",
                      "validation_level": "unknown",
                      "redacted": false
                    }
                  },
                  "validation": {
                    "browser_trusted": false,
                    "browser_error": "x509: certificate is self-signed and not a trusted root"
                  }
                }
              },
              "ja3s": "15af977ce25de452b96affa2addb1036",
              "handshake_completed_successfully": true
            }
          }
        }
      },
      "timestamp": "2026-06-11T20:56:39-04:00"
    }
  }
}

Actual Output

{
  "ip": "2.2.2.1",
  "data": {
    "http": {
      "status": "protocol-error",
      "protocol": "http",
      "port": 8443,
      "result": {
        "response": {
          "status_line": "400 Bad Request",
          "status_code": 400,
          "protocol": "HTTP/1.1",
          "protocol_major": 1,
          "protocol_minor": 1,
          "headers": {
            "content_length": [
              "45"
            ],
            "content_type": [
              "text/plain"
            ]
          },
          "content_length": 45,
          "request": {
            "url": {
              "scheme": "http",
              "host": "2.2.2.1:8443",
              "path": "/"
            },
            "method": "GET",
            "protocol": "HTTP/1.1",
            "protocol_major": 1,
            "protocol_minor": 1,
            "headers": {
              "accept": [
                "*/*"
              ],
              "user_agent": [
                "Mozilla/5.0 zgrab/0.x"
              ]
            },
            "host": "2.2.2.1:8443"
          }
        }
      },
      "timestamp": "2026-06-11T20:56:59-04:00",
      "error": "NGINX or Apache HTTP over HTTPS failure"
    }
  }
}

It detected the unique situation, but it didn't perform the retry, which means it ultimately didn't return any data

Notes & Caveats

• Unless there was some discussion since the initial implementation that I missed, I believe this is the desired behavior
• The way I chose to implement it is not dear to me, if the concept is good, I'm happy to tweak the details or have any maintainer tweak it for me

Issue Tracking

• Initial issue for this was [PR Merged] Improving retry-https by reading HTTP 400 and/or adding --retry-http #273
• Initial PR for this was Feature fail http to https for known status code and substrings #308
• It likely regressed some time during all the great enhancements being made over the past year or two

Screenshot (Regression)

Screenshot (Fixed)

[phillip-stephens]

Added an integration test, but otherwise this looks good! Seems reasonable and like you say, the code is already there. Thanks for this!

[mzpqnxow]

Added an integration test, but otherwise this looks good! Seems reasonable and like you say, the code is already there. Thanks for this!

Damn, you’re fast! Thanks 🙏