[BUG] Unclear error "appsecQuery:unreachable"

[ZzenlD]

Describe the bug 🐛
I am running crowdsec and traefik in the container, but traefik always displays the following error message:

ERROR: CrowdsecBouncerTraefikPlugin: 2025/05/09 09:55:14 appsecQuery:unreachable

Unfortunately, switching to debug level does not provide any more information.

Expected behavior 👀
The Traefik forwards the request to crowdsec-appsec, is successfully validated and the user is shown the website.

Context 🔎

# traefik.yml
log:
  level: DEBUG

accessLog: {}

experimental:
  plugins:
    bouncer:
      moduleName: github.com/maxlerebourg/crowdsec-bouncer-traefik-plugin
      version: v1.4.2

entryPoints:
  port-443:
    address: ":443"
    asDefault: true
    http:
      tls:
        certResolver: letsencrypt
      middlewares:
        - crowdsec-bouncer

#config.yml (dynamic config of traefik)
http:
  middlewares:
    crowdsec-bouncer:
      plugin:
        bouncer:
          enabled: true
          crowdsecAppsecEnabled: true
          crowdsecLapiScheme: https
          crowdsecLapiHost: <external-lapi-url>
          crowdsecLapiKey: <lapi-secret>
          crowdsecLapiTLSInsecureVerify: true

#acquis.yaml
source: appsec
listen_addr: 0.0.0.0:7422
appsec_config: crowdsecurity/appsec-default
labels:
  type: appsec

traefik-log:

ERROR: CrowdsecBouncerTraefikPlugin: 2025/05/09 09:55:14 appsecQuery:unreachable
10.89.1.2 - - [09/May/2025:09:55:14 +0000] "GET / HTTP/2.0" 403 0 "-" "-" 1 "test-server@file" "-" 72ms

crowdsec-log:

me="2025-05-09T11:55:14+02:00" level=error msg="UnmarshalJSON : invalid character 'E' looking for beginning of value" line="ERROR: CrowdsecBouncerTraefikPlugin: 2025/05/09 09:55:14 appsecQuery:unreachable"
time="2025-05-09T11:55:14+02:00" level=warning msg="failed to run filter : invalid character 'E' looking for beginning of value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=red-waterfall name=child-crowdsecurity/traefik-logs stage=s01-parse
time="2025-05-09T11:55:14+02:00" level=error msg="UnmarshalJSON : invalid character 'M' looking for beginning of value" line="May 09 11:55:14 knx-gateway traefik_traefik_1[264039]: ERROR: CrowdsecBouncerTraefikPlugin: 2025/05/09 09:55:14 appsecQuery:unreachable"
time="2025-05-09T11:55:14+02:00" level=warning msg="failed to run filter : invalid character 'M' looking for beginning of value (1:1)\n | UnmarshalJSON(evt.Parsed.message, evt.Unmarshaled, \"traefik\") in [\"\", nil]\n | ^" id=red-waterfall name=child-crowdsecurity/traefik-logs stage=s01-parse

I have successfully tested the following:

• crowdsec can reach the LAPI and log in
• The traefik bouncer is successfully registered with the LAPI and can reach it
• traefik can ping the crowdsec container and reach it on port 7422
• The following wget command executed inside the traefik-container is successful:

wget -O- \
  --header="x-crowdsec-appsec-api-key: <lapi-secret>" \
  --header="x-crowdsec-appsec-ip: 1.2.3.4" \
  --header="x-crowdsec-appsec-uri: /test" \
  --header="x-crowdsec-appsec-host: test.com" \
  --header="x-crowdsec-appsec-verb: GET" \
  --post-data='' \
  http://crowdsec:7422/

Version (please complete the following information):

• OS: OpenSuse
• Traefik version: 3.0.3
• Plugin version: 1.4.2
• Redis ? : not used

[mathieuHa]

Hi,
Could you provide a docker-compose, kubernetes manifest or something so we can reproduce this ?

Which version of Crowdsec are you using ?

[mathieuHa]

Just a thought, you mentionned crowdsecLapiHost: <external-lapi-url> which I guess means, your LAPI is not just a colocated container.
You didn't specify variable:

CrowdsecAppsecHost
    string
    default: "crowdsec:7422"
    Crowdsec Appsec Server available on which host and port. The scheme will be handled by the CrowdsecLapiScheme var.

If your appsec/crowdsec is remote and you are using Appsec, you have to specify it also

[ZzenlD]

Of course, I use the following compose file for traefik:

version: "3.9"
services:
  traefik:
    image: docker.io/traefik:v3.0.3
    hostname: traefik
    read_only: true
    security_opt:
      - no-new-privileges
    tmpfs:
      - /plugins-storage
    ports:
      - 80:80
      - 443:443
    volumes:
      - ./traefik/config:/etc/traefik:z,ro
      - ./traefik/var/letsencrypt/acme.json:/letsencrypt/acme.json:z
    labels:
      - io.containers.autoupdate=registry
    healthcheck:
      test: traefik healthcheck --ping
    networks:
      proxy:
        ipv4_address: 10.89.1.2
    environment:
      - NAMECHEAP_API_USER=<api-user>
      - NAMECHEAP_API_KEY=<api-key>
    restart: always

networks:
  proxy:
    external:
      name: proxy

I run the LAPI centrally on a host (crowdsec.example.com), while the log reading and the AppSec component runs separately on each host and can be accessed by traefik via crowdsec:7422.

If your appsec/crowdsec is remote and you are using Appsec, you have to specify it also

That means I have to specify crowdsecAppsecHost: crowdsec:7422 explicitly, because I have set a different host with crowdsecLapiHost and this is then also used as the AppSec host by default?

[mathieuHa]

Hi @ZzenlD,

If I understand correctly LAPI is working as https://domainlapi, and Traefik, appsec run in a host and can talk using docker dns style crowdsec:7422 from Traefik.

I see one thing that might be the issue:

From documentation for the CrowdsecAppsecHost var The scheme will be handled by the CrowdsecLapiScheme var.

You config mentionned crowdsecLapiScheme: https, but you accessed it with http above

wget -O-
--header="x-crowdsec-appsec-api-key: "
--header="x-crowdsec-appsec-ip: 1.2.3.4"
--header="x-crowdsec-appsec-uri: /test"
--header="x-crowdsec-appsec-host: test.com"
--header="x-crowdsec-appsec-verb: GET"
--post-data=''
http://crowdsec:7422/

Running http and https for both is not supported currently but will be in the future, it is tracked in the issue: #131

Please let me know if that helps you

[ZzenlD]

Yes, the LAPI is accessible via HTTPS, AppSec only locally via HTTP. So I fear that this is the problem.

However, everyone in a distributed setup is likely to have this problem, as probably very few have a central AppSec instance. After all, in such a case, every request would otherwise have to be transmitted over the network.
Is there already a time frame until when HTTPS and HTTP are possible in parallel? Or is there a workaround?

[gunpowder62]

Hello,

to solve the http issue, just create a dns record for Appsec like crowdsec-appsec.your-domain and use a dynamic routing on your traefik instance.

http:
routes:
crowdsec-appsec:
entryPoints:
- websecure
rule: "Host(crowdsec-appsec.your-domain)"
tls: {}
service: crowdsec-appsec-service
services:
crowdsec-appsec-service:
loadBalancer:
servers:
- url: "http://your-lapi-host-ip:7422"