fix stdin parsing #2021

dogancanbakir · 2024-12-03T13:45:10Z

Closes #2020

noaho · 2024-12-05T08:02:03Z

This pull request breaks -vhost-input for me, see #2020

dogancanbakir · 2024-12-05T13:57:30Z

@noaho I'll update this PR -was gatherin more info on this feature.

noaho · 2024-12-09T07:27:33Z

If it helps my understanding of the feature is this:

Sometimes you have hosts that you need to specify by IP, to do vhost discovery. (Send different Hosts: header for the same IP)

You might also want to process multiple hosts / vhosts in one session without starting up multiple HTTPX.

So you specify each vhost/IP pair on stdin like this:
cat inputfile.txt | httpx -vhost-input -title

You would of course have a input file with multiple vhosts you want to try on an IP (and multiple hosts you want to test) like this:

inputfile.txt

example.org,https://93.184.215.14
example2.org,https://93.184.215.14
www.google.com,https://74.125.68.147

This should give output like this:

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

                projectdiscovery.io

[INF] Current httpx version v1.6.9 (latest)
[WRN] UI Dashboard is disabled, Use -dashboard option to enable
https://93.184.215.14 [Example Domain]
https://74.125.68.147 [Google]
http://93.184.215.14 [404 - Not Found]

dogancanbakir · 2025-01-07T11:24:36Z

@noaho I've made some updates. Please give it another try and share your thoughts. Thanks!

This reverts commit ff00cee.

Mzack9999 · 2025-01-14T22:46:38Z

runner/runner.go

@@ -1302,6 +1302,29 @@ func (r *Runner) RunEnumeration() {
 	}
 }

+func parseVhostInput(input string) (hostname, ip string, err error) {
+	// Expecting format: host[ip]
+	if !strings.Contains(input, "[") || !strings.HasSuffix(input, "]") {


I'm not sure if we also should keep supporting the old format:

target,vhost

since we are introducing a breaking change. What do you think?

I know you weren't asking me, but my 2c anyway is that the old format with , was poorly (if at all) documented and I had to work it out from the source. I don't think anyone knows about it / is using it

So if the new format host[ip] could be documented somewhere then the new format is probably better

I believe we can go with the new format and +1 to document it.

noaho · 2025-01-21T03:51:36Z

Hi @dogancanbakir,

I tried the changes with both formats and it seems the new format isn't passing through the URL properly - the end web server seems to see only IP address, no Host: or sni.

While the old format seems to ignore the scheme, and always probe with https.

go install github.com/projectdiscovery/httpx/cmd/httpx@4dc7f90 # (this is the right commit, right? it's the last one you did)

# Make sure you re-resolve example.org if stealing my examples, the IP did change for me once
cat inputfile.txt
example.org,http://23.215.0.133
example.org,https://23.215.0.133

cat inputfile2.txt
http://example.org[23.215.0.133]
https://example.org[23.215.0.133]

cat inputfile.txt | httpx -vhost-input -json -no-fallback-scheme | jq

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

                projectdiscovery.io

[INF] Current httpx version v1.6.9 (latest)
[WRN] UI Dashboard is disabled, Use -dashboard option to enable
{
  "timestamp": "2025-01-21T03:49:49.243668565Z",
  "port": "443",
  "url": "https://23.215.0.133",
  "input": "example.org,https://23.215.0.133",
  "title": "Example Domain",
  "scheme": "https",
  "content_type": "text/html",
  "method": "GET",
  "host": "23.215.0.133",
  "path": "/",
  "time": "32.104988ms",
  "a": [
    "23.215.0.133"
  ],
  "tech": [
    "HTTP/3"
  ],
  "words": 298,
  "lines": 46,
  "status_code": 200,
  "content_length": 1256,
  "failed": false,
  "knowledgebase": {
    "PageType": "nonerror",
    "pHash": 0
  }
}
{
  "timestamp": "2025-01-21T03:49:49.24683302Z",
  "port": "443",
  "url": "https://23.215.0.133",
  "input": "example.org,http://23.215.0.133",
  "title": "Example Domain",
  "scheme": "https",
  "content_type": "text/html",
  "method": "GET",
  "host": "23.215.0.133",
  "path": "/",
  "time": "34.903798ms",
  "a": [
    "23.215.0.133"
  ],
  "tech": [
    "HTTP/3"
  ],
  "words": 298,
  "lines": 46,
  "status_code": 200,
  "content_length": 1256,
  "failed": false,
  "knowledgebase": {
    "PageType": "nonerror",
    "pHash": 0
  }
}

Note the scheme is https, twice. It didn't try on http.

cat inputfile2.txt | httpx -vhost-input -json -no-fallback-scheme | jq

    __    __  __       _  __
   / /_  / /_/ /_____ | |/ /
  / __ \/ __/ __/ __ \|   /
 / / / / /_/ /_/ /_/ /   |
/_/ /_/\__/\__/ .___/_/|_|
             /_/

                projectdiscovery.io

[INF] Current httpx version v1.6.9 (latest)
[WRN] UI Dashboard is disabled, Use -dashboard option to enable
{
  "timestamp": "2025-01-21T03:51:23.06037323Z",
  "port": "80",
  "url": "http://23.215.0.133",
  "input": "http://example.org[23.215.0.133]",
  "title": "Invalid URL",
  "scheme": "http",
  "webserver": "AkamaiGHost",
  "content_type": "text/html",
  "method": "GET",
  "host": "23.215.0.133",
  "path": "/",
  "time": "7.496144ms",
  "a": [
    "23.215.0.133"
  ],
  "words": 8,
  "lines": 8,
  "status_code": 400,
  "content_length": 310,
  "failed": false,
  "knowledgebase": {
    "PageType": "error",
    "pHash": 0
  }
}
{
  "timestamp": "2025-01-21T03:51:23.074694035Z",
  "port": "443",
  "url": "https://23.215.0.133",
  "input": "https://example.org[23.215.0.133]",
  "title": "Invalid URL",
  "scheme": "https",
  "webserver": "AkamaiGHost",
  "content_type": "text/html",
  "method": "GET",
  "host": "23.215.0.133",
  "path": "/",
  "time": "21.762277ms",
  "a": [
    "23.215.0.133"
  ],
  "words": 8,
  "lines": 8,
  "status_code": 400,
  "content_length": 310,
  "failed": false,
  "knowledgebase": {
    "PageType": "error",
    "pHash": 0
  }
}

Note it didn't pass the correct Host: or SNI

Cheers!

noaho · 2025-01-24T03:38:57Z

the other problem I have with this functionality is when you use the screenshot function, it saves the screenshots as the IP address only, so when you have multiple vhosts/IP pairs it just overwrites and corrupts them after the first one..

noaho · 2025-02-18T08:16:27Z

@dogancanbakir I tried the latest change (set customip) and it might be working but it breaks -proxy.. It seems to try to connect to the passed vhost on the proxy port as the proxy..

Was trying to use burp suite to verify that your changes covered all cases and then came across that issue.

inputfile2.txt
http://www.example.org[23.205.104.21]

Success:

cat inputfile2.txt | httpx -json -no-fallback-scheme -debug -vhost-input

[INF] Current httpx version v1.6.9 (outdated)
[WRN] UI Dashboard is disabled, Use -dashboard option to enable
[INF] Dumped HTTP request for http://www.example.org

GET / HTTP/1.1
Host: www.example.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1.1 Safari/605.1.15
Accept-Charset: utf-8
Accept-Encoding: gzip
<snip>
{"timestamp":"2025-02-18T08:10:39.204453175Z","port":"80","url":"http://www.example.org","input":"http://www.example.org[23.205.104.21]","title":"Example Domain","scheme":"http","content_type":"text/html","method":"GET","host":"23.205.104.21","path":"/","time":"10.905078ms","a":["23.218.217.177","23.218.217.154"],"aaaa":["2600:1408:ec00:36::1736:7f24","2600:1408:ec00:36::1736:7f2e"],"cname":["www.example.org-v2.edgesuite.net","a1519.dscr.akamai.net"],"words":298,"lines":46,"status_code":200,"content_length":1256,"failed":false,"knowledgebase":{"PageType":"nonerror","pHash":0},"resolvers":["1.1.1.1:53","1.0.0.1:53"]}

But with -proxy:

cat inputfile2.txt | httpx -json -no-fallback-scheme -debug -vhost-input -proxy http://127.0.0.1:8080
[INF] Current httpx version v1.6.9 (outdated)
[WRN] UI Dashboard is disabled, Use -dashboard option to enable
[INF] Dumped HTTP request for http://www.example.org

GET / HTTP/1.1
Host: www.example.org
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36
Accept-Charset: utf-8
Accept-Encoding: gzip

[DBG] Failed 'http://www.example.org': GET http://www.example.org giving up after 1 attempts: Get "http://www.example.org": read tcp <MY_IP>:39920->23.205.104.21:8080: read: connection reset by peer

Note that it connected to port 8080, which is the port of my proxy, not http://www.example.org

Also it seems to be always resolving DNS, even when we provide the IP we want to connect on. I'm not sure if that's a bug or not, but I was trying to verify that it will connect to the provided IP with burp (and not just the one in DNS)

fix stdin parsing

ff00cee

dogancanbakir requested a review from ehsandeep December 3, 2024 13:45

dogancanbakir self-assigned this Dec 3, 2024

dogancanbakir linked an issue Dec 3, 2024 that may be closed by this pull request

Scheme https is forced for -vhost-input even when using -no-fallback-scheme #2020

Open

dogancanbakir removed the request for review from ehsandeep December 5, 2024 13:57

dogancanbakir added 2 commits January 7, 2025 14:23

add vhost parsing

136ed07

Merge branch 'dev' into fix_stdin_parsing

ea54119

dogancanbakir requested review from Mzack9999 and ehsandeep January 7, 2025 11:24

Revert "fix stdin parsing"

4dc7f90

This reverts commit ff00cee.

Mzack9999 requested changes Jan 14, 2025

View reviewed changes

dogancanbakir added 2 commits January 27, 2025 11:56

Merge branch 'dev' into fix_stdin_parsing

e8f7294

set customip

725dbe8

dogancanbakir requested a review from Mzack9999 January 27, 2025 09:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix stdin parsing #2021

fix stdin parsing #2021

dogancanbakir commented Dec 3, 2024

noaho commented Dec 5, 2024

dogancanbakir commented Dec 5, 2024

noaho commented Dec 9, 2024

dogancanbakir commented Jan 7, 2025

Mzack9999 Jan 14, 2025

noaho Jan 21, 2025

dogancanbakir Jan 27, 2025

noaho commented Jan 21, 2025 •

edited

Loading

noaho commented Jan 24, 2025

noaho commented Feb 18, 2025

fix stdin parsing #2021

Are you sure you want to change the base?

fix stdin parsing #2021

Conversation

dogancanbakir commented Dec 3, 2024

noaho commented Dec 5, 2024

dogancanbakir commented Dec 5, 2024

noaho commented Dec 9, 2024

inputfile.txt

dogancanbakir commented Jan 7, 2025

Mzack9999 Jan 14, 2025

Choose a reason for hiding this comment

noaho Jan 21, 2025

Choose a reason for hiding this comment

dogancanbakir Jan 27, 2025

Choose a reason for hiding this comment

noaho commented Jan 21, 2025 • edited Loading

noaho commented Jan 24, 2025

noaho commented Feb 18, 2025

noaho commented Jan 21, 2025 •

edited

Loading