Gau vs Katana

Pros of waybackurls

• Simple and lightweight tool focused on a single task
• Easy to use with minimal setup required
• Integrates well with Unix pipelines and other tools

Cons of waybackurls

• Limited functionality compared to gau and katana
• Lacks advanced filtering options
• May miss some URLs that more comprehensive tools can find

Code Comparison

waybackurls:

resp, err := http.Get(fmt.Sprintf("http://web.archive.org/cdx/search/cdx?url=%s/*&output=json&fl=original&collapse=urlkey", url))

gau:

for _, source := range sources {
    urls, err := source.Fetch(ctx, domain, &opts)
    if err != nil {
        return err
    }
    for url := range urls {
        // Process URL
    }
}

katana:

crawler, err := crawler.New(&crawler.Options{
    MaxDepth:    options.MaxDepth,
    FieldScope:  options.FieldScope,
    BodyReadSize: options.BodyReadSize,
    // ... other options
})

The code snippets show that waybackurls focuses on fetching URLs from the Wayback Machine, while gau and katana offer more comprehensive crawling and URL discovery capabilities. gau supports multiple sources, and katana provides advanced crawling options, reflecting their broader functionality compared to waybackurls' simpler approach.

Pros of hakrawler

• Lightweight and fast, with minimal dependencies
• Supports custom headers and cookies for authenticated crawling
• Provides flexible output options (JSON, plain text, etc.)

Cons of hakrawler

• Limited depth control compared to Katana's advanced crawling features
• Lacks some advanced filtering options available in gau
• May miss some URLs that gau can discover through passive sources

Code Comparison

hakrawler:

func crawl(url string, depth int, c *colly.Collector) {
    c.Visit(url)
}

gau:

func getUrls(domains []string, providers []string, client *http.Client) {
    for _, domain := range domains {
        for _, provider := range providers {
            provider.Fetch(domain, client)
        }
    }
}

Katana:

func (c *Crawler) Start() error {
    c.navigateURL(c.options.URL, nil)
    return c.wait()
}

All three tools serve similar purposes but with different approaches. hakrawler is simpler and faster, gau focuses on passive URL discovery, while Katana offers more advanced crawling features. The choice depends on specific use cases and requirements.

Pros of gospider

• Supports crawling JavaScript files and extracting URLs from them
• Offers concurrent crawling for faster performance
• Provides flexible output options, including JSON and Markdown formats

Cons of gospider

• May have a steeper learning curve compared to gau and katana
• Less frequent updates and maintenance than the other two projects
• Limited built-in filtering options compared to katana

Code Comparison

gospider:

func (s *Spider) Start() error {
    s.pool.Run()
    var wg sync.WaitGroup
    for _, site := range s.C.Sites {
        wg.Add(1)
        go func(site string) {
            defer wg.Done()
            s.crawl(site)
        }(site)
    }
    wg.Wait()
    return nil
}

gau:

func getUrls(domains []string, providers []string, threads int) {
    var wg sync.WaitGroup
    for _, domain := range domains {
        wg.Add(1)
        go func(domain string) {
            defer wg.Done()
            for _, provider := range providers {
                getUrlsProvider(domain, provider)
            }
        }(domain)
    }
    wg.Wait()
}

katana:

func (c *Crawler) Start() error {
    for _, seed := range c.options.URLs {
        if err := c.AddSeed(seed); err != nil {
            return err
        }
    }
    c.startWorkers()
    return nil
}

Pros of ffuf

• Fast and efficient web fuzzing tool
• Highly customizable with numerous options for fine-tuning
• Supports multiple input sources and output formats

Cons of ffuf

• Limited to web fuzzing, not as versatile as Katana or gau
• Requires more manual configuration compared to gau's simplicity
• May generate more noise in results compared to Katana's targeted approach

Code Comparison

ffuf

matcher := ffuf.NewMatcher(options.Matchers, options.Filters, options.MatcherOptions)
for _, word := range wordlist {
    result := ffuf.NewResult(url, word, matcher)
    if result.IsValid() {
        output.WriteResult(result)
    }
}

gau

func fetchURLs(domain string, providers []string) ([]string, error) {
    var urls []string
    for _, provider := range providers {
        providerURLs, err := fetchFromProvider(domain, provider)
        if err != nil {
            return nil, err
        }
        urls = append(urls, providerURLs...)
    }
    return urls, nil
}

Katana

func (c *Crawler) Start() error {
    for {
        select {
        case <-c.options.Context.Done():
            return nil
        case url := <-c.queue:
            c.processURL(url)
        }
    }
}

Pros of httpx

• Fast and efficient HTTP probing and analysis
• Supports multiple protocols (HTTP, HTTPS, HTTP/2)
• Extensive output customization options

Cons of httpx

• Limited URL discovery capabilities compared to gau
• Less comprehensive web crawling features than katana
• Focused primarily on HTTP interactions, not full-scale web exploration

Code Comparison

gau:

urls := make(chan string)
go func() {
    gau.FromGoogleAnalytics(domain, urls)
    gau.FromWaybackMachine(domain, urls)
    close(urls)
}()

katana:

crawler, _ := katana.New(&katana.Options{
    Depth:       3,
    MaxURLs:     1000,
    Concurrency: 10,
})
results := crawler.Crawl(url)

httpx:

httpxRunner, _ := httpx.New(&httpx.Options{
    Methods: "GET",
    Threads: 50,
})
result, _ := httpxRunner.URL(url)

Summary

httpx excels in fast HTTP probing and analysis, supporting multiple protocols with customizable output. However, it lacks the extensive URL discovery capabilities of gau and the comprehensive web crawling features of katana. gau focuses on URL discovery from various sources, while katana offers in-depth web crawling functionality. httpx is best suited for HTTP-specific tasks, whereas gau and katana are more specialized for URL discovery and web crawling, respectively.

Pros of gobuster

• Simple and straightforward CLI tool for directory/file, DNS, and VHost busting
• Supports multiple wordlists and customizable output formats
• Lightweight and fast, with minimal dependencies

Cons of gobuster

• Limited to specific enumeration tasks (directories, DNS, VHosts)
• Lacks advanced crawling and content discovery features
• May require additional tools for comprehensive web reconnaissance

Code Comparison

gobuster:

func main() {
    globalopts, err := parseGlobalOpts()
    if err != nil {
        fmt.Fprintf(os.Stderr, "%s\n", err)
        os.Exit(1)
    }
    if globalopts.NoStatus {
        libgobuster.Ruler = false
    }
    // ... (additional setup and execution code)
}

gau:

func main() {
    flag.Parse()
    if *version {
        fmt.Printf("gau version: %s\n", Version)
        os.Exit(0)
    }
    // ... (URL processing and output handling)
}

katana:

func main() {
    options := &types.Options{}
    parseOptions(options)
    runner, err := runner.New(options)
    if err != nil {
        gologger.Fatal().Msgf("Could not create runner: %s\n", err)
    }
    runner.Run()
}

While gobuster focuses on specific enumeration tasks, gau specializes in URL discovery from various sources, and katana offers more comprehensive web crawling and content discovery features. Each tool serves different purposes within the web reconnaissance ecosystem.