04_fan_out_fan_in

🔄 Fan-Out & Fan-In Combined Pattern

🌟 Introduction

The Fan-Out & Fan-In Combined Pattern is a powerful concurrency pattern in Go, where tasks are distributed to multiple worker goroutines (Fan-Out), and their results are merged into a single channel (Fan-In). This pattern is ideal for scenarios where tasks need to be processed concurrently, and the results are required to be aggregated.

• Fan-Out: Distributes a single source of tasks to multiple worker goroutines.
• Fan-In: Collects results from multiple goroutines into a single channel for further processing.

🧩 How It Works

Fan-Out:

• A single input source is distributed to multiple goroutines (workers) for concurrent processing.
• Each worker handles a part of the task independently.

Fan-In:

• The results from all workers are sent to a single channel, consolidating data from multiple sources.

Combined Workflow:

The combined Fan-Out & Fan-In workflow ensures that tasks are distributed efficiently and their results are collected centrally.

🔑 Diagram

Here is a diagram illustrating the Fan-Out & Fan-In workflow, emphasizing the 1 → n → 1 structure:

         Input
           |
           v
         -----
       | Fan-Out |
         -----
        |  |  |
        v  v  v
  Worker  Worker  Worker
     |     |      |
     v     v      v
        ------
           |
           v
    Output (Fan-In)

• Input: A single source of tasks.
• Fan-Out: Tasks are distributed among multiple workers (goroutines) for concurrent processing.
• Workers: Process tasks independently and send results to a shared channel.
• Fan-In: Results from all workers are merged into a single channel for collection.

🚀 Example Code

Here is a Go implementation of the Fan-Out & Fan-In Combined Pattern:

package main

import (
	"fmt"
	"sync"
)

func worker(id int, jobs <-chan int, results chan<- int, wg *sync.WaitGroup) {
	defer wg.Done() // Ensure worker finishes its task

	for job := range jobs {
		fmt.Printf("Worker %d is processing job %d\n", id, job)
		results <- job * 2 // Simulate task: doubling the value
	}
}

func main() {
	jobs := make(chan int, 10)   // Channel for tasks
	results := make(chan int, 10) // Channel for collecting results
	var wg sync.WaitGroup

	// Create 3 workers
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go worker(i, jobs, results, &wg)
	}

	// Send tasks to the jobs channel (Fan-Out)
	for j := 1; j <= 5; j++ {
		jobs <- j
	}
	close(jobs) // Close the jobs channel after sending all tasks

	// Wait for all workers to finish their tasks
	wg.Wait()
	close(results) // Close the results channel after completion

	// Collect results from the results channel (Fan-In)
	for result := range results {
		fmt.Println("Result:", result)
	}
}

📌 Code Explanation

Channels:

• jobs channel: Holds the tasks (integer values) to be processed by the workers.
• results channel: Receives the results from the worker goroutines.

Worker Function:

• Each worker processes tasks received from the jobs channel and sends results to the results channel.

Synchronization:

• sync.WaitGroup: Ensures that the main program waits for all worker goroutines to complete before closing the results channel.

Goroutines:

• Each worker runs as a separate goroutine, allowing concurrent task processing.

🔑 Benefits of the Fan-Out & Fan-In Pattern

1. Concurrency: Multiple goroutines process tasks simultaneously, improving efficiency.
2. Result Aggregation: Results from multiple workers are consolidated into a single channel for streamlined processing.
3. Scalability: Easily add more workers to handle increased workloads without significant changes to the main program.
4. Efficiency: Optimizes task distribution and result collection, reducing processing time.

⚡ Things to Consider

1. Synchronization:
   • Use sync.WaitGroup or other synchronization mechanisms to ensure all workers complete their tasks before exiting.
2. Channel Management:
   • Ensure channels are properly closed after usage to avoid deadlocks or data loss.
3. Worker Coordination:
   • Properly coordinate worker goroutines to send results into the shared channel without conflicts.

🎯 Conclusion

The Fan-Out & Fan-In Combined Pattern is a robust design for concurrent processing in Go. It efficiently distributes tasks across multiple goroutines (Fan-Out) and consolidates their results into a single channel (Fan-In). This pattern is especially useful when you need to balance workload distribution and result collection in a seamless manner.