02_fan_out

➡️ Fan-Out Pattern

🌟 Introduction

The Fan-Out Pattern is a parallel programming design pattern where a single source of work (typically a goroutine) distributes tasks to multiple worker goroutines for concurrent processing. This pattern is particularly useful when you want to optimize the processing of similar tasks that can be handled concurrently.

• Fan-Out refers to "distributing" tasks to multiple entities for processing.
• In Go, this pattern is implemented using channels.

🧩 How Fan-Out Pattern Works

1. Source of work: A channel will hold the data (or tasks) that need to be processed.
2. Task distribution: One or more worker goroutines receive tasks from the channel and process them.
3. Results: The workers can either return results through another channel or process them further.

🔑 Fan-Out Pattern Diagram

The diagram helps visualize the Fan-Out pattern:

  Source of work
         |
         v
     ---------
    |       |
    |       |
  Worker   Worker
    |       |
    v       v
  Results   Results

• Source of work: A channel that holds tasks.
• Worker: Each worker receives tasks from the channel and processes them concurrently.
• Results: The results from the workers are returned via channels or processed further.

🚀 Example Code

Here’s a simple example of how to implement the Fan-Out pattern in Go, where each worker processes a task from the channel:

package main
package main

import (
	"fmt"
	"sync"
)

func worker(id int, jobs <-chan int, wg *sync.WaitGroup) {
	defer wg.Done() // Ensure worker completes the task

	for job := range jobs {
		fmt.Printf("Worker %d is processing job %d\n", id, job)
	}
}

func main() {
	jobs := make(chan int, 10)  // Channel for tasks
	var wg sync.WaitGroup

	// Create 3 workers
	for i := 1; i <= 3; i++ {
		wg.Add(1)
		go worker(i, jobs, &wg)
	}

	// Send tasks to the jobs channel
	for j := 1; j <= 5; j++ {
		jobs <- j
	}
	close(jobs) // Close jobs channel after sending tasks

	// Wait for all workers to complete
	wg.Wait()

	fmt.Println("All jobs processed.")
}

📌 Explanation of the Code:

1. jobs channel: This channel holds the jobs (integer values) that need to be processed. Each job represents a task that a worker will process.
2. worker function:
   • Each worker receives jobs from the jobs channel and processes them. In this example, the worker simply prints a message indicating that it's processing a job.
   • The workers are goroutines that operate concurrently, meaning they process tasks at the same time.
3. sync.WaitGroup: Ensures the main function waits for all workers to complete before exiting.
4. Goroutines: Each worker is run as a separate goroutine, allowing for concurrent processing of tasks.

🔑 Benefits of Fan-Out Pattern:

1. Improved Performance: Tasks are distributed among multiple goroutines, which can run concurrently and utilize CPU resources more efficiently.
2. Easy to Scale: You can easily adjust the number of workers based on the task load and system resources.
3. Reduced Latency: With tasks processed concurrently, the overall time required for processing reduces.

⚡ Things to Consider When Using Fan-Out Pattern:

1. Synchronization:
   • Use sync.WaitGroup or other synchronization techniques to ensure that the main program waits for all goroutines to complete their work before exiting.
2. Resource Management:
   • Be mindful of the number of workers to avoid resource exhaustion or overloading the system.
3. Channel Management:
   • Ensure that channels are used correctly (buffered or unbuffered) to avoid issues like deadlocks or congestion.
4. Using select for Task Cancellation:
   • You can use select to monitor multiple channels and handle cancellation or timeout if tasks take too long.

🎯 Conclusion

• The Fan-Out Pattern is an effective way to process similar tasks concurrently in Go, improving performance and reducing latency.
• This design pattern can easily be extended and combined with other patterns like Fan-In to collect results from multiple workers.