Selenium and Go ( Proxy, User Agent, Screen Shot)

Selenium and Go ( Proxy, User Agent, Screen Shot)

Selenium WebDriver, combined with Go, opens up a wide range of possibilities for web-based. In this guide, we will walk through a Go script that automates browser interactions using Selenium, with a focus on utilizing proxies, user agents, and handling tasks like searching on Google and taking screenshots. We’ll also explore how to manage logs, load environment variables, and securely handle proxy authentication using extensions in Chrome.

What is Selenium and Why Use it?

Selenium is a widely used tool for automating browsers. It allows you to simulate real user interactions with websites, which is invaluable for tasks such as web scraping, testing web applications, or automating repetitive online tasks. When paired with the Go programming language, it becomes a robust tool for building scalable, automated workflows.

Key Features of Selenium:

1. Browser Automation: Automate interactions with browsers such as Chrome or Firefox.
2. Cross-Browser Compatibility: Write tests or tasks that run across multiple browsers.
3. Support for Different Languages: Selenium supports a variety of programming languages, including Go.

Setting Up Your Selenium Task in Go

The Go script we’re about to explore automates the process of launching a Chrome browser, setting a proxy, defining a user agent, and performing a Google search. Let’s break it down.

*You can access the full code at the bottom of the page.

Environment Setup

Before you start working with Selenium in Go, you’ll need to ensure that the necessary dependencies are installed:

1. Go Programming Language: Ensure Go is installed by running go version.
2. Selenium WebDriver: Download the Selenium standalone server JAR file and ChromeDriver.
3. Required Go Packages:
   • github.com/joho/godotenv for loading environment variables.
   • github.com/tebeka/selenium for controlling the browser.

Loading Environment Variables
The script loads environment variables from a .env file using the godotenv package. This allows for easy configuration of sensitive information, such as API keys or search queries, without hardcoding them into the script.

## .env
SEARCH_KEY="tolga karabulut"
DEBUG=false

err := godotenv.Load()
if err != nil {
 logger.Printf("Error loading .env file: %v", err)
 return
}

Initializing the Logger
Logging is crucial in automation tasks, especially when dealing with long-running processes. The script sets up a logger that writes to a log file, providing a persistent record of the task’s activities.

logFile := setLogger()
defer func(logFile *os.File) {
 err := logFile.Close()
 if err != nil {
  panic("Failed to close log file")
 }
}(logFile)

The setLogger function initializes the logger and logs all actions taken during the Selenium task.

Starting the Selenium Service
The core of the automation process is the Selenium service. This script configures and starts Selenium with a series of options, such as enabling frame buffer and logging output to the terminal

opts := []selenium.ServiceOption{
 selenium.StartFrameBuffer(),
 selenium.Output(os.Stderr),
}
service, err := selenium.NewSeleniumService(seleniumPath, port, opts...)
if err != nil {
 log.Printf("Error starting the ChromeDriver service: %v", err)
 return
}
defer service.Stop()

Loading Proxies and User Agents

To simulate diverse browsing conditions, the script loads proxies and user agents from external JSON files. This allows you to rotate proxies and user agents, which is useful when making multiple requests to websites that may block repeated access from a single IP or browser identity.The function randomIndex ensures that the proxy and user agent are selected randomly from the respective lists, adding variability to each session.

## user_agents.json
[
  "Mozilla/5.0 (Linux; Android 13; SM-G998B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.5790.171 Mobile Safari/537.36",
]

## proxy.json
[{"host":"tolgakarabulut.com.tr","port":80,"username":"user","password":"password", "type":"http"}]

proxies, _ := loadProxyByFile("proxy.json")
userAgents, _ := loadUserAgentsByFile("user_agents.json")
proxy := proxies[randomIndex(len(proxies))]
userAgent := userAgents[randomIndex(len(userAgents))]

Configuring Chrome with Proxy and User Agent
The script uses a Chrome extension to configure the proxy settings. This extension is dynamically generated with the specific proxy information, including the type, host, port, and authentication credentials.The setProxyExt function creates a manifest.json and background.js file that define the proxy configuration. These files are zipped into an extension, which is then loaded into Chrome during startup.

setProxyExt(proxy.Type, proxy.Host, proxy.Port, proxy.Username, proxy.Password)

Running the Google Search Task
With the proxy and user agent set, the browser navigates to Google, where it performs a search using the search term provided in the .env file.

searchKey := os.Getenv("SEARCH_KEY")
err = wd.Get(google)
if err != nil {
 log.Printf("Failed to load page: %v", err)
 return
}

The search is executed by locating the search input box using a CSS selector and sending the search query.

searchBox, err := wd.FindElement(selenium.ByCSSSelector, "textarea[name='q']")
err = searchBox.SendKeys(searchKey + selenium.EnterKey)

Capturing a Screenshot
Once the search is completed, the script takes a screenshot of the result page.

screenShot(wd)

The screenshot is saved with a timestamped filename for easy reference later.

Conclusion

This Go and Selenium script provides a versatile automation solution for web tasks. By leveraging proxies, user agents, and logging, it ensures that tasks are executed efficiently and securely. Whether you’re scraping data, automating tests, or performing repetitive web interactions, this script serves as a strong foundation for your automation needs.

Important Note :
This code can only be used for proxies that do not contain a user password. Selenium WebDriver in Go allows you to configure proxies directly when starting a browser instance. You can set a proxy using the Selenium.Capabilities structure and provide proxy settings in goog:chromeOptions or moz:firefoxOptions depending on the browser.

proxy := selenium.Proxy{
  Type: selenium.Manual, // Use manual proxy setup
  HTTP: "http://proxyserver:8080",  // HTTP proxy address
  SSL:  "http://proxyserver:8080",  // HTTPS proxy address
 }

 // Chrome capabilities
 caps := selenium.Capabilities{
  "browserName": "chrome",
  "proxy":       proxy, // Setting the proxy in capabilities
  "goog:chromeOptions": map[string]interface{}{
   "args": []string{
    "--ignore-certificate-errors", // Ignore SSL errors (optional)
    "--start-maximized",           // Start Chrome maximized (optional)
   },
  },
 }

Docker Compose

services:
  chrome:
    image: selenium/node-chrome:4.7.2-20221219
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  firefox:
    image: selenium/node-firefox:4.7.2-20221219
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443

  edge:
    image: selenium/node-edge:4.7.2-20221219
    shm_size: 2gb
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - SE_NODE_OVERRIDE_MAX_SESSIONS=true

  selenium-hub:
    image: selenium/hub:4.7.2-20221219
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

Full Code Example

package main

import (
 "archive/zip"
 "crypto/rand"
 "encoding/base64"
 "encoding/json"
 "fmt"
 "github.com/joho/godotenv"
 "github.com/tebeka/selenium"
 "log"
 "math/big"
 "os"
 "time"
)

var logger *log.Logger

const (
 seleniumPath = "vendor/selenium-server-standalone-3.4.jar"
 seleniumHost = "http://127.0.0.1:4444/wd/hub"
 port         = 4444
 google       = "https://google.com"
 fileName     = "proxy.zip"
)

type Proxy struct {
 Host     string `json:"host"`
 Port     int    `json:"port"`
 Username string `json:"username"`
 Password string `json:"password"`
 Type     string `json:"type"`
}

func main() {
 logFile := setLogger()
 defer func(logFile *os.File) {
  err := logFile.Close()
  if err != nil {
   panic("Failed to close log file")
  }
 }(logFile)

 err := godotenv.Load()
 if err != nil {
  logger.Printf("Error loading .env file: %v", err)
  return
 }
 runSeleniumTask()
}

// runSeleniumTask starts the Selenium task and handles the WebDriver interactions.
func runSeleniumTask() {
 defer func() {
  if r := recover(); r != nil {
   fmt.Println("Throw panic:", r)
  }
 }()
 logger.Println("Starting selenium task")

 opts := []selenium.ServiceOption{
  selenium.StartFrameBuffer(),
  selenium.Output(os.Stderr),
 }
 selenium.SetDebug(os.Getenv("DEBUG") == "true")
 service, err := selenium.NewSeleniumService(seleniumPath, port, opts...)
 if err != nil {
  log.Printf("Error starting the ChromeDriver service: %v", err)
  return
 }
 defer service.Stop()

 proxies, _ := loadProxyByFile("proxy.json")
 userAgents, _ := loadUserAgentsByFile("user_agents.json")
 proxy := proxies[randomIndex(len(proxies))]
 userAgent := userAgents[randomIndex(len(userAgents))]

 setProxyExt(proxy.Type, proxy.Host, proxy.Port, proxy.Username, proxy.Password)

 time.Sleep(2 * time.Second)
 fmt.Println("Proxy server: ", proxy)
 fmt.Println("User agent: ", userAgent)

 extensionFile, _ := os.ReadFile(fileName)
 encodedExtension := base64.StdEncoding.EncodeToString(extensionFile)
 userAgentText := fmt.Sprintf("--user-agent=%s", userAgent)

 caps := selenium.Capabilities{
  "browserName": "chrome",
  "goog:chromeOptions": map[string]interface{}{
   "extensions": []string{
    encodedExtension,
   },
   "args": []string{
    userAgentText,
    "--ignore-certificate-errors",
    "--start-maximized",
    "--lang=tr",
    "disable-infobars",
    "disable-gpu",
    "--no-sandbox",
   },
   "prefs": map[string]interface{}{
    "profile.managed_default_content_settings.images": 2,
   },
  },
 }

 wd, err := selenium.NewRemote(caps, seleniumHost)
 if err != nil {
  log.Printf("Error connecting to the WebDriver: %v", err)
  return
 }
 defer func() {
  logger.Println("Quitting WebDriver")
  err := wd.Quit()
  if err != nil {
   fmt.Printf("Error quitting WebDriver: %v\n", err)
  }
 }()
 searchKey := os.Getenv("SEARCH_KEY")
 err = wd.Get(google)
 if err != nil {
  log.Printf("Failed to load page: %v", err)
  return
 }
 fmt.Println("Visited Google")
 time.Sleep(3 * time.Second)

 fmt.Println("Searching for: ", searchKey)
 err, _ = Search(err, wd, searchKey)
 if err != nil {
  return
 }
 screenShot(wd)

}

// setLogger initializes the logger and returns the log file.
func setLogger() *os.File {
 logFileName := fmt.Sprintf("logs/bot-%s.log", time.Now().Format("2006-01-02"))
 logFile, err := os.OpenFile(logFileName, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0666)
 if err != nil {
  log.Printf("Failed to open log file: %v", err)
 }

 logger = log.New(logFile, "", log.LstdFlags)
 return logFile
}

// Search performs a search on Google using the provided WebDriver and search key.
// Parameters:
// - err (error): The error object to handle any errors.
// - wd (selenium.WebDriver): The WebDriver instance to interact with the browser.
// - searchKey (string): The search keyword to use.
func Search(err error, wd selenium.WebDriver, searchKey string) (error, bool) {
 searchBox, err := wd.FindElement(selenium.ByCSSSelector, "textarea[name='q']")
 if err != nil {
  logger.Printf("Error finding the search box: %v", err)
  return nil, true
 }
 err = searchBox.SendKeys(searchKey + selenium.EnterKey)
 if err != nil {
  logger.Printf("Error typing into the search box: %v", err)
  return nil, true
 }
 time.Sleep(3 * time.Second)
 return err, false
}

// loadProxyByFile loads proxy configurations from a JSON file.
// Parameters:
// - filename (string): The name of the file to load proxies from.
func loadProxyByFile(filename string) ([]Proxy, error) {
 file, err := os.Open(filename)
 if err != nil {
  return nil, err
 }
 defer func(file *os.File) {
  err := file.Close()
  if err != nil {
   fmt.Printf("Failed to close file: %v", err)
   return
  }
 }(file)
 var proxies []Proxy
 decoder := json.NewDecoder(file)
 err = decoder.Decode(&proxies)
 if err != nil {
  return nil, err
 }

 return proxies, nil
}

// loadUserAgentsByFile loads user agents from a JSON file.
// Parameters:
// - filename (string): The name of the file to load user agents from.
func loadUserAgentsByFile(filename string) ([]string, error) {
 file, err := os.Open(filename)
 if err != nil {
  return nil, err
 }
 defer func(file *os.File) {
  err := file.Close()
  if err != nil {
   fmt.Printf("Failed to close file: %v", err)
   return
  }
 }(file)
 var userAgents []string
 decoder := json.NewDecoder(file)
 err = decoder.Decode(&userAgents)
 if err != nil {
  return nil, err
 }

 return userAgents, nil
}

// randomIndex generates a random index within the given maximum value.
// Parameters:
// - max (int): The maximum value for the random index.
func randomIndex(max int) int {
 nBig, err := rand.Int(rand.Reader, big.NewInt(int64(max)))
 if err != nil {
  logger.Printf("Failed to generate random index: %v", err)
 }
 return int(nBig.Int64())
}

// screenShot takes a screenshot using the provided WebDriver.
// Parameters:
// - wd (selenium.WebDriver): The WebDriver instance to interact with the browser.
func screenShot(wd selenium.WebDriver) {
 screenshot, err := wd.Screenshot()
 if err != nil {
  logger.Printf("Failed to take screenshot: %v", err)
  return
 }

 screenShotName := fmt.Sprintf("ss/screenshot-%s.png", time.Now().Format("2006-01-02 15:04:05"))
 err = os.WriteFile(screenShotName, screenshot, 0644)
 if err != nil {
  logger.Printf("Failed to save screenshot: %v", err)
  return
 }
 return
}

// setProxyExt creates a Chrome extension to set the proxy configuration.
// Parameters:
// - proxyType (string): The type of the proxy (e.g., "http").
// - proxyHost (string): The host of the proxy server.
// - proxyPort (int): The port of the proxy server.
// - proxyUser (string): The username for proxy authentication.
// - proxyPass (string): The password for proxy authentication.
func setProxyExt(proxyType string, proxyHost string, proxyPort int, proxyUser string, proxyPass string) {
 manifestJson := `
{
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
}
`
 backgroundJs := fmt.Sprintf(`
var config = {
        mode: "fixed_servers",
        rules: {
        singleProxy: {
            scheme: "%s",
            host: "%s",
            port: parseInt(%d)
        },
        bypassList: ["localhost"]
        }
    };
console.log(config);
chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
    return {
        authCredentials: {
            username: "%s",
            password: "%s"
        }
    };
}

chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ['blocking']
);
`, proxyType, proxyHost, proxyPort, proxyUser, proxyPass)
 zp, err := os.Create(fileName)
 if err != nil {
  fmt.Println("Error creating zip file:", err)
  return
 }
 defer zp.Close()

 zipWriter := zip.NewWriter(zp)
 defer zipWriter.Close()

 writeFileToZip(zipWriter, "manifest.json", manifestJson)
 writeFileToZip(zipWriter, "background.js", backgroundJs)
}

// writeFileToZip writes a file to the provided zip writer.
// Parameters:
// - zipWriter (*zip.Writer): The zip writer to write the file to.
// - filename (string): The name of the file to write.
// - content (string): The content of the file to write.
func writeFileToZip(zipWriter *zip.Writer, filename, content string) {
 fileWriter, err := zipWriter.Create(filename)
 if err != nil {
  fmt.Println("Error creating file in zip:", err)
  return
 }
 _, err = fileWriter.Write([]byte(content))
 if err != nil {
  fmt.Println("Error writing content to file in zip:", err)
 }
}

Tolga Karabulut
tolga.karabulut@teknasyon.com
Teknasyon | Senior PHP Developer