Advanced Selenium Automation: Real-World Experiments with Sikuli, Appium, Python & Linux

Selenium is often introduced as a simple browser automation tool. Click here, assert there, move on. But once you start working on real-world enterprise applications, Selenium reveals its true personality. Flexible, extensible, and surprisingly powerful when combined with the right tools.

Over the years, I’ve used Selenium not just for UI automation, but as the central orchestrator in complex automation frameworks spanning image-based testing, OS-level dialogs, backend systems, performance testing, and even mobile OTP workflows.

In this blog, I’ll walk you through some of my hands-on Selenium experiments, explaining the why, the how, and sharing code snippets you can adapt in your own projects.

1. Automating a Map-Based Web Application Using Selenium + Sikuli

The Problem

Map-based applications (Google Maps–like UIs) are notoriously difficult to automate. Many controls are:

  • Canvas-rendered

  • Dynamic

  • Not accessible via DOM locators (XPath, CSS)

The Solution

I integrated Sikuli with Selenium WebDriver (Java).

Sikuli works on image recognition, making it perfect when DOM-level automation fails.

Architecture Overview

  • Selenium handles browser navigation and page-level flows

  • Sikuli handles map clicks, drag-drop, and visual validations


Visual Automation Layer (Sikuli Integration)

This layer is invoked when:

  • UI elements are not accessible via DOM

  • Canvas-based components are used

  • Map-based or image-driven interfaces exist

How It Fits

Flow:

  1. Selenium navigates to the relevant page

  2. Control is handed over to Sikuli

  3. Sikuli performs image-based actions (click, hover, drag)

  4. Control returns to Selenium

Architectural Benefit

  • Zero dependency on fragile locators

  • Ideal for maps, charts, video players

  • Keeps Selenium scripts clean and readable


Sample Code (Java + Sikuli)

java
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.sikuli.script.Screen;
import org.sikuli.script.Pattern;

public class MapAutomation {

    public static void main(String[] args) throws Exception {
        // Set ChromeDriver path (if not in PATH)
        // System.setProperty("webdriver.chrome.driver", "path/to/chromedriver.exe");
        
        WebDriver driver = new ChromeDriver();
        driver.get("https://example-map-app.com");

        Screen screen = new Screen();
        Pattern mapIcon = new Pattern("map_marker.png");

        // Click on map marker using image recognition
        screen.wait(mapIcon, 10);
        screen.click(mapIcon);

        driver.quit();
    }
}

Why This Worked

  • Selenium alone could not “see” the map elements

  • Sikuli allowed pixel-level interaction

  • The combination delivered stability without hacks

2. Handling Windows Dialogs Using Selenium + WinIT

The Problem

Browser-native dialogs like:

  • File upload popups

  • Authentication dialogs

  • OS-level alerts

are outside Selenium’s control.

The Solution

I integrated WinIT with Selenium to interact with Windows dialogs directly.


OS-Level Interaction Layer (WinIT Integration)

Selenium cannot interact with:

  • Windows file upload dialogs

  • Authentication popups

  • Native OS alerts

Architecture Flow

  1. Selenium triggers an action that opens an OS dialog

  2. WinIT attaches to the dialog window

  3. Performs required actions (set text, click buttons)

  4. Returns execution to Selenium

Design Principle

This layer ensures:

  • Browser automation does not break due to OS limitations

  • No manual intervention in regression suites

  • End-to-end flows remain fully automated


Sample Code (Java + WinIT)

java
import com.lazerycode.selenium.util.Query;
import com.lazerycode.selenium.util.SeleniumUtils;

Query uploadDialog = new Query.Builder()
        .usingTitle("Open")
        .usingText("File name:")
        .build();

// Set file path in dialog
SeleniumUtils.waitForWindowAndSetText(uploadDialog, "C:\\test\\data.csv");
SeleniumUtils.waitForWindowAndClickButton(uploadDialog, "Open");

Key Takeaway

Selenium does not need to be replaced for OS-level testing. It just needs the right companion tool.

3. Performance Testing Using Selenium WebDriver with Python

The Problem

Traditional performance tools don’t always reflect real user journeys.

The Solution

I used Selenium WebDriver with Python to measure:

  • Page load times

  • Feature-level response times

  • User-action latency

This helped catch performance regressions from a user’s perspective.


Sample Code (Python)

python
import time
from selenium import webdriver

driver = webdriver.Chrome()
start_time = time.time()

driver.get("https://example-app.com/login")

login_time = time.time() - start_time
print(f"Login page load time: {login_time:.2f} seconds")

driver.quit()

Why This Approach Is Powerful

  • Combines functional + performance testing

  • Works well in CI pipelines

  • Highlights slow UX flows early

4. Frontend + Backend Automation Using Selenium + Paramiko

The Problem

Some test scenarios required:

  • UI actions on the frontend

  • Backend validations on AIX/Linux servers and vice versa.

The Solution

I integrated Python’s Paramiko module with Selenium.


Backend Automation Layer (Paramiko via SSH)

This layer enables frontend + backend automation in a single test case.

Typical Use Cases

  • Log validation after UI actions

  • Database or file system checks

  • Triggering backend jobs

  • AIX/Linux command execution

Architectural Flow

  1. Selenium performs UI action (e.g., submit form)

  2. Paramiko connects to backend server via SSH

  3. Executes commands and fetches output

  4. Results are validated within the same test execution

Why This Is Powerful

  • Eliminates manual backend verification

  • Enables true end-to-end testing

  • Speeds up root-cause analysis


Sample Code (Python + Paramiko)

python
import paramiko
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# Initialize
driver = webdriver.Chrome()
driver.get("https://example-app.com")

# Trigger backend command via SSH
with paramiko.SSHClient() as ssh:
    ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
    ssh.connect("backend.server.com", username="user", password="pass")
    
    # Execute backend action
    ssh.exec_command("systemctl restart app-service")
    
    # Verify effect in logs
    stdin, stdout, stderr = ssh.exec_command("systemctl status app-service")
    print("Backend status:", stdout.read().decode())

# Validate UI effect
WebDriverWait(driver, 30).until(
    EC.visibility_of_element_located((By.ID, "status-indicator"))
)
status = driver.find_element(By.ID, "status-indicator").text
print(f"UI Status: {status}")

driver.quit()

5. Automating OTP Flows Using Selenium + Appium

The Problem

OTP-based authentication breaks traditional automation.

The Solution

I combined Selenium (Web) with Appium (Mobile).


Mobile Automation Layer (Appium Integration for OTP)

OTP-based authentication traditionally blocks automation.

Architecture Flow

  1. Selenium triggers OTP generation on the web app

  2. Appium connects to the mobile device or emulator

  3. Reads OTP from SMS or notification

  4. Selenium enters OTP into the web UI

  5. Authentication completes automatically

Key Advantage

  • Web and mobile automation run as one logical transaction

  • Enables full regression of secure flows

  • Extremely useful for fintech, banking, and telecom apps


Sample Flow (Sample Code)

java
import io.appium.java_client.AppiumDriver;
import io.appium.java_client.android.AndroidDriver;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.support.ui.ExpectedConditions;
import org.openqa.selenium.support.ui.WebDriverWait;
import java.net.URL;
import java.time.Duration;
import java.util.regex.Matcher;
import java.util.regex.Pattern;

public class OTPAutomation {
    
    private static WebDriver webDriver;
    private static AppiumDriver mobileDriver;
    private static final int MAX_RETRIES = 5;
    private static final int POLL_INTERVAL_SECONDS = 2;
    
    public static void main(String[] args) throws Exception {
        try {
            // Step 1: Initialize Selenium WebDriver for Web Application
            setupWebDriver();
            
            // Step 2: Initialize Appium for Mobile Device
            setupMobileDriver();
            
            // Step 3: Navigate to web application and trigger OTP
            triggerOTPGeneration();
            
            // Step 4: Fetch OTP from mobile SMS
            String otp = fetchOTPFromMobile();
            
            // Step 5: Enter OTP in web application
            enterOTPInWebApp(otp);
            
            // Step 6: Verify successful authentication
            verifyAuthentication();
            
        } finally {
            // Cleanup
            if (webDriver != null) webDriver.quit();
            if (mobileDriver != null) mobileDriver.quit();
        }
    }
    
    private static void setupWebDriver() {
        System.setProperty("webdriver.chrome.driver", "path/to/chromedriver");
        webDriver = new ChromeDriver();
        webDriver.manage().window().maximize();
        webDriver.manage().timeouts().implicitlyWait(Duration.ofSeconds(10));
    }
    
    private static void setupMobileDriver() throws Exception {
        // Appium server URL
        URL appiumServerUrl = new URL("http://127.0.0.1:4723/wd/hub");
        
        // Desired capabilities for Android device
        DesiredCapabilities capabilities = new DesiredCapabilities();
        capabilities.setCapability("platformName", "Android");
        capabilities.setCapability("deviceName", "Android Emulator");
        capabilities.setCapability("automationName", "UiAutomator2");
        capabilities.setCapability("appPackage", "com.google.android.apps.messaging");
        capabilities.setCapability("appActivity", "com.google.android.apps.messaging.ui.ConversationListActivity");
        capabilities.setCapability("noReset", true);
        
        mobileDriver = new AndroidDriver(appiumServerUrl, capabilities);
    }
    
    private static void triggerOTPGeneration() {
        // Navigate to login page
        webDriver.get("https://example-app.com/login");
        
        // Enter phone number
        webDriver.findElement(By.id("phoneNumber")).sendKeys("+1234567890");
        
        // Click "Send OTP" button
        webDriver.findElement(By.id("sendOtpButton")).click();
        
        // Wait for OTP sent confirmation
        WebDriverWait wait = new WebDriverWait(webDriver, Duration.ofSeconds(10));
        wait.until(ExpectedConditions.visibilityOfElementLocated(
            By.xpath("//div[contains(text(),'OTP sent successfully')]")
        ));
        
        System.out.println("OTP generation triggered successfully");
    }
    
    private static String fetchOTPFromMobile() throws InterruptedException {
        String otp = null;
        int retryCount = 0;
        
        // Wait for SMS to arrive and extract OTP
        while (retryCount < MAX_RETRIES && otp == null) {
            System.out.println("Checking for OTP SMS (Attempt " + (retryCount + 1) + ")");
            
            try {
                // Navigate to SMS app and get the latest message
                mobileDriver.findElement(By.id("conversation_list")).click();
                
                // Wait for messages to load
                Thread.sleep(2000);
                
                // Get the latest SMS body
                String smsText = mobileDriver.findElement(
                    By.id("message_text")
                ).getText();
                
                // Extract OTP using regex pattern (6-digit OTP)
                Pattern pattern = Pattern.compile("\\b\\d{6}\\b");
                Matcher matcher = pattern.matcher(smsText);
                
                if (matcher.find()) {
                    otp = matcher.group();
                    System.out.println("OTP found: " + otp);
                }
                
            } catch (Exception e) {
                System.out.println("Error reading SMS: " + e.getMessage());
            }
            
            if (otp == null) {
                Thread.sleep(POLL_INTERVAL_SECONDS * 1000);
                retryCount++;
                
                // Refresh SMS view
                mobileDriver.navigate().back();
            }
        }
        
        if (otp == null) {
            throw new RuntimeException("Failed to retrieve OTP after " + MAX_RETRIES + " attempts");
        }
        
        return otp;
    }
    
    private static void enterOTPInWebApp(String otp) {
        // Wait for OTP input field to be visible
        WebDriverWait wait = new WebDriverWait(webDriver, Duration.ofSeconds(10));
        wait.until(ExpectedConditions.visibilityOfElementLocated(By.id("otpInput")));
        
        // Enter OTP digit by digit (for individual input fields)
        WebElement otpInput = webDriver.findElement(By.id("otpInput"));
        otpInput.clear();
        otpInput.sendKeys(otp);
        
        System.out.println("OTP entered successfully: " + otp);
        
        // Click verify button
        webDriver.findElement(By.id("verifyOtpButton")).click();
    }
    
    private static void verifyAuthentication() {
        // Wait for successful authentication
        WebDriverWait wait = new WebDriverWait(webDriver, Duration.ofSeconds(10));
        
        try {
            // Check for successful login indicator
            WebElement successElement = wait.until(
                ExpectedConditions.visibilityOfElementLocated(
                    By.xpath("//div[contains(text(),'Login Successful') or contains(text(),'Welcome')]")
                )
            );
            
            System.out.println("Authentication successful! Message: " + successElement.getText());
            
        } catch (Exception e) {
            // Check for error message
            try {
                WebElement errorElement = webDriver.findElement(
                    By.xpath("//div[contains(@class,'error')]")
                );
                System.out.println("Authentication failed! Error: " + errorElement.getText());
            } catch (Exception ex) {
                System.out.println("Authentication status unknown");
            }
        }
    }
}

Final Thoughts: Selenium Is a Platform, Not Just a Tool 

What I learned through these experiments is simple but powerful:

Selenium is not limited by what it can do, but by how creatively you use it.

By integrating Selenium with tools like Sikuli, WinIT, Python, Paramiko, and Appium, you can:

  • Solve non-standard automation problems

  • Build resilient enterprise frameworks

  • Cover UI, OS, backend, mobile, and performance testing together

If you’re still using Selenium only for basic UI checks, you’re barely scratching the surface.

Suggested Articles


--
Author
Vaneesh Behl
Quality Engineering Leader | Automation Enthusiast

Popular posts from this blog

Mastering Selenium Practice: Automating Web Tables with Demo Examples

Image
1. Introduction Automating web tables using Selenium WebDriver is one of the most essential and frequently asked skills in automation testing interviews and real-world QA projects. From dashboards and admin panels to reports and data grids, web tables are everywhere — and handling them dynamically is a must for Selenium testers. In this tutorial, you’ll learn how to automate static and dynamic web tables in Selenium using real demo examples , including row and column traversal, dynamic XPath creation, and data validation techniques. Whether you’re preparing for interviews or working on enterprise UI automation, this guide will help you master Selenium web table automation with practical, reusable code examples . Table of Content 1. What is Web Table? Types of Web Table Static Table Dynamic Table 2. Automate Reading data from Static Web Table with Selenium Practice Exercises for automating Static Table Solution Code for Static Table Code Explanation 3. Automate Handling Dynamic Web Ta...
Read more

18 Demo Websites for Selenium Automation Practice in 2026

Image
Looking for safe and free dummy sites for testing your Selenium WebDriver automation skills? In this updated guide, we’ve curated the best demo websites where you can practice automation confidently, build portfolio-ready projects, and simulate real-world testing scenarios without touching live systems. Testing on production applications can be risky, which is why dummy sites for testing are ideal for learning Selenium, Cypress, and other automation tools. This post features free, reliable demo websites across multiple complexity levels, allowing you to practice UI automation, functional validation, and even performance testing in a secure, controlled environment. Why Practice Selenium on Demo Websites? Before working on live applications, mastering automation on demo websites helps you: Learn risk-free : Experiment without breaking production systems Simulate real-world scenarios : From simple forms to complex e-commerce checkouts Build portfolio projects : Demonstrate skills...
Read more

Selenium Automation for E-commerce Websites: End-to-End Testing Scenarios

Image
Automating e-commerce websites using Selenium WebDriver is a critical skill for modern QA engineers and test automation professionals. E-commerce platforms involve complex user journeys such as product search, dynamic filtering, cart management, checkout flows, and payment gateway integrations, all of which require reliable end-to-end automation testing. In this tutorial, you will learn how to automate real-world e-commerce scenarios using Selenium WebDriver with practical examples, covering user registration, login validation, product selection, shopping cart operations, and checkout automation. By the end of this guide, you’ll understand how to build stable, scalable Selenium test scripts for e-commerce applications and apply best practices to handle dynamic elements, waits, and real-time user interactions. Table of Contents Why Automate E-commerce Testing? Key Challenges in E-commerce Testing Prerequisites for Automation Test Scenarios for E-commer...
Read more

14+ Best Selenium Practice Exercises to Master Automation Testing (with Code & Challenges)

Image
Boost Your Automation Skills with Practical Selenium Exercises Looking to build real, hands-on experience with Selenium? This curated set of practical automation exercises helps you master Selenium WebDriver by working through realistic testing scenarios. Each assignment mirrors challenges faced in professional QA environments, helping you move beyond theory and develop job-ready automation skills. Why These Exercises Matter 🚀  Career-ready skills : Learn automation techniques used in top tech firms 💡  Progressive difficulty : From beginner forms to advanced e-commerce workflows ✅  Complete solutions : Reference code for all exercises included 📈  Skill validation : Test your knowledge against industry standards Top Selenium Practice Projects E-Commerce Website Automation with Selenium (Amazon-like) In this self-learning project, you will automate key e-commerce workflows using Selenium and Java. The assignment covers end-to-end scenarios such as user sign-up and l...
Read more

Top 10 Highly Paid Indian-Origin CEOs in the USA

Image
Top Indian CEOs in the USA Indian Americans have been making significant contributions to the US economy for decades. Many Indian-origin executives have reached the pinnacle of success in American corporations and hold some of the highest-paid positions in the country. In this post, we will take a look at the top 10 highly paid Indian-origin CEOs in the USA, their history, academics, salary, and net worth. 1. Sunder Pichai   Salary US$ 199.7 million Sundar Pichai is the CEO of Google and Alphabet Inc. Born in Tamil Nadu, India, Pichai grew up in a middle-class family and earned a Bachelor’s degree in Metallurgical Engineering from the Indian Institute of Technology in Kharagpur. He went on to earn a Master’s degree in Material Sciences and Engineering from Stanford University and an MBA from the Wharton School of the University of Pennsylvania. Pichai joined Google in 2004 and rose through the ranks to become CEO in 2015. His annual salary in 2021 was $2 million, and his net ...
Read more

What is Java Class and Object?

Image
In this Java Class and Object tutorial, we will learn about the fundamentals of Java Programming. Java Class and Object are the base of Java. And we have tried to explain it in a simpler way. For your better understanding, we have also provided example Java programs for concepts of Java Classes and Object and Java Constructors. Classes and Objects are the fundamental building blocks of OOPS (Object Oriented Programming). A Java object is a physical and logical entity whereas a Java class is a logical entity. Table of Content 1. What is a Java Class? Syntax of java Class Explanation of example code Types Of Java Classes 2. What is a Java Object? How to create Java Object? Example Code and it's explanation What is the use of Java Objects? 3. Write your first Java Program - "Hello World" 4. What is Java Constructor? Non-Parameterized Constructor (Default) Parameterized Constructor 1. What is a Java Class? Before creating an object in Java, you need to define a class. A class...
Read more

A Complete Software Testing Tutorial: The Importance, Process, Tools, and Learning Resources

Image
  What is Software Testing? Software testing is a crucial process in software development that ensures that software is free from defects, meets user requirements, and performs as expected. In today's fast-paced world, software testing has become even more important because of the growing demand for high-quality software that is delivered on time and within budget. In this blog, we'll explore the various types of software testing, the software testing process, the benefits of software testing, common tools and techniques used in software testing, and steps to learn software testing. Why Software Testing is important? Testing helps identify bugs and errors in software before it is released to users. It is a critical step in the development process, ensuring that the software meets the requirements and functions as intended. Additionally, testing helps prevent downtime and ensures that users have a positive experience with the software.
Read more

12 AI Tools You Should Start Using Today to Save Time & Work Smarter (2026)

Image
Artificial Intelligence is no longer a futuristic buzzword. It has quietly moved into our daily workflows, helping professionals save time, automate repetitive tasks, and make smarter decisions. From writing emails to managing meetings, coding, research, and automation, AI tools are reshaping how we work. In this guide, you’ll discover practical AI tools you should start using today , grouped by real-world use cases so you can quickly find what fits your needs. AI Tools for Productivity & Daily Work 1. ChatGPT / Claude / Gemini These AI assistants help with: Writing emails, documents, and reports Brainstorming ideas Explaining complex topics Debugging code They are ideal for professionals, students, and creators who want a reliable thinking partner. Example:  If you’re preparing a report, ask ChatGPT for a draft or detailed research summary to save hours of effort. Its adaptability makes it useful for creating business proposals, summarizing long texts, or...
Read more

Behavior-Driven Development (BDD) with Python Behave: A Complete Tutorial

Image
This comprehensive Python Behave tutorial teaches you Behavior-Driven Development (BDD) step by step using real examples. You’ll learn how to write Gherkin feature files, implement step definitions in Python, structure a Behave project, and run automated tests — preparing you for real-world test automation using BDD. Table of Content 1. What is BDD? 2. What is Gherkin Language in BDD? 3. What is Behave in Python? 4. Installing and Setup Behave 5. Project Structure for Behave Python (BDD) 6. Feature Files, Step Functions and Other Files in Behave Feature File Step Functions Step Parameters Passing String Param from Feature File Passing Integer Param from Feature File environment.py file behave.ini file 7. Run Feature files in Behave Run a single feature file Run multiple feature files Generate JUNIT xml reports with behave Run feature files with Python suite file Upcoming Tutorials: 8. API Automation Testing with BDD Python (Behave) 9. Selenium Automation with BDD Python (Behave) 1. Wha...
Read more

Top 7 Web Development Trends in the Market (2026)

Image
  Technology continues to advance as humans discover new methods to innovate, doing tasks quicker and more creatively than before. Developers are continuously on the lookout for new technologies that will help them move forward towards a better future.  The contemporary industry is constantly evolving, and new online technologies emerge daily. Entrepreneurs who want to interact with more consumers and stay ahead in this competitive industry may take advantage of these new trends. There aren't many industries that have as much innovation as bespoke web development. Hire a WordPress developer to help entrepreneurs interact with more consumers and stay ahead. Websites, without a question, play an important part in today's commercial environment. A WordPress design agency will build you a website that functions as a separate brand, organization, and individual. Doing business on a worldwide scale has become conceivable and attainable as firms and enterprises have transcended all b...
Read more