Flaky tests: causes, diagnosis, artifacts and Allure
A flaky test passes and fails on the same code. It destroys trust in the suite, so you cure it instead of "rerunning until green".
Typical causes in Selenium tests:
- wrong waits:
Thread.sleep, a click before the element is clickable, a check before an asynchronous request finishes;
- dependencies between tests: shared data, execution order, someone else's state in the database;
- unstable locators: indexes, generated classes, text that gets changed;
- the environment: a slow CI, little browser memory in a container, external services, time and time zones;
- animations, overlays, remote resources without stubs.
Diagnosis: reproduce in a loop (run the test 20-50 times), compare failed and passed runs, look at the screenshot, the DOM, browser and server logs and step timing. Only after finding the cause do you decide: fix the wait, isolate the data, stabilise the environment or quarantine the test with a ticket.
Artifacts on failure: a screenshot (((TakesScreenshot) driver).getScreenshotAs(OutputType.BYTES)), the URL, the page source, logs, and in Grid also a video. They are captured automatically: in JUnit 5 with an extension implementing TestWatcher (testFailed), in TestNG with ITestListener.onTestFailure. A "screenshot if failed" in @AfterEach without an extension gets clumsy and is forgotten.
Allure gathers this into a report: add allure-jupiter (formerly allure-junit5) or allure-testng, mark steps with @Step, add attachments with Allure.addAttachment(...), results land in allure-results, and allure generate builds the HTML. The run history shows which tests are unstable.
An automatic retry (repeating failures) is acceptable as a temporary measure and always with a flaky mark in the report. If it silently turns everything green, real defects get hidden.
BiDi instead of CDP, Actions, files, iframes and windows
CDP versus BiDi. Selenium 4 gives access to the Chrome DevTools Protocol (((HasDevTools) driver).getDevTools()): network interception, emulation, console events. But CDP works only in Chromium and is tied to the browser version, and Selenium ships bindings for specific versions, so after a Chrome update the code may not find a matching implementation. The Selenium documentation calls CDP support temporary until WebDriver BiDi is implemented: a W3C standard for two-way communication over WebSocket, cross-browser (Chrome, Firefox, Edge).
ChromeOptions options = new ChromeOptions();
options.setCapability("webSocketUrl", true);
WebDriver driver = new ChromeDriver(options);
try (LogInspector logs = new LogInspector(driver)) {
logs.onConsoleEntry(entry -> System.out.println(entry.getText()));
driver.get("https://example.com");
}
For new code BiDi is the default choice (console events and JS errors, network, scripts); CDP only when a capability is needed that BiDi does not have yet. The old driver.manage().logs() works only in Chromium.
The Actions API composes complex gestures: hover, drag and drop, double/context click, key combinations, scrolling.
new Actions(driver).moveToElement(menu).pause(Duration.ofMillis(200)).click(sub).perform();
new Actions(driver).scrollToElement(footer).perform();
new Actions(driver).dragAndDrop(card, column).perform();
Without perform() nothing happens.
Files. Upload: sendKeys with an absolute path into input type="file" (no click on the button; Selenium does not see the OS dialog); on Grid add setFileDetector(new LocalFileDetector()). Download: the folder and the dialog suppression are set in browser options, then wait for the file to appear.
iframe. An element inside a frame is invisible to lookups until you switch: driver.switchTo().frame(...), back with parentFrame() or defaultContent(); ExpectedConditions.frameToBeAvailableAndSwitchToIt combines waiting and switching.
Windows and tabs. Remember the current handle, wait for a new one (numberOfWindowsToBe), switch to the other; driver.switchTo().newWindow(WindowType.TAB) opens a tab. After close() you must switch back to the needed window explicitly, otherwise you get NoSuchWindowException.