To solve the problem of Cypress flaky tests, here are the detailed steps:
Skip the hassle and get the ready to use 100% working script (Link in the comments section of the YouTube Video) (Latest test 31/05/2025)
Check more on: How to Bypass Cloudflare Turnstile & Cloudflare WAF – Reddit, How to Bypass Cloudflare Turnstile, Cloudflare WAF & reCAPTCHA v3 – Medium, How to Bypass Cloudflare Turnstile, WAF & reCAPTCHA v3 – LinkedIn Article
0.0 out of 5 stars (based on 0 reviews)
There are no reviews yet. Be the first one to write one. |
Amazon.com:
Check Amazon for Cypress flaky tests Latest Discussions & Reviews: |
Start by analyzing the nature of your flaky tests.
Are they occasional failures in CI but pass locally? Do they fail on specific elements or actions? Understanding the pattern is the first step. Next, isolate the problematic test case.
Sometimes, flakiness is due to race conditions, network delays, or an unstable test environment.
Implement intelligent waiting strategies using cy.wait
sparingly, as it can slow tests or, better yet, cy.should'be.visible'
, cy.should'have.text'
, and other assertions that automatically retry. Ensure your selectors are robust and unique.
Avoid relying on dynamic IDs or classes that change frequently.
Utilize Cypress’s retryAbility
feature, which can be configured globally or per test to re-run failed tests a specified number of times.
Debug with cy.pause
and cy.debug
to observe the test execution step-by-step and identify exactly where the test diverges from the expected behavior.
Consider using cy.intercept
to mock network requests, reducing dependency on external APIs and their inherent latencies or inconsistencies.
Finally, stabilize your test environment by ensuring it’s consistent across local and CI runs, and clear any persistent state before each test if necessary.
Understanding the Enemy: What Are Flaky Tests?
Flaky tests are those enigmatic beasts in your test suite that sometimes pass and sometimes fail without any code changes. Think of them as the random number generator of your CI/CD pipeline—utterly unpredictable and incredibly frustrating. They can masquerade as genuine bugs, leading to wasted hours of investigation, or cause a false sense of security when they randomly pass, allowing real issues to slip through. For any development team striving for efficiency and reliability, flaky tests are a significant impediment. They erode trust in your test suite, slow down deployment cycles, and ultimately cost time and resources. According to a 2021 survey by Applitools, over 50% of development teams spend at least 15% of their time dealing with flaky tests. That’s a staggering amount of time that could be better spent building features or improving user experience.
The True Cost of Flakiness
The ripple effect of flaky tests extends far beyond just debugging time.
- Developer Frustration and Burnout: Constantly re-running tests or sifting through false positives can lead to significant frustration among developers, impacting morale.
- Reduced Confidence in Test Suite: When tests are unreliable, developers start to ignore failures, potentially missing critical issues.
- Slower Release Cycles: Teams become hesitant to deploy code when the build pipeline is constantly red due to flakiness, leading to delayed releases.
- Increased Infrastructure Costs: More retries mean more CI/CD build minutes consumed, directly increasing operational expenses.
- Misleading Bug Reports: Flaky tests can generate bug reports for non-existent issues, diverting valuable engineering resources.
Common Causes of Flakiness in Cypress
Cypress, despite its powerful features, is not immune to flakiness.
Several factors contribute to it, often stemming from the asynchronous nature of web applications and the inherent challenges of testing UIs.
- Race Conditions: This is arguably the most prevalent cause. Your test tries to interact with an element before it’s fully rendered, visible, or actionable. For example, clicking a button that hasn’t finished its animation, or asserting text that’s still being fetched from an API.
- Network Latency and External Dependencies: Tests relying on external APIs or slow network conditions can fail if the data doesn’t arrive in time, or if the API itself is intermittently unavailable.
- Asynchronous Operations: JavaScript’s event loop means elements might not appear immediately after an action. If Cypress tries to assert or interact too quickly, it fails.
- State Management Issues: If your application’s state isn’t reset properly between tests, or if tests depend on the order of execution, flakiness can emerge.
- Poorly Written Selectors: Selectors that are too brittle e.g., relying on dynamic IDs or too broad e.g., selecting the wrong element can lead to unpredictable behavior.
- Environmental Inconsistencies: Differences between local development environments and CI/CD environments e.g., browser versions, screen resolutions, network speed can introduce flakiness.
- Animation and Transitions: Elements that are animating or transitioning can be difficult for Cypress to interact with reliably if the test doesn’t wait for them to settle.
- Implicit Waits vs. Explicit Waits: Over-reliance on Cypress’s implicit waits for element existence can sometimes fall short when more explicit conditions are needed.
Strategic Waiting: Taming Asynchronous Behavior
One of the most effective strategies for combating flaky tests in Cypress is mastering intelligent waiting. While Cypress automatically retries assertions and commands that query the DOM, there are scenarios where you need more explicit control. The goal isn’t just to cy.wait
for an arbitrary amount of time, which is often a “smell” for a deeper issue and significantly slows down your tests. Instead, it’s about waiting for specific conditions to be met. This disciplined approach ensures your tests are robust without becoming excessively slow. Action class selenium python
Leveraging Cypress’s Built-in Retries and Assertions
Cypress is designed to be asynchronous-aware.
Its core commands and assertions automatically retry for a default timeout typically 4 seconds. This means when you do cy.get'.my-element'.should'be.visible'
, Cypress isn’t just looking for the element once.
It’s continuously polling the DOM and re-running the .should
assertion until the element becomes visible or the timeout expires.
This is your first line of defense against flakiness.
-
cy.should
: This is your best friend. Instead ofcy.get'.loading-spinner'. cy.wait2000. cy.get'.content'.
, you should aim for something like: Enterprise application testingcy.get'.loading-spinner'.should'not.exist'. // Wait for the spinner to disappear cy.get'.content'.should'be.visible'. // Wait for content to appear
This approach is declarative, readable, and highly resilient.
It ensures the test only proceeds when the necessary conditions are met.
-
Chaining Assertions: You can chain multiple assertions to a single command:
cy.get’#submit-button’
.should’be.enabled’
.and’contain’, ‘Submit’.Cypress will retry the entire chain until all assertions pass.
Explicit Waiting Strategies When and How to Use Them
While implicit retries are powerful, sometimes you need explicit control. Game testing platforms
-
cy.waitmilliseconds
Use Sparingly!: This is the blunt instrument. It pauses the test for a fixed duration. While easy, it’s brittle what if the operation sometimes takes longer? and inefficient what if it completes faster?.- When to use: Only as a last resort, usually for external factors completely outside your control e.g., a known, fixed delay in a third-party API that you cannot mock. Even then, try to find a better alternative.
- Example Bad:
cy.wait5000. // Hope the data loads
- Example Slightly better, but still a smell:
cy.wait500. // For a quick, known animation transition
-
cy.wait'@alias'
for Network Requests: This is a crucial strategy for handling asynchronous data loading. By intercepting network requests usingcy.intercept
, you can explicitly wait for a specific request to complete before proceeding. This is far superior tocy.waitms
.Cy.intercept’GET’, ‘/api/users’.as’getUsers’.
cy.visit’/users’.Cy.wait’@getUsers’. // Test waits until /api/users request completes
Cy.get’.user-list’.should’have.length.gt’, 0.
This ensures your tests aren’t racing ahead of data that’s still being fetched from the backend. A report by CircleCI in 2022 highlighted that network-related flakiness accounts for approximately 30% of CI failures in web applications.cy.intercept
is your weapon against this. Elementor breakpoints -
Waiting for Element Properties/States: Sometimes you need to wait for an element’s attribute to change or a specific class to be applied/removed.
Cy.get’.modal’.should’have.class’, ‘is-open’.
Cy.get’.button-group’.children.should’have.length’, 3. // Wait for children to render
-
Custom Commands for Complex Waits: For recurring complex waiting scenarios, encapsulate them in a custom Cypress command.
// commands.jsCypress.Commands.add’waitForAnimationEnd’, selector => {
cy.getselector.should$el => { Mobile compatibility testing// Example: check if a CSS transition has ended const computedStyle = window.getComputedStyle$el. if computedStyle.transitionDuration !== '0s' && parseFloatcomputedStyle.transitionDuration > 0 { // This is a simplified check.
A more robust one might involve listening to ‘transitionend’ event
// or checking transform values.
return new Promiseresolve => $el.addEventListener'transitionend', resolve, { once: true }.
}
return true. // No transition or already ended
}.
}.
// In your test:
cy.get'.animated-panel'.waitForAnimationEnd.
By prioritizing cy.should
and cy.wait'@alias'
over arbitrary cy.waitms
, you build a test suite that is significantly more resilient to the inherent asynchronous nature of modern web applications, leading to a dramatic reduction in flakiness.
Robust Selectors: The Foundation of Stable Tests
The selectors you use in your Cypress tests are the bedrock of their stability. If your selectors are brittle, dynamic, or too generic, your tests will inevitably become flaky, breaking with minor UI changes. Imagine trying to find a specific book in a library by its color alone – you might get the right one sometimes, but often you’ll pick the wrong one, or the color might change. That’s what happens with poor selectors. The goal is to create selectors that are unique, resilient to typical UI changes, and reflective of the element’s purpose rather than its transient properties. Industry best practices suggest that brittle selectors are a primary cause of test flakiness in over 40% of UI test suites.
The Pitfalls of Brittle Selectors
Brittle selectors are those that are prone to breaking easily due to minor, non-functional changes in your application’s HTML structure or styling.
- IDs generated dynamically: Frameworks like React, Angular, or Vue might generate unique IDs e.g.,
id="app-root-12345"
on each render or deployment. These are highly unreliable.- Bad Example:
cy.get'#app-root-12345'
- Bad Example:
- Class names used for styling only: While class names are common, if they’re purely for visual styling and change frequently, they become unreliable.
- Bad Example:
cy.get'.btn-primary-red-v2'
What if it changes to.btn-primary-blue-v3
?
- Bad Example:
- Deep, nested CSS selectors: Relying on a long chain of parent-child relationships makes tests fragile. A small change in the DOM structure can break the entire selector.
- Bad Example:
cy.get'div > div > ul > li:nth-child2 > a'
- Bad Example:
- Text content selectors use with caution: While useful, relying solely on text content can be problematic if the text changes for internationalization, typos, or minor wording adjustments.
- Use with caution:
cy.contains'Submit Order'
What if it becomes ‘Place Order’?
- Use with caution:
Best Practices for Resilient Selectors
The key is to use attributes that are stable and meaningful within the application’s structure. Nightwatchjs tutorial
-
Prioritize
data-test
or similar custom attributes: This is the gold standard. Add specificdata-*
attributes to elements that you intend to interact with or assert against in your tests. These attributes are explicitly for testing purposes and are unlikely to be changed by styling or structural refactoring.- Why it’s good: It clearly signals the element’s testability, decouples tests from styling/structure, and provides unique identifiers.
- Example:
<button data-cy="submit-button">Submit</button> <input type="text" data-testid="username-input">
cy.get''.click. cy.get''.type'johndoe'.
- Many organizations, including Google, advocate for this approach. A report by the National Institute of Standards and Technology NIST on software testing best practices emphasizes the use of stable, semantic selectors for automated UI tests.
-
Use
id
attributes if stable and unique: If an element has a stable, uniqueid
that is not dynamically generated, it’s a good candidate.- Example:
cy.get'#main-navigation'
- Example:
-
Leverage
cy.contains
for text content with context: When you need to find an element by its text,cy.contains
is excellent, especially when combined with a parent selector to narrow the scope.cy.get’.user-profile’.contains’Edit Profile’.click. Cypress visual testing components
- This is less brittle than
cy.get'button'.contains'Edit Profile'
if there are multiple buttons with similar text on the page.
- This is less brittle than
-
Target semantic HTML elements and attributes: Use HTML5 semantic elements e.g.,
nav
,header
,footer
,article
,section
and attributes likename
,type
,role
,aria-label
where appropriate.- Example:
cy.get'input'
orcy.get'button'
- This also has the added benefit of improving accessibility.
- Example:
-
Avoid relying on
nth-child
oreq
for dynamic lists: If the order of elements in a list can change, usingnth-childN
or.eqN
will lead to flakiness. Instead, find a unique attribute or text within the specific list item you want.- Bad Example:
cy.get'ul li'.eq1.click.
- Better Example: If list items have unique IDs or text, find by that:
cy.get'ul li:contains"Task A"'.click.
orcy.get''.click.
- Bad Example:
By investing time in crafting robust, stable selectors, you are building a resilient test suite that won’t crumble at the slightest UI tweak.
This proactive approach significantly reduces the time spent on debugging and maintaining flaky tests, allowing your team to focus on meaningful development.
Environment Consistency: Bridging the Local-CI Gap
One of the most insidious causes of flaky tests is inconsistency between local development environments and Continuous Integration CI environments. Localization testing using appium
A test that passes flawlessly on your machine might fail mysteriously in CI, or vice-versa. This disparity often stems from differences in:
- Browser versions: CI might run an older or newer version of Chrome/Electron than your local machine.
- Operating systems: Differences between Windows, macOS, and Linux can affect rendering, file paths, or even timing.
- Screen resolution and viewport sizes: A test might rely on elements being visible, but if the CI environment runs a smaller resolution, elements could be off-screen or rendered differently.
- Network speed and latency: CI environments often have faster, more consistent network access, but sometimes they can be slower or introduce different routing.
- Resource availability: CI machines might have less CPU or memory, leading to slower page loads or rendering issues.
- System time/timezone: Date and time-dependent features can fail if timezones differ.
- Database state/seed data: If your tests interact with a database, inconsistent seed data between environments can lead to failures.
- Caching: Browser caches or application caches can behave differently.
A 2023 report from a leading CI/CD provider indicated that environmental discrepancies account for approximately 25% of all CI failures not directly attributable to code bugs. Bridging this gap is crucial for a stable test suite.
Standardizing Your Cypress Environment
The key is to minimize variables.
Make your local and CI environments as similar as possible for test execution.
-
Specify Browser Versions: How to analyze appium logs
- Cypress Configuration: You can configure Cypress to run specific browsers. For CI, it’s common to use
cypress run --browser chrome
orcypress run --browser electron
. - Docker: The most effective way to ensure browser consistency is to run Cypress in a Docker container. Official Cypress Docker images
cypress/browsers
come pre-installed with specific versions of Chrome, Firefox, and Electron. This guarantees the exact browser version and environment across all runs.- Example Dockerfile snippet for CI:
FROM cypress/browsers:node16.14.0-chrome99-ff97 WORKDIR /app COPY . . RUN npm ci # ... run your tests
- Example Dockerfile snippet for CI:
- Cypress Configuration: You can configure Cypress to run specific browsers. For CI, it’s common to use
-
Control Viewport Size: Ensure your
cypress.config.js
ore2e.js
specifies a consistentviewportWidth
andviewportHeight
. This prevents issues where elements are not visible due to differing screen sizes.
// cypress.config.js
const { defineConfig } = require’cypress’.module.exports = defineConfig{
e2e: {
setupNodeEventson, config {
// implement node event listeners here
},
baseUrl: ‘http://localhost:3000‘,
viewportWidth: 1280, // Consistent width
viewportHeight: 720, // Consistent height
}, -
Manage Application State Before Each Test:
-
cy.clearCookies
andcy.clearLocalStorage
: Always clear cookies and local storage before each test or at least each spec file to ensure a clean slate. This prevents state from previous tests from interfering.// cypress/support/e2e.js or in your beforeEach hook
beforeEach => {
cy.clearCookies.
cy.clearLocalStorage.
}. Incident in software testing -
Database Seeding: For tests interacting with a backend database, implement mechanisms to reset the database to a known state before each test or suite. This is often done via API calls to your application’s backend or direct database manipulation from
setupNodeEvents
.// Example in cypress.config.js Node.js context
on’task’, {
async resetDb {// Your Node.js code to clean/seed the database
console.log’Resetting database…’.
return null.
}
}.
// In your test:
cy.task’resetDb’.
cy.visit’/’.
-
-
Handle Time-Sensitive Tests: Chrome compatibility mode
- Use libraries like
cypress-real-events
orcypress-sinon
to control the system clock if your application has time-dependent logic. This removes external time variability. - Avoid relying on
new Date
directly in your application for critical paths that tests interact with. instead, inject time.
- Use libraries like
-
Network Mocking
cy.intercept
: As discussed,cy.intercept
is vital. It allows you to consistently control network responses, preventing flakiness from slow or unreliable APIs. This also helps isolate your frontend tests from backend changes.
By meticulously managing these environmental factors, you can significantly reduce the “works on my machine” syndrome and achieve a high degree of confidence that if your tests pass locally, they will pass in CI, and vice-versa.
This consistency is a cornerstone of a reliable test automation strategy.
Test Isolation: A Clean Slate for Every Run
Test isolation is a fundamental principle in automated testing, and it’s particularly crucial for combating flakiness in Cypress. The core idea is that each test should run independently of others, starting from a known, clean state. If tests share or modify global state, or if the outcome of one test influences the next, you’re setting yourself up for unpredictable failures – the definition of flakiness. A study by IBM found that poor test isolation contributes to approximately 35% of all reported test flakiness.
Why Test Isolation Matters
- Predictability: When each test runs in isolation, you know exactly what the starting conditions are, making it easier to predict outcomes and debug failures.
- Reproducibility: A failure in an isolated test is much easier to reproduce, as you don’t need to consider the context of previous tests.
- Reliability: Eliminates the ripple effect where one failing test causes a cascade of unrelated failures down the line.
- Maintainability: Tests become easier to understand, modify, and refactor without worrying about side effects on other tests.
- Parallelization: Isolated tests can be run in parallel, significantly speeding up your CI/CD pipeline, without fear of race conditions between tests.
Achieving Isolation in Cypress
Cypress provides several mechanisms to help you achieve excellent test isolation. Defect clustering in software testing
-
Before Each Test Hook
beforeEach
: This is your primary tool for setting up a clean state before every single test.-
Visiting the Base URL: Always start by visiting the page your test needs.
cy.visit’/’. // Or specific path like ‘/login’
-
Clearing Browser State:
cy.clearCookies
: Removes all cookies. Essential for login/session-related tests.cy.clearLocalStorage
: Clears all data from local storage. Useful for user preferences, cached data, etc.cy.clearSessionStorage
: Clears session storage. Note: Cypress automatically clearssessionStorage
before each test, but explicit calls can be useful if managing state across multiple tabs or origins.
cy.visit’/dashboard’.
-
-
Backend State Reset Critical for Data-Driven Tests: View website from another country
If your application interacts with a database or persistent backend state, merely clearing browser state isn’t enough.
You need to reset your backend to a known, clean state.
* Using `cy.request` to Seed/Reset Data: Make an API call to your backend specifically designed for testing purposes to reset the database or create test data. This is often the most robust approach.
// Assuming your backend has a /api/test/reset endpoint
cy.request'POST', '/api/test/reset-db'.then => {
cy.visit'/login'.
cy.login'testuser', 'password123'. // Custom command for login
* Using `cy.task` for Node.js Backend Operations: If your backend is Node.js, you can use `cy.task` to execute arbitrary Node.js code from your tests, allowing direct database manipulation.
// In cypress.config.js
const { defineConfig } = require'cypress'.
const db = require'./db-helper'. // Your database helper functions
module.exports = defineConfig{
e2e: {
setupNodeEventson, config {
on'task', {
async seedDatabasedata {
await db.seeddata. // Implement this helper
return null.
},
async cleanDatabase {
await db.clean. // Implement this helper
}
}.
},
baseUrl: 'http://localhost:3000'
}
// In your test file
cy.task'cleanDatabase'. // Clean before each test
cy.task'seedDatabase', { users: }. // Seed specific data
cy.visit'/login'.
// ... login using the seeded user
This approach offers granular control over your backend state.
-
Mocking Network Requests
cy.intercept
: For tests that don’t need to interact with a live backend, mocking network requests usingcy.intercept
provides perfect isolation. Your frontend tests don’t depend on the backend’s availability or data, making them faster and less flaky.
beforeEach => {cy.intercept’GET’, ‘/api/users’, { fixture: ‘users.json’ }.as’getUsers’.
cy.visit’/users’.
cy.wait’@getUsers’. -
Avoid Global Variables in tests: Do not rely on
Cypress.env
or global JavaScript variables that persist across tests within the same spec file unless absolutely necessary and carefully managed. How to write a good defect report
By consistently applying these isolation techniques, you create a testing environment where each test runs in its own pristine sandbox.
This drastically reduces the likelihood of intermittent failures caused by lingering state, making your Cypress test suite robust, reliable, and a true asset to your development process.
Retrying Flaky Tests: A Smart Approach
While the ideal scenario is to eliminate flakiness at its root, some degree of flakiness can persist, especially in complex, real-world applications with external dependencies or subtle timing issues. Cypress offers a powerful built-in feature: retries
. This allows you to configure Cypress to automatically re-run a failing test a specified number of times before marking it as a definitive failure. This isn’t a silver bullet to fix flakiness, but it’s an excellent pragmatic solution to mitigate its impact, especially in CI environments, preventing unnecessary red builds and developer interruptions. A survey by Testim in 2020 showed that teams using intelligent test retries reduced their CI build failures by up to 40%.
Understanding Cypress Retries
Cypress retries work by re-executing a test from the very beginning including beforeEach
hooks if it fails. This ensures a clean slate for each retry attempt.
Configuration Options:
You can configure retries at different levels: What is test harness
-
Global Configuration in
cypress.config.js
:This applies retries to all tests in your project.
It’s suitable for projects where a low level of flakiness is generally acceptable and you want to prevent most intermittent failures from breaking the build.
retries: {
runMode: 2, // Number of retries when running cypress run e.g., in CI
openMode: 0 // Number of retries when running cypress open e.g., locally
* `runMode`: This is typically set to 1 or 2. More retries increase build time.
* `openMode`: Setting this to 0 default is recommended for local development. You want to see failures immediately when developing, not have them magically pass after retries, which hides the underlying flakiness.
-
Test-Specific Configuration:
You can override the global setting for individual tests or suites using
it
ordescribe
options.
This is useful when you know a specific test is inherently more flaky due to external dependencies or complex interactions.
describe'Payment Flow', { retries: 3 }, => { // All tests in this suite will retry 3 times
it'should process a successful payment', => {
// ... test steps
it'should handle payment failure', => {
it'should submit form very flaky test', { retries: 5 }, => { // This test retries 5 times
// ... test steps
it'should show welcome message not flaky', { retries: 0 }, => { // This test won't retry
Best Practices for Using Retries
- Don’t Use Retries to Hide Bugs: Retries are a mitigation strategy, not a solution for consistently failing or genuinely buggy tests. If a test consistently fails even with retries, it indicates a real problem that needs investigation.
- Balance Between Reliability and Speed: Each retry adds time to your CI/CD pipeline. Use the minimum number of retries necessary. Typically,
runMode: 1
or2
is sufficient to catch most transient network or timing issues. A test still failing after 2 retries is highly likely to be a genuine bug or a deeply rooted flakiness issue. - Monitor Retried Tests: Implement reporting that highlights which tests were retried and how many attempts they took to pass. Tools like Cypress Dashboard automatically provide this information. If a specific test consistently requires retries to pass, it’s a prime candidate for refactoring and debugging to eliminate its flakiness. This data-driven approach helps you prioritize your flakiness eradication efforts.
- Combine with Other Strategies: Retries are most effective when combined with robust selectors, intelligent waits, and thorough test isolation. They act as a safety net, catching the occasional unpredictable hiccup that the other strategies might not entirely prevent.
- Educate Your Team: Ensure everyone on your team understands how retries work and when to use them. Emphasize that passing on retry doesn’t mean the underlying flakiness is gone. it just means the build didn’t fail.
By strategically implementing Cypress retries, you can significantly improve the perceived stability of your CI builds, reduce developer frustration, and maintain a faster feedback loop, while still identifying areas for deeper flakiness investigation.
It’s a pragmatic tool in your arsenal against the unpredictable nature of UI automation.
Debugging Flaky Tests: Pinpointing the Problem
Debugging flaky tests can feel like chasing ghosts. They appear, disappear, and re-appear with maddening inconsistency. However, Cypress provides a rich set of debugging tools that, when used effectively, can help you pinpoint the exact moment and reason for a test’s unpredictable failure. The key is to be systematic and to leverage Cypress’s unique ability to “time travel” through your application’s state during a test run. Studies show that effective debugging practices can reduce the time spent on flaky test resolution by up to 60%.
Leveraging Cypress’s Time-Travel Debugging
Cypress’s Test Runner UI is your most powerful debugging asset.
When a test runs especially in cypress open
mode, it captures snapshots of the DOM at each step.
-
Stepping Through Commands:
- When a command like
cy.get
,cy.click
,cy.type
executes, a snapshot of the DOM before and after the command is taken. - Click on a command in the command log on the left. The application under test AUT on the right will revert to the state it was in at that exact moment. This allows you to see the DOM, network requests, and console logs at any point in the test.
- Look for:
- Element visibility: Was the element truly visible when Cypress tried to interact with it?
- DOM changes: Did an element disappear, reappear, or change attributes unexpectedly?
- Network requests: Were all expected network requests completed before the failing command?
- When a command like
-
cy.pause
:This command is like hitting the pause button during a live action movie.
It pauses the test execution at that specific point, allowing you to manually inspect the application, open developer tools, and poke around the DOM.
cy.get’.my-element’.should’be.visible’.
cy.pause. // Test pauses here
cy.click.
* Use case: Insert cy.pause
right before a command that’s frequently flaky. When the test pauses, open your browser’s developer tools F12 and inspect the element.
* Is it present?
* Is it visible?
* Is it disabled?
* Are there any overlays preventing interaction?
* Are there any console errors?
* Pro tip: In the dev tools console, type Cypress.$'.my-element'
to directly query the element using jQuery and inspect its properties.
-
cy.debug
:Similar to
cy.pause
, but instead of pausing the entire test, it pauses execution and opens the browser’s developer tools, specifically highlighting the command in question.
You can then use debugger.
statements within your test code or cy.debug
to step through code execution.
cy.get'.my-element'.debug.click. // Pauses, opens dev tools, highlights .click
* Use case: When you need to inspect the *value* of variables or debug custom commands or complex logic within your test file.
Advanced Debugging Techniques
-
Console Logs
console.log
,cy.log
:- Use
console.log
in your test code to print variable values, messages, or states. These logs appear in the browser’s developer console F12. cy.log
adds messages directly to the Cypress command log, making them visible in the Cypress Test Runner UI, which is great for readability and documenting test flow.
cy.log’Attempting to click submit button’.
cy.get’#submit-button’.click.
- Use
-
Network Tab Dev Tools and
cy.intercept
Insights:-
When debugging network-related flakiness, keep the Network tab open in your browser’s developer tools. Look for slow requests, failed requests, or unexpected response codes.
-
cy.intercept
provides powerful debugging capabilities. You can log intercepted requests and responses directly to the console:Cy.intercept’GET’, ‘/api/users’, req => {
console.log’Intercepted users request:’, req.
}.as’getUsers’. -
This helps you verify if the correct data was sent or received.
-
-
Video Recordings and Screenshots CI:
When tests run in
runMode
especially in CI, Cypress automatically records videos of test runs and takes screenshots on failure.
These are invaluable for debugging CI flakiness that you can’t reproduce locally.
* Review the video to see the exact state of the application when the failure occurred. Did an element pop up unexpectedly? Was an animation still playing? Was the page still loading?
* Analyze the failure screenshot. Is it blank? Is it an unexpected error state?
-
Error Messages:
Don’t ignore the error messages Cypress provides.
They are often highly descriptive and point directly to the problem, e.g., “Timed out retrying: Expected to find element ‘.my-element’ but never found it.” This tells you the selector was bad or the element never appeared.
By systematically applying these debugging tools and techniques, you can transform the frustrating task of chasing flaky tests into a more manageable, data-driven investigation.
The more precisely you can reproduce and observe the failure, the faster you can implement a robust solution.
Best Practices for Preventing Flakiness: A Proactive Approach
While debugging and retries are crucial for managing existing flaky tests, the ultimate goal is to prevent them from occurring in the first place. Adopting a proactive mindset and integrating best practices into your development and testing workflow can significantly reduce the likelihood of flakiness, leading to a more stable, reliable, and efficient test suite. Organizations that invest in preventative measures report a 20-30% reduction in new flaky tests within a year.
Architectural and Code-Level Practices
-
Introduce
data-test
Attributes ordata-cy
: As discussed, this is paramount. Make it a team standard to add uniquedata-*
attributes to all interactive and assertable elements. This decouples your tests from CSS class names, IDs, and structural changes.- Action: Enforce this in code reviews. Consider linting rules to flag elements missing
data-test
attributes in key components.
- Action: Enforce this in code reviews. Consider linting rules to flag elements missing
-
Modularize Tests and Use Custom Commands:
- Break down complex user flows into smaller, manageable tests.
- Extract repetitive actions into Cypress Custom Commands. This makes your tests more readable, maintainable, and reusable, reducing errors and inconsistencies.
// cypress/support/commands.js
Cypress.Commands.add’login’, username, password => {
cy.get’#username’.typeusername.
cy.get’#password’.typepassword.
cy.get’#login-button’.click.
cy.url.should’include’, ‘/dashboard’.cy.login’testuser’, ‘password123’.
-
Prioritize API Testing and Component Testing:
- Not everything needs to be an end-to-end E2E test. E2E tests are slower and more prone to flakiness.
- API Tests: Test your backend APIs directly using
cy.request
to ensure data integrity and business logic before involving the UI. This is faster and much more stable. - Component Tests: Cypress has excellent support for Component Testing. Test individual UI components in isolation e.g., a button, a form, a data table. This is faster and less flaky than E2E tests because you’re not dealing with a full application stack. A balanced test pyramid more unit/component tests, fewer E2E tests is key.
-
Isolate State and Avoid Inter-Test Dependencies:
- Always clear browser state
cy.clearCookies
,cy.clearLocalStorage
before each test. - Reset backend database state before each test or suite.
- Never assume the state from a previous test will be available or correct for the current test.
- Always clear browser state
-
Mock External Dependencies
cy.intercept
:- If your application relies on third-party APIs payment gateways, analytics, social media integrations, use
cy.intercept
to mock their responses. This removes external network latency and unreliability as sources of flakiness. - Only use real external services for dedicated integration tests, not for regular E2E suites.
- If your application relies on third-party APIs payment gateways, analytics, social media integrations, use
Testing and Development Workflow Practices
-
Early and Frequent Testing:
- Run Cypress tests frequently during development, not just before merging code. The faster you get feedback, the easier it is to identify the source of flakiness.
- Encourage developers to write Cypress tests for new features and bug fixes.
-
Consistent Environments:
- Ensure your CI/CD environment mirrors your local development environment as closely as possible browser versions, viewport, dependencies. Use Docker for consistency.
-
Review Flakiness Reports Regularly:
- Integrate Cypress with a CI dashboard or Cypress Cloud that tracks test flakiness.
- Regularly review which tests are flaky. Don’t just dismiss them. investigate and fix the root causes. Prioritize fixing the most consistently flaky tests.
-
Code Reviews with a Testing Lens:
- During code reviews, specifically look for potential flakiness triggers:
- Are new
data-test
attributes being used? - Are there hardcoded
cy.wait
calls that could be replaced with assertions? - Are selectors brittle?
- Does the new feature introduce complex animations or race conditions?
- Are new
- During code reviews, specifically look for potential flakiness triggers:
-
Use Per-Test Retries Strategically:
- While global retries can be helpful, use per-test retries only for known, deeply entrenched flaky tests that you’ve already tried to fix. This highlights problem areas.
-
Educate the Team on Async Behavior:
- Ensure all developers understand how modern web applications handle asynchronous operations and how Cypress’s retry-ability works. This helps them write more resilient application code and tests.
By embedding these proactive practices into your development lifecycle, you don’t just react to flaky tests.
You build a culture and a system that actively works to prevent them.
This leads to a more robust and efficient development process, ultimately saving time and resources.
Frequently Asked Questions
What exactly is a flaky test in Cypress?
A flaky test in Cypress is an automated test that produces inconsistent results—sometimes passing and sometimes failing—without any changes to the application code or the test code itself.
These failures are typically intermittent and hard to reproduce consistently.
Why are my Cypress tests flaky?
Cypress tests can be flaky due to several common reasons:
- Race conditions: The test attempts an action before the application is ready e.g., element not yet visible or clickable.
- Asynchronous operations: Delays in network requests or UI rendering causing the test to proceed too quickly.
- Poorly written selectors: Brittle selectors that rely on dynamic IDs or unstable CSS classes.
- Environmental inconsistencies: Differences between local and CI environments e.g., browser versions, network speed.
- Lack of test isolation: Tests leaving behind state cookies, local storage, database changes that affects subsequent tests.
- Animations and transitions: Tests failing because they interact with elements that are still animating.
How can I make my Cypress tests less flaky?
To make Cypress tests less flaky, focus on:
- Intelligent waiting: Use
cy.should'be.visible'
,cy.should'exist'
,cy.wait'@alias'
instead of arbitrarycy.waitms
. - Robust selectors: Prefer
data-test
attributes over brittle CSS selectors or dynamic IDs. - Test isolation: Clear browser state
cy.clearCookies
,cy.clearLocalStorage
and reset backend data before each test. - Network mocking: Use
cy.intercept
to control API responses and remove network variability. - Environment consistency: Ensure local and CI environments are as similar as possible e.g., use Docker.
Should I use cy.wait
to fix flaky tests?
No, using cy.waitmilliseconds
is generally discouraged as a primary solution for flaky tests.
While it can temporarily resolve some timing issues, it often masks the root cause, makes tests slower, and can break if the delay changes. It’s a “smell” for a deeper problem.
Prioritize explicit assertions like cy.should
or waiting for network requests with cy.wait'@alias'
.
What are data-test
attributes and why are they important for Cypress?
data-test
or data-cy
, data-testid
attributes are custom HTML attributes added to elements specifically for testing purposes.
They provide stable, unique identifiers that are resilient to changes in CSS classes or HTML structure.
This decouples your tests from styling and refactoring, making your selectors robust and less prone to flakiness.
How does cy.intercept
help with flaky tests?
cy.intercept
allows you to mock, stub, or spy on network requests made by your application. By intercepting requests, you can:
- Control responses: Return consistent data, preventing flakiness from unreliable APIs.
- Simulate network conditions: Test loading states or error scenarios reliably.
- Wait for requests: Use
cy.wait'@alias'
to ensure all necessary data has loaded before proceeding, addressing race conditions.
What is the role of beforeEach
in preventing flakiness?
beforeEach
is a Cypress hook that runs before every test in a spec file. It’s crucial for test isolation by:
- Visiting a clean URL: Ensures each test starts from a defined entry point.
- Clearing browser state:
cy.clearCookies
andcy.clearLocalStorage
remove lingering state from previous tests, preventing unexpected interactions. - Resetting backend state: Can be used to trigger backend API calls or tasks to reset database data, ensuring a clean slate.
Can Cypress automatically retry flaky tests?
Yes, Cypress has a built-in retries
feature.
You can configure it globally in cypress.config.js
e.g., runMode: 2
or for specific tests/suites, allowing Cypress to re-run a failing test a specified number of times before marking it as a definite failure.
This mitigates the impact of transient flakiness in CI.
Should I always use retries for my flaky tests?
No, using retries should be a mitigation strategy, not a primary fix.
While they can prevent intermittent failures from breaking your CI builds, they don’t address the root cause of flakiness.
Continually passing tests on retry indicates an underlying issue that needs investigation and permanent resolution.
Use retries judiciously and monitor which tests consistently require them.
How do I debug a flaky test in Cypress?
Debugging flaky tests involves using Cypress’s powerful tools:
- Time-travel debugging: Click on commands in the command log to see the DOM state before and after each action.
cy.pause
: Temporarily halt execution to inspect the application in the browser’s developer tools.cy.debug
: Likecy.pause
, but highlights the command in the dev tools.- Console logs: Use
console.log
orcy.log
to output values and messages. - Video recordings/screenshots: Review artifacts from CI runs to see the application’s state at the moment of failure.
- Network tab: Monitor network requests in developer tools for unexpected delays or failures.
What is test isolation and why is it important?
Test isolation means each test should run independently, starting from a known, clean state, without affecting or being affected by other tests.
It’s critical because it ensures predictability, reproducibility, and reliability.
Without it, lingering state from previous tests can lead to unpredictable outcomes and flakiness.
How does environment consistency affect test flakiness?
Differences between your local development environment and your CI/CD environment e.g., browser versions, screen resolution, network speed, OS can cause tests to pass in one place but fail in another.
Ensuring consistency across environments helps eliminate this source of flakiness.
Docker containers are highly effective for achieving this.
Can animations cause Cypress tests to be flaky?
Yes, animations and CSS transitions can definitely cause flakiness.
If your test tries to interact with an element that is still animating or transitioning, Cypress might not be able to find it or click it correctly.
You need to explicitly wait for animations to complete before interacting, often by asserting on element visibility or specific CSS properties.
Is it better to write many small tests or a few large tests?
Generally, it’s better to write many small, focused tests. Smaller tests are:
- Easier to understand and maintain.
- Faster to run.
- More isolated, reducing flakiness.
- Provide more granular feedback on failures.
While a few large tests might cover a full user flow, they are harder to debug and more prone to flakiness due to accumulated state and dependencies.
How can I handle dynamic IDs in Cypress tests?
Avoid using dynamic IDs directly.
Instead, look for stable attributes on or near the element, such as:
data-test
attributes recommended.- Stable
name
attributes for form fields. - Unique text content using
cy.contains
. - Semantic HTML tags with specific roles or attributes.
What is the “Test Pyramid” and how does it relate to flakiness?
The Test Pyramid is a strategy that advocates for writing many low-level, fast, and stable tests unit and component tests, fewer mid-level integration tests, and very few high-level end-to-end E2E tests.
E2E tests are the most expensive, slowest, and flakiest.
By having more tests at lower levels, you catch most bugs earlier and faster, reducing the reliance on, and thus the flakiness impact of, E2E tests.
Should I test third-party integrations with Cypress?
For regular E2E suites, it’s generally better to mock third-party integrations using cy.intercept
to remove external dependencies and their inherent unreliability.
You can have separate, dedicated integration tests that run against actual third-party services, but these should be run less frequently.
How do I prevent state from leaking between tests?
Prevent state leakage by:
- Using
beforeEach
tocy.clearCookies
,cy.clearLocalStorage
, and potentiallycy.clearSessionStorage
. - Resetting backend data via
cy.request
orcy.task
calls to a test-specific API endpoint. - Avoiding global variables in your test files that persist across
it
blocks.
What logging strategies are useful for debugging flaky tests?
Use cy.log
to add custom messages to the Cypress Command Log, helping you trace test execution.
Use console.log
within your test code or custom commands for deeper introspection and variable inspection in the browser’s developer console.
Combine this with cy.debug
and cy.pause
for interactive debugging.
Is Cypress Dashboard helpful for identifying flaky tests?
Yes, Cypress Dashboard Cypress Cloud is incredibly helpful. It provides analytics on test runs, including:
- Flakiness detection: Identifies tests that consistently fail and then pass on retry.
- Run history: Shows trends and historical data for test stability.
- Video recordings and screenshots: Provides visual evidence of failures in CI, which is invaluable for debugging intermittent issues.
- Performance metrics: Helps identify slow tests or bottlenecks.
Leave a Reply