A Node.js tool to automate end-to-end web testing.

Are you requesting a feature or reporting a bug?

Feature

What is the current behavior?

You can run test only in single window of the browser

What is the expected behavior?

It's useful to run test in multiple windows to simultaneously observe results from different users perspective. E.g. we test online chat or online game. The idea is to spawn new windows in test and switch test context to that window (It also we'll play well with Roles feature):

test('Multi-window', async t => {
     await t
           .switchRole(user1)
           .click('#start-game')

           .switchToWindow(1)   // Spawns new browser window if it wasn't used before
           .switchRole(user2)
           .click('#start-game')
           .click('#shoot')

           .switchToWindow(0); // Return to default window

    expect(await healthbar.value).eql(0);   
});

The syntax is still a subject for bikeshedding. \cc @DevExpress/testcafe

\cc @DevExpress/testcafe-docs This thing is quite big. Overall, it's finished but not quite polished in each and every corner. That's because I expect you to find some major drawbacks that will require rewriting or rearranging some parts.

I understand you'll want to read it bit by bit, during the next couple of days.

closes #285

This is a general place for the APIv2 discussion. I'll move proposals from wiki to here. Any suggestions/ideas/proposals are welcome. \cc @DevExpress/testcafe

This PR will add a new section to the config file: "clientScripts"

example:

 {
    "src": "test.js",
    "browsers": [
        "chrome",
    ],
    "clientScripts": [
        "./node_modules/@testing-library/dom/dist/@testing-library/dom.umd.js"
    ]
}

adding this to your config will cause testcafe to inject the above scripts into the browser as a script tag.

This will be very useful for testcafe-testing-library which needs to inject DomTestingLibrary into the dom, on each page-load, and will thus now only need to be configured once, and will then work on each page.

resolves https://github.com/DevExpress/testcafe/issues/1739

https://github.com/DevExpress/testcafe/issues/481

Changes

1. Add t.browser.

t.browser examples

{
  alias: 'saucelabs:Internet [email protected]:Windows 7',
  headless: false,
  name: 'Internet Explorer',
  version: '11.0',
  platform: 'desktop',
  os: { name: 'Windows', version: '7' },
  engine: { name: 'Trident', version: '7.0' },
  prettyUserAgent: 'Internet Explorer 11.0 / Windows 7',
  userAgent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko'
}

{
  alias: 'chrome',
  headless: false,
  name: 'Chrome',
  version: '77.0.3865.120',
  platform: 'desktop',
  os: { name: 'Windows', version: '10' },
  engine: { name: 'Blink' },
  prettyUserAgent: 'Chrome 77.0.3865.120 / Windows 10',
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36'
}

{
  alias: 'chrome:headless --no-sandbox',
  headless: true,
  name: 'Chrome',
  version: '77.0.3865.120',
  platform: 'desktop',
  os: { name: 'Windows', version: '10' },
  engine: { name: 'Blink' },
  prettyUserAgent: 'Chrome 77.0.3865.120 / Windows 10',
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/77.0.3865.120 Safari/537.36'
}

{
  alias: 'firefox:headless:disableMultiprocessing=true',
  headless: true,
  name: 'Firefox',
  version: '69.0',
  platform: 'desktop',
  os: { name: 'Windows', version: '10' },
  engine: { name: 'Gecko', version: '20100101' },
  prettyUserAgent: 'Firefox 69.0 / Windows 10',
  userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0'
}

{
  alias: 'saucelabs:iPad Air [email protected]',
  headless: false,
  name: 'Safari',
  version: '12.1',
  platform: 'tablet',
  os: { name: 'iOS', version: '12.2' },
  engine: { name: 'WebKit', version: '605.1.15' },
  prettyUserAgent: 'Safari 12.1 / iOS 12.2',
  userAgent: 'Mozilla/5.0 (iPad; CPU OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1 Mobile/15E148 Safari/604.1'
}

{
  alias: 'saucelabs:[email protected]:macOS 10.14',
  headless: false,
  name: 'Chrome',
  version: '77.0.3865.75',
  platform: 'desktop',
  os: { name: 'macOS', version: '10.14.0' },
  engine: { name: 'Blink' },
  prettyUserAgent: 'Chrome 77.0.3865.75 / macOS 10.14.0',
  userAgent: 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.75 Safari/537.36'
}

{
  alias: 'saucelabs:Samsung Galaxy S9 Plus WQHD GoogleAPI [email protected]',
  headless: false,
  name: 'Chrome',
  version: '67.0.3396.87',
  platform: 'mobile',
  os: { name: 'Android', version: '8.1.0' },
  engine: { name: 'Blink' },
  prettyUserAgent: 'Chrome 67.0.3396.87 / Android 8.1.0',
  userAgent: 'Mozilla/5.0 (Linux; Android 8.1.0; Android SDK built for x86 Build/OSM1.180201.026) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.87 Mobile Safari/537.36'
}

Are you requesting a feature or reporting a bug?

Reporting a possible bug

What is the current behavior?

Some requests in chrome seem to return 222 from the proxy and no json content, however the same tests work correctly in firefox and safari. There don't seem to be major differences investigating curl differences which sort of leads me to believe it's a chrome fetch related issue

What is the expected behavior?

That a page with a third party xhr request works consistently in browser.

How would you reproduce the current behavior (if this is a bug)?

Do not have a repro currently, it seems that xhr requests in chrome are not currently working

Specify your

• operating system: macOS Sierra 10.12.1
• testcafe version: 0.11 and 0.12-alpha5
• node.js version: 7.1

/cc @AlexanderMoskovkin @AndreyBelym @MargaritaLoseva please help with comment: https://github.com/helen-dikareva/testcafe-phoenix/commit/3526257d02f8d3cbb520413fcfb9863570eaecb5#diff-5b7bb7364f1a533c7d2d73407a451457R352

I see that there are some bug reports, that describe similar (or the same) issue, e.g. # 2406, however they have been closed for inactivity or other reasons, and I still don't see a solution. Hopefully, my feedback could help identify the issue.

What is your Test Scenario?

I am trying a very simple test - verifying that some text, for example "Host Status", exists on a page.

What is the Current behavior?

I see the following error:

1) Content of the iframe in which the test is currently operating did not load.

when using testcafe ver. 1.2.1. In addition to this, I see:

(node:1842) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.

None of these errors were present in testcafe ver. 0.20.0.

Here's the entire output:

 Running tests in:
 - Firefox 67.0.0 / Ubuntu 0.0.0

 Nagios XI Tests
(node:1842) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
 ✖ Host Status

   1) Content of the iframe in which the test is currently operating did not load.

      Browser: Firefox 67.0.0 / Ubuntu 0.0.0

         30 |  await t
         31 |    .useRole(admin)
         32 |    .click( Selector('a').withText('Home') )
         33 |    .click( Selector('a').withText('Host Status') )
         34 |    .switchToIframe( '#maincontentframe' )
       > 35 |    .expect( Selector('h1').withText('Host Status').exists ).ok()
         36 |    .expect( Selector('.hoststatustar_').withText('All hosts').exists ).ok()
         37 |    .expect( Selector('.xi-table-recordcount').withText('Showing').exists ).ok()
         38 |});
         39 |

         at ok (/home/ludmil/testcafe/XI/Home/Left Menu/Details/host_status.js:35:62)
         at <anonymous> (/home/ludmil/testcafe/XI/Home/Left Menu/Details/host_status.js:29:1)
         at <anonymous> (/usr/lib/node_modules/testcafe/src/api/wrap-test-function.js:17:28)
         at TestRun._executeTestFn (/usr/lib/node_modules/testcafe/src/test-run/index.js:243:19)
         at TestRun.start (/usr/lib/node_modules/testcafe/src/test-run/index.js:292:24)



 1/1 failed (38s)

What is the Expected behavior?

I expect my test to succeed as it used to in testcafe ver. 0.20.0.

Example:

 Running tests in:
 - Firefox 61.0.0 / Ubuntu 0.0.0

 Nagios XI Tests
 ✓ Host Status


 1 passed (29s)

What is your web application and your TestCafe test code?

Public link: http://nagiosxi.demos.nagios.com/nagiosxi

host_status.js

import { Role } from 'testcafe';
import minimist from 'minimist';

const args = minimist(process.argv.slice(2));
const baseurl = args.baseurl;

fixture ('Nagios XI Tests')
  .page( baseurl )

// Roles

const admin = Role(baseurl, async t => {
  await t
    .typeText('#usernameBox', 'nagiosadmin')
    .typeText('#passwordBox', 'nagiosadmin')
    .click('#loginButton')
});

// Tests

/* Template
test('Test Name', async t => {
  await t
    .useRole(admin)
});
*/

test('Host Status', async t => {
  await t
    .useRole(admin)
    .click( Selector('a').withText('Home') )
    .click( Selector('a').withText('Host Status') )
    .switchToIframe( '#maincontentframe' )
    .expect( Selector('h1').withText('Host Status').exists ).ok()
    .expect( Selector('.hoststatustar_').withText('All hosts').exists ).ok()
    .expect( Selector('.xi-table-recordcount').withText('Showing').exists ).ok()
});

Your Environment details:

testcafe version: 1.2.1

node.js version: v12.4.0

command-line arguments: testcafe -c 1 firefox:headless host_status.js --baseurl='http://nagiosxi.demos.nagios.com/nagiosxi' -e

browser name and version: Firefox Quantum 67.0.2 (64-bit)

platform and version: Ubuntu 18.04.2 LTS

Typedefs will be updated in a separate PR.

Feature has 2 parts: in testcafe and in hammerhead. Firstly, the testcafe part will be reviewed. After that - the hammerhead (https://github.com/DevExpress/testcafe-hammerhead/pull/1527).

For convenience,in this PR the hammerhead part used as module - see https://github.com/DevExpress/testcafe/compare/master...miherlosev:request_hooks?expand=1#diff-b9cfc7f2cdf78a7f4b91a753d10865a2R106.

Changes:

• add RequestMock, RequestLogger and RequestHook classes
• rename ClientFunctionResultPromise to ReExecutablePromise.
• add API for add/removing request hooks on fixture, page and testcontroller levels
• add RequestHookConfigureAPIError class
• move assertThrow to testing helpers

Are you requesting a feature or reporting a bug?

Feature

It's hard for browser test frameworks to know when a page has fully loaded. Therefore developers of the tests should define this case.

If I have a test like this:

test('example', async t => {
    await t.navigateTo('http://example.com');
   // Page should be ready here
});

I would like to let TestCafe know when my page is ready. So far, as a hack, I was thinking of hooking into the beforeEach callback, overwriting the test.navigateTo method, and ensuring the page is ready before test execution continues.

Instead, do you have any better ideas? I'm trying to avoid doing this:

test('example', async t => {
   await t.navigateTo('http://example.com');
   await pageToBeReady();
});

Because I'd have to repeat that in each test. Ideally, I'd only have to define a page load strategy once.

What is your Test Scenario?

I'm trying to run testcafe in our pipeline (Semaphore) using a docker image based on the official one, where the only additions are copying our tests inside it and install some other additional npm packages used by them. Those tests run against a test environment, that for security reasons can only be accessed either via VPN or a proxy. I'm using the --proxy flag but the test run fails.

From that same container, if I run curl https://mytestserver.dev --proxy $HTTP_PROXY it connects without problems, so does a Cypress container running in the same pipeline.

What is the Current behavior?

Tests fail to run, with the message

ERROR Unable to establish one or more of the specified browser connections.
1 of 1 browser connections have not been established:
- chromium:headless

Hints:
- Use the "browserInitTimeout" option to allow more time for the browser to start. The timeout is set to 2 minutes for local browsers and 6 minutes for remote browsers.
- The error can also be caused by network issues or remote device failure. Make sure that the connection is stable and the remote device can be reached.

As per suggestion here I set

DEBUG=hammerhead:*,testcafe:*

And the log is:

021-07-07T15:23:44.572Z testcafe:utils:temp-directory Found temp directories: []
2021-07-07T15:23:44.574Z testcafe:utils:temp-directory Temp directory path:  /tmp/testcafe/chrome-profile-128Nvnnsbk1AEpG
2021-07-07T15:23:44.587Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:46.423Z testcafe:compiler:typescript version: 3.9.10
2021-07-07T15:23:46.425Z testcafe:compiler:typescript options: [Object: null prototype] {
  experimentalDecorators: true,
  emitDecoratorMetadata: true,
  allowJs: true,
  pretty: true,
  inlineSourceMap: true,
  noImplicitAny: false,
  module: 1,
  moduleResolution: 2,
  target: 3,
  jsx: 2,
  suppressOutputPathCheck: true,
  skipLibCheck: true
}
2021-07-07T15:23:48.119Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.120Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.121Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.122Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.124Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.221Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.222Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.223Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.224Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.225Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.226Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.227Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.228Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.230Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.232Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.232Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.234Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.235Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.236Z testcafe:compiler:typescript path: "typescript"
2021-07-07T15:23:48.245Z testcafe:bootstrapper tests compilation took 3.67 s
2021-07-07T15:25:44.563Z testcafe:browser:connection:469Ca~Y status changed to 'closing'
2021-07-07T15:25:44.585Z testcafe:browser:connection:469Ca~Y TypeError: Cannot read property 'config' of undefined
    at BrowserProviderPluginHost.isHeadlessBrowser (/app/node_modules/testcafe/src/browser/provider/built-in/dedicated/base.js:65:51)
    at BrowserProvider.canUseDefaultWindowActions (/app/node_modules/testcafe/src/browser/provider/index.ts:260:53)
    at BrowserProvider.closeBrowser (/app/node_modules/testcafe/src/browser/provider/index.ts:323:44)
    at BrowserConnection._closeBrowser (/app/node_modules/testcafe/src/browser/connection/index.ts:208:13)
    at BrowserConnection.close (/app/node_modules/testcafe/src/browser/connection/index.ts:410:9)
    at Function._closeConnection (/app/node_modules/testcafe/src/runner/browser-set.ts:55:9)
    at async Promise.all (index 0)
    at BrowserSet.dispose (/app/node_modules/testcafe/src/runner/browser-set.ts:163:9)
    at Function.from (/app/node_modules/testcafe/src/runner/browser-set.ts:108:13)
    at async Promise.all (index 0)
2021-07-07T15:25:44.586Z testcafe:browser:connection:469Ca~Y status changed to 'closed'

I had the chance to get the logs in our proxy server, and found out this entry coming from the ip in semaphore where the container is running:

TCP_DENIED/403 4464 GET http://172.17.0.2:1337/browser/connect/sYNp3vU - HIER_NONE/- text/html

Denying access to that request is logical as it points to a local network address.

Incidentally I tried using firefox:headless just in case but found out that the --proxy option is completely ignored.

What is the Expected behavior?

I would expect the hammerhead proxy to issue requests to our proxy with the urls that our tests are visiting, not a local ip address.

What is your Scenario?

Run tests using the latest TestCafe version (2.2.0)

What is the Current behavior?

There is an error that prevents to run tests for all browsers when using the latest TestCafe version (2.2.0).

The error is:

ERROR TypeError: dns_1.default.setDefaultResultOrder is not a function at new Proxy (/Users/ayemelyanenko/testcafe-examples/node_modules/testcafe-hammerhead/lib/proxy/index.js:75:23) at new TestCafe (/Users/ayemelyanenko/testcafe-examples/node_modules/testcafe/src/testcafe.js:30:30) at Object.construct (/Users/ayemelyanenko/testcafe-examples/node_modules/import-lazy/index.js:19:20) at createTestCafe (/Users/ayemelyanenko/testcafe-examples/node_modules/testcafe/src/index.js:93:22) at processTicksAndRejections (internal/process/task_queues.js:93:5) at runTests (/Users/ayemelyanenko/testcafe-examples/node_modules/testcafe/src/cli/cli.js:92:22) at cli (/Users/ayemelyanenko/testcafe-examples/node_modules/testcafe/src/cli/cli.js:196:13)

What is the Expected behavior?

User should be able to run tests on all browsers when using the latest TestCafe version (2.2) and when using either npm or yarn.

What is your public website URL? (or attach your complete example)

https://github.com/DevExpress/testcafe-examples

What is your TestCafe test code?

The test examples are present in the attached repo from DevExpress

Your complete configuration file

No response

Your complete test report

No response

Screenshots

No response

Steps to Reproduce

1. Clone https://github.com/DevExpress/testcafe-examples
2. npm uninstall existing testcafe version (2.1)
3. npm install testcafe to get the latest TestCafe version (2.2)
4. Attempt to run tests by using npm test

TestCafe version

2.2.0

Node.js version

14.15.0

Command-line arguments

npm test

Browser name(s) and version(s)

No response

Platform(s) and version(s)

No response

Other

No response

What is your Scenario?

Unstable tests are an unfortunate truth about E2E testing. During my experience with TestCafe, I discovered a technique to reliably check the test stability and reproduce a decent amount of rare failures: wrap the test in a for-loop, only the test, run the parent fixture with medium to high concurrency.

What this helps to uncover:

1. Application race conditions. Actual bugs to fix, the tests did their job.
2. Rate limiting issues (for example, Stripe).
3. Suboptimal lazy content loading assertions in tests.

What are you suggesting?

The ability to run a specific test an arbitrary number of times in order to sample the test stability seems like a good fit for any test runner. Add a new runner option repeat (or something like that). Since TestCafe comes with its own test runner, it would be really handy to have such a feature.

What alternatives have you considered?

• Manually modify the test suite. Pros: can be done with the current TestCafe. Cons: not suitable for CI; limited discoverability by other developers, resulting in an overall more unstable suite.
• Modify the test suite at the AST level. Pros: can be automated; can be done with the current TestCafe. Cons: requires custom instrumentation.
• Run the test suite multiple times with shell scripting. Pros: can be automated; can be done with the current TestCafe. Cons: significant overhead, increasing overall run time; reduced CPU utilization.

Additional context

No response

What is your Scenario?

I am trying to click a button that opens up an Integry modal but during automation I keep receiving this error:

It also doesn't work when unlocking the window and clicking it manually, but it does work fine when clicking it manually outside of automation.

What is the Current behavior?

An error gets thrown at t.setAttributeCore (http://localhost:65062/hammerhead.js:10:28517) when clicking the button.

What is the Expected behavior?

Clicking the button opens the Integry modal without any issues.

What is your public website URL? (or attach your complete example)

DevExpress has access to our repo. Please email me if re-authentication is necessary.

What is your TestCafe test code?

integrations_page_tests.js

test
    .meta({ created_on: '12/27/22', created_by: 'Justin Grabenbauer', updated_on: '', updated_by: '' })
    .meta({ scope: 'sanity', priority: 'low', showstopper: 'no', github_ID: '1585' })
    ('user_can_add_an_integration', async t => {
        
        // Purpose: To verify that a user can add an integration

        const integration_name = 'ClickFunnels Classic'

        await WorkspaceMainNavPageObjects.navigate_to_workspace_settings()
        await SettingsPageObjects.navigate_to_integrations_page()
        await t.debug()
        await IntegrationsPageObjects.connect_integration(integration_name)
})

integrations_page_repo.js

import { Selector, t } from 'testcafe'


class IntegrationsPageObjects {
    constructor() {
        this.btn_connect_integration = Selector('button').withText('Connect integration')
    }

    async connect_integration(integration_name) {

        // Purpose: To connect an integration

        await t
            .click(this.btn_connect_integration)
            .click(Selector('div').withText(integration_name))
    }
}

export default new IntegrationsPageObjects()

Your complete configuration file

require('dotenv').config()
const data = require('./data/data.json')
const { RequestLogger } = require('testcafe')

// TODO: See if we can make this part local to a function instead of out in the open, globally.
// Set the URL for where we want to start logging
let logger_url = data.api_endpoints.notifications_dropdown

switch(process.env.ENV) {
    case 'staging':
        logger_url = data.authenticated_urls.staging.concat(logger_url)
        break

    case 'production':
        logger_url = data.authenticated_urls.production.concat(logger_url)
        break
    }
    
const notifications_dropdown_logger = RequestLogger(logger_url)

// Agnostic for all environments
let skipped_tests = [

    // Located in all_contacts_page_tests.js
    'user_can_modify_contacts_table_columns', // 12-19-22
    
    // Located in all_products_page_tests.js
    'user_can_create_a_subscription_product', // 09-27-22
    'user_can_set_a_price_for_physical_products', // 09-27-22

    // Located in apps_page_tests.js
    'user_cannot_access_sites_when_feature_is_disabled', // 07-28-22
    'user_cannot_access_settings_when_feature_is_disabled', // 07-28-22
    'user_cannot_access_analytics_when_feature_is_disabled', // 09-16-22
    'user_cannot_access_marketing_when_feature_is_disabled', // 09-16-22
    'user_cannot_access_products_when_feature_is_disabled', // 09-16-22
    'user_cannot_access_contacts_when_feature_is_disabled', // 09-16-22
    'user_cannot_access_orders_when_feature_is_disabled', // 09-16-22
    'user_cannot_access_sales_when_feature_is_disabled', // 09-16-22

    // Located in billing_page_tests.js
    'user_can_navigate_to_billing_page', // 08-22-22

    // Located in blog_page_tests.js
    'user_is_able_to_change_blog_template', // 12-06-22
    'user_can_add_a_gif_to_blog', // 11-30-22
    'user_can_change_blog_post_style', // 12-06-22
    'user_can_clone_a_blog_post', // 11-30-22
    'user_can_create_a_blog_post', // 11-30-22
    'user_can_edit_blog_settings', // 07-06-22
    'user_can_edit_existing_blog_post_information', // 11-30-22
    
    // Located in courses_page_tests.js
    'courses_back_button_navigates_to_previous_page',
    'optin_user_can_navigate_through_course', // 12-06-22
    'user_can_log_into_customer_center_with_magic_link',

    // Located in dashboard_page_tests.js
    'user_can_navigate_to_forms_page_from_dashboard', // 11-22-22
    'user_can_navigate_to_payouts_page_from_dashboard', // 11-22-22

    // Located in funnels_page_tests.js
    'user_can_clone_a_funnel',
    'user_can_navigate_through_funnel',
    
    // Located in marketing/broadcasts_page_tests.js
    'user_can_create_new_broadcast', // 06-20-22

    // Located in marketing/templates_page_tests.js
    'user_can_create_broadcast_template', // 06-13-22

    // Located in orders/all_orders_page_tests.js
    'user_can_refund_order', // 10/18/22

    // Located in orders/order_page_tests.js
    'user_is_able_to_create_new_order', // 09-27-22
    'user_is_able_to_complete_payment_on_order', // 09-27-22
    'user_is_able_to_partially_pay_order', // 09-27-22
    'user_is_able_to_refund_order', // 09-27-22
    'user_is_able_to_set_delayed_status_on_order', // 09-27-22
    'user_is_able_to_set_processing_status_on_order', // 09-27-22
    'user_is_able_to_set_packed_status_on_order', // 09-27-22
    'user_is_able_to_set_ready_status_on_order', // 09-27-22
    'user_is_able_to_set_delivered_status_on_order', // 09-27-22
    'user_is_able_to_set_shipped_status_on_order', // 09-27-22
    'user_is_able_to_set_backorder_status_on_order', // 09-27-22
    'user_is_able_to_set_pending_status_on_order', // 09-27-22
    'user_is_able_to_set_lost_status_on_order', // 09-27-22
    'user_can_add_multiple_fulfillments_to_order', // 09-27-22
    'user_can_partially_fulfill_order', // 08-11-22

    // Located in pages_page_tests.js
    'user_can_pin_a_page_from_edit_page', //07-29-22

    // Located in pipelines_page_tests.js
    'user_can_navigate_to_pipelines_page', // 09/14/22

    // Located in reporting_page_tests.js
    'opt-ins_by_landing_page_report_displays', // 06/21/22

    // Located in sites_overview_page_tests.js
    'user_can_add_header_to_site', // 07-06-22
    'user_can_change_site_style', // 10-31-22
    'user_can_edit_site_name', // 10-04-22
    'user_can_make_site_password_protected', // 10-03-22

    // Located in themes_page_tests.js
    'user_can_edit_theme_name', // 08-08-22

    // Located in url_redirects_page_tests.js
    'user_can_delete_a_url_redirect', // 11-30-22
    'user_is_able_to_edit_url_redirect' // 11-30-22
]

// Cleanup scripts that need to be skipped, but can't be in the skipped tests
let clean_up_tests = [
    // Located in clean_up_scripts.js
    'clean_up_copies_of_funnels',
    'clean_up_copies_of_funnel_pages',
    'clean_up_copies_of_pages', 
    'clean_up_products',
    'clean_up_site_pages',
    'clean_up_workflows',
    'dismiss_notifications',
    'clean_up_collaborators',
]

// Production specific scripts needing to be skipped for various reasons
let production_skipped_tests = [
    'user_can_navigate_to_affiliate_center_page', // Disabling because the link is disabled in production as of 12/07/22
    'user_can_navigate_to_order_tags_page', // Link disabled through Launch Darkly as of 08/29/22
    'user_can_navigate_to_payments_ai_dashboard', // Production PAI account not set up as of 08/29/22
    'user_is_able_to_navigate_to_the_inbox_page', // Inbox link is not activated in production as of 08/31/22
    'customer_can_place_order_through_funnel', // Disabling until we have PAI set up in production as of 10/04/22
    'user_can_navigate_to_sequences_page', // Disabling because it's behind a feature gate in produciton as of 10/17/22
    'user_can_navigate_to_events_and_deadlines_page', // Disabling because the link is disabled in production as of 11/28/22
    'user_can_navigate_to_community_page' // Disabling because the link doesn't display in production as of 12/21/22
]

// Scripts that need to run with a single concurrency
let isolated_tests = [
    // Located in sites_overview_page_tests.js
    'user_can_make_site_public',
    'user_can_edit_site_name',
    'user_can_make_site_password_protected',
    'user_can_make_site_private',
    'user_can_edit_site_domain',
    'changing_site_domain_updates_funnel_urls',
    'changing_site_domain_updates_blog_urls',
    'changing_site_domain_updates_course_urls',
    'user_can_add_header_to_site',

    // Located in account_details_page_tests.js
    'user_can_edit_account_name',
    'user_can_edit_account_password',
    'user_can_change_account_language',

    // Located in general_page_tests.js
    'user_can_edit_workspace_details',

    // Located in blog_page_tests.js
    'user_can_edit_blog_settings',

    // Located in themes_page_tests.js
    'user_can_promote_theme_to_live_from_index_page',
    'user_can_promote_theme_to_live_from_overview_page',

    // Located in the workflow_builder_page_tests.js
    'broadcast_room_category_pulls_back_expected_pages',
    'downsell_category_pulls_back_expected_pages',
    'landers_category_pulls_back_expected_pages',
    'my_funnel_pages_category_pulls_back_expected_pages',
    'opt_in_category_pulls_back_expected_pages',
    'order_category_pulls_back_expected_pages',
    'order_confirmation_category_pulls_back_expected_pages',
    'sales_category_pulls_back_expected_pages',
    'terms_of_service_category_pulls_back_expected_pages',
    'thank_you_category_pulls_back_expected_pages',
    'upsell_category_pulls_back_expected_pages',
    'webinar_confirmation_category_pulls_back_expected_pages',
    'webinar_registration_category_pulls_back_expected_pages'
]

// Filter that joins the skipped scripts with the production skipped tests
let production_filter = skipped_tests.concat(production_skipped_tests).concat(clean_up_tests)

// Filter that joins ALL filtered scripts so that they are not run for a full suite
let full_filter = skipped_tests.concat(isolated_tests).concat(clean_up_tests)

let build_filter = () => {

    // Purpose: To set the suite that we want to run. (isolated, regression, sanity)

    var filter = {}
    
    switch(process.env.ENV) {
        case 'production':
            if(process.env.SCOPE) {
                filter = {
                    testGrep: `^(?!.*(${production_filter.join('|')})).*$`,
                    testMeta: {
                        scope: process.env.SCOPE
                    }
                }
            }
            break
        case 'staging':
            if(process.env.SCOPE) {
                filter = {
                    testGrep: `^(?!.*(${skipped_tests.join('|')})).*$`, 
                    testMeta: {
                        scope: process.env.SCOPE
                    }
                }
            } else {
                filter = {
                    testGrep: `^(?!.*(${full_filter.join('|')})).*$`
                }
            }
            break
    }

    return filter
}

let determine_concurrency = () => {

    // Purpose: To set concurrency dependant on the scope that is passed
    
    let concurrency = 5
    if (process.env.SCOPE == 'isolated') {
        concurrency = 1
    } 
    return concurrency
}

module.exports = {
    assertionTimeout: 5000,
    browsers: [
        "chrome:headless --window-size=1920,1159"
    ],
    concurrency: determine_concurrency(),
    cache: true,
    clientScripts: [
        "scripts/click_expand_all_for_courses_details_page.js"
    ],
    disableMultipleWindows: true,
    filter: build_filter(),
    hooks: {
        fixture: {
            before: async ctx => {
                // Set the notifications dropdown logger globally
                ctx.notifications_dropdown_logger = notifications_dropdown_logger
            }
        },
        // Set the request hook for the notifications dropdown logger
        request: notifications_dropdown_logger
    },
    hostname: "localhost",
    pageLoadTimeout: 30000,
    quarantineMode: {
        successThreshold: 1,
        attemptLimit: 2
    },
    reporter: [
        {
            name: "spec",
            output: "artifacts/reports/spec_results"
        },
        {
            name: "xunit",
            output: "artifacts/reports/xunit_results.xml"
        }, 
        {
            name: "testrail"
        }
    ],
    runExecutionTimeout: 1800000,
    screenshots: {
        path: "artifacts/screenshots",
        pathPattern: "${TEST}_${DATE}_${TIME}.png",
        takeOnFails: true,
        thumbnails: false
    },
    selectorTimeout: 5000,
    skipJsErrors: true,
    skipUncaughtErrors: true,
    src: "tests/**",
    testExecutionTimeout: 240000
}

Your complete test report

Running tests in:

• Chrome 108.0.0.0 / Ventura 13

settings - integrations ✖ user_can_add_an_integration (screenshots: /Users/justingrabenbauer/cf2-test-cafe-poc/artifacts/screenshots/user_can_add_an_integration_2022-12-27_10-49-53.png)

1. The specified selector does not match any element in the DOM tree.

     | Selector('div')  > | .withText('ClickFunnels Classic')

   Browser: Chrome 108.0.0.0 / macOS 10.15.7 Screenshot: /Users/justingrabenbauer/cf2-test-cafe-poc/artifacts/screenshots/user_can_add_an_integration_2022-12-27_10-49-53.png

   11 | 12 | // Purpose: To click the Connect integration button 13 | 14 | await t 15 | .click(this.btn_connect_integration)

   16 | .click(Selector('div').withText(integration_name)) 17 | } 18 |} 19 | 20 |export default new IntegrationsPageObjects() 21 |

   at (/Users/justingrabenbauer/cf2-test-cafe-poc/pom/settings/integrations/integrations_page_repo.js:16:14) at asyncGeneratorStep (/Users/justingrabenbauer/cf2-test-cafe-poc/pom/settings/integrations/integrations_page_repo.js:2:258) at _next (/Users/justingrabenbauer/cf2-test-cafe-poc/pom/settings/integrations/integrations_page_repo.js:2:596) at (/Users/justingrabenbauer/cf2-test-cafe-poc/pom/settings/integrations/integrations_page_repo.js:2:761) at (/Users/justingrabenbauer/cf2-test-cafe-poc/pom/settings/integrations/integrations_page_repo.js:2:502) at IntegrationsPageObjects.connect_integration (/Users/justingrabenbauer/cf2-test-cafe-poc/pom/settings/integrations/integrations_page_repo.js:10:49) at (/Users/justingrabenbauer/cf2-test-cafe-poc/tests/settings/integrations_page_tests.js:45:39) at asyncGeneratorStep (/Users/justingrabenbauer/cf2-test-cafe-poc/tests/settings/integrations_page_tests.js:7:315) at _next (/Users/justingrabenbauer/cf2-test-cafe-poc/tests/settings/integrations_page_tests.js:7:653)

1/1 failed (1m 17s)

Warnings (2):

The "browsers", "concurrency", and "filter" options from the configuration file will be ignored.

The file at "/Users/justingrabenbauer/cf2-test-cafe-poc/artifacts/screenshots/user_can_add_an_integration_2022-12-27_10-49-53.png" already exists. It has just been rewritten with a recent screenshot. This situation can possibly cause issues. To avoid them, make sure that each screenshot has a unique path. If a test runs in multiple browsers, consider including the user agent in the screenshot path or generate a unique identifier in another way.

Screenshots

No response

Steps to Reproduce

1. Run script
2. Check console

TestCafe version

2.0.0

Node.js version

18.12.0

Command-line arguments

ENV=staging testcafe "chrome '--window-size=1920,1159'" -t 'user_can_add_an_integration' -c1

Browser name(s) and version(s)

Chrome 108

Platform(s) and version(s)

macOS Ventura 13

Other

No response

What is your Scenario?

Enabling the experimentalProxyless option breaks any selectors from the @testing-library/testcafe.

What is the Current behavior?

Just created a simple project based on the NextJS and passed experimentalProxyless: true to the testcafe options. And I get the following error from the simple test. Error: Cannot call method 'queryByRole' of undefined

What is the Expected behavior?

Selectors from @testing-library/testcafe should work experimentalProxyless option.

What is your public website URL? (or attach your complete example)

https://github.com/Roy412/msw-testcafe-error/tree/testing_library_compatibility

What is your TestCafe test code?

import { screen } from "@testing-library/testcafe";

fixture("Rest").page`http://localhost:3000/`;

test("should render the page correctly", async (t) => {
  await t
    .expect(screen.findByRole('heading',{ name: 'Loading...' }).exists).ok('Page loaded')
});

Your complete configuration file

No response

Your complete test report

No response

Screenshots

Steps to Reproduce

1. Clone the repo: https://github.com/Roy412/msw-testcafe-error/tree/testing_library_compatibility
2. Yarn install: yarn
3. Run testcafe test: yarn testcafe

TestCafe version

2.2.0-rc.1

Node.js version

v16.16.0

Command-line arguments

NEXT_PUBLIC_IS_TESTCAFE=1 node runtestCafe.js

Browser name(s) and version(s)

No response

Platform(s) and version(s)

No response

Other

No response

What is your Scenario?

I'm running tests on a page that takes a long time to load. I use the pageLoadTimeout option to delay running tests until the page is loaded.

What is the Current behavior?

Tests are run without waiting for either the timeout specified by the pageLoadTimeout option or the window.onload event.

What is the Expected behavior?

Tests should be run within the specified timeout, according to the documentation.

What is your TestCafe test code?

import { ClientFunction } from 'testcafe';

fixture `New Fixture`
    .page `localhost:3000`;

test(`New Test`, async t => {
    await ClientFunction(() => console.log('TestCafe test started:', Date.now()))();

    await t.debug();
});

Your complete test report

Steps to Reproduce

git clone https://github.com/felis2803/testcafe-temp-9 cd testcafe-temp-9 npm i npm test

Once the browser is launched, open the DevTools console. Watch for messages in the console. There should be 4 of them. If pageLoadTimeout is working properly, the test launch message should be the last one. However, you may notice that it comes second.

TestCafe version

2.1.0

Node.js version

16.18.0