Fixes #632. Ref #534.

~~I'm not loving how un-DRY the Test#run function is... but that's not new, only exacerbated by this PR.~~

Updated: This makes the setup, teardown, and test methods all aware that their return value may be a Promise in need of asynchronous resolution (resolve-ing).

Criticism or other suggestions welcomed, as usual.

cc: @domenic @stefanpenner

Uses google's Diff Patch Match Library to replace the current diff algorithms.

Solves: https://github.com/jquery/qunit/issues/348

Discussed (and screenshots) in https://github.com/jquery/qunit/pull/764

@leobalter @jzaefferer Please Review

We still want to provide better support for async testing. We have several options for implementing this, though they all come down to getting rid of methods within the test context.

Some ideas:

// async() returns done callback
test(..., function(assert) {
  // async() can be invoked multiple times, each callback has to be invoked
  var done = assert.async();
  setTimeout(function() {
    done();
  });
});
// this makes integration of Promises very easy
test( ..., function( assert ) {
    /* ... */
    APP.foo().always( assert.async() ).done( function () {
        assert.ok(true);
    } );
    APP.bar().always( assert.async() ).done( function () {
        assert.ok(true);
    } );
});

// all tests are async, always have to explicitly end the test, async or not
test(..., function(assert) {
  assert.end();
});

// expect sets up async test, finishes when number of assertions ran
test(..., function(assert) {
  // makes this test async
  expect(1);
  setTimeout(function() {
    // expected 1 assertion, continue with other tests
    // any other assertions will fail this test
    assert.ok(true);
  });

// return a promise from the test
[...]

For the last one, see #634

Frameworks that are good references: mocha, nodeunit. We could force calling done like nodeunit, always or depending on configuration.

Related: #374

This replaces #331, to have a fresh start on this discussion.

This PR tries to close #782.

I make the pull request now so that anyone can see how I'm approaching the task and can tell me whether I'm doing something wrong. As it is work in progress, it may not pass the tests

I think it would be handy to be able to filter tests right in the source code. This feature is essential when Karma is used and the test suite is large.

Mocha and Jasmine already support this feature via '.only' and 'iit' respectively.

What do you guys think?

Addressing #707. This will enable us to use ES6 via Babel and build our application via Rollup, meaning no more reliance on specified concat order.

Some interesting things to note:

• The code in src should mostly be free from QUnit expressions, other than in src/core and src/export which is where the definition and export of the global QUnit object occurs.
• JSHint now runs on the individual source files and is more strict to ensure that we are explicit about dependencies.
• Anything in reporter/runner does not depend on any code in src other than src/core which exports QUnit and src/globals which provides access to global objects.
• There are some files in reporter/runner that have their contents wrapped in an IIFE to allow them to return early. These files should be converted to export functions that execute similar code. This will eventually make it easy to ship QUnit separately from the runner and reporter which are specific to the browser.

Improvements for future PRs:

• Remove all IIFEs, these shouldn't be needed anymore

• Breakup some of the larger modules to simplify dependency graph (e.g., src/test and src/html)

• Move all source code into src. This might be contentious, but I'd like to see a structure similar to the following:

  qunit/
    src/
      qunit.js
      qunit.css
      core/
        index.js
      reporter/
        index.js
      runner/
        index.js
  

  The qunit.js file will import core/index.js, reporter/index.js, and runner/index.js and those will then be responsible to act as entry points for the functionality encompassed within. index.js could also be something like main.js or exports.js. The idea here is to have just one directory for our source code that can also mirror how we'd like to think about our dependency graph.

Something that came out of a discussion with Yehuda: During a big refactoring they have hundreds of failing tests. They turned them off using QUnit.skip, but that doesn't tell them which of the skipped tests starting passing again at some point. To quickly test changes in passing tests, they ended up with this:

QUnit.skip = QUnit.test;

There's probably a better way to deal with that situation. One idea is to have something like a QUnit.pending method, that has the same signature as QUnit.test(), but will reverse the result. So if all assertions pass, the test fails, if at least one fails, the test passes.

The HTML reporter could use a similar marker as QUnit.skip.

Thoughts?

Update 2015-04-15:

After some discussion, we came up with this:

// This test will show up with a "todo" badge, with a similar style as the "skipped" badge
// It will show as passed when at least one assertion fails, or fail when all assertions pass
QUnit.todo( "this isn't implemented, yet" function( assert ) {
  assert.ok( featureTest(), "waiting for feature" );
} );

QUnit.testDone( function( details ) {
  console.log( details.name, details.todo );
  //> "this isn't implemented, yet" true
} );

QUnit.log( function( details ) {
  console.log( details.message, details.todo );
  //> "waiting for feature" true
} );

Work in progress: Just fixing minor issues at this point.

TODOs:

• [x] Fix indentation issues
• [x] Use eslint-disable-next-line comments instead of eslint-disable-line comments

Why

I was missing a feature that would mark tests as "skipped" so that I can see tests that aren't implemented, or features lacking in that browser, or to temporarily skip a test with a reason so I could remember to come back to it.

How

The general implementation is to make your final result a 3-state variable instead of 2-state. Right now there is true for passing and false for failing. This adds a thirds state of null for skipped. Calling skip() or skipIf() will mark that test as "skipped" which will color it yellow, and add to the "skipped" counter (note that skipIf() has a test param and will only assert "skipped" if that param is truthy).

Use cases

Mocked Test

The most basic use case that I started with was me mocking out tests, and implementing random ones out of order. I didn't want to have "failing" tests for unimplemented, nor did I want to use expect(0) and have a passing test because I will forget to go back to it. So the most basic case was something like:

module('Utils');

test('#utilFunction', function() {
    skip();
});

which gives me output like:

SkipIf based on Feature Detection

This was brought up in #189 that it would be nice if we could detect a browser feature, and conditionally skip the assertions depending on if it had it or not. Something like:

test("that the function will return the element passed by a selector", function() {
    skipIf(!document.querySelectorAll, "Browser doesn't support querySelectorAll. Implement test for alternative interface if we're going to support this browser.", function () {
        var result =document.querySelectorAll('#container a');
        ok(result);
    });
});

which on FF19 gives the result:

but in IE7 gives the result:

Final Notes

Obviously there a many reasons that skipping a test is preferred to it failing. I may have implemented it in a way that is not ideal, and if so please let me know and I will try to fix it because this is a feature I would really like to see land. The final thing I want to note is that if you have failed assertions in a test with a skip() call as well, the test is still marked as failed:

If you have tests you know will fail if some condition isn't met, then that is what skipIf() is for. You can do the test and only run code inside if you need to, otherwise you get a skip assertion.

This allows for error recovery in async conditions.

I've written the logic and added a basic test. I still need to write tests for a few edge cases (mostly along the lines of "what will happen if pushFailureAndResume() has been called already but other assertions come back after the test runner has moved on?"). I just wanted to put this up to get some initial feedback.

Open to feedback on the function names I've used.

We're using QUnit and RequireJS. We've defined a require.onError handler as follows:

define([], function () {
    require.onError = function (e) {
        if (QUnit && QUnit.config) {
            if (QUnit.config.current) {
                var currentTest = QUnit.config.current;

                currentTest.pushFailure("Died on test #" + (currentTest.assertions.length + 1) + " " +
                    currentTest.stack + ": " + (e.message || e));
            }

            // Restart the tests if they're blocking
            if (QUnit.config.blocking) {
                QUnit.start();
            }
        }
    };
});

This is heavily inspired by how the global error handler used to handle these issues. However, QUnit.start() itself now throws an error (in 2.0.0-rc1), so I need a new way to resume the test runner. What is the best way to do that?

disclaimer: I'm not sure if Test.finish is the best place to put such logic, however it seemed most logic place for me.

The problem description: In Ember.js we have ember-qunit package that assigns reference to the currently running application instance to this.owner, where this is the reference to testEnvironment property in Test class.

During test execution (here https://github.com/qunitjs/qunit/blob/2.19.3/src/test.js#L752), we persist references to the timeoutHandler in the shared config object.

After the test execution, config.timeoutHandler keeps reference to the last test that was executed (as it's closure created in https://github.com/qunitjs/qunit/blob/2.19.3/src/test.js#L738).

As a result, after tests execution there is a reference to the application instance persisted in heap, which makes it harder to find memory leaks in the application code.

I'm not sure if there is a need/way to write test for this, so please advice.

related issue: https://github.com/emberjs/ember-qunit/issues/476

Tell us about your runtime:

• QUnit version: 2.19.1
• Which environment are you using? (e.g., browser, Node): browser
• How are you running QUnit? (e.g., QUnit CLI, Grunt, Karma, manually in browser): browser, jsfiddle

Code that reproduces the problem:

QUnit.test( "demo", async assert => {
	assert.timeout( 10 );
	const done = assert.async();
	assert.true( true );
} );

https://jsfiddle.net/fasjpd49/

This PR introduces further refactors and optimizations based on https://github.com/qunitjs/qunit/pull/1700

Certain changes probably require QUnit to update its browser policy and deprecate support for certain browsers.

I'm confident that each commit should result in faster deepEqual checks, except the last one. The last commit particularly needs to be benchmarked I think.

I seperated each refactor to its seperate commit. Have a look, I think you'll find it very interesting! :)

During my code removals I already saw that this method is deprecated and planned to be removed in v3. This PR could serve as the removal commit(s) if it gets merged, otherwise feel free to close the PR ;)

While reading the source code I realized this internal function isn't needed anymore if we drop the IE support:

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/includes#browser_compatibility

Tell us about your runtime:

• QUnit version: 2.19.1
• Which environment are you using? (e.g., browser, Node): Node
• How are you running QUnit? (e.g., QUnit CLI, Grunt, Karma, manually in browser): Terminal

What are you trying to do?

Code that reproduces the problem:

Running qunit -w. On first run it successfully outputs the results of the tests, upon making a change to the test file, the error occurs:

What did you expect to happen?

Test should just re-run. Any help would be appreciated!