Test suite improvements

[bsamuel-ui]

Some improvements I'm looking for:

Make the test suite faster

We should be able to run tests in parallel locally and, ideally, easily run a single test when we're debugging a specific problem. And that same setup should allow github to run tests in parallel.

Easy and obvious to run tests individually

When I've mucked with the tests, I've written stupid things to skip all the tests I don't care about. Maybe I'm missing a better way to do this, but tape seems set up to expect only a few tests in a given file.

Avoid contention when merging tests

As long as your feature is adding a test, you should never be dealing with merge conflicts.

While I'm trying to express these as requirements, this largely means "put tests into individual files" to me.

Tests should be independent.

Right now, there are some existing files we use to set up tests. We're also running in these directories, so we're mucking with our own directories and have to clean it with:

const setup = () => {
  removeSync(getUserCachePath());
  process.chdir(initialWorkingDir);
};

const teardown = () => {
  const cwd = process.cwd();
  if (!cwd.startsWith(initialWorkingDir)) {
    throw new Error(`Somehow cd'd into ${cwd}`);
  }
  if (cwd != initialWorkingDir) {
    [
      'puck',
      'puck2',
      'puck3',
      'node_modules',
      '.serverless',
      '.requirements.zip',
      '.requirements-cache',
      'foobar',
      'package-lock.json',
      'slimPatterns.yml',
      'serverless.yml.bak',
      'module1/foobar',
      getUserCachePath(),
      ...glob.sync('serverless-python-requirements-*.tgz')
    ].map(path => removeSync(path));

Gross.

Ideally, creating a new test should be mostly "copy an existing test that does what I want and change it" and this is a bit harder when you're depending on files. So, having all the tests explicitly denote what files they're using is verbose, but makes this easy.

Idea to do this

Putting the tests in separate files is a bit of refactoring.

Moving the support files into code could be a simple fixture:

filesystemFixture({
  folder: {
     "simple_file.py": "def some_python(fool): pass",
     "big_file.py": "file:big_file.py",
     another_folder: {
       "some_symlink": (path) => fs.symlinkSync(path, ...);
     }
  }
})

There are three files in the test file directories that are > 1K, so I think describing them in code is quite doable.

What I like here is if you have chmoded a file or something, it's plainly there in code rather than being a detail in github because you pointed at one folder vs. another.

The fixture can then spin up a temporary directory to construct the files. This might even solve the "busy resource" issue on the Windows test runners. Maybe?

I think that sets us up to then run tests in parallel without conflicts.

Existing tooling

I'd especially like to tap people more knowledgeable about the Javascript universe to recommend idioms / packages that are well established.

Thoughts?

[miketheman]

You're 100% sharing the feeling I've had running testing locally, and it was a pain.

I'm on board with splitting the test suite up - hell, even ready to start, using the popular javascript testing tool jest.
We should be able to group/classify things together and run them separately - absolutely!

Where we have to do our research is that we're looking for on-disk/shell manipulation - and sometimes we're installing a lot of things, and polluting a filesystem, so whatever we choose has to accommodate that being done in parallel - since we can also saturate a computer's resources if we're not careful.

But we can probably start by carving up test.js into smaller pieces and figure out what shared state or functions we have - and then look at what tests are longer durations, and what tests can be parallel.