Using ember-backstop in a CI pipeline

[shibulijack-fd]

The documentation says:

You will need the backstop-remote service running for visual tests. In a seperate window run...
ember backstop-remote

While it's feasible in local development environment, how does one go about achieving the same results in a CI like travis or Github Actions?

[garris]

Simply start the service prior to running tests. E.g. ember backstop-remote & ember test

[shibulijack-fd]

Simply start the service prior to running tests. E.g. ember backstop-remote & ember test

This works with ember backstop-stop in posttest hook, but in case of a failure, the process never exits and the CI builds timeout.

More importantly, how do we address this: https://github.com/garris/BackstopJS#using-docker-for-testing-across-different-environments

[ming-codes]

Maybe we can work start-server-and-test into the doc somewhere.

[garris]

@ming-codes Happy to add a link or a example callout in the docs if you have a gist or a project template or an article that describes how this integration can work.

[ming-codes]

Just this one liner in scripts should work.

// package.json
{
  "scripts": {
    ...
    "test": "ember test",
    "ci": "start-server-and-test 'ember backstop-remote' http://localhost:3000/version test"
    ...
  },
  "devDependencies": {
    ...
    "start-server-and-test": "^1.12.5"
    ...
  }
}

start-server-and-test takes a series of serve and resource pairs and then a final test command to run the test. Here's the breakdown of the script above.

start-server-and-test \
  'ember backstop-remote' \          # start backstop server
  http://localhost:3000/version \    # ping this endpoint to make sure server successfully started
  test                               # run the test!

We can even insert the above as part of the default blueprint

[garris]

Does this address the hang-forever issue?

[ming-codes]

Should be. The command kills the server regardless the status of the test.

[Sticksword]

Mildly related, how would you go about storing/approving changes? I setup my CircleCI pipeline and I'm getting this:

BackstopJS found differences. See: http://localhost:7357/backstop/backstop_data/html_report/

I see that the bitmap references are stored in ember-backstop/backstop_data/bitmaps_reference so why isn't the test picking them up in the CI pipeline?

these are my versions for things

ember-backstop 1.5.0
ember-cli: 3.25.3
node: 14.16.0

[Sticksword]

Update: I learned that things were working fine, it just so happens that the Chrome version Circle CI was using is off by 1.5% and my test threshold was set to 0.1 by default

Main lesson is that Chrome can behave differently based on which image you use and how you install Chrome, resulting in slightly different screen captures.