test: Introduce Chromatic for visual regression tests

[gnapse]

Short description

This is work in progress, not even sure it's going to make it to main yet.

Chromatic is a visual regression testing system provided via an external service (SaaS). It is based on Storybooks (in fact, it's provided by the creators and maintainers of Storybooks). What it does is, it runs your Storybooks stories and creates screenshots of them, which are then used as the basis for comparing when new screenshots are taken in the future.

This PR integrates Chromatic with Reactist's PR builds, so that whenever a PR is created, Chromatic is run and runs the visual regression tests. Via Chromatic's UI, you can review the changes and approve or reject any visual changes that the new PR may have created. More details about this below.

Here’s a sneak peek of what we get with Chromatic:

• Snapshot changes need to be approved

  • Checks in the CI build that come from Chromatic. If there are snapshot differences, they need to be approved in Chromatic’s UI first

  • Once all changes are approved in Chromatic, the CI step is marked as passed

  • If the reviewer in Chromatic does not accept all visual changes, the CI step is marked as failed instead

• Storybooks instance per PR

  • In all PRs, we also get a hosted Storybook instance reflecting the changes in the PR. See the link also provided in the CI checks. This will allow us to involve designers and other stakeholders in seeing changes live before merging.

Related references

• test: Introduce visual regression tests using Playwright #732
• https://twist.com/a/1585/ch/607587/t/3963022/

Test plan

• Expand the successful CI checks UI below to see the individual steps. See that there are some steps corresponding to Chromatic (https://share.cleanshot.com/gZ9JKqBW)
• Open the "Details" link to the right of the CI step labelled "Storybook publish"
  • From what I understood, anyone should be able to see it, and you do not need the Chromatic account that you'll be creating in the next steps. But if that's not the case, then let me know, and leave this step for after you create the Chromatic account.
• Create an account at https://chromatic.com, using GitHub as your authentication provider.
  • It's absolutely important that you use GitHub.
• Check if you automatically see the Reactist project, or maybe I need to invite you to it. Don't know, as this is all new to me. Let me know. We might even sync via a video call next week if needed.
  • Maybe a good way to do this is that instead of visiting chromatic.com manually, click the "Details" link in one of the Chromatic CI steps below.

We'll take it from there once you see the project in Chromatic. This is mostly to validate that we can all use Chromatic for collaboration.

PR Checklist

• Added tests for bugs / new features
• Updated docs (storybooks, readme)
• Executed npm run validate and made sure no errors / warnings were shown
• Described changes in CHANGELOG.md

Versioning

This does not need to generate a new release, as it's all inward improvements for maintainers only. No new user-facing features are included here.

[frankieyan]

@gnapse I'm going to re-roll as I'll be off the first half of next week. Super interesting though, I'll definitely check it out when I'm back 👍

[scottlovegrove]

Was able to view the storybook without the chromatic account.
Was able to create a chromatic account and once in, it showed me the Reactist project. There were two there, one is Reactist the other is reactist. I think the former is the one we are really expecting, correct?

[gnapse]

There were two there, one is Reactist the other is reactist. I think the former is the one we are really expecting, correct?

Yup. I had not noticed that, and do not know how that other one came into being, but I now removed it.