Skip to content

e2e-minimal-testing-philosophy.md

Latest commit

 

History

History
177 lines (140 loc) · 5.56 KB

e2e-minimal-testing-philosophy.md

File metadata and controls

177 lines (140 loc) · 5.56 KB

E2E Minimal Testing Philosophy

Overview

This document outlines our approach to end-to-end testing, emphasizing simplicity and reliability over comprehensive coverage. This philosophy emerged from solving Playwright test flakiness in CI environments.

Core Principles

1. Minimal is Better

  • Start with the absolute minimum - single test, single assertion
  • Add complexity incrementally - one check at a time
  • Remove until it works - when debugging, strip everything down first

2. Focus on Critical Flows Only

E2E tests should validate:

  • ✅ Homepage loads successfully
  • ✅ Core navigation works
  • ✅ Essential user journeys function

E2E tests should NOT validate:

  • ❌ Specific text content (too fragile)
  • ❌ Complex user interactions (unit tests handle this)
  • ❌ Edge cases or error scenarios
  • ❌ Visual styling or layout details

3. Reliability Over Coverage

  • Better to have 2 tests that always pass than 20 tests that are flaky
  • CI failures should indicate real problems, not test instability
  • Simple assertions are more stable than complex ones

Implementation Guidelines

Test Structure

// ✅ Good - Minimal and focused
test('homepage loads', async ({ page }) => {
  await page.goto('/');
  await expect(page.locator('body')).toBeVisible();
  await expect(page.locator('input')).toBeVisible();
});

// ❌ Bad - Too complex and fragile
test('homepage loads with all content', async ({ page }) => {
  await page.goto('/', { waitUntil: 'networkidle' });
  await page.waitForLoadState('domcontentloaded');
  await expect(page.getByText('Analyze GitHub Repository Contributors')).toBeVisible();
  await expect(page.locator('input[placeholder*="Search repositories"]')).toBeVisible();
  await expect(page.locator('h1, h2, h3')).toBeVisible();
});

Configuration Principles

// ✅ Good - Simple configuration
export default defineConfig({
  workers: 1, // Avoid resource conflicts
  timeout: 30000, // Use defaults
  webServer: {
    command: 'npm run dev',
    url: 'http://localhost:5173',
    reuseExistingServer: !process.env.CI,
  },
});

// ❌ Bad - Over-engineered configuration
export default defineConfig({
  workers: process.env.CI ? 8 : undefined,
  timeout: process.env.CI ? 120000 : 30000,
  webServer: process.env.CI ? {
    command: 'npm run build && npm run preview',
    url: 'http://localhost:4173',
    // ... complex CI-specific logic
  } : {
    // ... different local logic
  },
});

Troubleshooting Strategy

When e2e tests fail in CI:

1. Simplify First

  • Remove all but one test
  • Remove all but one assertion
  • Use default timeouts
  • Single worker only

2. Build Up Incrementally

  • Once one test passes, add one more
  • Add one assertion at a time
  • If something breaks, you know exactly what caused it

3. Common CI Issues

  • Vite dev server flakiness - Use simple server setup
  • Resource conflicts - Always use workers: 1
  • Timing issues - Avoid custom timeouts and wait strategies
  • Content loading - Check for body existence, not specific text

What We Learned

From Reddit Community

A developer with the exact same issue found that:

  • Tests worked locally but timed out in GitHub Actions
  • Problem was multiple workers and complex server setup
  • Solution: "Remove things until it works, then add back 1 by 1"

From Vite Issues

  • Vite dev server can be unreliable in CI environments
  • Preview builds are more stable but add complexity
  • Simple dev server setup works fine for minimal tests

From Playwright Docs

  • Use CLI instead of GitHub Action
  • Install browsers with --with-deps
  • Keep test scenarios focused and simple

Current Test Suite

Our minimal test suite validates:

test.describe('Critical User Flows', () => {
  test('homepage loads', async ({ page }) => {
    await page.goto('/');
    await expect(page.locator('body')).toBeVisible();
    await expect(page.locator('input')).toBeVisible();
  });

  test('repository page loads', async ({ page }) => {
    await page.goto('/facebook/react');
    await expect(page.locator('body')).toBeVisible();
  });
});

This covers:

  1. Homepage accessibility - Site loads, search input present
  2. Routing functionality - Can navigate to repository pages

Benefits

✅ Advantages

  • Reliable in CI - Tests consistently pass
  • Fast execution - Minimal overhead
  • Clear purpose - Catches real breakages
  • Easy debugging - Simple assertions, obvious failures
  • Low maintenance - Rarely needs updates

⚠️ Trade-offs

  • Limited coverage - Doesn't test complex interactions
  • Surface-level - Won't catch subtle bugs
  • Requires discipline - Team must resist adding complexity

Guidelines for New Tests

Before adding a new e2e test, ask:

  1. Is this a critical user flow? (Can users complete their main task?)
  2. Would this break the entire application? (Not just a feature)
  3. Can this be tested more reliably at the unit level?

If yes to 1 & 2, and no to 3, then add a minimal e2e test.

Success Metrics

A successful e2e test suite:

  • ✅ Passes consistently in CI (>99% reliability)
  • ✅ Runs quickly (<2 minutes total)
  • ✅ Catches real deployment issues
  • ✅ Rarely produces false positives
  • ✅ Requires minimal maintenance

Conclusion

E2E tests are deployment smoke tests, not comprehensive feature validation. Keep them minimal, focused, and reliable. Use unit and integration tests for detailed functionality validation.

"Perfect is the enemy of good" - A working minimal test suite is infinitely better than a comprehensive but flaky one.