Implement multi-patch support in patch-package

[roryabraham]

Problem

patch-package is a simple and useful package that we use to implement bug fixes and new features in 3rd-party libraries without creating a fork. However, it's limited in that each package can have only one monolithic patch file. This makes it difficult to keep track of separate changes in a 3rd-party package, which is one of the reasons we've opted for forks in the past. You can track pull requests in the fork in parallel with their upstream equivalent and use that to reconcile all the custom changes we've made to a 3rd-party dependency. This also helps indicate when our code has landed upstream and the fork/patch is no longer needed.

Solution

Upgrade patch-package to support multiple patch files per dependency. To ground this with an example, let's say we wanted to patch an image bug in React Native, and also wanted to build a new feature in the TextInput component.

For each of these features, we could have a separate patch file for each patch, and those patches would be applied in sequence:

patches/react-native+0.72.0+ImageBug.patch
patches/react-native+0.72.0+TextInputFeature.patch

We could have a separate PR in E/App to introduce each of these patches, and those PRs would link to their upstream equivalent.

Furthermore, we should have error reporting on a per-patch basis, and a way to resolve conflicts in specific patches while upgrading the base version.

Additional context

• Original slack thread
• Upstream issue
• Accounting note: Time spent on this should be considered R&D from a CAP SW perspective.

[melvin-bot]

Current assignee @mallenexpensify is eligible for the NewFeature assigner, not assigning anyone new.

[roryabraham]

@shannonmccallexfy I'm assigning you here so that you can help make sure this project gets counted as CAP SW. I guess the Accounting auto-assigner doesn't work in E/App

[ds300]

Hi! 👋🏼 I'm David Sheldrick, the author of patch-package.

I had a call with @roryabraham a couple of days ago to hash out a prospective scope for this feature. He asked me to write up the following proposal:

Scope

The project can broadly be broken down into three core features:

1. Patch sequence application.

   This would give the ability to load and apply multiple ordered patches to a single package, and for patch-package to convey to the user which patches failed, if any, during application. This would work the same way as cherry-picking a series of commits in git. The patch files should support semantic descriptions in the filenames, and might look something like:

   patches/react-native+0.72.0+01.ImageBug.patch
   patches/react-native+0.72.0+02.TextInputFeature.patch
   

   patch-package will attempt to apply all patches, even if one at the start or in the middle of the sequence fails to apply. And it will not leave merge conflicts in files. This matches the current single-patch behavior.

2. Patch sequence generation.

   This would give the ability to generate a new patch file on top of existing patch files, and to amend the 'latest' patch file. This would be achieved with an extra cli option, e.g.:

   • patch-package react-native --append 'FixPressableEventDispatch' to generate a new patch file with any extra changes added on top of existing patches.

     It would create a patch file named something like

     patches/react-native+0.72.0+03.FixPressableEventDispatch.patch
     

   • patch-package react-native to amend the latest patch file (this syntax mimics the current patch generation behavior)

   The existing workflow of simply editing the relevant files inside node_modules and running patch-package to generate the patch file, whether to amend or append, would remain unchanged.

3. Error recovery via --rebase

   Let's say you have a sequence of five patches for react-native which build on one another. If the second patch in the sequence fails to apply after upgrading react-native, but you don't want to simply delete it, you should be given the opportunity to resolve the merge conflicts manually. patch-package can enable this by allowing you to run a command like

   patch-package react-native --rebase
   

   Which will

   • Undo any patches that were previously successfully applied
   • Re-apply the patches in order until one of them fails
   • Insert merge conflict markers into any files where conflicts were detected, as git does during merge/rebase.
   • Print a list of the files with conflicts, and the name of the patch that caused them.
   • Allow you to run patch-package react-native --rebase-continue to update the failing patch with the resolved changes and continue the rebase.

All features will have appropriate unit and integration tests, in line with the existing patch-package codebase.

Integration with existing patch-package features

The --create-issue flag will not be supported.

The .dev.patch patch file suffix will be supported. This allows patch files created for transitive dev dependencies to be safely ignored if the package is not present in node_modules.

Budget allocation

The total budget for this project is $20,000, and I'd propose allocating it between two milestones as follows

1. Patch sequence application + generation: $15,000
   These two features are the bulk of the work, and it doesn't make sense to ship one without the other so I'd propose combining them.
2. Error recovery via --rebase: $5,000

This would include a two-month period of asynchronous support + bug fixing while you battle-test the features, starting on the day of final delivery of milestone 2. I can also provide up to 2 hours of synchronous zoom support in the EST time zone over the same two-month period if needed, scheduled ad-hoc and subject to availability.

Timeline

I expect to work on this in 1–2-hour chunks over the period of a month or so. I'll be working on it in my spare time during the height of summer so I can't commit to a fixed timeline, but I believe it's a realistic estimate. I'm obviously happy to keep you updated on my progress on a regular schedule (e.g. on Mondays and Thursdays, or whatever works for you), and I'll do the work in the open on the patch-package repo so you can follow along.

Payment method

If convenient you can pay me via one-time GitHub sponsors donations, which I already have set up on the patch-package repo. Otherwise Rory mentioned that you normally use Upwork, which I am happy to register for.

Thanks,
David

[melvin-bot]

📣 @ds300! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:

1. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
2. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
3. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
   
   Format:

Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>

[ds300]

Contributor details
Your Expensify account email: d.j.sheldrick@gmail.com
Upwork Profile Link: https://www.upwork.com/freelancers/~015445808e5ddd1173

[melvin-bot]

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

[roryabraham]

Patch sequence generation.

@ds300 What if you want to edit a specific patch file rather than the latest one? Maybe there should be three options:

• npx patch-package my-package – creates a new patch file if one does not exist, or edits the latest patch if one does exist
• npx patch-package my-package --append 'PatchName' – creates a new patch file to be applied after any existing patch files. Fails if there's already a patch w/ name PatchName
• npx patch-package my-package --update 'PatchName' – looks for any patch file matching PatchName. Fails if no such file exists, otherwise it overwrites that specific patch file.

The use-case for that might be that you introduced a patch associated with an upstream PR, but then you make changes to the upstream and want to apply the same changes to your patch. Rather than creating a second patch associated with the same upstream PR, you would update the existing one.

Integration with existing patch-package features

👍🏼 to both of the items there.