Device test runner by pictos · Pull Request #1444 · CommunityToolkit/Maui

pictos · 2023-10-04T23:48:31Z

This PR adds the Device test runner to our repo. It's WiP but it's nice to have some visibility of you all. It's working and if you want you can start to add tests. I added two tests to be a guide for you.

I'll need help of someone who knows yaml in order to create a step on our build to run these tests, I know that Maui does that so we can grab for them (?).

Right now these bits are in preview, and uses a private feed, I added a nuget.config file adding the source to the needed packages, I would say it's fine to use it since is the one that .NET MAUI uses and it the team is working in making this stable. Also, using it on our repo is a good way to help them provide early feedback.

pictos · 2023-10-04T23:49:48Z

cc: @mattleibow just in case you have something to add (:

VladislavAntonyuk · 2023-10-05T06:01:00Z

+  </ItemGroup>
+
+  <ItemGroup>
+    <MauiXaml Update="Tests\UITests\StatusBarTestPage.xaml">


Can be removed

VladislavAntonyuk · 2023-10-05T06:07:35Z

+public class BehaviorTest
+{
+	[Fact]
+	public void SuccessfulTest()


Can be removed

Let's make it run on your CI first

* Run tests on Android * fix path * skip warnings * install android sdk * update nuget.config * fix build * use x86-x64 android emulator * use android x64 * ios maccatalyst * fix path * fix ios build * use xcode 14.3.1 * macos-13 * comment android * Run only on Mac

VladislavAntonyuk · 2023-12-08T21:54:02Z

@pictos can we merge this PR as it is (with comments fixes)?

This saves time by not attempting to re-build projects that were previously compiled when the Unit Tests ran. It also ensures that the project build step will fail alerting us immediately to the root cause of the CI pipeline failure.

bijington · 2024-10-01T19:13:28Z

It looks like it is forcing us to upgrade to Xcode 16 and iOS 18? Is this supported via Azure Pipelines?

TheCodeTraveler · 2024-10-01T19:22:06Z

It looks like it is forcing us to upgrade to Xcode 16 and iOS 18? Is this supported via Azure Pipelines?

Yup! You need to do two things to add iOS 18 support to the library:

Update CommunityToolkitLibrary_Xcode_Version to '16'

Maui/azure-pipelines.yml

Line 32 in 372d873

CommunityToolkitLibrary_Xcode_Version: '15.4'
Update TOOLKIT_NET_VERSION to 8.0.402

net8.0-ios only support Xcode 16 in .NET SDK v8.0.402 and higher

Maui/azure-pipelines.yml

Line 9 in 372d873

TOOLKIT_NET_VERSION: '8.0.303'

Copilot

Copilot reviewed 27 out of 42 changed files in this pull request and generated 3 suggestions.

Files not reviewed (15)

samples/CommunityToolkit.Maui.Sample.sln: Language not supported
src/CommunityToolkit.Maui.DeviceTests/App.xaml: Language not supported
src/CommunityToolkit.Maui.DeviceTests/AppShell.xaml: Language not supported
src/CommunityToolkit.Maui.DeviceTests/CommunityToolkit.Maui.DeviceTests.csproj: Language not supported
src/CommunityToolkit.Maui.DeviceTests/MainPage.xaml: Language not supported
src/CommunityToolkit.Maui.DeviceTests/NuGet.config: Language not supported
src/CommunityToolkit.Maui.DeviceTests/Platforms/Android/AndroidManifest.xml: Language not supported
src/CommunityToolkit.Maui.DeviceTests/Platforms/Android/Resources/values/colors.xml: Language not supported
src/CommunityToolkit.Maui.DeviceTests/Platforms/MacCatalyst/Info.plist: Language not supported
azure-pipelines.yml: Evaluated as low risk
src/CommunityToolkit.Maui.DeviceTests/App.xaml.cs: Evaluated as low risk
src/CommunityToolkit.Maui.DeviceTests/AppShell.xaml.cs: Evaluated as low risk
src/CommunityToolkit.Maui.DeviceTests/Platforms/Tizen/Main.cs: Evaluated as low risk
src/CommunityToolkit.Maui.DeviceTests/Platforms/Android/MainApplication.cs: Evaluated as low risk
src/CommunityToolkit.Maui.DeviceTests/Platforms/Android/MainActivity.cs: Evaluated as low risk

Comments skipped due to low confidence (2)

src/CommunityToolkit.Maui.DeviceTests/BaseUITest.cs:25

The TaskCompletionSource should specify the type explicitly, e.g., TaskCompletionSource<object?>.

var tcs = new TaskCompletionSource();

src/CommunityToolkit.Maui.DeviceTests/MainPage.xaml.cs:5

[nitpick] The variable name 'count' is ambiguous. It should be renamed to 'clickCount' or something more descriptive.

int count = 0;

…pDelegate.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mattleibow · 2025-06-13T09:55:53Z

I have been iterating on the repo and I have added some features for running on the cli without xharness. It uses tcp connections.

Once I finish the final ios test runner, then I will ship a version and get it to work with skia as a first dogfooder.

There are some great Android and Apple cli tools from @Redth, so I am using that under the hood.

I also want to make is easier to have a test app and not need all the boilerplate with things like icons and resources. I am hoping to almost emulate a app project with nothing in it except your tests.

This latter idea is maybe for next time I get to it, but hopefully the new cli tool can just replace the xharness way and I can deprecate/remove that dependency.

I see this PR has been here for a while, what has been the main blockers to adding testing support to the toolkit?

pictos · 2025-06-13T15:06:53Z

I have been iterating on the repo and I have added some features for running on the cli without xharness. It uses tcp connections.

Once I finish the final ios test runner, then I will ship a version and get it to work with skia as a first dogfooder.

There are some great Android and Apple cli tools from @Redth, so I am using that under the hood.

I also want to make is easier to have a test app and not need all the boilerplate with things like icons and resources. I am hoping to almost emulate a app project with nothing in it except your tests.

This latter idea is maybe for next time I get to it, but hopefully the new cli tool can just replace the xharness way and I can deprecate/remove that dependency.

I see this PR has been here for a while, what has been the main blockers to adding testing support to the toolkit?

@mattleibow I was working on this one, and one thing that I can remember was, how to configure it to run on CI.

Other than that I believe everything was working as expected

mattleibow · 2025-06-13T16:48:43Z

Ah, nice. OK, so my repo now has workflows (iOS is almost done) and it is pretty simple - especailly with Jon's cool CLI tools to create, start, launch emus and sims.

The workflows are a couple steps long, so you might just be able to copy and make it work.

I really would like this all to be as simple as possible, so if anything is more that 3 chars deep and 2 lines long, let me know and I can try make it a single stop shop.

mattleibow · 2026-05-01T21:25:18Z

How is this going? Do we still like this? So far, I have been using it in SkiaSharp and the last updates have made it very reliable. I still use xharness as a host, but the TCP runner is working and I am investigating a dotnet run-ability for things like mac and windows.

The dream is still a dotnet test, but that may still be a net11 or later thing. There is hope. But if there is still interest in this I can have a look at all the updates and ensure the device tests can run.

I have workflows that you can copy in the repo if need be. I think now all the nugets are on nuget.org, making things nicer.

I also have a cli tool that does the work, but not required - you just have to capture the logs if you don't want it. I discovered that there is a wasdk nuget that does dotnet run now: https://www.nuget.org/packages/Microsoft.Windows.SDK.BuildTools.WinApp

That is very cool and with dotnet run now working: mattleibow/DeviceRunners#94

TheCodeTraveler · 2026-05-01T23:49:12Z

The dream is still a dotnet test

That'd be great! To be honest, I never really got this working.

added runner project

69ec49d

pictos requested a review from a team October 4, 2023 23:49

pictos force-pushed the pj/device-test-runner branch from b8415ae to f818c9d Compare October 4, 2023 23:51

Added UITest capabilities

5513c8e

pictos force-pushed the pj/device-test-runner branch from f818c9d to 5513c8e Compare October 4, 2023 23:54

VladislavAntonyuk reviewed Oct 5, 2023

View reviewed changes

VladislavAntonyuk added the hacktoberfest-accepted A PR that has been approved during Hacktoberfest label Oct 5, 2023

VladislavAntonyuk and others added 2 commits October 5, 2023 20:21

Merge branch 'main' into pj/device-test-runner

9f1b099

ghost added stale The author has not responded in over 30 days help wanted This proposal has been approved and is ready to be implemented labels Nov 17, 2023

bijington mentioned this pull request Nov 19, 2023

Fix for MediaElement Android Resource file not found #1537

Merged

6 tasks

Merge branch 'main' into pj/device-test-runner

2b2c1f9

VladislavAntonyuk and others added 15 commits December 8, 2023 23:54

Merge branch 'main' into pj/device-test-runner

f3af8fc

Merge branch 'main' into pj/device-test-runner

9d3383b

Merge branch 'main' into pj/device-test-runner

a3b5683

Merge branch 'main' into pj/device-test-runner

0770d65

Update to .NET 8

d030aa6

Update azure-pipelines.yml

a1314f4

Update azure-pipelines.yml

64b65ce

Remove Unnecessary Code

c4b6dfb

Update install step

ff05284

Update azure-pipelines.yml

3a0f4d2

Remove Redundant Folder

62fc85a

Update azure-pipelines.yml

a1bebd8

Update azure-pipelines.yml

cc8d4c3

Run Unit Tests after all projects have been compiled

8050559

This saves time by not attempting to re-build projects that were previously compiled when the Unit Tests ran. It also ensures that the project build step will fail alerting us immediately to the root cause of the CI pipeline failure.

Fix App Identifier

a6853e5

TheCodeTraveler removed the stale The author has not responded in over 30 days label Sep 21, 2024

TheCodeTraveler marked this pull request as draft September 21, 2024 19:21

TheCodeTraveler mentioned this pull request Oct 1, 2024

Make use of a specific iOS + MacCatalyst TFM #2243

Closed

6 tasks

Merge branch 'main' into pj/device-test-runner

073f310

This comment was marked as outdated.

Sign in to view

Update to xcode 16

9aaec9b

dotnet-policy-service Bot added the stale The author has not responded in over 30 days label Nov 1, 2024

VladislavAntonyuk requested a review from Copilot December 10, 2024 23:11

Copilot AI reviewed Dec 10, 2024

View reviewed changes

Comment thread src/CommunityToolkit.Maui.DeviceTests/BaseUITest.cs

Comment thread src/CommunityToolkit.Maui.DeviceTests/Platforms/MacCatalyst/AppDelegate.cs Outdated

Comment thread src/CommunityToolkit.Maui.DeviceTests/MainPage.xaml.cs Outdated

TheCodeTraveler and others added 5 commits December 10, 2024 15:33

Update src/CommunityToolkit.Maui.DeviceTests/Platforms/MacCatalyst/Ap…

7dd4b5c

…pDelegate.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Update src/CommunityToolkit.Maui.DeviceTests/MainPage.xaml.cs

c0b64c6

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Merge branch 'main' into pj/device-test-runner

7a035ae

Merge branch 'main' into pj/device-test-runner

6c7fb61

Merge branch 'main' into pj/device-test-runner

76db87e

bijington removed help wanted This proposal has been approved and is ready to be implemented stale The author has not responded in over 30 days labels Feb 4, 2025

dotnet-policy-service Bot added stale The author has not responded in over 30 days help wanted This proposal has been approved and is ready to be implemented labels Mar 7, 2025

Conversation

pictos commented Oct 4, 2023

Uh oh!

pictos commented Oct 4, 2023

Uh oh!

VladislavAntonyuk Oct 5, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

VladislavAntonyuk Oct 5, 2023

Choose a reason for hiding this comment

Uh oh!

pictos Oct 5, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

VladislavAntonyuk commented Dec 8, 2023

Uh oh!

bijington commented Oct 1, 2024

Uh oh!

TheCodeTraveler commented Oct 1, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mattleibow commented Jun 13, 2025

Uh oh!

pictos commented Jun 13, 2025

Uh oh!

mattleibow commented Jun 13, 2025

Uh oh!

mattleibow commented May 1, 2026

Uh oh!

TheCodeTraveler commented May 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

TheCodeTraveler commented Oct 1, 2024 •

edited

Loading