post-build-status-update (maybe all pipeline jobs?) is broken

[richardlau]

Was just on this week's release mentoring session and we noticed that starting CI's wasn't triggering comments/status back to the pull requests. I've dug into it and it looks like https://ci.nodejs.org/job/post-build-status-update/ (which posts status to the github-bot) is currently broken.

The console output for these jobs is useless (they're empty) but "Pipeline Steps"->"Start of pipeline" (e.g. https://ci.nodejs.org/job/post-build-status-update/1062413/execution/node/2/, but I don't expect that link to live long given that the job is executed multiple times for a requested CI):

Error

java.lang.NoSuchMethodError: No such DSL method 'pipeline' found among steps [addBadge, addErrorBadge, addHtmlBadge, addInfoBadge, addShortText, addWarningBadge, ansiblePlaybook, ansibleVault, build, checkout, compareVersions, createSummary, emailext, emailextrecipients, findFiles, git, httpRequest, input, jiraComment, jiraIssueSelector, jiraSearch, junit, library, libraryResource, load, lock, milestone, nodesByLabel, nvm, parallel, properties, publishHTML, readCSV, readJSON, readManifest, readMavenPom, readProperties, readTrusted, readYaml, realtimeJUnit, removeBadges, removeHtmlBadges, removeSummaries, resolveScm, setGitHubPullRequestStatus, sha1, sshagent, stage, svn, tee, timestamps, tm, touch, unzip, withCredentials, writeCSV, writeJSON, writeMavenPom, writeYaml, xunit, zip] or symbols [AUnit, BoostTest, CTest, CUnit, Check, CppTest, CppUnit, Custom, FPCUnit, GoogleTest, JUnit, MSTest, MbUnit, NUnit2, NUnit3, Number, Open, PHPUnit, QtTest, UnitTest, Valgrind, all, allBranchesSame, allowRunOnStatus, always, ant, antFromApache, antOutcome, antTarget, apiToken, architecture, archiveArtifacts, artifactManager, asIsGITScm, audit-trail, authorizationMatrix, batchFile, bitbucket, bitbucketBranchDiscovery, bitbucketForkDiscovery, bitbucketPublicRepoPullRequestFilter, bitbucketPullRequestDiscovery, bitbucketSshCheckout, bitbucketTagDiscovery, bitbucketTrustEveryone, bitbucketTrustNobody, bitbucketTrustProject, bitbucketTrustTeam, bitbucketWebhookConfiguration, bitbucketWebhookRegistration, booleanParam, branchCreated, branches, brokenBuildSuspects, brokenTestsSuspects, buildButton, buildDiscarder, buildDiscarders, buildParameter, buildRetention, buildSelector, buildUser, caseInsensitive, caseSensitive, certificate, choice, choiceParam, clock, close, command, commentPattern, commit, commitChanged, commitMessagePattern, compressBuildLog, configFile, configFileProvider, copyArtifactPermission, copyArtifacts, copyartifact, created, credentials, cron, crumb, cssText, cssUrl, culprits, default, defaultFolderConfiguration, defaultView, deleted, demand, description, developers, disableConcurrentBuilds, disableResume, dockerCert, dockerServer, dockerTool, downstream, dumb, durabilityHint, email-ext, embUnit, envInject, envVars, executor, failed, faviconUrl, file, fileParam, filePath, findText, fingerprint, frameOptions, freeStyle, freeStyleJob, fromDocker, fromScm, fromSource, git, gitBranchDiscovery, gitHub, gitHubBranchDiscovery, gitHubBranchHeadAuthority, gitHubEvents, gitHubExcludeArchivedRepositories, gitHubForkDiscovery, gitHubPRStatus, gitHubPlugin, gitHubPullRequestDiscovery, gitHubSshCheckout, gitHubTagDiscovery, gitHubTopicsFilter, gitHubTrustContributors, gitHubTrustEveryone, gitHubTrustNobody, gitHubTrustPermissions, gitParameter, gitTagDiscovery, github, githubAccessToken, githubBranches, githubPRAddLabels, githubPRClosePublisher, githubPRComment, githubPRMessage, githubPRRemoveLabels, githubPRStatusPublisher, githubPlugin, githubPullRequests, githubPush, globalConfigFiles, gtester, hashChanged, headRegexFilter, headWildcardFilter, hyperlink, hyperlinkToModels, inheriting, inheritingGlobal, installSource, javadoc, jdk, jdkInstaller, jgit, jgitapache, jnlp, jobBuildDiscarder, jobDsl, jobName, jsUrl, labels, labelsAdded, labelsExist, labelsNotExist, labelsPatternExists, labelsRemoved, lastCompleted, lastDuration, lastFailure, lastGrantedAuthorities, lastStable, lastSuccess, lastSuccessful, lastWithArtifacts, latestSavedBuild, legacy, legacySCM, list, local, location, logParser, logRotator, loggedInUsersCanDoAnything, mailer, masterBuild, maven, maven3Mojos, mavenErrors, mavenMojos, mavenWarnings, message, modernSCM, myView, namedBranchesDifferent, noGITScm, nodeProperties, nodejs, nodejsci, nonInheriting, nonMergeable, none, overrideIndexTriggers, paneStatus, parameters, passed, password, pattern, permalink, permanent, pipelineTriggers, plainText, platformlabelerconfig, plugin, pollSCM, projectNamingStrategy, proxy, pruneTags, pullRequest, pullRequests, queueItemAuthenticator, quietPeriod, rateLimit, rateLimitBuilds, realTimeJUnitReports, recipients, requestor, resourceRoot, restriction, restrictions, rule, run, runParam, sSHLauncher, schedule, scmRetryCount, script, scriptApproval, scriptApprovalLink, search, security, separator, shell, simple-theme-plugin, simpleBuildDiscarder, skipped, slave, sourceRegexFilter, sourceWildcardFilter, specific, ssh, sshPublicKey, sshPublisher, sshPublisherDesc, sshTransfer, sshUserPrivateKey, standard, status, statusOnPublisherError, string, stringParam, suppressAutomaticTriggering, swapSpace, tag, tags, teamSlugFilter, text, textFinder, textParam, timestamper, timestamperConfig, timezone, tmpSpace, toolLocation, unsecured, untrusted, upstream, upstreamDevelopers, url, userSeed, usernameColonPassword, usernamePassword, viewsTabBar, weather, withAnt, workspace, x509ClientCert, xUnitDotNet, zfs, zip] or globals [currentBuild, env, manager, ownership, params, scm]
	at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:202)
	at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
	at sun.reflect.GeneratedMethodAccessor300.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:163)
	at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:157)
	at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:142)
	at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:161)
	at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:165)
	at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
	at WorkflowScript.run(WorkflowScript:9)
	at ___cps.transform___(Native Method)
	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
	at sun.reflect.GeneratedMethodAccessor299.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
	at com.cloudbees.groovy.cps.impl.ClosureBlock.eval(ClosureBlock.java:46)
	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

On https://ci.nodejs.org/manage there's this big red box:

Pipeline: Nodes and Processes (2.36)
    Jenkins (2.248) or higher required

I've no idea how we got into this state (I assume it happened overnight as we were getting status updates in pull requests yesterday). We're on the most recent LTS version 2.235.5 (which is lower than 2.248).

[richardlau]

"Pipeline: Nodes and Processes" doesn't show up in manage plugins, probably because it failed to load, so I can't roll it back via the Jenkins UI.

[rvagg]

there was a security release of jenkins yesterday so I did that during quiet hours and ran plugin updates afterward, I suppose this got tangled up during that process

sorry, I really should remember to post in issues when I do these updates

[mmarchini]

@rvagg anything I can do to help with this?

[rvagg]

What a tedious beast Jenkins has become. The number of plugins we use is partly to blame of course.

So, the plugin named Pipeline: Nodes and Processes (a.k.a. "workflow-durable-task-step") has a new version that breaks with the LTS version of Jenkins. It's apparently fine with the latest release of Jenkins but we use LTS and it's incompatible, for whatever reason. I've had to manually downgrade the plugin (rename the workflow-durable-task-step.bak to workflow-durable-task-step.jpi in /var/lib/jenkins/plugins/ and restart) to make Jenkins work properly. While trying to fix this the Github Auth plugin even broke. It appears to be back to "normal" now, I'll do a bit of testing to make sure.

@nodejs/jenkins-admins DO NOT UPGRADE "Pipeline: Nodes and Processes" in the plugin manager until we know the version works with the version of Jenkins we're running.

[rvagg]

seems to be working OK

also ... post-build-status-update is up to build # 1,064,243, that's a lota builds.

[mmarchini]

Looks like it's not working properly (not all machines are posting job status update):

nodejs/node#34836 / https://ci.nodejs.org/job/node-test-pull-request/

nodejs/node#34862 / https://ci.nodejs.org/job/node-test-pull-request/32877/

Reopening, but if this is a different issue I can open a new one instead.

[richardlau]

I suspect it's another issue -- https://ci.nodejs.org/job/post-build-status-update/ runs are now passing.

For nodejs/node#34836 https://ci.nodejs.org/job/node-test-pull-request/32876/ is a resumed build, so not all platforms were run (only the ones that didn't pass in the build that was being resumed).

For nodejs/node#34862 I see:

which would appear to almost match https://ci.nodejs.org/job/node-test-pull-request/32877/ with the exception of the test-commit-linux failures.

[mmarchini]

Right, on my PR it wasn't showing while running, and on the bot logs there were no start events for those platforms as well. I'll keep an eye tomorrow and if things are ok I'll close here, might've been an intermittent issue.

[mmarchini]

Haven't seen this anymore and haven't seen anyone complaining, so closing.