Highland v3.0.0

[vqvu]

This issue tracks any changes we want for v3.0.0. See the 3.0.0 branch.

The biggest change for now is a reimplementation of the Highland engine. See the original PR at #175.

Breaking changes

• Can no longer call fork after consume. Must call fork at the start.
• Can never consume a stream twice.
• The (undocumented) stream.source no longer exists.
• pipe passes along errors if piping to a Highland stream instead of calling this.emit('error') (Modifies pipe to pass along highland errors #166).
• wrapCallback and stream constructor passes an array for the default mapping hint when there are more than one argument is passed to the callback (Implement mappingHint for wrapCallback, fixes #246 #247, hintMapper default mapper should return array when more than one arg #335). See Reconsider making mappingHint's default mapper wrap in an array. #360 (comment)
• Renames and reargs (Highland v3.0.0 #179 (comment))
  • zipAll renamed to zipEach.
  • zipAll0 renamed to zipAll.
  • reduce and scan argument order reversed
• map will now throw an error if the argument is not a function. The old behavior instead mapped all stream elements to the argument if it was not a function.

Before release

• Implement onDestroy (see back-propagation of _.nil #172)
• Disable multiple consumes without a fork.
• Make sure all transforms correctly destroy their parent. Anything consume-based should get this for free, but not all transforms use consume. Maybe we should add a createDownstream method that does this for us.
  • merge
  • parallel
  • latest
  • sequence
  • flatten
  • reduce/reduce1
  • scan/scan1
• Create a test like noValueOnError to test the backpropagation behavior.
• Make pipe pass along errors to a Highland stream (Modifies pipe to pass along highland errors #166).
• Add a statement to the pipe docs pointing to through for the case when users want to pipe along errors as well as values to another Highland stream (Highland v3.0.0 #179 (comment)).
• Remove ConsumeStream and PipelineWrapperStream (to make it easier to port Extensibility 3.0 #337).
  • ConsumerStream
  • PipelineWrapperStream
• Port use (see Extensibility 3.0 #337).
• Create a test like noValueOnError to test that core transforms always return an object of the appropriate extended type.
• Docs for use.
• Fix default mapping hint behavior (hintMapper default mapper should return array when more than one arg #335). Remove the note in the docs that say we are changing the default behavior.
• Rename & reargs
  • reduce and scan
  • zipAll and zipAll0
  • pickBy
• Make map throw an error if not given a function (Map shouldn't accept a value. Move that to value() #404).
• Test that destroyed fork does not contribute backpressure (Highland v3.0.0 #179 (comment))
• Create highland-2 module for reverting the transform rename/reargs (Highland v3.0.0 #179 (comment))
• Split index.js into highland/core and highland/transforms
• Release the pending 2.x version.
• Merge 2.x into 3.0.0 once released.

Nice to have but can be delayed until after release

• Port over sync optimizations.
• Port over iterative consume and redirect.
• Check for memory leaks.
• Create some benchmarks.

[vqvu]

Ref #176.

[jeromew]

@vqvu thanks for creating the branch. I haven't had time to zoom into it before the holiday season. That could be on the new year's resolutions ;-) happy holidays !

[LewisJEllis]

@vqvu how are iterative resume/redirects going? Is there a way I can be helpful? I was contemplating features/changes for 3.0.0 a bit, but @jeromew wisely suggested that I don't get carried away while the engine is still up in the air.

Added a 3.x potential label to track related items.

[jeromew]

@LewisJEllis I am not sure @vqvu has had time to look into this over the holiday season. I think that the changes needed to flatten the stack are still up for discovery. Last time @vqvu mentioned it he said he had no idea how convoluted this could get.

I unfortunately could not find time to look deeply at the new branch. If you feel like diving into it, it would be great and @vqvu will certainly appreciate to be able to discuss potential solutions with someone.

I was planning on working on the merge issue #133. @vqvu has rewritten merge in 3.0.0 but, from my understanding, this has nothing to do with the new engine. So I would like to take his new implementation and adapt it to have it meet #133's requirements.

This way we will be able to remove this from the 3.0.0 branch.

After this, we should make sure that 3.0.0 and 2.x are in sync feature wise (I have seen a few commits that need to be checked)

If possible, we should have strictly the same tests between 2.x and 3.0.0

[vqvu]

I haven't had time to do any work on Highland for the last few weeks...

@LewisJEllis I'm happy to discuss ideas on how iterative resume/redirect would be done. We also need some sort of a "memory leak checker" and better performance testing. Also the back-prop stuff (this is pretty useful behavior that's not at all implemented) and pipe passing errors to Highland streams.

[jeromew]

I modified the PR for merge on #133 ; if @vqvu and @LewisJEllis you have time to review it we could remove the merge refactoring from 3.0.0

[jeromew]

@LewisJEllis would it make sense to backport ESLint from 3.0.0 to 2.x ? I understand that caolan was ok for the change and don't see a reason to wait for 3.0.0 on this.

[LewisJEllis]

Yes, definitely. I originally did only 3.0.0 because I didn't want to make all the silly little changes twice, and because I wasn't sure how much would be done on 2.x in the meantime. It's now evident that 2.x needs it too, and a couple regexes did most of the work anyway, so I'll go ahead and backport it. That should also help avoid silly conflicts when 3.0.0 lands.

Regarding keeping 2.x and 3.0.0 in sync as mentioned in #189 - most things being done on 2.x in the meantime shouldn't cause any major conflicts, so I think an occasional rebase should work just fine. I'll try rebasing 3.0.0 onto master after I backport the linting stuff.

I'm not too familiar with merge or the new engine yet, but I'll get up to speed soon so I can take a look at #133 and try to think about the iterative stuff.

[jeromew]

ok thanks. We'll have to agree with @vqvu and the 'rebase or not rebase'. I'm all for removing

• eslint
• merge
• other simple things

from the 3.0.0 potential conflict footprint. I can't say for sure but I think that it will take some time before we get approval from caolan on 3.0.0 so we'd better not freeze 2.x in the meantime and have an easy workflow for merging/rebasing and later have a clean merge from 3.0.0 to master.

[vqvu]

@jeromew Why was this issue closed?

As for "rebase vs merge", I suppose it doesn't really matter too much if we don't expect non-collaborators to contribute. Only issue is that if we start getting a large-ish number of commits on 3.0, it'll start getting more annoying and error-prone to do rebases. That said, I haven't actually tried to do the rebase yet, so maybe this is a non-issue.

Agreed that 2.x should still be the branch that gets all back-compat improvements for now.

[jeromew]

Oops sorry my mistake. I thought I was closing the issue regarding ESLint.

[jeromew]

@vqvu maybe we could :

• backport to 2.x what can be backported
• make a new 2.x release
• merge or rebase or re-create the branch

It is always a pain when there are 2 active branches but we did not have a choice here given the surface area of the refactoring.

do you see other things that can be backported without breaking back-compat on 2.x ? (eslint and merge is done)