Avoid manually ending read stream - #346
Open
jeremypvx wants to merge 1 commit into
Open
Conversation
We do not need to emit 'end' here, it will be called automatically by the node stream implementation.
The problem is the following:
Lets assume we have a read stream on a ZIP file, that we want to parse. We use the following construct:
const zip = unzip.Parse();
myStream.pipe(zip);
zip.on('data', data => {
// do something with the data
});
zip.on('end', () => {
// finish processing the data
});
What happens is that zip contains a write stream and a read stream. myStream.pipe(zip) will pipe the myStream stream into zip's write stream,
while zip's read stream will be consumed by the 'data' event listener.
The 'finish' event pertains to the write stream, while 'end' pertains to the read stream.
Thus, with the assumption that finishing the write stream means that the read stream should also finish,
as any data that was processed by the write stream before finishing would have resulted in a 'data' event being emitted by the read stream,
we emit 'end' when the write stream finishes.
However, the autodrain feature seems to interfere with this assumption. As described in the documentation:
'If you do not intend to consume an entry stream's raw data, call autodrain() to dispose of the entry's contents. Otherwise the stream will halt.'
This can cause the following sequence of events:
- The read stream is paused, because the current entry is not being consumed, or rather is taking a while to consume (for instance, because it is a large file)
- The 'finish' event is emitted by the write stream, because it has finished parsing the zip file, which results in the 'end' event being emitted by the read stream
- The reader finishes consuming the current entry, possibly calling autodrain(). Then it stops because it receives the end event, and does not expect any more data to come.
- more 'data' events are emitted, because the read stream resumed sending them, after the entry was consumed / autodrained() was called
This effectively causes the reader to not be able to rely on the 'end' event, as more data may come afterwards.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We do not need to emit 'end' here, it will be called automatically by the node stream implementation. The problem is the following:
Lets assume we have a read stream on a ZIP file, that we want to parse. We use the following construct:
const zip = unzip.Parse();
myStream.pipe(zip);
zip.on('data', data => {
// do something with the data
});
zip.on('end', () => {
// finish processing the data
});
What happens is that zip contains a write stream and a read stream. myStream.pipe(zip) will pipe the myStream stream into zip's write stream, while zip's read stream will be consumed by the 'data' event listener. The 'finish' event pertains to the write stream, while 'end' pertains to the read stream. Thus, with the assumption that finishing the write stream means that the read stream should also finish, as any data that was processed by the write stream before finishing would have resulted in a 'data' event being emitted by the read stream, we emit 'end' when the write stream finishes.
However, the autodrain feature seems to interfere with this assumption. As described in the documentation: 'If you do not intend to consume an entry stream's raw data, call autodrain() to dispose of the entry's contents. Otherwise the stream will halt.' This can cause the following sequence of events:
This effectively causes the reader to not be able to rely on the 'end' event, as more data may come afterwards.