- Fix BroadcastDispatcher delivering duplicate demand in some cases
- Fix ordering when there are many buffered info messages
This release addresses corner cases in both demand and broadcast dispatcher where they would sending events when the pipeline became temporarily empty.
- Send events to dispatchers even if there are no consumers
- Hold demand in broadcast dispatcher until asking
- Support
:on_cancelinGenStage.from_enumerable/2
- Shuffle the demands on first dispatch for load balancing across consumers
- Allow explicitly specifying
:max_demandinGenStage.DemandDispatcher
- Do not dispatch when accumulating demand
- Log a clear error messages when stage terminates due to noproc on non-temporary subscription
v1.1 requires Elixir v1.7+.
- Fix warnings on latest Elixir versions
- Support more process specifications in the
:subscribe_tooption - Add callback to allow instrumenting discarded count and function to get estimated buffer size
- Allow events to be discarded in PartitionDispatcher by returning
:none - Raise for unknown partitionis in PartitionDispatcher
- Improvements to typespecs and error messages
- Add
GenStage.demand/1
- Fix code_change callback implementation
- Fix warnings on Elixir v1.8
This version requires Elixir v1.5+.
- Ensure a
:producer_consumerstops asking the producer if downstream demand is 0 - Enforce the :hash option for non-int partitions in PartitionDispatcher
Note: this is the last version to support Elixir v1.3 and v1.4.
- Log error on unknown partitions
- Do not allow restart: :permanent in
ConsumerSupervisorto avoid common pitfalls - Improve and fix types to avoid dialyzer warnings
- Avoid conflict with user specified
@opts
- Mirror
ConsumerSupervisor.init/2afterSupervisor.init/2 - No longer define default implementations for callbacks, instead declare them as
@optional_callbacks
- Ensure
ConsumerSupervisordoes not send demand when restarting a child
- Support Elixir v1.5 supervisor childspecs in ConsumerSupervisor
- Mark
GenStage.child_spec/1overridable
- Define Elixir v1.5 childspecs in GenStage and ConsumerSupervisor
- Fix a bug where info messages would be sent out of order in producer consumers
- Fix a bug where handle_cancel would be invoked out of order in producer consumers
- Add
cancel: :transientto subscription options which does terminate if the exit is:normal,:shutdown, or{:shutdown, _} - Add
GenStage.sync_info/3andGenStage.async_info/2which queues an information message to be delivered once the current queue is consumed
- Remove
:max_dynamicfrom ConsumerSupervisor - The notification mechanism has been removed from GenStage. For termination, GenStage now uses proper exit signals and
cancel: :transienthas been added as a subscription option.
- Remove the Experimental namespace
- Rename DynamicSupervisor to ConsumerSupervisor
- Move Flow to a separate project: https://github.com/elixir-lang/flow
Except by the module name changes, all APIs remain exactly the same.
- Accumulate demands but don't sum them together. This provides a better ramp up time for producers with multiple consumers
- Add
Flow.group_by/3andFlow.group_by_key/3as conveniences aroundFlow.reduce/3 - Add
Flow.map_values/2for mapping over the values in a key-value based state - Add
Flow.take_sort/3that efficiently sorts and takes the top N entries
- Ensure BroadcastDispatcher sends demand to itself when custom selector discards events
- Ensure flows started with
Flow.start_link/2properly terminate if producers terminate - Ensure flows exit locally instead of relying on linked processes exits. With this change,
Flow.run(flow)andEnum.to_list(flow)no longer start stages linked directly to the caller but does so through a supervisor
- Add
GenStage.sync_resubscribe/4andGenStage.async_resubscribe/4 - Improve logs, specs and docs
- Ensure
Flow.departition/4works onFlow.start_link/1 - Make sure no lingering monitors or messages on the inbox after GenStage.stream/1
- Support a
:selectoroption in theBroadcastDispatcher
- Ensure PartitionDispatcher does not create more partitions than necessary
- Pass the events
lengthto dispatchers for more performant dispatching
- Introduce count-based windows, process-time windows and session-based windows on Flow
- Support resetting or keeping buffer on Flow watermarks
- Remove
:milliseconds,:seconds,:minutesand:hoursfor units in favor of:millisecond,:second,:minuteand:hour. You will get an error if you use the previous values. - Specifying shortcuts to
:hashhas been removed in favor of the:keyoption. You will get an error if you use the previous values.
- Ensure uneven partitions emit all windows on
Flow.departition/4 - Properly emit the beginning of the window time on triggers for fixed windows
- Properly count the most recent entry for each fixed window
- Introduce
Flow.departition/5 - Include examples of broadcasters and rate limiters in the documentation
- Allow custom-named, non-integer partitions
- Ensure consumer supervisor respects
min_demandand does not send demand too soon
- Remove
Flow.new/0,Flow.new/1andFlow.new/2in favor of passing options tofrom_enumerable/2andfrom_stage/2 - Remove
Flow.partition/3andFlow.merge/3in favor of passing the:windowoption toFlow.partition/2andFlow.merge/2
This release moves Flow from under the GenStage namespace and into Experimental.Flow.
- Add
Flow.uniq/2andFlow.uniq_by/2 - Add
Flow.start_link/2andFlow.into_stages/3 - Add
Flow.window_join/8 - Unify window and partition APIs
- Support
Flow.Window.global/0andFlow.Window.fixed/3
- Add
Flow.inner_join/6 - Add
GenStage.demand/2that allows a producer to accumulate demand as a synchronization mechanism - Improve performance for the partition dispatcher and allow it to change the partitioned event
- Fix a bug where a flow wouldn't terminate if a source stream halts
- Add
Flow.trigger/3andFlow.trigger_every/4supporting custom, count and processing-time triggers. Event-time triggers can be implemented viatrigger/3. Event-time triggers will be added once windows support is included
- Introduce
Flowwith enumerable/stream based operations - Include more information on
:sys.get_status/1calls for GenStage
- Fix a bug where a
:producer_consumerstage which filtered events would eventually halt - Fix
format_status/2results when inspecting GenStage in:observer
- Support notifications
- Introduce
GenStage.stream/1to stream events as a consumer from a stage - Introduce
GenStage.from_enumerable/2to start a producer stage that emits events from an enumerable (or a stream)
- Add
GenStage.PartitionDispatcher - Set default
:max_demandto 1000 - Use buffer based
:producer_consumerto avoid escalating demand
- Support
:producer_consumertype - Support
:infinityas:buffer_size(useful for:producer_consumer)
- Namespace all modules under
Experimental - Ensure
:cancelreason does not cascade through the pipeline
- Include GenStage with
:producerand:consumertypes - Include ConsumerSupervisor implemented as a
GenStageconsumer and that provides the:simple_one_for_onefunctionality