External code usage

[alkuzad]

Hi

I was thinking if it's possible to include feature for including external code into code of aggregate. I've done some fast test with include and it works - I can use new functions from external file.

The reason for this is that I currently have a lot of code that does some checks, merging on map and calculations (like time duration). It starts to be very very messy and I would like to create some general functions so I can pass whole event and map and do necessary calculations. Therefore the code will not look like:

map['foo'] ||={}; map['foo2'] ||={}; map['start'] = 0; map['foo'] = event['stop'] - map['start'] if map['start']; map['start'] = event['start'] if event['start'] and map['start'] == 0

but like:

start_stop_calculate(map, event)

What do you think about such idea ? Basic include is trivial but ensuring that only proper functions can be called is more complicated (i.e move aggregate call to separate class)

[fbaligand]

You say you managed to include external class or method into aggregate call.
How did you do that ?
Globally, how do you see the feature ?

[alkuzad]

@fbaligand
Just for test I've added includes option to aggregate:
config :includes, :validate => :array, :required => false, :default => []

and required files on register:

@includes.each do |i|
  require i
end

Then I've added includes to aggregate filter and I could use new functions.

I've experimented with separate aggregate call class to dynamically include another module but I do not have this code right now. It shouldn't be too hard though.

[fbaligand]

Thanks for these details.

Actually, I think you can already do what you need with aggregate plugin as it is :

aggregate {
    code => "require 'your-module'
             callToYourMethod(event, map)"
     ...
  }

[alkuzad]

@fbaligand yes it's true. The problem is that you can easily mess up the imported code with internal aggregate code.

I.e one can declare simple "filter" function to do whole task. It can't be done as filter is already declared as main function in LogStash::Filters::Aggregate and it's declaration will be first on the list to use. Of course I can use Module::Function syntax but still it's quite cumbersome (until you made some note in README about that problem).

This is why I wanted to remove @codeblock from Logstash::Filter::Aggregate and move it to smth like Logstash::Filter::Codeblock. This way method running inside class will not have access to Aggregate internals.

[fbaligand]

Hi @alkuzad,
Firstable, sorry for my late answer.

OK, so I think I understand well your need.

To reach your need, I can :

• add a new independant class inside aggregate.rb called LogStash::Filters::Aggregate::CodeBlockHolder
• at register time, I create a new CodeBlockHolder instance that I keep, and I put @codeblock inside this instance.
• and at filter time, I call codeblock which is inside codeBlockHolder instance.
• for your information, I have checked that inside CodeBlockHolder class, you have no direct access to aggregate methods, so you can't call an aggregate method mistakenly.

Does this reach your need ?

[alkuzad]

@fbaligand right, this is it :)

Currently I have such code:

require '/home/alkuzad/scripts/aggregate_scripts/arch_calculations.rb';
a = Foo::Bar::Baz.new(map, event); 
a.initialize_event('arch1');
map.merge!(a.map)

So I have require, initialization of object (passed values of map and event), calculations and merge back map from object. The problems are two:

1. Every code needs require, even if my 30 aggregate uses same class
2. Every code needs map/event assignment or merge (dependent on map_action) as I can only pass values

The best solution would require to require/include only once and extend the class code as it can use the event/map directly.