Feature extraction analyzer
The feature extraction analyzer creates attributes out of event data based on different extraction plugins.
Note Please be aware that this analyzer does not extract ipv4, email addresses and similar from all events, but only those that match the definitions configured for the plugins explained below!
This analyzer is helpful to extract additional data from events as separate attributes. Those extracted attributes can then be used in search, lookups, correlations, aggregations or with analyzers.
For example: In the default configuration, the analyzer will extract
email_addresses from the message field of events with the source
matching the regular expression.
Regex Extraction Plugin
This feature extraction plugin uses regular expression to extract matching strings from an existing event attribute (e.g. message) and adds it as a new attribute to the event.
Features are defined in data/regex_features.yaml
A regex based feature extraction definition looks like this:
name: # Define either a query_string or query_dsl. query_string: * query_dsl: # Mandatory fields. attribute: store_as: re: # Optional fields. re_flags:  emojis:  tags:  create_view: False aggregate: False overwrite_store_as: True overwrite_and_merge_store_as: False store_type_list: False keep_multimatch: False
Each definition needs to define either a
query_string or a
re_flags is a list of flags as strings from the
re module. These include:
emojis are optional.
store_as defines the name of the attribute the feature is stored as.
create_view is an optional boolean that determines whether a view should be created if there are hits.
aggregate is an optional boolean that determines if we want to create an aggregation of the results and store it (ATM this does nothing, but once aggregations are supported it will).
overwrite_store_as is an optional boolean that determines if we want to overwrite the field
store_as if it already exists.
overwrite_and_merge_store_as is an optional boolean that determines if we want to overwrite the field
store_as and merge the existing values.
store_type_list is an optional boolean that determines if we want to store the extracted data in List type (default is text).
keep_multimatch is an optional boolean that determines if we want to store all matching results (default store first result).
The feature extraction works in the way that the query is run, and the regular expression is run against the attribute to extract a value. The first value extracted is then stored inside the "store_as" attribute. If there are emojis or tags defined they are also applied to that event. In the end, if a view is supposed to be created a view searching for the added tag is added (only if there are results).
Winevt Extraction Plugin
This feature extraction plugin uses configured mappings to create new attributes for Windows Event Log events that were parsed using Plaso.
The mapping is based on the
strings array, that gets generated by Plaso for
the event data entries.
Note The winevt extraction plugin does not map all Windows Event Log fields. It does only map the ones configured in data/winevt_features.yaml!
Features are defined in data/winevt_features.yaml
A mapping for a Windows Event uses the yaml format and looks like this:
name: source_name: Type: list[str] | REQUIRED | case-insensitive A list of source names to match against. Multiple entries will be checked with OR. provider_identifier: Type: list[str] | OPTIONAL | case-insensitive A list of provider identifiers to match against. Multiple entries will be checked with OR. event_version: Type: int | REQUIRED The event version to match against. event_identifier: Type: int | REQUIRED The event identifier to match against. references: Type: list[str] | OPTIONAL A list of references to provide as context and source for the event mapping. E.g. a URL to the official Microsoft documentation on the event. mapping: Type: list[dict] | REQUIRED A list of dicts that define the new attribute name and the string index of the event to extract the value from. Additonally it can also contain an alias list to add multiple attributes with the same value but different names. name: Type: str | REQUIRED The name of the new attribute to create. string_index: Type: int | REQUIRED | Starting at index 0 The string index of the event to extract the value from. Based on the plaso extracted "strings" attribute with Windows eventlog entries. aliases: Type: list[str] | OPTIONAL A list of aliases to add additionally to the offical name of the attribute. This can be used to add different field names matching individual field name ontologies. E.g. srcIP, domain, etc.
Checkout the preconfigured mappings for some examples: data/winevt_features.yaml