Filtering

Filters can be applied in one of two ways, either immediately after an annotator in the flow or at the very end of all flows. Conditions may exist that require annotation filtering before the next annotator runs in the flow. In this case you would define your filter with the specific annotator within the flow element. However, if you do not have unique conditions or you do not know which annotator to add a filter definition in the flow, you can just add your filters to run at the end of all the flows. To do so you define a global configuration as an element of the AnnotatorFlow. (Note: Because of the nature of the Hypothetical and Negation annotators, more consideration may be needed to determine desired filter usage. See Hypothetical and Negation Annotator Filtering below for more information.) Below is an example of several annotators defined in the flow with a global filter to run at the end of all flow elements.

The globalConfigurations filter example below filters out any SymptomDiseaseInd annotations where either negated=true or hypothetical=true - i.e. remove any negated or hypothetical SymptomDiseaseInd annotations.

The following filter examples illustrate filtering that is defined to be executed after specific annotators complete processing.

Sample request invoking concept_detection with a filter defined to exclude any annotations detected from concept_detection where the semanticType field does not equal neop.

The definition of filtering includes target and condition sections. The target section specifies type of entities to be filtered (defined via object model path to desired entities) e.g. unstructured.data.concepts. The condition section includes filtering criteria (Match Condition) or filtering criteria set (Grouped Condition) to be applied to target entities. The following snippets illustrates the target section of the filtering definition. The data section will be added to the unstructured section after ACD has invoked an annotator for the unstructured text.

The condition section of the filtering definition specifies different matching conditions. Field-level filtering criteria is based on a set of possible values listed in the following table. The basic field-level filtering can be either grouped as a grouped condition or nested as a nested condition.

Table 1. Field-level filtering criteria

The following is an example of Field Equals scenario.

The following is an example of Field Not Equals scenario. Notice that not field has the value of true.

The following is an example of Field Contains. Notice that the field is set to be section, and the ‘values’.

The following is an example of Field Exists. Notice that the field is set to be hccCode, and there are no ‘values’ required.

fieldExists scenario looks for the existence of the specified field in the annotation and ensures that the field contains a valid value. Valid values are not null or empty values.

The ‘type’ field has the default value of match. Grouped and nested conditions can be implemented using different values in the ‘type’ field. The following table list set of conditions to be evaluated.

Table 2. Conditions to be evaluated

The following snippet illustrates the grouped condition with the all type. The returned annotators will satisfy both of the listed conditions.

The following snippet illustrates the grouped condition with the any type. The returned annotators will satisfy either of the conditions.

The following snippet illustrates the grouped condition with the notAny type. The annotators will only be returned if they don’t match the condition.

Nested condition refers to the include and exclude annotators with different criteria. Nested condition, similar to to the grouped condition, can be implemented by modifying the value of the type field. The following example illustrates the situation where one wants to include annotators with the semanticType of ‘podg’ however, one wants to exclude annotators with the semanticType of ‘neop’ and annotators with the cui equals ‘C0684249’.

Because the Hypothetical and Negation Annotators can affect annotations created by annotators earlier in a flow, filters constructed for these annotators are typically run immediately after the Hypothetical or Negation annotator. If these filters were run in a global manner, after all of the annotators in the flow have completed, the results returned may include annotations with incorrect feature values for “hypothetical,” “hypotheticalType,” or “negated.”

Consider the scenario where the annotator flow contains both the Procedure annotator followed by the Negation annotator, and the text “She did not have an MRI” is analyzed. A ProcedureInd annotation over “MRI” will get created with the “negated” feature value set to “true.” There will also be a NegatedSpan annotation over “MRI” returned in the results. If a global filter was used to filter away NegatedSpans where the trigger.coveredText equals “not,” the NegatedSpan annotation would no longer show up in the results, however, the ProcedureInd annotation would still have it’s “negated” feature value set to “true.” This can lead to confusing results. If the same filter was used with the Negation annotator, the “negated” feature would remain “false.”

To avoid potential inconsistencies, filters created for Hypothetical or Negation annotators should be declared within the Hypothetical or Negation configurations so they are executed before the effects of these annotators carry over to existing annotations, and not as a global filters.