TokenMatchers and Tokenizers (Legacy)

You can use Token Matchers and Tokenizers with some Refiner functions.

• Tokenizers provide a way to break text into multiple pieces, while
• TokenMatchers provide a way to score and clean each particular piece of text with knowledge of the semantic category that it belongs to.

For example, without context, it is difficult for a computer to interpret the value of 1o Novembr 200B, but knowing that this value is supposed be a date changes everything: it is clearly 10 November 2008.

TokenMatcher and Tokenizer usage

Token Matchers and Tokenizers can be used with the following Refiner functions (documented in the Refiner/Sheet Functions::NLP Functions section):

• nlp_token_clean cleans a piece of text
• nlp_token_score provides a semantic validation score, from 0 to 1.0
• nlp_token_find finds the best scoring token in a string
• nlp_token_find_all finds all the tokens in a string with a score higher than 0.8.
• nlp_token_select finds the best scoring string among its argument list

Each of these refiner functions take in a model_config argument, which is a dictionary of parameters that can be used by your chosen TokenMatcher. The nlp_token_find and nlp_token_find_all functions also take in tokenizer and tokenizer_config arguments to define how text should be broken up before each token is scored (default to unigrams, or splitting text at whitespace).

At minimum, these nlp_token functions take in the model argument. Most TokenMatchers are automatically paired with a default Tokenizer, usually the recommended Tokenizer for that TokenMatcher.

Here are a few examples of how these functions can be used:

nlp_token_find('Due on 20IB-0A-1B', 
               model='only-digits', 
               separator=' ') # -> '20IB-0A-1B'

Here is an example of how to pass a file path to the Lexicon TokenMatcher:

  nlp_token_find('Your input string', 
                 model='dataset', 
                 tokenizer_config=map_create([['dataset', 'your_dataset.csv']]))

Available TokenMatchers

The following token matcher can be used within Refiner formulas and UDFs.

• ISO 3 country code
• Machine-readable zone

Available Tokenizers

The TokenMatcher framework allows for different ways to break up text before searching for tokens. The following tokenizers can be used. (The default for most operations is Unigram).

Creating custom TokenMatchers and Tokenizers

Creating custom TokenMatchers and Tokenizers is similar to the process of registering custom Refiner formulas. Import the TokenMatcher and Tokenizer interfaces, and create subclasses that implement the required functionality. The custom classes are then registered by defining register_token_matchers and register_tokenizers functions. See the example below:

from instabase.model_utils.tokenizer import Tokenizer
from instabase.model_utils.token_matcher import TokenMatcher
from instabase.provenance.tracking import Value

class MyCustomTokenizer(Tokenizer):
  TOKEN_TYPE = u'my-custom-tokenizer'
  HUMAN_NAME = u'Custom - From a UDF'

  def tokenize(self, text, **kwargs):
    # type: (Value[Text]) -> List[Value[Text]]
    # Just return the first word
    return [Value(text.value().split(' ')[0])]


class MyCustomTokenMatcher(TokenMatcher):
  MATCHER_TYPE = u'my-custom-token-matcher'
  HUMAN_NAME = u'Custom - From a UDF'
  DEFAULT_TOKENIZER = u'my-custom-tokenizer'

  def score(self, token, **kwargs):
    # type: (Value[Text], **Any) -> float
    # For this example, give all tokens the same score
    return 0.8


def register_token_matchers(name_to_fn):
    name_to_fn.update({
      'MyCustomTokenMatcher': {
        'class': MyCustomTokenMatcher
      }
    })
def register_tokenizers(name_to_fn):
  name_to_fn.update({
    'MyCustomTokenizer': {
      'class': MyCustomTokenizer
    }
  })

The tokenizer takes in a Value object of type Text and returns a list of Value objects.

This list of value objects can then be used in a Refiner formula:

  nlp_token_find('Some example text', 
                 model='my-custom-token-matcher', 
                 tokenizer='my-custom-tokenizer')

or within a UDF using REFINER_FNS:

scan_result, err = REFINER_FNS.call('nlp_token_find_all', 
                                    val, 
                                    model='my-custom-token-matcher', 
                                    tokenizer='my-custom-tokenizer', 
                                    threshold=0.7, 
                                    **kwargs)

Make sure to include the **kwargs parameter that comes from the calling UDF function. The **kwargs parameter includes an important context for running TokenMatchers.