How Natural Language Processing Works?

As we all know that Apple has introduced an API for natural language processing from iOS 5, which allows us to tokenize text, detect the language and determine parts of speech.

Basically Natural Language Processing (NLP) is used to either predict your next word or suggest a correction while typing a word. NLP is likely used in Siri.

The main API is NSLinguisticTagger which is used in analyzing and tagging text, segmenting content into paragraphs, sentences, and words. In iOS 11, NSLinguisticTagger becomes more powerful. It is used for following schemes,

• Identifying the specific language
• Tokenization: classifies each character as either a word, punctuation, or whitespace.
• Lemmatization: Identifies all the forms of a word.
• Parts of Speech: Breaks a sentence into nouns, verbs, and adjectives.
• Named entity recognition: It helps us identify whether the token is named entity like person name and place name.

Let’s take experiment with the new NLP API. At first, we need to do is create a tagger. In NLP, a tagger can read the text and give different information to it such as part of speech, recognize names and languages, perform lemmatization etc.

When you initialize NSLinguisticTagger, you have to pass in the tagSchemes in which you want to perform. Let’s do it:

1. Language Identification

You can retrieve this language by accessing the dominantLanguage property of the NSLinguisticTagger. NSLinguisticTagger analyzes the text to get the dominant language.

2. Tokenization

The punctuations and whitespaces are omitted with NSLinguisticTagger Options. Tokenization is the process of classifying the text into paragraphs, sentences, and words. We call them tagger.enumerateTags function to tokenize.

It will be splitting the above string into words and then we get the list of each word that is there in the sentence.

3. Lemmatization

Deriving the dictionary form of the word is called Lemmatization. For example, a user wants the result for word ‘run’. if you were able to consider base forms of the word, you would be able to also get results for ‘running’, ‘ran’, ‘will run’, etc.

Here the raw value of the tag is lemma of a particular word. So In output we got is stem form of each word token.

4. Parts of Speech

This is used to get each token’s lexical class. It will return each word and its part of speech. You can see the noun, preposition, verb, adjective, or determiner.

In the output, You can see the verbs, nouns, prepositions, adjectives, etc.

5. Named Entity Recognition

Named Entity Recognition is allowing you to recognize any names, organizations, or places. You would have seen certain keywords highlighted like numbers, names when you use some iPhone application.

In the following example we identify whether the token is named entity like personal name and place name.

Here we have used tag variable which is used by tagger to look for particular tags in sentence.

Conclusion

Natural Language Processing is already a powerful tool, growing exponentially and that can be widely used in the Applications. Apple’s Siri and Facebook’s messenger bots are the best examples of NLP.

In this article, we, 9series, have covered about NLP, its terms and how it works. If you have more experience with it or find more features then feel free to share your own experience with us.

Stay tuned for upcoming articles.