Neural network in predictive text software
Using predictive text on smartphones is sometimes a great way to cut down on time spent typing words or sentences that are particularly common and/or long. One of the early predictive text companies to work on predictive text since 2010, SwiftKey just this early October announced their new software, SwiftKey Neural Alpha – the first keyboard software that uses an artificial neural network to predict text. Essentially, like most other predictive text software we are used to, this one suggests 3 words/phrases as you are typing a sentence; however, what’s new with Neural Alpha is that the context and meaning of the sentence is starting to be understood in order to provide better suggestions. Prior to this development, there was their “n-gram” technology, which predicted words well, but was limited because it used just the most recent two words to suggest words and could only offer words that it has “seen” before. With Neural Alpha, it looks at clusters of words instead of strings, to understand that some words are essentially very similar to others that it has seen been used before.
For example, if the software has trained with the sentence “Let’s meet at the airport”, using relations of clusters in its network, it will understand that “hotel” and “office” are appropriate replacements for “airport” because they are related by being in the same cluster in the network of words (nouns that are locations?), even though it hasn’t seen that exact combination before. And with the new ability to consider earlier parts of the sentence, Neural Alpha understands that “Let’s meet at the airport” is of similar sentence structure to “Let’s chat at the office.”
So in a simplified manner, consider you are typing a sentence and Neural Alpha is trying to decide what words to suggest as you are typing it. It would first recognize your sentence structure, create a sort of code using the codes of each word in the sentence so far, go through the network via clusters that make sense with this structure, and look at similar nodes with closely matched prediction codes so that the word makes sense in context. This software is not fully released yet, but I would assume that if in the conversation you’ve been typing about restaurants a lot, then it would suggest “restaurant” to finish “let’s eat at the” since it already knows “let’s meet at the airport.” (n-gram would suggest things like “moment” and “time” perhaps, since it only recognizes “at the”). And of course, when SwiftKey offers a few suggestions, it must determine which ones are more likely to be the one the user wants, so it ranks them in the order to display.
(Figure 1: n-gram only considers the most recent two words, so it thinks “take a look”. The neural network model instead sees the whole sentence structure so it decides the user wants something related to time)
In class, we’ve talked about networks in which some nodes are more closely associated with some nodes as opposed to others (i.e. strengths of ties or distances between nodes), and in cases where some are interconnected, we can consider as a “component”. In statistics and data mining, often one can use data and generate a graph of nodes with edges as some sort of relationship; sometimes we can see natural groups of nodes closely related to each other (short distances between, some sort of characteristic, etc.), and these are clusters. We haven’t talked too much about clusters, but in class, we have talked about strong and weak ties, as well as knowing that some groups of nodes are more strongly connected than others.
(Figure 2: Example of clusters! Notice that all the months form a cluster, and all the days of the week also form their own tight cluster. The two clusters have some connection because they’re both related to dates.)
We’ve also started discussing pagerank to determine importance and relevance in a search. Neural Alpha, like its predecessor, ranks its results before listing a few suggestions to the user. In order to give likely words or phrases, there may be some sort of system of ranking in which if a certain word is mentioned a lot or clearly makes more sense, then its point values increase (being referenced, or has a strong connection with something that has been referenced before). SwiftKey does not offer a lot of information on specifically how this works, but there should be a sort of “pagerank”-like thing going on after it finds some desirable results, some more usable or common than others based on what other words tend to point to this word or type of word. The more this word tends to be used after a certain sentence structure, or is being used in general as a subject/action, the higher it would rank as an option to use.
Sources:
https://blog.swiftkey.com/neural-networks-a-meaningful-leap-for-mobile-typing/