Bayes and Big Data
Big data is growing trend that will only continue to become more prevalent. Big data refers to data sets so large that parsing the information, keeping it secure, and analyzing trends within the set has become more difficult. With increasing use of big data come concerns about privacy and security of the information collected by various groups. Data scientists have conducted studies that revealed that 90% of shoppers could be correctly identified from only a few dates and locations of random purchases. This analysis was also done on anonymous taxi cab records in New York City; the results revealed the identities of the users, some of which included actors and celebrities. This ease of identification extraction raises the concern for when criminals come to utilize the same analysis to commit fraud or kidnappings.
Currently combining Bayes theorem and big data, scientists are attempting to halt the activity of fraudsters and online criminals. Bayes theorem attempts to show how current beliefs are modified in the presence of new information. Financial institutions are creating these Bayesian classifiers, which parse through large data sets to understand trends and pinpoint any peculiar or fraudulent behavior. For example in the UK, Darktrace uses software that utilizes this strategy. It understands what normal behavior should be and looks for anomalies. Using recursive Bayesian estimation, this software prioritizes the possible threats by deviations from the norm.
Plainly stated, Bayesian principles combine with the program by understanding when given certain extracted trends what is the likelihood this data set is compromised. The increase of use in big data has given rise to the many problems associated with maintaining this data and information assurance. This strategy is not novel, but it is becoming more important as it is one of the more effective methods in these macroscopic problems. These issues become the responsibility of those who choose to maintain these data sets to maintain the integrity of the information, including utilizing techniques like Bayes theorem and other strategies.
http://www.thenational.ae/uae/technology/how-big-data-is-an-inescapable-force-that-is-shaping-the-future#page1