Sentiment Analysis: Mining the Web for Feelings

I read in The New York Times a nice article about Sentiment Analysis, written by Alex Wright, where he commented about some new sentiment analysis companies that are trying to tap into the growing business interest in what is being said online. "The rise of blogs and social networks has fueled a bull market in personal opinion: reviews, ratings, recommendations and other forms of online expression. An emerging field known as sentiment analysis is taking shape around one of the computer world’s unexplored frontiers: translating the vagaries of human emotion into hard data", he said.

“Social media used to be this cute project for 25-year-old consultants. Now, top executives are recognizing it as an incredibly rich vein of market intelligence”, said Margaret Francis, vice president for product at Scout Labs. “Our algorithm is about 70 to 80 percent accurate,” said Ms. Francis, who added that its users can reclassify inaccurate results so the system learns from its mistakes."

Jodange, another company, uses a sophisticated algorithm that not only evaluates sentiments about particular topics, but also identifies the most influential opinion holders, based on research by computer science professor Claire Cardie.

Newssift uses an experimental program that tracks sentiments about business topics in the news, coupled with a specialized search engine that allows users to organize their queries by topic, organization, place, person and theme. It is used by The Financial Times.

Bo Pang developed software that looks at several different filters, including polarity (is the statement positive or negative?), intensity (what is the degree of emotion being expressed?) and subjectivity (how partial or impartial is the source?). He is researcher at Yahoo, and co-wrote Opinion Mining and Sentiment Analysis, one of the first academic books on sentiment analysis.

“Sentiments are very different from conventional facts,” said Seth Grimes, the founder of Alta Plana, who points to the many cultural factors and linguistic nuances that make it difficult to turn a string of written text into a simple pro or con sentiment. “I see sentiment analysis becoming a standard feature of search engines,” he said, and suggests that such algorithms could begin to influence both general-purpose Web searching and more specialized searches in areas like e-commerce, travel reservations and movie reviews.

With the quick growth of social networks, the use of Sentiment Analysis is becoming increasingly important to get the sentiment right from human emotions and opinions, translating into data to make better decisions.

Interviews with Eric Schmidt, CEO of Google

Recently, I read and watched two interesting interviews with Eric Schmidt, CEO of Google.


The first is a video interview, for McKinsey Quaterly, where he talks about several subjects, divided in frames: Change competition, making money, the long tail, evolving management, the nature of innovation, and global standards. This interview was conducted by James Manyika, a director in McKinsey’s San Francisco office.

About innovation, he said: "Google's objective is to be a systematic innovator at scale. Scale means more than one. And innovator means things tha make you go, 'Wow'", and about the long tail, he said: "You can have a long tail strategy, but you better also have a head, 'cause that is where the revenue is".


The second interview, is for The New York Times, conducted by Miguel Helft. It is a Q&A Interview (it has a video too), where he talks about his plans for managing Google in a downturn, the unraveling of an advertising partnership with Yahoo, green energy and his support for President-elect Barack Obama.

I would like to highlight the below question, his answer is very interesting:
Q. Isn’t it less fun to run a company that has to watch its spending more carefully?
A. I think it is actually more fun. The reason is that it is very easy to be a successful executive in high-growth times. It is much more challenging, but in my view much more rewarding to be a leader in times where you have to make really hard choices.

How Wall Street Lied to Its Computers

The New York Times published yesterday an article in its section of Technology, called How Wall Street Lied to Its Computers, written by Saul Hansell, where he wrote about the crisis in Wall Street.

He begins the article puzzled about what happened: "That’s what has been running through my head as I watch some of the oldest and seemingly best-run firms on Wall Street implode because of what turned out to be really bad bets on mortgage securities."

He talks about the quantitative analysts (”quants”): mathematicians, computer scientists and economists who were working on Wall Street to develop the art and science of risk management. They were developing systems that would comb through all of a firm’s positions, analyze everything that might go wrong and estimate how much it might lose on a really bad day. We’ve had some bad days lately, and it turns out Bear Stearns, Lehman Brothers and maybe some others bet far too much. Their quants didn’t save them.

In the article has some comments and statements, I highlight:
Some analysts said that most Wall Street computer models radically underestimated the risk of the complex mortgage securities. That is partly because the level of financial distress is “the equivalent of the 100-year flood,” in the words of Leslie Rahl, the president of Capital Market Risk Advisors, a consulting firm. But she and others say there is more to it: The people who ran the financial firms chose to program their risk-management systems with overly optimistic assumptions and to feed them oversimplified data. This kept them from sounding the alarm early enough.

“There was a willful designing of the systems to measure the risks in a certain way that would not necessarily pick up all the right risks,” said Gregg Berman, the co-head of the risk-management group at RiskMetrics, a software company spun out of JPMorgan. “They wanted to keep their capital base as stable as possible so that the limits they imposed on their trading desks and portfolio managers would be stable." He also said: "They continued to trade very complex securities concocted by their most creative bankers even though their risk management systems weren’t able to understand the details of what they owned."

"So some trading desks took the most arcane security, made of slices of mortgages, and entered it into the computer if it were a simple bond with a set interest rate and duration. This seemed only like a tiny bit of corner-cutting because the credit-rating agencies declared that some of these securities were triple-A. (20/20 hindsight: not!) But once the mortgage market started to deteriorate, the computers were not able to identify all the parts of the portfolio that might be hurt.

Lying to your risk-management computer is like lying to your doctor. You just aren’t going to get the help you really need."

Reality Mining: Predicting Where You’ll Go and What You’ll Like

The New York Times published an interesting news today, in its section of technology, about a predictive analytics tool, called Macrosense, released earlier this month by Sense Networks, a software analytics company based in New York City.

According Sense Networks: "Macrosense is the world's first platform capable of collecting and analyzing massive amounts of anonymous, aggregate location data in real-time" and "Macrosense applies complex statistical algorithms to sift through the growing heaps of data about location and to make predictions or recommendations on various questions — where a company should put its next store, for example".

The Key Features of Macrosense are: Real-Time Activity Analysis, Powerful Analytics, Historical Data Normalization, Contextual Data Inputs,and Flexible Interfaces and Visualizations.

Sandy Pentland,co-founder of Sense Networks, said that Macrosense tool lets companies engage in “reality mining”, term coined by her.

Reality mining raises questions about privacy, but according the company, it is interested only in aggregate data and that it’s looking for broad patterns, not the specific behavior of individuals.

Sense Networks also announced another tool, called Citysense, defined by them as: an innovative mobile application for local nightlife discovery and social navigation, answering the question, "Where is everybody?" Citysense shows the overall activity level of the city, top activity hotspots, and places with unexpectedly high activity, all in real-time. Then it links to Yelp and Google to show what venues are operating at those locations.

They are testing Citysense in the city of San Francisco, California; and it is currently available on BlackBerry devices and will be released for the Apple iPhone soon.

I think the use of predictive analytics will allow several kind of applications like that, and that is just beginning.

Pursuing the Next Level of Artificial Intelligence

Today, The New York Times published in its section of technology, a nice article about Artificial Intelligence, entitled Pursuing the Next Level of Artificial Intelligence.

It is about the work of Daphne Koller, a researcher at Stanford University, in the field of Artificial Intelligence.

I think Artificial Intelligence is one of the most fields of research nowadays, and its advance is amazing in the last years.

In Business Intelligence and Performance Management, the AI together its relationed fields of study, mainly Machine Learning and Data Mining, will leverage the predictive analysis.


In this article, you can notice the power of Google, when Daphne said that many of her graduate students have gone to work at Google, although she tries to persuade undergraduates to stay in academia; and also when she said: “My husband still berates me for not having jumped on the Google bandwagon at the beginning”.

After all, Peter Norvig, a famous research of AI, is Director of Research of Google.

For those interested in AI, Peter Norvig and Stuart Russell wrote a very good book: Artificial Intelligence: A Modern Approach (2nd Edition)