WildHacks 2015: Real-time Sentiment Analysis of Tweets with Node.js framework

[#1]

The dilemma and the goal:

During the thanksgiving season of 2015, I decided to work on an application which will perform tweet analysis in real-time and thus started working on language modeling for WildHacks 2015.

Twitter's trending topics only go as far as showing what people are talking about. To find out what's happening, you have to read through all the tweets about that trend and try to figure out whether people are happy or sad about that topic. In addition to that, many of the top trends are about the same topic but are still separated because they are tagged with different keywords. I wanted to solve this problem. I wanted people to be able to just think about one topic and they will automatically be shown tweets what that keyword and other related keywords, and tell them what people are thinking about that topic presently.

I decided to use HPE Haven OnDemand to implement these features because they have strong APIs that assist obtaining related topics and analyzing sentiment.

Building your own language modeling app using HPE Haven OnDemand’s Find Related Concepts API and Sentiment Analysis API integrated with Twitter Streaming API

We are going to walk through the following steps:

• Setting up the app
• Getting keyword from the user using websockets
• Finding Related Concepts using HPE Haven OnDemand API
• Getting tweets from Twitter using Streaming API
• Analyzing sentiment using HPE Haven OnDemand API
• Posting the average sentiment using websockets
• Customizing the graph using Highcharts

Setting up the application

First, we want to initialize a new project and create a server which will serve all the requests. The http module is used to create the server. Though http module comes with the Node.js installation, other modules need to be downloaded separately using the package.json file.

We install the following dependencies available in npm:

• js: A lightweight web framework built on top of Node.js
• io: A websocket framework which allows us to push data in real time without refreshing the page
• Haven OnDemand: Node.js client library for various HPE Haven OnDemand API
• Twitter: Node.js client library for streaming the tweets

We go to Twitter to create our application and get all the credentials. We also create an account on Haven OnDemand and get the developer API key. We initialize the Twitter and Haven OnDemand client in our server.

//initialize Haven OnDemand client
var havenondemand=require('havenondemand')
var HPClient= new havenondemand.HODClient('http://api.havenondemand.com', process.env.hpe_apikey)

//initialize twitter client
var Twitter=require('twitter')
var twitterclient= new Twitter({
	consumer_key:process.env.consumer_key,
	consumer_secret:process.env.consumer_secret,
	access_token_key:process.env.access_token,
	access_token_secret:process.env.access_token_secret
});

Getting topics from the user using websockets

Firstly, we setup the input form on the dashboard which takes a topic from the user. We also include the libraries for jQuery and socket.io.

<form>
        <div class="form-group col-md-10">
          <input type="text" class="form-control live_command" name="stream" id="stream0" placeholder="What do you want to know about?">
        </div>
        <div class="col-md-2">
          <button type="submit" class="btn btn-default button1" onclick="execute()">Execute</button>
        </div>
</form>

After this, we initialize socket.io and write a JavaScript function for the event when the execute button is clicked. The function emits the value in the form by attaching it to the topic attribute for the ‘streamTopic’ we are sending.

<script>
  var socket=io();
function execute() {
      var topicValue = document.getElementById('stream0').value;
      socket.emit('streamTopic',{topic: topicValue});
  }
</script>

Finally, on the server side we receive the topic value emitted from the dashboard.

socket.on('streamTopic',function(msg){
	var data = {text: msg.topic}
}

 

Finding Related Concepts using HPE Haven OnDemand API

Now that we have the topic entered by the user from the dashboard, we need to find all the related entities. We call the Find Related Concepts API through the Haven OnDemand client which returns a JSON response with all the entities. We extract each entity and emit it to the dashboard using socket.io.

HPClient.call('findrelatedconcepts', data, function(err, resp){	
			var relatedKeys={text:resp.body.entities};
		 	var i;
		 	for(i=0;i<relatedKeys.text.length;i++){
		 		 io.emit('relatedKey',relatedKeys.text[i].text)
		 	}
}

On the dashboard, after socket.io is initialized we create a JavaScript function which listens for incoming ‘relatedKey’ data from the server. When the data arrives, the callback function creates a JSON object with the received data, parses it, and then displays it to the user.

socket.on('relatedKey',function(msg){
    var entities={text:msg}     
});

 

Getting tweets from Twitter using Streaming API

We set up a stream which tracks all the posts related to the entities received above. It receives a response of incoming tweets which is saved in a JSON object.

twitterclient.stream('statuses/filter', {track:relatedKeys.text[i].text}, function(stream){
stream.on('data',function(tweet){
		var data = {text: tweet.text}
	});
stream.on('disconnect', function (disconnectMessage) {
		console.log(disconnectMessage);
	});
	stream.on('error',function(error){
		throw error;
	});
});

 

Analyzing sentiment using HPE Haven OnDemand API

After receiving the tweets from twitter, we perform sentiment analysis on them using the Sentiment Analysis API. We use the Haven OnDemand node.js client and call the sentiment analysis API which returns a JSON response with sentiment as positive, negative or neutral of the tweet. We are interested particularly in the confidence score of the sentiment which will be used as the data source for drawing the graph.

HPClient.call('analyzesentiment', data, function(err, resp){
	var aggregate = resp.body.aggregate.score
	var positive= resp.body.positive.score
	var negative= resp.body.negative.score
var sentiment = resp.body.aggregate.sentiment
});

Posting the average sentiment using websockets

After getting the tweet and the sentiment we emit it to the dashboard using socket.io. We first create an object with all the data and then send it over to the browser.

var tweet_Data = {tweet: tweet, positive: positive, negative:negative, aggregate:aggregate};
io.emit('tweetData',tweet_Data)

On the dashboard, after socket.io is initialized we create a JavaScript function which listens for incoming ‘tweetData’ data from the server. When the data arrives, the callback function creates a json object with the received data.

socket.on('tweetData',function(msg){
var plotData={count:msg.aggregate, coordinates:msg.tweet.place.bounding_box.coordinates}      
});

Customizing the graph using Highcharts

Next, we use the data received from the server to create a graph updating each second. This shows the sentiment of an incoming tweet in real time. Highcharts is used to implement this: http://www.highcharts.com/docs/working-with-data/live-data.

Sample graph created when “coffee” is searched:

Final Thoughts

It was my first hackathon and working with HPE Haven OnDemand was a lot of fun. I saw a lot of potential uses of the different APIs and looking forward to further developing the application. Storing all the sentiment regarding a topic in a database and using Haven OnDemand Predictive Analysis API to predict what people might be thinking about a topic around the world and representing it using WebGL Globe is next on my radar.

That's it. I would love to get your opinions. Feel free to comment.

Thanks,

Shivam Bharuka

Full Stack Web Developer, Student at University of Illinois at Urbana-Champaign

I am an aspiring entrepreneur who loves combining technology with arts. I love coding and reading about technology. I hope to live a future which will bring my passion for startups, programming and science together. I plan on attending few hackathons this year to expose myself more in these areas.

Twitter @shivambharuka

Useful Haven OnDemand Links:

Developer Documentation

Developer resources for your hackathon

Client Libraries

• Python Client Library
• Ruby Client Library
• Node.js Client Library
• iOS Swift Client Library
• Windows 8.1 Universal Client Library
• Android Client Library
• PHP Client Library