DIMACS

General Information

me
Student: Kevin Wong
Office: CoRE 450
School: Rutgers University
E-mail: kevinw@reu.dimacs.rutgers.edu
Project: Visualizing Twitter Trends

Project Description

Twitter offers a window into the minds of millions. What are people talking about? Are there any emerging trends? In this project we will collect and analyse twitter streams, cluster the tweets as well as graphical relations between people, and summarise and visualize the results to get a big picture of what's being talked about. In the process the student will learn topics such as graph algorithms, machine learning (text model) and visualization. The eventual goal of the project is a website that allow users to enter a search term and get a dynamic, clustered and graphical view of tweets related to the term.

Weekly Log

Week 1:
I moved in to Busch Campus on Sunday. After opening presentations, I was able to meet my professors for a debriefing on a number of proposed projects. I chose the one on Visualizing Twitter Trends. The purpose of the project is to combine the information stored in tweets with Twitter's geolocation to map the world in a different way. I took my first steps towards this goal by reading up on cURL, a popular tool I can use to pull tweets off of the twitter stream. Currently the plan is to implement this using Microsoft Visual C++.
Week 2:
After reading through a number of cURL tutorials and installation guides, I decided to switch from Visual C++ to using GCC in Cygwin; the UNIX-like environment makes cURL much easier to handle. However, as this was my first experience with anything UNIX, I spent the greater part of this week learning the various commands and rules. After a few hours of troubleshooting with Dr. Hu, I was able to compile and run a regular C++ program written in Notepad, but not yet one that implemented cURL. I also created an account with Twitter, as they require a login before granting access to their stream.
Week 3:
My progress this week started on Sunday with the successful execution of a simple C program making calls to functions provided by the cURL library. Soon afterwards, I was able to modify the program (and the cURL installation settings) to tap into the twitter stream. Next, I started looking into a way to use this data with the assistance of Dr. Abello. The route I set down this week had me considering whether or not we could glean any information regarding slang usage from the tweets. Thus, I spent the latter half of this week thinking about English slang terms from a mathematical perspective. The main question to consider is: Can we formulate some definition of slang that is both mathematically rigorous and reasonably accurate?
Week 4:
During my weekend hunt for the solution to last week's question, I found that the topic of slang itself is largely untreaded territory, even among linguists. Whether a term is considered slang or not is a matter of opinion; thus, an attempt at a rigorous formulation of slang would be rather fruitless. I have moved this idea to the back burner for the time being. In other news, this week I was able to further modify my twitter program to save the incoming raw tweets onto my hard drive.
Week 5:
Using the cJSON library, I was able to parse the raw tweets and extract all relevant information - tweet text, retweet count, follower count, and language - in real time. I drafted a program that parses all of my previously collected tweets. Also, using a modified version of Ahmed's tokenizer I am able to run my parsed tweets through it to clean them up (punctuation, stop words, URL's, user_mentions, etc).
Week 6:
This week I wrote a Java program to process the output of the tokenizer. It can generate a word count of every word that appears in the tweets. It also tries to match each location as specified in each tweet to a country/US state with latitude and longitude. For this project, I am focusing on slang terms, thus I also filter out any posts that do not contain slang words. This program outputs information about the most used word in each location over 24 hours.
Week 7:
Using GraphStream, I have generated a visualization of the twitter program output by plotting the latitude/longitude points of each location with the top word alongside it. The points are also colored and sized to reflect certain aspects of its 'trendiness'. Final presentations were also this week. Slides are below.
Week 8:
Last week of the program. This week I modified my visualization so that the color of each point encodes a particular term's trendiness in one location with respect to the others, while its size encodes its current usage at that point in time. I also finished writing up my final paper and wrote a readme doc for the project.

Presentations

Additional Information