DAsH

Learning Goals

A common use for network graphs is to analyze social networks, or how groups of people interact and are connected. Twitter is sometimes mined for these connections between people: who is retweeting who, who is using a given hashtag, and how those users are connected to each other, directly or indirectly. 

With this exercise you'll find out how to pull data from Twitter on a previous viral trend "Laurel/Yanni", the audio clip where some hear a voice saying "Laurel" and others hear "Yanni". By importing Twitter data, you'll see how that trend is spreading, in a smaller section of Twitter at least, by searching for instances where those words were used alongside a tweet with a video linked to it. It's important to note that you aren't going to get the exact same data as me, due to the limitations of NodeXL's basic import functions of Twitter, but you'll learn the principles of how to import and visualize Twitter data so that you can use it for your own projects.

Getting Started

• If you haven't already downloaded NodeXL from their site, visit their download center here. Unzip the resulting folder you receive and run the installer.
• Locate NodeXL Excel Template in your Start Menu and click to open it. 
• It will open like a normal Excel Template with some column headers already set up. However the rows so far contain no data. Using the tabs at the bottom, scroll through the different sheets included in the template to get a sense of how the data is going to be organized. 
  • It opens to the sheet Edges. This is where the data on the connections between one user and another is stored. It starts with columns called Vertex 1 and Vertex 2 where each endpoint of a connection is stored. Vertex is a fancy word for nodes or actors, or the two people, places, or organizations that you want to see the connections of. In the case of this exercise, the edge is an individual tweet that matched your search parameters, and each vertex is a user who mentioned or retweeted that tweet as well as its originator. The Edges sheet will store the information about which two vertices (vertices is the plural of vertex) are connected, and it has additional columns for how to represent those connections on the graph, as in what color to make the line, as well as other information downloaded from Twitter about when the relationship occurred, what kind of relationship it is, and what geographic locations or websites are represented.
  • The next sheet is Vertices. This is where the data on the different nodes goes. The vertices are the items that are connected, in this case Twitter users. The additional columns allow for instructions on what color to make each node that can either be generated by the program, or customized by you. There is also additional information downloaded from Twitter about that user. 
  • After that comes Groups and Group Vertices. In Groups, you can set up the different categories you want the vertices to go into, and how the vertices in that group will be symbolized (square? yellow?). In Group Vertices, you assign the vertices to a particular group. This can be done manually, or the program can do so for you. For instance, you may want to separate the Twitter users in this group into men, women and institutions. Or you might want the program to separate out depending on a vertex's properties in the graph, how connected that vertex is or other measurements.
  • The last sheet in this workbook Overall Metrics is where measurements from looking at the graph as a whole will be kept, such as the graph's density or other measures. 
• If you don't already have a Twitter account, please sign up for one now at Twitter.com. This will be needed for the next step. 

Importing a Network from Twitter

There are a bunch of different options visible to you from the menu in NodeXL about where you can import from, but some of them are only available to Pro users. In this instance you'll be using the import function from Twitter. It's crucial to note that there's a limit of about 2,000 tweets that you can gather at one time, and that Twitter's API has a limit that you can only query it about once every 15 minutes so you'll want to be careful about your search terms so you get the information you want on your first try.

You'll be using this import function to gather tweets containing the words laurel and yanni that link to a video in an attempt to see how that viral challenge spread. 

• Click on the menu on the Excel menu ribbon marked NodeXL Basic and go to the dropdown labeled Import on the left side 
• From that menu, select Import from Twitter Search Network. 
• There's a box below the initial search box that says Your Twitter Account. From there , if it's your first time using NodeXL, select the top option that you want to be taken to the authorization page on Twitter to sign into your account. Log in to Twitter from that page to let NodeXL login to your Twitter account. It will only use it to access Twitter, it won't post, but if you're feeling a little paranoid, you can always create an account just for this purpose, and authorize it to use that account.
• Once you've authorized it, consult the Twitter Advanced Search Options to help you come up with the operators you can use in your Twitter search. It contains options to look for different word combinations, different users, different locations, and different filters for kinds of links associated with the tweet.
• 
• In this case laurel yanni filter:videos is what you'll enter in. Stick with the basic network in this case. Your network will contain different information than mine, but when it finishes processing (give it 5-10 minutes), you'll get a pop up saying it'll need to turn on text wrapping. Click that you still want to continue the import
• You'll get information in your NodeXL sheet that looks something like this
  
• Edges contains the information on who tweeted, replied or mentioned whom.  It also contains dates and times. Note that they are all recent as the program starts recently and then counts backwards as a date was not given. When I've tried to use it lately, the date function has been a little buggy, so I simplified it for this tutorial and didn't include one.
• Vertices contains the nodes or twitter handles of users along with their description, number of tweets and information on how many followers and follows they have, if you want to use that information to filter or visualize them.
• The Groups and Group Vertices tabs will be blank until you generate groups (which you will later).

Getting Metrics and Deciding Visual Properties

What you get after importing the tweet is what amounts to a table. In order for this to become something that you'll be able to look at and analyze, you'll want to graph it, but as there are going to be a lot of individual tweets and users represented in this graph, you'll want to make sure that the different properties about those users and connections are indicated visually in your graph. With the below instructions you'll calculate which nodes are connected to the most other nodes (meaning they tweeted to the most or were retweeted by the most other users), and color them in accordingly.

• In the NodeXL tab click where it says Graph Metrics, this is how the program will calculate the different measurements for each node. 
• On the box that pops up, sure you've checked the boxes for Overall Graph Metrics, Vertex Degree, Vertex In-Degree, and Vertex Out-degree and click on Calculate Metrics. Degree is a measure of how many other nodes a given node is connected to. In-Degree and Out-Degree are measures on a directed graph like this one, where connections aren't always reciprocal. In-Degree is how many connections that other nodes have made to a certain node, so in this case something like time other users have retweeted, tweeted at or mentioned that twitter user. Out-degree is how many times that user retweeted, tweeted at or mentioned another user. Overall Graph Metrics will take measurements that apply to the entire graph, like the total number of vertices and edges.
• When this calculation finishes you'll see that the tab Overall Metrics now contains information in it and the Vertices sheet now has information in the columns of In-Degree and Out-Degree
• On the Vertices sheet, you can make the graph show nodes that have a higher in-degree as (the ones being tweeted to, retweeted or mentioned) a different color than ones with a lower in-degree using the Visual Properties columns. Click on the In-Degree column and then sort it largest to smallest
• Highlight the rows with an in-degree of 50 or over in the column Color (column B). Right-click the dropdown at the top,choose Select Color and click on the red swatch. Select the rows with an in-degree of 49 to 10, choose Select Color in the same way and click on one of the orange swatches. You'll see this time it populates with three numbers separated by commas. These are that shade of orange's RGB values. Select the rows remaining and choose the the light blue on the Select Color Menu
• Next go to the Edges sheet. Go over to the Relationship column, and sort by that column alphabetically. The same way that choosing different colors for nodes based on the categories they fall in lets you tell more about the nodes in the graph, this will let you see more about the relationships in the graph. Highlight the column Color (column C) where the Relationship is Mentions make and choose the purple swatch on the Select Color menu. For Replies choose green, and for Tweets choose black. 

Graphing Sheet

Now that your graph will have some symbology indicating which nodes belong to which category, you can create a graph. You'll see at the top of the top of the sheet is a window called Document Actions which by default shows information about Node XL. This is where your graph will be displayed. 

• In the NodeXL tab, you'll see a couple of dropdowns listing Type and Layout.  This is where you can make decisions about what method will be used to design your graph. For now, we'll just go with the options of Directed and Fruchterman. Click on Show Graph.
• When the display in your Document Actions window changes, it will look something like this:
  
• Based on the colors that you decided upon, you can see that most of the nodes have rather low degrees, but a few have higher degrees. This makes sense if you take into account the structure of Twitter. A few people will make a tweet that becomes popular and is passed on by other users. Most of the connections are mentions, 
• If you want to explore where the connections are, you can click on a given vertex (node) and it will highlight in red all the different connections made to it on the graph. Hovering will show you the twitter handle and the tweet associated with that point.
  
• When you do this it will also highlight the affiliated lines on the Edges and Vertices sheet of the edges associated with that node as well as the Vertex.
• This is just one layout possibility though. You can try one of the others by clicking on the Layout dropdown,  choosing Harel- Koren Fast-Multiscale and clicking Refresh Graph
  
• You can see this layout emphasizes the distances between the points and that they aren't in many cases interconnected at all. It also makes a bit clearer which tweets are the ones that connect the different more separated groups
• You can try out the different settings for lauouts but the top two (Fructerman-Reingold and Harel-Koren) seem to be the most illustrative of what is going on with a group that contains this many datapoints
• Using Dynamic Filters, you can change the dates that the graph displays by changing the dates on the calendar, or by using the sliding arrows. 
• You can also go to either the Vertices and Edges sheets and use the filters at the top of a column to change what is showing up, for instance if you just want to filter to texts in the Pacific Time Zone, click on the filter arrow at the top of the Time Zone column and choose that designation
• And then refresh the graph, and you'll see that it will have changed, and that all of the tweets that are filtered out are now black. It looks like a lot of the tweets that were gathered were not in the Pacific Time Zone. 
  

Grouping

If it isn't very clear on the graph which tweets are most densely related to each other, you can use NodeXL's analysis mechanisms to figure out which node share commonalities in who they are connected to.

• In the NodeXL ​tab, click on Groups and choose Group by Cluster 
• You have a couple of options for what method to use, choose Clauset-Newman-Moore. Give it a few minutes to process, and when it finishes it will open up to the Groups tab, and you'll see that it now has information printed in it. 
  
• Groups contains the numbering of the group and what color and shape the vertexes within it will be. Group Vertices lists which  vertices make up which group. In this case it's created 272 groups
• Click Refresh Graph and you'll see that colors and shapes have changed based on which group the vertices are placed in. 
  
• To highlight which nodes are in which group, click on each row in the Group sheet and that group's nodes and edges will  be highlighted in red on the graph.
• You probably noticed that there are a whole lot of groups and hence a whole lot of extra shapes. This is in part because of items like tweets that are not connected to any other node - those with the circular black arrow. You don't really need each of those to have a separate group, so you can redo the Group by Cluster and this time check the box for Put all neighborless vertices into one group. This will mean that instead of each of those individual users that did not get mentioned or followed being a separate group, they'll all be in one group and hopefully your graph will be easier to read. Click OK
  
• When it is done processing, go back to the Groups tab and you'll see that at least now there are 115 groups instead of 272. 
• When you click on Refresh Graph, you'll hopefully see a much more organized picture. All of the nodes that are only connected to themselves now share one color.
  
• If you change the graph layout to Harel-Koren you'll see that the groupings are made a bit clearer
• There is a second choice for how NodeXL can group by cluster and that's Wakita-Tsurumi. This is a method designed for communities with a large number of nodes, so it may help you to make more sense of this graph. To see how this method differs, go to the option for  Group by Cluster again, choose that Wakita-Tsurumi and click okay
• You have a couple of more groups this way, but overall, at least in my graph, this doesn't tell me any more than the existing color choices that I made in the last section of this tutorial.
• If you want to create the groups yourself, you can also group by an attribute of the different vertices. Like, say, Time Zone. In the NodeXL tab, click on Groups > Group by Vertex Attribute. From the dropdown, select which of the attributes you want it to be grouped by. In this case, select Time Zone and click OK.
  I
• The Groups sheet in your workbook now has group names designated based on the Time Zone reported for the tweet. Click on Refresh Graph. 
  
• You'll see most are now light blue (the color for unknown) so this isn't as useful as you might have liked, but in a graph where you have greater control over attributes that are written in and making sure that you always have information for that column, this can be a useful thing
• You can also group by how many tweets overall a person has done.
  • First, sort by smallest to largest to see what kind of number distribution you have. 
  • Click on Group > Group by Vertex Attribute and choose Tweets. Choose that the column's values are Numbers. Add in the ranges that you'd like the new groups to start at. I've chosen 200, 500, 1,000, 2,500, 5,000, 10,000, 20,000, 50,000, 100,000. If you don't put in a range it will just make one group. Click OK.
  • Then Refresh Graph and you'll see the graph changes. If you want to make it more evident which each belongs to which group, you can choose your colors more deliberately to move from one side of the spectrum to another, so cooler tones to warmer tones
  • They aren't in order so you'll need to go to the Vertices tab while highlighting each group in the Groups tab and then to the Vertices tab to see which are highlighted. I'll have it so cooler (blues, greens) tones mean lower tweets and warmer (reds, oranges) tones mean a larger number of tweets.
  • Refresh Graph. From this graph  you'll see that nodes that have their tweets being mentioned a lot, tended to be ones with a larger number of overall tweets.

Analyzing Your Network Graph

• You can zoom in and out on the graph on certain points. For instance , maybe you are most interested in users that tweeted two different groups to see what may be unique about them.
  
• You can click on these nodes and see if there are any common characteristics that tweets connecting two different clusters seem to have (i.e. are they jokes? do they offer additional commentary?) You can look for these connecting nodes, and use the sheet to label them, or make them larger or change their color.
  • For instance by clicking through this exchange you could see that this user both mentioned a video link of an audio engineer explaining the difference, and that of a tv show host interviewing a musician on which they heard, and different groups picked up on those tweets.
• While currently the colors and shapes of the dots are determined by the group they are in, you can also change the color or shape scheme depending on the tweeter's time zone, by the time of the tweet by going over to the Edges sheet. Or if you were interested in whether the tweet was a joke, an explanation or some other commentary, you could change the coloring or shape based on that by editing the Visual Properties for that tweet on the Edges sheet.
• If you want to continue adding to this graph but are stymied by the 2000 tweet limit imposed by using the basic level of NodeXL, you can open a new sheet, import a new set of data, copy and paste it to this sheet, and use the remove duplicate function in case you do get some of the same tweets. 

NodeXL is a relatively simple and free way to play around with analyzing Twitter networks.