Mind Mapping a Conference

If you are a regular user of Twitter and follow folk who go to conferences you might have noticed that they have a tendency to highlight points they feel are noteworthy. It is a real change from just taking notes and communicating them to people upon return, now you can follow a conference even if you are not there. The information arriving in your Twitter stream though is raw, unsorted and delivered unevenly over time. What if you wanted to make some sense of it? One of my favourite tools to help with such a challenge is FreeMind, which is open source software that deals with mind maps. The only problem is getting the information out of Twitter and in to FreeMind. So why FreeMind and not another mind mapping tool? In the end it comes down to preference, you might just be able to use another tool you prefer here, it might even make your life easier. The reason I picked FreeMind though, is that it is quite an easy way to get into mind mapping techniques, it is fairly easy to use and, rather significantly, it is quite easy to generate files that it can use. FreeMind cannot read directly from Twitter though, so we need something to search Twitter and make those results into a file that FreeMind can use. Once we have the results in a mind map we can delete the ones we don't need and rearrange the others to get a coherent picture of what was said.

There are various ways in which you can extract Twitter search results into a Freemind file. The one I've picked here is to do a command line script in PHP. This will strike many people as an ususual choice for a scripting language, but PHP has come on greatly over the last few years and has become a powerful language that is quite easy to learn. It is mostly used for web applications, but it works very well on the command line, and as it has a lot of built in functionality we can use this to help us. The script works like a bit of glue, bringing the functionality of Twitter search and Freemind together.

To use this script and the method outlined here you will need PHP 5.2, Freemind 0.8.1 (if you are an Ubuntu user you can grab this version from my PPA) and a text editor! The script will do the following things:

Search Twitter for tweets about a provided tag
Remove re-tweets and replies
Build a Freemind XML file containing:
A root node with the search term
Child nodes of the root with the name of the user who made the tweet
Nodes containing the actual tweet attached to the user name

Why remove the re-tweets and replies? This is an attempt to keep the total number of nodes down, and also to filter out some content we might not want. I'm basing this on the assumption that people will tweet sound bites from a conference they find interesting, but a re-tweet will just reproduce the same point, and replies might just be background discussion. You could also modify the script to take into account the number of times a point is re-tweeted to get a feel for how important conference attendees thought it was, another opportunity for sense making.

The first part of the script adds a bit of robustness by making sure that the script has been supplied with a tag to analyse, the command line arguments are passed through the PHP variable $argv, the first element contains the name of the script as it was run, the rest contain any arguments passed. Something that can trip us up if using this script on Linux and Mac systems with the bash shell is if you pass the tag to the script with a hash '#' character at the start, but not in quotes, unfortunately this gets interpreted as a comment so the tag gets ignored! To prevent this from happening make sure you put quotes around the tag.

At around line 17 we start to interact with the Twitter API, specifically the method to perform a search (which is documented at: http://apiwiki.twitter.com/Twitter-Search-API-Method%3A-search). For convenience we're getting the results from Twitter in JSON format, which in PHP 5.2 onwards is very simple to read into an array using the json_decode() function; saving us from having to write a complex routine to parse the results. There are a couple of things to take into account here, firstly, the search may not return all of the results for your search, it will return a maximum number, then if there are any more it will return a link to the next set of results. I've put in a request limit as well, which is useful when experimenting with the script. This limits the number of pages of search that will be processed, there is a maximum limit of one hundred pages of search results anyway.

The filtering out of the replies and re-tweets happens at line 44, so you can or enhance that here; you might for example want to add in some more filtering to get down to the really useful information. The results get loaded into an array and at line 54 the array starts to get converted into a Freemind XML file. On line 61, the root node is built which contains the term you searched for, connected to this root nodes are nodes that contain the user names of people who made tweets at the conference. In the foreach loop on line 76 we attach the tweets they made and a link to the original tweet page.

Once the script has output the Freemind XML file, you can fire up Freemind and open it, you can then rearrange the tweets to make sense of what was said at the conference and construct a story. If it is a very large file on a slow machine you might notice that it can be quite slow to use. In this case you might want to export the file as an SVG image and open it in Mozilla Firefox or another application that understands SVG files. The advantage of this file format is that you can zoom in and out without losing image quality. At the top of the script you will notice a “shebang” line, this tells Unix systems to run the script though a PHP interpreter, so you can run it like this (where #opened09 is being used as a tag):

./php2mm.php “#opened09”

If you are using a system that doesn't understand this sytax you can run it with:

php php2mm.php “#opened09”

Mind mapping can be a great way to sort out thoughts and ideas that might seem a bit uncoordinated, and being able to extract those ideas from a microblogging service could be very useful. There is a lot that could be done with this script to make better sense of the ideas being expressed, or to make better mind maps, but it is just meant to be a starting point to demonstrate the technique. Once thing the script does not attempt to do though is any sort of network analysis, it is more geared towards finding out what was said. Tony Hirst recently wrote a blog post on network analysis and visualising the network around the recent Open Education Conference, his post can be found at: http://ouseful.wordpress.com/2009/08/13/preliminary-thoughts-on-visualising-the-opened09-twitter-network/.

The listing for the script file is below, copy and paste it into a text editor and save it as "tag2mm.php", if you are using a Linux or Mac system don't forget to make the script executable with "chmod u+x tag2mm.php".

#!/usr/bin/env php
<?php
/*
* Twitter tag to FreeMind script
*/

// print out a help message if needed
if (sizeof($argv) == 1 || $argv[0] == '--help') {
echo "Usage ".$argv[0]." TAGNAME\n";
echo "Remember if you put '#' in front of the tag you should enclose it in quotes to stop it being ignored.\n";
exit();
}

$request_limit = 20;
$search = $argv[1];

$search_page = 'http://search.twitter.com/search.json';
// build the URL for the search, this will be updated with the URL for the next page of results
$fetch_url = $search_page.'?q='.urlencode($search);
// build an array with all of the tweets for this tag (there is a limit of 100 pages of search results)
$sorted_tweets = array();
while ($fetch_url != 'stop' && $request_limit != 0 ) {
echo "Processing $fetch_url \n";
// set URL and other appropriate options
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $fetch_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, 0);

// grab URL and pass it to the browser
$results_raw = curl_exec($ch);
// exit this loop if the request was not successful
if (curl_getinfo($ch, CURLINFO_HTTP_CODE) != 200) {
    echo "Error: Could not make request.\n";
}
else {
    $tweets = json_decode($results_raw, true);
    // close cURL resource, and free up system resources
    curl_close($ch);

// sort the results
    foreach ($tweets['results'] as $tweet) {
      // Don't include retweets or replies in our mind map
      if (substr($tweet['text'], 0, 1) != '@' && substr($tweet['text'], 0, 2) != 'RT') {
        $sorted_tweets[$tweet['from_user']][] = $tweet;
      }
    }
    // if there is a next page of results, update the fetch url and run this loop again
    $fetch_url = isset($tweets['next_page']) ? $search_page.$tweets['next_page'] : "stop";
}
// decrease the request limit as we have used up a request
$request_limit--;
}
// Now to start writing the mind map
// make a filename for the mindmap containing only aphanumric characters and space
$filename = preg_replace("/[^a-zA-Z0-9\s]/", "", $search).'.mm';
$handle = fopen($filename, 'w');
// write out the base element
$doc = new SimpleXMLELement('<map version="0.8.1"/>');
// this is the root node of the map, it will display the tag being searched for
$root = $doc->addChild('node');
$root->addAttribute('CREATED', time());
$root->addAttribute('ID', 'Freemind_Link_'.time());
$root->addAttribute('MODIFIED', time());
$root->addAttribute('TEXT', $search);
$tweets = array();
// cycle through results
foreach ($sorted_tweets as $username => $tweets) {
// add a node for who the tweet came from
$user = $root->addChild('node');
$user->addAttribute('CREATED', time());
$user->addAttribute('ID', 'Freemind_Link_'.md5($username));
$user->addAttribute('MODIFIED', time());
$user->addAttribute('TEXT', $username);
$user->addAttribute('LINK', sprintf('http://twitter.com/%s', $username));
foreach($tweets as $tweet) {
    // add a node for what they said
    $child = $user->addChild('node');
    $child->addAttribute('CREATED', strtotime($tweet['created_at']));
    $child->addAttribute('ID', 'Freemind_Link_'.$tweet['id']);
    $child->addAttribute('MODIFIED',strtotime($tweet['created_at']));
    $child->addAttribute('TEXT', $tweet['text']);
    $child->addAttribute('LINK', sprintf('http://twitter.com/%s/status/%s', $tweet['from_user'], $tweet['id']));
}
}
fwrite($handle, $doc->asXML());
fclose($handle);

Open Learning Network

Recent Posts

Follow Us

Subscribe to RSS

Open Learning Network

Recent Posts