All of the scripts I've developed for this project, as well as a copy of the Compass corpus, are now here
All of the scripts I've developed for this project, as well as a copy of the Compass corpus, are now here
Up until now, I've been looking at document vectors which had either no knowledge of authorship (the feature vectors) or knowledge of the authors being paired (class vectors). Dr. Kantor and I wonder: what would knowledge of authors in general, but not these particular authors, do to the shared-authorship accuracy?
To test this, I selected 12 of the 24 authors in Shenzhi Li's corpus to hold out for testing. I then created a pair of author id models, one using all 24 authors, and one using only the 12 authors I had not selected. I then used these models to identify the authors, and did the usual analysis on the resulting vectors.
Area-under-the-curve for the pair data generated with the 12-author model fell smack in the middle of the AUC for the features-only and 24-author-model pairs. If this holds up, it would tend to indicate what we can use author id probability vectors to improve shared author analysis, even if we cannot create a model which includes the authors being paired.
After a number of attempts at getting the recursive merge sort to work properly, I tracked down a non-recursive implementation of the algorithm and swapped that in. This took me from sort times of 3+ hours ending in seg faults to sub-two-minute sorts of the 2.7 million records. I suspect that my seg faults were being caused by trying to recurse too deeply and overflowing the stack.
My talk went well, if a bit longer than expected. The slides, in PowerPoint 2003 format, are available here.
I met with Dr. Kantor to go over what I have done so far in my presentation. At his suggestion, I'm going to be adding a number of additional slides, though most of these are going to be graphs rather than textual content. Some Photoshop magic will be required to get the graphs ready to use.
I've settled on Merge Sort for dealing with the data from Shenzhi. This algorithm is O(n log n) in terms of processing time, and requires constant memory when applied to linked lists. I have tracked down a recursive implementation of the algorithm and tested it on the Compass data set with promising results. However, it's suffering segmentation faults after about three hours of crunching on Shenzhi's data, so it will require further debugging.
Started work on the PowerPoint presentation to accompany my talk on Thursday. I'm about eight slides in at the moment, though the two for the introduction to Bayesian Analysis are still blank. It looks like the presentation will get rather longer before it's done, though.
As was the case with the Compass corpus, I am using a 75/25 training/test split on the corpus from Shenzhi. While BMR/BXR can deal with this just fine, my existing scripts are having a lot of trouble with it. There are 2353 documents in the testing set, which gives a total of 2.7 million pairs of vectors. At that volume of data, Perl's memory management forces the use of multiple gigabytes of RAM, which is not possible on my current hardware.
That being the case, I'm rewriting my analysis scripts to be a bit smarter about memory usage, at the cost of slowing the processing rather significantly. I am also starting work on some C++ to do the ROC area under the curve computations because Excel can't deal with 2.7 million rows of data. I need to do some research into sorting algorithms that are efficient on linked lists.
This morning I received an email from Shenzhi with a new corpus attached to it. She sent me feature vectors rather than raw text, so I have no idea what the documents were, but I think the corpus would have been rather difficult to email without passing it through the feature extractor first.
The corpus contains 9,377 documents from 24 authors, which makes is rather substantially larger that my corpus' 130 documents and 16 authors. I shall have to overhaul my analysis and support scripts to deal with the sheer volume of data generated from this new data set.
Meeting with Dr. Kantor, I got a crash course in Bayesian Analysis, which gave me at least a passing understanding of what is going on 'under the hood' in BMR.
We also generated an ROC curve of my initial shared-authorship, and discovered a potentially strong correlation between documen Euclidean distance/dot product and document authorship. The results are promising, but I do not have enough data (yet) to determine if these results are, in fact, meaningful.
Also on the agenda was what I would be discussing in my talk on the 19th. The new results provide me with a topic to discuss, though I still have to actually pull together the presentation.
My tasks for the next few days are to generate more data using the Compass corpus, and pull together a presentation. I should also contact Shenzhi Li about potentially borrowing a different corpus from her to make sure the shared-authorship results I'm seeing on the Compass data are a general thing, rather than a function of particular documents I've been working on.
After supervising test set runs most of the weekend, I've gathered enough data to put together a graph of overall attribution accuracy for a number of dataset.
I also successful assembled a Perl script to take the dot produce and Euclidean distance of sets of document class probabilities. Yes, Perl is sub-optimal for math like this, but I have neither Matlab nor Mathematica on my personal computer at the moment, and don't want to deal with moving data onto and off of one of the DIMACS computers just to do relatively simple math.
Initial analysis shows a correlation between dot product and likelihood that documents are attributed to the same author. Further research will be required to determine if there is a detectable correlation between dot product (or Euclidean distance) and actual shared authorship.
Meeting with my advisor prompted new work in two directions: pretty pictures and document author comparison.
I had worked up a graph of the results from my first ten data sets, but it wasn't terribly useful from either an organization or visualization point of view. In response, I'm going to be generating enough new test sets to put together some meaningful statistics and then graph those. Hopefully this will make it easier to look at the graph and figure out what's going on with the data.
We're also looking to switch from strictly identifying authors to determining if two documents have the same document. BMR/BXR generates an internal vector of probabilities that a document belongs to each of the possible authors. We hope that there will be some kind of detectable correlation between feature vectors for documents with the same author. Initial tests will look at dot products and the Euclidean distance between the vectors.
I've worked through the action items from 28 June, and have arrived at some rather interesting results.
New training/test splits on my inital corpus show much lower accuracy on wordPOS attribution than was acheived on the first test. Subsiquent examination reveils that the perfect results acheived on the first test were in error, as I inadvertantly used the testing set for both training and testing. Retraining on the correct set brings accuracy into line with the results from other splits.
Analyzing short documents using a long-document-trained model shows reduced accuracy when compared with long-document-only testing. However, accuracy improves when short documents are also used in the training set. Short-document-only training and testing actually shows better accuracy than either long-only or mixed, which contradicts results elsewhere indicating that author identification is not as effective on documents around 300 words.
A script has been assembled which is able to determine the highest-weighted features for the authors in a model. This returns some suprising (to me) important works, such as 'like' and 'as', which do not seem, at first glance, to be useful for indentifying authorship.
We received some data from the researchers in Arizona. Unfortunately, the raw data has apparently gone walkabout, so we're limited to the Weka-encoded feature sets they used. I've assembled a Perl script that can convert Weka feature sets to feature sets for either BMR or BXR, but it looks like we'll need to find something else to work on.
Results of analysis of the initial training/test split three different feature sets are summaried below:
In meeting with Dr. Kantor, we decided that the perfect score on the first feature set, combined with the only-slightly-better-than-random results on the other two feature sets probably indicate that some over fitting is going on. (The best possible random strategy results from assigning all articles to Doug, and results in 4 of 21 articles correctly attributed.) At this point, two different routes of investigation are open on this test set:
I should also look into finding a way to determine which features the model has decided are important. This will likely require building a script to parse the model file and match individual feature number to actual feature labels.
Dr. Kantor feels that I have made a good start on the project, having effectively gotten the "machine plugged in and running." Additional work with this corpus will help determine a number of things, including whether article length affects attribution accuracy. (Item 2a, above)
It looks as though we may be receiving feature data from Arizona "by the end of this week." If so, that may give me an additional corpus of data work on.
Using a trio of custom-built Perl scripts (one each for stripping text from Word documents, adding XML markup required by the feature extractor, and constructing a training/test split of the documents) I have successfully assembled a test corpus from the Compass archives. Of the original 413 documents in the archive, 78 were at least 500 words long, and could be readily assigned to an author. Of these, 61 had authors with at least two documents in the corpus. I used these articles to contruct a training set with 40 documents, and a test set with the remaining 21 docs. floor(.75 * |author's doc|) articles were chosen from each author to include in the training set.
I've decided to try and assemble a testing corpus using the archive of articles printed in the Compass last year. There are 413 articles in the archive, though I imagine that many of these will be unsuitable for use, for some reason. Possible reasons for exclusion include insuffient length, list-based content (calendars, etc), and insuffient works by that author (we need at least one article each in the training and testing sets).
With a bit of help from Dr. Lewis, I finally understand the file format required by the feature extractor. Armed with this knowledge, I threw together a training/test split using an email conversation with Doug Anger as my corpus. BBR managed a 50% accuracy rate. I shall need a new (larger) corpus to really try things out.
Dr. Kantor arrived back in New Jersey after two weeks in Israel. This is the first time we met in person. I should start familiarizing myself with the software (BMR/BXR and the feature extractor), which Dr. Kantor gets in touch with researchers in Arizona to see if they can send us a copy of their test data, so we can compare our results to those returned by their Genetic Algorithm approach to feature selection.
CSS is up and running, and everything is looking good.
Development finally gets underway on the project web page. Look for CSS to be implemented tomorrow.
