Key Clouds for the four-volume Leveson Inquiry Report

And just in case you haven't had time to read all 1,078,416 words in the Leveson Inquiry Report, here are the Key Clouds produced by Wmatrix for all four volumes conflated together. I took the reports available from the Inquiry website (http://www.levesoninquiry.org.uk/), downloaded the PDFs, converted them to text using PDFbox ExtractText tool (http://pdfbox.apache.org/commandlineutilities/ExtractText.html) and then loaded them in to Wmatrix (http://ucrel.lancs.ac.uk/wmatrix/) and compared them to the BNC written sampler.

The statistically most significant key words (i.e. not just the most frequent words) are:

And the statistically most significant key semantic categories are:

All these items are significant, but the larger the font, the more significant they are. If you want to have a look at these texts in Wmatrix, just let me know and I can share the folders with you. You'll then be able to click through the clouds to see each example of the word and tags in context.

Clouds for Leveson Inquiry Executive Summary

If you don't have time to read even the Executive Summary of the Leveson report (26345 words) published today, then here's what Wmatrix shows as the key words compared to the BNC sampler written reference corpus:

And now for the key semantic tags:

Libdem manifesto key concepts

And finally the key concepts for the Libdem manifesto.

Labour key concept cloud

And here's the key concept cloud from the Labour manifesto ... there seems to be much more variety of concepts here and hence the cloud is much bigger. I had to shrink it further to get it all in one screenshot.

Conservative key concept cloud

In addition to key words, Wmatrix can produce key concepts by comparing a frequency list of semantic fields automatically tagged in the data with a reference corpus, again here the BNC written sampler. This shows statistically key concepts in the Conservative manifesto

TEI versions of UK election manifestos

Meanwhile, somewhere deep in France with a laptop, Lou Burnard has created TEI encoded versions of the UK election manifestos, tagged and lemmatised them with TreeTagger. Download from http://ucrel.lancs.ac.uk/wmatrix/ukmanifestos2010/TEIversion/

Thanks Lou!

Updated Libdem manifesto and cloud

Lou Burnard spotted some conversion errors in the Libdem manifesto (extra spaces after ligatures e.g. 'fi') and Martin has now fixed smart quotes to straight ones. The new text version of the Libdem manifesto is at http://ucrel.lancs.ac.uk/wmatrix/ukmanifestos2010/ and here is the updated key word cloud. You'll notice the main difference is that "Britains" is no longer key because it was actually "Britain's" and the apostrophe now being fixed means that it combines its frequency with "Britain".

Libdem manifesto key words

And finally the Liberal Democrat manifesto key word cloud.

Labour key word cloud

And here's the Labour key word cloud ...

Conservative key word cloud

Unlike tag clouds or those produced by Wordle where the size of a word depends on its frequency, the Wmatrix key word clouds show words where their size is related to their statistical keyness i.e. how different their frequency is from what it is expected to be (based on a large reference corpus). Here's the conservative key word cloud.

UK election manifestos 2010

This week I've been reading the UK election manifestos, or rather I've set Wmatrix to read them for me. First, you have to convert the online versions in PDF or HTML into plain text. Saving automatically from Acrobat as plain text leaves unwanted headers and footers and some lost capitalisation. Thanks to Martin Wynne for editing the Libdem and Conservative files. I've edited the Labour manifesto by taking the HTML version from their website and marking the chapter boundaries with a pseudo-XML tag. The edited full plain text versions are available to download at: http://ucrel.lancs.ac.uk/wmatrix/ukmanifestos2010/

Labour's manifesto is 29,508 words long. The Conservative manifesto is 27,562 words and the LibDem one is shorter at 18,433 words.

Next, I loaded the files into Wmatrix and compared them to a general reference corpus for written British English. Key word clouds coming up ...

Leadership and Diffusion in Online Communities

David Huffaker's using text analysis methods such as LIWC and Wmatrix to analyse "leadership and diffusion in online communities by exploring the relationship between communication processes, social structure and language use".

http://www.davehuffaker.com/blog/?page_id=3