After seeing mass media’s strong response to the extremists’ attack against Charlie Hebdo, I started thinking what can I do for this increasingly important topic? One simple work is making a dictionary containing keywords related to terrorism, so the Terrorism Dictionary 2014 is created. This dictionary is made from newswires submitted by the Associated Press […]
Left-right policy position dictionary
The Latent Semantic Scaling (LSS) not only works well with positive-negative sentiment but with left-right position on economic policy. The seed words for this dimension are {deficit, austerity, unstable, recession, inflation, currency, workforce} for the light and {poor, poverty, free, benefits, prices, money, workers} for the left. Left-right policy position dictionary was created from UK […]
Immigration dictionary
This is probably the final version of my immigration dictionary. This text analysis dictionary was created using technique called the Latent Semantic Scaling, which is based on the Latent Semantic Analysis, from British newspaper corpus. The result of the automated content analysis by this dictionary is strongly corresponds to manual coding by Amazon’s Mechanical Turks […]
Text analysis dictionary on psychology
My automated dictionary creation project is making good progress, and I created a psychology dictionary from a large corpus of UK news on psychology from 1990 to 2011. Scores given to each entry word is interpreted as strength of association to psychology, and the list can be truncated based on the scores. The words are […]
Factor analysis in R and Python
Python has a number of statistical modules that allows us to perform analysis without R, but it is always good idea to compare the outputs of different implementations. I performed factor analysis using Scikit-learn module of Python for my dictionary creation system, but the outputs were completely different from that of R’s factanal function just […]
Testing immigration dictionary
After making some changes in my automated dictionary creation system, I ran a test to validate the word choice for the new immigration dictionary. Latest version contains fewer intuitively negative words with positive scores, unlike the original version. The test was performed by comparing the computer content-analysis with human coding of the 2010 UK manifestos. […]
Text analysis dictionary on immigration policy
Dictionary-based text analysis has a number of good properties, but it is always difficult to make a new dictionary and text analysts often use existing dictionaries that include the General Inquirer dictionaries, which are originally created decades ago, or their derivatives. However, I believe that it is time to create new dictionaries from scratch using […]
News coverage of Ukraine crisis by Russian media
The political unrest in Ukraine became a major political crisis in Europe by the intervention of Russia. Capitalized on the crisis, I performed a quantitative analysis of impartiality of news coverage by Russian news agencies, and wrote a paper, Political influence on news agency coverage of Ukraine crisis . The outcomes of the analysis and […]
Mitigating performance agony of R
R users often complain about the language’s performance. For shorter running time, we should avoid computationally loops, but we sometimes need to use long loops since there is only a few core functions in R for textual analysis. I wish the developers of the R core to address this issue, but we can mitigate the […]
Import UK parliamentary debate data in Python
I tried to import UK parliamentary debates into R, but it seems that Hansard reports are too large for R. R also has very poor in handling different character coding, so I gave up with R and wrote an importer in Python. The Python script imports the XML into MySQL database. #!/usr/bin/python # -*- coding: […]