I recently submitted my chapter in ICA Handbook of Computational Communication Research to the editors recently. Among the wide range of topics covered in the volume, my chapter, Time-dynamic Analysis, explains how to analyze textual data collected from over an extended period: In communication research, scholars often analyze news articles, speech transcripts or social media […]
Latent Semantic Scale based on Word2vec
Latent Semantic Scaling (LSS) has been used in many research projects to analyze polarity of documents. LSS is useful in research because it assigns polarity scores (e.g., sentiment) to documents based on user-provided seed words. I was trying to further improvement of the technique but it appeared to be difficult because of its algorithm is […]
Align word vectors of multiple Word2vec models
I have been developing a new R package called wordvector since last year. I started it as a fork of the Word2vec package but made several important changes to make it fully compatible with quanteda. By training the Word2vec model on quanteda’s tokens object using the package, it becomes easier to use the trained models […]
AI products and text analysis methods
A few days ago, I received an email from a researcher asking if text analysis is becoming irrelevant because of artificial intelligence (AI). I replied to her briefly saying that text analysis methods and AI products serve different purposes, but I thought I need to write more to answer this question. In fact, I had […]
A new topic model for analyzing imbalanced corpora
I have been developing and testing a new topic model called Distributed Asymmetric Allocation (DAA) because latent Dirichlet allocation (LDA) takes a long time to fit to a large corpus, but does not always discover topics that I am interested in. I know that these are also problems for many other users, so I decided […]
Automatically adjusting alpha for small and large topics
Since the release of the seededlda package v1.0 which implements the distributed LDA algorithm, I have been applying topic models on many different corpora. In doing so, I became increasingly aware that it is important to optimize the Dirichlet prior the document-topic distribution, alpha, to identify topics that are interesting for social scientists. Alpha, as […]
Develop efficient custom functions using quanteda v4.0
The most important change in quanteda v4.0 is the creation of the external pointer-based tokens object, called tokens_xptr, that allows us to write efficient custom functions. In earlier versions of the package, modification of tokens using tokens_*() required long execution time and large memory space because they transfer data between R and C++. Such inefficiency […]
New tokens object in quanteda v4.0
I am very happy that we could release quanteda v4.0 after more than a year of development. For this release, I improved the package’s efficiency by creating a new tokens object and writing many internal functions in C++ to allow users to process millions of documents on a laptop (or tens of millions on a […]
Perform context-specific operations on tokens using a new argument
The bag-of-words approach is common in text analysis, but it has problems in distinguishing between meanings of words that depend on their contexts. In order to address this issue, we added an argument that allows users to select, remove or lookup tokens that occur in specific contexts in the new version of quanteda (v4.0). We […]
Group polarity words with different colors in LSS
We should always ensure that we are measuring what we want to measure in text analysis. In Latent Semantic Scaling (LSS), we can asses the validity of measurement by inspecting polarity scores of words using LSX::textplot_terms(). This function automatically selects words with very high or low polarity scores and highlights them. We can confirm that […]