After the initial hype about the promises of Big Data for social science research subsumed, scholars are now increasingly thinking seriously about the new possibilities that digital research methods afford. My question is simple: Is there a place for digital methods also in the sociology of finance?
In this post, I address this question by looking at a very narrow area of digital research methods: the possibility of using machine-learning type of techniques (topic modelling) to obtain knowledge from a large corpus of digitised documents (academic articles).
I got interested in topic modelling after reading a paper that used topic modelling techniques to analyse transcripts from the Federal Reserve board meetings (Fligstein, Brundage, and Schultz 2014). At the time, it wasn’t clear to me how I could use topic modelling in my own research, which is a historcal study of actuarial practices, and so I put it on the back burner. Recently, however, after reading Goldstone and Underwood’s (2012) write-up on an analysis of the journal PMLA using topic modelling techniques and that provides some useful tips for how to go about it, I became curious again, and started exploring possibilities for analysing this vast amount of actuarial writing without having to plough my way through each and every single article.
In this post, I describe my experiments with topic modelling using a corpus of 6,906 articles that have been derived from 4 journals: The Journal of the Institute of Actuaries (JIA), The Transactions of the Faculty of Actuaries (TFA), The British Actuarial Journal (which replaced the previous 2 since 1995), and The Journal of Risk and Insurance (JRI – documents were available from 1964-onwards).
Many of the writings of topic modelling discuss the technical aspects of the method; my purpose, however, is to explore some of the possibilities that topic modelling affords; specifically, I seek to reflect on the possibility of using topic modelling in research designs that use qualitative methods such as mine.
What is topic modelling? Simply put, topic modelling is a statistical method that allows researchers to analyse the content of a large corpus of documents without having to read each and every single one of them. Treating a single document as a ‘bag of words’, a topic model tries to infer what a document is about. Topics are collections of words, some of which carry more weight than others, that frequently occur together. A topic, in more formal terms, is a distribution of the frequency in which we expect a word to occur in a given piece of writing about a specific set of topics. The model then estimates for each document the probabilities that its author intended to write about each of the topics contained in the model.
To achieve this, a topic model makes an admittedly unrealistic but fundamental assumption about how authors write. It assumes that before writing anything, authors first decide on which topics they want to write an article and then attach a relative weight to each of these topics. This yields a distribution of topics, which, as noted above, are themselves a distribution of words. To write an article, authors then randomly pick words from the ‘distribution of distributions’ hence derived. At least, that’s the assumption.
This assumption, however strange it may seem, enables a topic model to reconstruct the process of writing to arrive at a predefined number of topics for a given corpus including a distribution of the relative probabilities for each word within a topic and the relative probability for each topic in a document. This backward tracing process is done using a computational technique called Bayesian inference. In plain words, a topic model arrives at a set of topics and their distribution across documents through a (sometimes rather lengthy) process of trial and error. With each iteration of this process, the fitness of the model with the data improves.
The above explanation of topic modelling is based on my own (rather intuitive) understanding. For more technical expositions of how topic modelling works, I recommend reading articles, for instance, by David Blei who is credited with the invention of the method (e.g., Blei and Lafferty 2007; Blei 2012).
Building a Topic Model
There are several possibilities for doing topic modelling. For my own experiment with topic modelling, I used Mallet, a piece of software written by Andrew MacCallum from the University of Massachusetts at Amherst. This application is relatively user friendly and requires only several parameters to be set.
Deciding on the parameters, however, is no easy task. Perhaps the most important parameter is the number of topics that you want to include in your model. This choice is naturally of great importance for the topics that will come out. Select only a few topics, and they will likely be very general, not teaching you anything you didn’t already know. Select too many, and the topics themselves might be overly specific. There are, however, no hard and fast rules for how to determine the optimal number of topics, and experience is a good guide (Zhao et al. 2015). The model I use here contains 30 topics.
Other parameters that need to be set include the number of iterations, and the interval at which Mallet attempts to optimise something called the ‘hyperparameter’. You can read more about these parameters here. For the purposes of this blog it suffices to say that I set the number of iterations at 1,000 and the optimisation interval at 50.
Many of the topics that were returned to me by Mallet immediately made some intuitive sense. The five most prevalent words in Topic 2, for instance, are ‘health’, ‘care’, ‘medical’, ‘insurance’, and ‘costs’, which clearly indicates that the topic is about health insurance. Nevertheless, there were also topics that seemed at first rather ambiguous. Less clear, for instance, was Topic 23 (‘years’, ‘time’, ‘made’, ‘point’, ‘fact’). In these cases, I looked at some of the documents to which the topic model had assigned the highest probability for the topic to see what these articles had in common. In the case of Topic 23, I learned that the articles that scored high for this topic were mostly historical essays and memorial lectures in which the authors looked back at the history of the profession. Hence, I have labelled it as “reflections”. Analysing each of the topics this way, I managed to assign a label to each of these topics.
I assigned a label to them and subdivided them in four ‘types’ of topics: institutional, knowledge domain, language and business type topics.
|institution||Life insurance companies||Topic 4|
|Actuarial profession||Topic 6|
|Computer systems||Topic 22|
|Actuarial journals||Topic 25|
|Life office||Topic 28|
|knowledge domain||Valuation of Business||Topic 1|
|Mortality factors||Topic 7|
|Policy valuation||Topic 10|
|Risk management||Topic 12|
|Options pricing||Topic 16|
|Capital/corporate finance||Topic 20|
|Mortality experience||Topic 21|
|products||Retail, financial services||Topic 0|
|Health insurance||Topic 2|
|Catastrophe insurance||Topic 3|
|Life annuities||Topic 11|
|Automobile insurance||Topic 14|
|Social security||Topic 17|
Reviewing these topics, this is roughly what one could reasonably expect insurance-related journals to be about. Nevertheless, there were a number of surprises. Observing the list, for instance, we can identify two mortality related topics, which, if we simply look at the words contained within those topics, each consist of markedly similar words (topic 7: ‘mortality’, ‘population’, ‘age’, ‘rates’, ‘life’; and topic 21: ‘mortality’, ‘age’, ‘table’, ‘rates’, ‘experience’). Even after close scrutiny, it is not clear to me how these two topics differ. The only clear difference between the topics seems to be that topic 21 spiked in 2009 when there was a special issue in BAJ on mortality (see also figure 3 below).
Another surprising aspect of the model distributions is the conflation of the books/education topic. While it is clear that many of the articles covered with this topic are book reviews, discussions of actuarial education seem to have been grouped under the same denominator. Overall, however, the topics make good sense.
Differences Between British Actuarial Journals and The Journal of Risk and Insurance
Topic modelling allows you to look at several things. First, it allows for the comparison between two different types of documents. Let’s look at the difference between the journals.
Because the three British journals in the corpus (JIA, TFA and BAJ) are all professional actuarial journals (English, Scottish and UK-wide respectively), I have grouped them together as the ‘British Journals’. These journals can then be compared to The Journal of Risk and Insurance, which is an American journal affiliated with the American Risk and Insurance Organisation, an industry organisation unrelated to the American actuarial profession. It is thus reasonable to expect some clear differences.
A first method of comparison is to plot the average distribution of topics for the British journals and the JIR. Because I have only used data from JIR from 1964 onwards, I have left out the documents from JIA and TFA in the preceding years.
The figure indicates some clear differences between the two bodies of articles, most of which are not very surprising. For example, while the topic containing words specific to the JIR is relatively prominent in JIR, the reverse is true for the actuarial journals. This, of course, tells us little we didn’t already know.
The figure shows that articles in JRI tend to talk more about the various types of insurance products, such as ‘health insurance’, ‘catastrophe insurance’, ‘car insurance’ and ‘retail finance’. Moreover, these articles tend to focus on microeconomic issues and are more likely to be about the capital structure of insurance firms. Articles in the British actuarial journals, however, tend to focus more on the various aspects of running a life insurance office, which is indicated by the relative importance of topics such as ‘mortality factors’, ‘indices’, ‘risk management’, ‘investment’, ‘mortality experience’, and ‘life office’. These differences reflect the general concern of JRI with the ‘sales’ side of insurance, while the actuarial journals tend to be more concerned with the financial management’ of life offices.
Another notable difference is that modelling language (‘model’, ‘distribution’, ‘probabilities’, ‘models’, ‘values’) seems, according to my interpretation of the topic model, to be more prominent in the UK actuarial journals, while the language revolving around evidence (‘data’, ‘results’, ‘variables’, ‘model’, ‘table’) is relatively prominent in JRI. But does this point to an epistemological difference between the journals?
Differences Through Time
Topic models also allow to trace the developments of journals through time. We can, for instance, look at what an average issue of a British actuarial journal for each of the years looks like. Figure 2 plots the topic distributions for the British journals in three different years.
Looking at this figure, it becomes immediately clear that the ‘risk management’ topic (‘risk’, ‘management’, ‘capital’, ‘risks’, ‘financial’) has become incredibly important. While virtually a non-existent topic in 1980, it was already a relatively important topic in 1995. In 2011, however, the topic model indicates that almost a third of the journals’ content was about risk management. This makes perfect sense though it is perhaps surprising that it had already become an important topic in 1995. The regulatory landscape for the UK insurance companies has changed significantly since the early 2000s; the role of regulators has become increasingly important, and there are extensive regulations for capital reserving. Moreover, insurance companies have seen the ascendancy of enterprise risk management as a new governance technology. These factors help explain the enormous differences through time for the risk management topic, though the extent of which as indicated by the topic model is perhaps surprising.
Plotting line graphs for the topics over time allows us to compare how the prevalence of the topics developed over time in each of the two sets of journal articles.
Here we see how the average annual probability for each of these topics developed through time for both the British journals, and for the JRI. Some interesting differences come to the fore. The business valuation topic, while having a similar presence in the late 1970s, increased significantly in the British journals since, but did not gain in importance in the JRI. The two mortality topics, while each not important in the JRI, showed different developments; while the mortality experience topic slowly declined, the mortality factors topics remained at a relatively steady level, before peaking significantly in 2009 with the special issue on mortality and longevity. Does this indicate a shift in how actuaries talk about mortality? The investment topic shows a roughly similar pattern in the JRI and the British journals, though its overall importance is higher in the British journals.
Noticeable is also the difference in the developments between the two journals when it comes to the risk management and options pricing topics. These topics are, in light of my interest in the role of financial economics in actuarial science, of particular interest. In the following section, I’ll look at these topics more closely.
Risk Management and Options Pricing
For better comparison, figure 4 plots the development of the ‘risk management’ and ‘options pricing’ topic in the British journals through time.
The figure shows that the ‘risk management’ and ‘options pricing’ topics slowly gained significance in the late 1980s. But while the rise of the options pricing theory peaked in 2004 and decreased again to stabilise at a much lower level (around 0.025), the risk management topic maintained its upward trend and became a hugely important topic in actuarial science (nearing an average probability of 0.3).
Evaluation of Topic Modelling and Potential Uses for the Social Studies of Finance
The analysis of the topic model presented here is of course quite crude. My aim of experimenting with topic modelling, however, was to see whether and how topic modelling could become a useful part of my research design. Others have provided much more sophisticated analysis of the technical aspects of topic modelling and I recommend reading about their experiences before trying out topic modelling yourself.
In the light of my own research, I can see several potential uses for topic modelling. But let me first say something about what I think it is not useful for at the risk of stating the obvious: topic modelling, I think, does not replace getting your hands dirty with an in-depth engagement with (some of) the data. The visualisations of the data that I used in this relatively simple analysis do provide little information that could not have been retrieved by other means, even though they are nice representations of the history of actuarial science. Moreover, as some have argued in relation to digital research methods in the social sciences, aggregation and quantification of enormous heaps of data brings with it the risk of ‘flattening’ the object of study; specificities and, perhaps, crucial though minor deviations may get lost in the process and become obfuscated behind an aura of rigor that comes with computational methods. When making claims using topic models, it thus seems crucial to stay vigilant and point out where nuances could be made were the object of study to be approached with different methods.
Having said that, I also see some potential for using topic models in my own research, which may also be relevant for research in the social sciences and humanities more generally (these points may have been made elsewhere, but I’m not immediately aware of that).
- While much of the findings in topic modelling may not be surprising and even seem superficial, it is, I think, particularly those findings that come across as strange that are interesting. The dissonance between expectations and what comes out of the topic model is not of value in itself, but rather can usefully prompt further engagement with the body of texts. The danger of this, of course, is that valuable information remains hidden within the modelling output that does not seem surprising.
- Topic modelling affords a technique to ‘zoom-in’ and ‘zoom-out’ between aggregated generic trends and specific cases. As some have argued, this capacity to shift zoom is perhaps the most significant characteristic of what might be called ‘big data’ (Latour 2009). Indeed, topic modelling, if treated carefully, allows aggregated trends to be investigated without the necessity to give up the detail of unique and individual specific cases. Research to these ends would seek not so much to make statistically robust claims about the aggregated trends, but would rather add extra depth to the qualitative research conducted by situating descriptions of specific events and instances in those trends and vice versa.
- Topic modelling could be used simply as a search tool that allows you to browse through a large corpus of documents. While the search functions of online databases remain opaque, a topic modelling analysis allows to search through a given set of documents with much more control.
In conclusion and based on this small experiment, I would argue that making valuable scholarly contributions to research in the sociology of finance (and related fields of research) using topic modelling as a standalone method will be extremely difficult. However, I see sufficient potential uses for topic modelling within my own research to keep experimenting with it and refining the types of analysis that I can perform. This would require, however, to approach the topic modelling with a little bit more rigour as I have done here.
Blei, David M. 2012. “Probabilistic Topic Models.” Communications of the ACM 55 (4): 77–84.
Blei, David M., and John D. Lafferty. 2007. “A Correlated Topic Model of Science.” The Annals of Applied Statistics 1 (1): 17–35. doi:10.1214/07-AOAS114.
Fligstein, Neil, John Stuart Brundage, and Michael Schultz. 2014. “Why the Federal Reserve Failed to See the Financial Crisis of 2008: The Role of ‘Macroeconomics’ as a Sense Making and Cultural Frame. IRLE Working Paper #111-14.,” no. 111. doi:10.1017/CBO9781107415324.004.
Goldstone, Andrew, and Ted Underwood. 2012. “What Can Topic Models of PMLA Teach Us about the History of Literary Scholarship?” https://tedunderwood.com/2012/12/14/what-can-topic-models-of-pmla-teach-us-about-the-history-of-literary-scholarship/.
Latour, Bruno. 2009. “Tarde’s Idea of Quantification.” In The Social after Gabriel Tarde: Debates and Assessments, edited by M Candea, 174–164. London: Routledge.
Zhao, Weizhong, James J Chen, Roger Perkins, Zhichao Liu, Weigong Ge, Yijun Ding, and Wen Zou. 2015. “A Heuristic Approach to Determine an Appropriate Number of Topics in Topic Modeling.” BMC Bioinformatics 16 (Suppl 13). BioMed Central Ltd: S8. doi:10.1186/1471-2105-16-S13-S8.