For the past 9 years I have been working with mobile phones — particularly with their wireless performance (the ratio of erroneously received bits to the total number of transmitted bits) in a variety of situations. When I have analysed the performance of two phones, the inevitable question has always been ‘which is best?’ After I have carefully explained how one is better under some circumstances while the other is better in other circumstances, the next question has always been, ‘yes, but which one is best . . . overall?’
But how are we supposed to compare?
Should we compare the very best that one phone is capable of to the very best that another is capable of? Or perhaps we should compare the worst to the worst — for which phone does the largest fraction of test scenarios fail completely? Or should we perhaps compare the bulk of the data? In statistical terms the median or the mode? If we want a definitive answer, we can only compare one set of numbers.
I start with this illustration of a professional problem because it is, to my scientifically conditioned mind, very similar to the problem of saying something about the value of source criticism of Tolkien's work.
Based on the argumentation in Tolkien and the Study of His Sources edited by Jason Fisher, it is my impression that many would wish to dismiss source criticism as a legitimate critical approach because Tolkien himself disliked it, but I have never agreed with this argument: there are many points where I would disagree with Tolkien; in some cases I even appreciate Tolkien's views as an integral part of his sub-creation and despite disagreeing with him, I wouldn't have any adaptation that took a different position on it (this was my main complaint against the New Line Cinema adaptation of The Lord of the Rings). If new technology and new machines appear to make my life easier, I don't really care if it is at the cost of a few trees . . ..
There is, of course, the question of purpose: if we wish to do source studies, it is fair to ask why, and what we would wish to achieve by it. Verlyn Flieger, at the panel debate on source criticism at the Return of the Ring conference in Loughborough this August, suggested that the purpose was to understand the mind of the author (an endeavour I find no less daunting than Kristine Larsen's wish to “improve our individual chances of holding our own, if only for a brief moment, in a lively discussion with the Good Professor in whatever version of the Eagle and Child awaits the Second Born beyond the Walls of the World.”) In the panel, the conclusion was that we should do source criticism in order to better understand the “mental landscape” of the author. As landscapes can be understood in various levels of detail and abstraction (just play around for a bit with Google maps), I have no problem with that definition, and I would agree that it is a worthwhile effort: even if you wish to understand the story-internal origin of the Orkish race, you will need to understand Tolkien's mental landscape, and understanding it ever better can only help us in our pursuits to also achieve a better understanding of his sub-creational work.
The main issue that I have had with source criticism of Tolkien's work has been that so much of it has been so very badly conducted. I have, in my earlier post about source criticism, listed a number of the problems I have encountered in such studies and I will not expand on that here. One thing that Jason Fisher's book has done for me has been to open my eyes also to those few excellent to perfect examples of source studies that are also there while the bulk of the source studies are still poor. I've tried to illustrate this with a figure — the majority of source studies of Tolkien's work lie between the abominable and the tolerable (with most being merely poor), but a small fraction are good, excellent or even better. In the top we have a few studies such as Verlyn Flieger's Splintered Light.
The following paragraph is inserted after first publication in response to Jason's insightful comments below:
I should add that this represents my personal impression of “Tolkien Source Criticism” alone. As Jason Fisher points out in comments below, this is likely to also be true of other approaches, or indeed of all approaches (I do believe I could find other approaches for which I would say that the same picture to be true). This is also based on what is, after all, a limited sample (I have not read all Tolkien source studies) which may of course not be representative, and it is based on my personal ideas and preferences. I have targeted source studies specifically in response to reading Jason Fisher's book in which he sets out to improve on Tolkien source studies, not because I find that this approach shows particular problems compared to other approaches.
Looking at this, the big problem is upon what we should base our evaluation of source studies as a method for Tolkien criticism. Should we base it on the fact that a select few can use this approach for sublime results? I would venture that this is probably the result of the people rather than the method. Similarly I would claim that those conducting the abominable studies would probably do so regardless of the method. On the other hand, why should we judge the method on what the average guy can make of it?
When we realise that the quality of the resulting study will depend at least as much on the scholar and the specific topic as it does on the chosen method, why should we expect to be able to say anything general about the method at all? As in so many situations, the only answer is, ‘It depends!’
Jason Fisher's book, and in particular his own contribution to it, ‘Tolkien and Source Criticism: Remarking and Remaking’, gives a set of rules and sets up a standard that may help shift the distribution towards something less bottom-heavy — wouldn't it be wonderful if the bulk of Tolkien source-studies were good rather than poor? Certainly in Fisher's book itself the skew is the other way around and the bulk of the essays are better than tolerable.
While I shall probably still approach source studies with a certain degree of scepticism (so as to avoid disappointment), I will also do so with the small hope that this one might be one of the excellent studies, knowing that such studies do exist and are possibly not quite as rare as I had previously thought.
Oh! I do hope to have more to say about Fisher's book at a later point — this is not meant to be a review, but rather some further thoughts on my personal perception of source studies as a critical approach to Tolkien's work.
 The point where exactly half the results are better and half are worse. Somewhere between ‘Poor’ and ‘Tolerable’ in the figure.
 The point where the most results are concentrated — where the function peaks at ‘Poor’ on the figure.