Evaluating Digital Scholarship, or Confessions of an External Reviewer

Jason Mittell's picture

At the Society for Cinema & Media Studies conference in Seattle, I am part of a workshop on “Making Digital Scholarship Count,” where we are discussing how to frame digital projects for hiring, tenure, and promotion. One of the points that I am making is that external reviewers in the tenure process are important figures in framing a candidate’s digital work as part of a scholar’s portfolio. But until you are asked to do these reviews, you never see examples of the form, and once you do write them, they disappear into a locked black box of internal use only.

So I have decided to share some of this hidden genre of writing, offering some quotations from external letters that I have written about tenure and promotion candidates’ digital work. I have obscured the identity of the scholars, and I believe that in each case the promotion case was successful. I hope these examples provide models and frames of reference for other reviewers, for junior scholars putting together dossiers to frame their own work, and for T&P committees to see the type of benefits for doing digital work within media studies (or other fields). I’ve sequenced the excerpts in order of prominence of the digital within the portfolios, from lesser to greater.

This excerpt is about an assistant professor who was a blogger:

I do also want to mention X’s work on his blog, XXX. The rise of blogging within media studies, especially among untenured faculty talking about their research, pedagogy, and critical response to contemporary culture, has been a key development over the past few years, and X has been an active participant in that community. While he is not the most frequent poster, his comments are invariably thoughtful and detailed, and I believe his blog has helped him establish a solid reputation within media studies as an emerging scholar. While self-published commentary is not “tenurable” work per se, I do believe it is part a broader part of a scholar’s commitment to disseminating knowledge and promoting critical engagement with culture, and as such should be commended and encouraged.

This next candidate was an even more active blogger whose online writings helped establish her academic identity and reputation:

In addition to her more traditional scholarship, I believe it is important to consider X’s online writing in assessing her place in the field. Her blog is written in a casual and humorous tone, and certainly makes no claims to be formally academic. However, the quality of her insights into film and other aspects of popular culture shine through the humorous tone. In fact, reading her blog before seeing any of X’s scholarship made me want to seek out her formal publications and read what she contributed to sites like Flow, as I was impressed with the depth of thinking and quality of insight within blog posts. Additionally, I firmly believe that within the contemporary media environment, the best way for a scholar to boost their reputation is to maintain a vibrant online presence, putting their work and ideas out there for people to discover and engage with. X does this in a way that allows her to explore various writing voices and get feedback for ideas in progress, as well as developing scholarly networks across subfields and different regions. I have no doubt that without her active online writing, X would have a less significant reputation and it would take longer for readers to discover the value of her formal scholarly publications.

This next excerpt comes from a candidate who ran a significant scholarly website—I was specifically asked only to review her online materials (not her formal published scholarship), which I took as a good sign that the committee was trying to evaluate her work on its own terms, not treating the digital work as just a “side bonus project”:

In looking at X’s work on XXX, it is tempting to try to compare the website to a more traditional form of scholarly communication. I could certainly highlight how it resembles a well-regarded journal (with an excellent editorial board and regular publication of scholarship that has been edited and reviewed), or a scholarly conference (with ongoing conversations in response to authors presenting academic material). However, I think XXX is best understood as a scholarly community, hosting multiple platforms of publication and conversation, and fostering a growing population of contributors and readers around shared interests and expertise. The efforts to develop and maintain such a community, while ensuring high-quality content and active participation, are enormously difficult, and thus she must be commended for the consistent quality of the site’s content and active level of participation of its readers and authors. It is a unique resource within the broader field of media studies, and one that I would like to see other subfields emulate… Within media studies, scholarly communication is increasingly conducted digitally and XXX is a model for how to create a high-quality, rigorous, and academically validated venue for online scholarly discourse.

I would like to single out one of X’s posts as indicative of the strengths of both her scholarship and the site that she edits.  The essay itself is an engagingly written, effectively illustrated piece that opens up numerous questions through a close analysis of [topic]—it is not a full-fledged research article, but a 2,000 word piece of critical writing that is poised to engage a broad array of readers better than a longer research article. The 17 comments on the essay highlight the nature of open peer review better than nearly any example of online scholarship I have seen—commenters range from highly supportive to skeptical to hostile, with some referencing other source material and others engaging in constructive debate. Given that [the topic] frequently raises the hackles of readers resistant to cultural criticism, and [the subject] often prompts defensive posturing from fans, I was not surprised that the discussion got heated; however the heat became productive via constructive dialogue, with X contributing clarifications of her argument and other scholars joining to support her claims. In short, this type of dialogue and debate is what scholarly review should look like when conducted in public, providing a range of perspectives for readers to use in making sense of academic discourse and argumentation.

Finally, this candidate’s dossier was fully immersed in digital projects, both born-digital scholarly pieces and creative digital work:

X has demonstrated a commitment to publishing his work in open venues, whether via open access journals, books with free online versions, or via his blog as public scholarship; I share his belief in such open access publishing, as it reinforces how faculty should make our work serve more than insider academic conversations or acquiring another line on a personal C.V., but actually be read and disseminated across a wider range of readerships. As a member of [his program], such a commitment to open access is particularly important to spread and share the work that comes out of the collaborations between your faculty and students…

His creative works… all use digital techniques to explore new forms of creative expression and critical commentary. Such experimental work is difficult to situate in a tenure dossier, as it lacks peer review and often is not even framed as a scholarly act. But X’s online presence… models an attitude that embraces experimentation and boundary crossing that can produce scholarly insights and critical reflection…. While such experiments might not merit tenure on their own right, I think it is important to include such experimental work as part of X’s scholarly profile and recognize such creative explorations as a vital part of the professional output of a scholar of digital media.

For me, the takeaways of these excerpts is that framing matters, and the job of the external reviewer is to give departments and committees references and analogues that are more familiar, while highlighting what makes digital scholarship unique and important. I always try to imagine what skepticism or confusion might arise, and preemptively explain those elements away without becoming defensive or hostile to skeptics.

I hope these excerpts are useful to people in various capacities – and I encourage discussion in the comments about issues they raise, further questions, or other examples that might help illuminate this highly secretive and anxiety-saturated process.

Filed under: Academia, Conferences, Media Studies, New Media Tagged: digital humanities, digital publishing, tenure

Jason Mittell

Publication date (from feed): 

Sun, 23 Mar 2014 16:14:31 +0000