DHThis: An Experiment in Crowdsourcing Review in the Digital Humanities

Version of Record: Koh, Adeline (2014). DHThis: An Experiment in Crowdsourcing Review in the Digital Humanities. Ada: A Journal of Gender, New Media, and Technology, No.4. doi:10.7264/N3RX99C5

1 Leave a comment on paragraph 1 0 Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 Unported License.

1 Leave a comment on paragraph 1 0 Adeline Koh
Director, DH@Stockton, Assistant Professor of Literature
Richard Stockton College of New Jersey

2 Leave a comment on paragraph 2 0  

3 Leave a comment on paragraph 3 1 Abstract: This essay describes the launch and reactions to the DHThis experiment (DHThis.org), a platform that sources the best digital humanities content from a base of users, rather than from a select group of editors. DHThis was launched in September 2013 and is built on the goals of open access and open engagement. This essay describes the platform functions, the background behind its creation, critiques and counterarguments it has received, and plans for the future.

4 Leave a comment on paragraph 4 0 —————————————

5 Leave a comment on paragraph 5 2 In 2013, a group of friends [1] and I launched DHThis.org, an experiment in crowdsourcing the best work in the digital humanities. DHThis is the first entirely crowdsourced outlet for digital humanities (DH) news. There are no comparable models to DHthis. All existing sites which aggregate DH content still run on an editorial model. [2] DHThis flips this model, shifting control of new developments in DH to wider publics. Using a Slashdot-style system of user engagement, DHThis gives registered users the opportunity to upvote and downvote articles and give karma points that reward active (and useful) participation in the community. We built DHThis on an ethos of open access and open engagement, and to provide an ongoing forum for defining DH in the moment.

6 Leave a comment on paragraph 6 3 DHThis originated in a set of conversations among the team about how knowledge is produced, distributed, and consumed within DH. The “digital humanities” has come under fire in the last five years for its exclusivity [3], despite its attempts to redefine itself under the metaphor of a “big tent” which able to encompass much broader forms of work. Our team decided to create DHThis to answer the research question: is the digital humanities community growing to the extent that it is much more diverse than it used to be? If so, can crowdsourced curation serve as a means to illustrate the diversity present within the “big tent”? We created DHThis to attempt to answer these questions.

7 Leave a comment on paragraph 7 1 We launched DHThis on September 13 2013, using a basic installation of the Pligg CMS. People were invited to register for free and to submit content. Any registered user can vote on the content using a thumbs-up or thumbs-down function. To appear on the front page of DHThis, a submission needs to have 5 thumbs-up votes. Votes are also accorded by “karma”—which users can accrue through actions including submission, up voting and comments—all aspects of functioning well within a community. Users need a minimum of 100 karma points before a post they submit will appear on the front page. Once users register, they are allowed to submit links and add their own descriptions as to why their posts should be upvoted.

8 Leave a comment on paragraph 8 0 The “New” page is where posts first appear after they have been submitted:

9 Leave a comment on paragraph 9 0 Image 1

10 Leave a comment on paragraph 10 0 If a post gets 5 “thumbs up” from other users (also known as up votes), it will then be published on the Home page of the system. Posts that do not accrue 5 or more upvotes in ninety days will not be published on the home page.

11 Leave a comment on paragraph 11 0 Image 2 

12 Leave a comment on paragraph 12 0 DHThis allows users to view which users have upvoted or down voted particular stories, and to leave comments about the post. This becomes visible after a user clicks on a link on #DHThis. Clicking on the “Who Upvoted” tab will reveal a screen similar to the one below, which lists the users who have up voted the story. There will also be a “Who Downvoted” tab if the story has received downvotes.

13 Leave a comment on paragraph 13 0 Image 3

14 Leave a comment on paragraph 14 0 To foster the creation of communities and subcommunities, DHThis also allows users to create their own groups. By creating a group, users can share specific content to a self-selected group of people:

15 Leave a comment on paragraph 15 0 Image 4 

16 Leave a comment on paragraph 16 0 When the project was first launched, the following criticisms and their counter-arguments were made [4]:

17 Leave a comment on paragraph 17 0 Critique 1: Upvoting and downvoting is overly simplistic, and an insufficiently complex metric for the complexities of academic work.

18 Leave a comment on paragraph 18 2 Counterargument: What is the end result of peer review other than a thumbs-up or a thumbs-down? When a peer reviewer, editor or editorial board judges whether your article or book deserves to be published, these decisions also boil down to thumbs-ups or thumbs-downs. One might argue that these decisions come at least with comments for revision, but there is also a comments function within the DHThis site that allows users to exchange commentary on the various links which they have up voted or down voted.

19 Leave a comment on paragraph 19 0  

20 Leave a comment on paragraph 20 0 Critique 2: Is DHThis really just a popularity contest?

21 Leave a comment on paragraph 21 0 Counterargument: Traditional peer-reviewed academic work, unfortunately, is not immune to the popularity contest. Academic publishing is tied with the larger economic market. However, it would be useful to observe whether the content that is highlighted on DHThis actually diverges from what has been showcased in more traditional outlets. This information will only become viable, however, when the DHThis community reaches sufficient critical mass and activity.

22 Leave a comment on paragraph 22 0  

23 Leave a comment on paragraph 23 0 Critique 3. DHThis claims to be original, but the team that started DHNow also began with a crowdsourcing model, and only switched back to an editorial model upon realizing that the selected content was not diverse enough.

24 Leave a comment on paragraph 24 1 Counterargument: We are grateful for the historical precedent that DHNow has set. However, we launched DHThis to test if the digital humanities community has grown to the point where a crowdsourced platform will actually reflect the diversity of this community. Arguably, during the period where DHNow was first launched, the community was less diverse than it is today. However, with the growing popularity of the digital humanities field, and the flowering of multiple digital humanities, the tastes of the community may have changed. We are thus using DHThis as an experiment to see if this hypothesis—that the DH community has grown more diverse—holds true.

25 Leave a comment on paragraph 25 0  

26 Leave a comment on paragraph 26 0 Critique 4. The downvoting function of DHThis is not encouraging to junior scholars, who might feel more embarrassed about their work being critiqued in public. Also, shouldn’t downvoting be anonymous to encourage honesty?

27 Leave a comment on paragraph 27 1 Counterargument: Junior scholars will have to ultimately deal with “downvoting” in the traditional research arena, which means either rejections or revise-and-resubmits. Downvoting is “anonymous” in the sense that one can sign up for a user profile with whatever information one chooses. However, whatever votes or comments a user makes will be tied to their user profile. We employ this adjusted form of anonymity to help mitigate some of the venom that can come from the anonymity within the traditional peer review process.

28 Leave a comment on paragraph 28 0  

29 Leave a comment on paragraph 29 0 Critique 5:  DHThis is supposed to be a place which encourages community building. If so, why does it borrow from the Slashdot and Reddit platforms for a model, given their infamous reputations for trolling and Internet flaming?

30 Leave a comment on paragraph 30 0 Counterargument: Our intentions with DHThis were to experiment with crowdsourcing in the digital humanities. The entire idea is focused around the idea of a crowd rather than a small elite group—and Slashdot and Reddit are some of the existing few models out there for this. As our platform grows we might have to implement some kind of moderation system to work on the possibilities of trolling and flaming. Currently,  however we do not have enough users for this to be an issue (our biggest problem currently is with spam).

31 Leave a comment on paragraph 31 0 In Conclusion

32 Leave a comment on paragraph 32 3 As of January 26 2014, DHThis has currently 208 users. These users have submitted a total of 148 submissions, 45 of which have been published to the front page.

33 Leave a comment on paragraph 33 0 Given these statistics, the DHThis community has not achieved sufficient mass to decide whether the actual research question of its creators—whether the digital humanities has grown and is diverse enough as a community to make content curation through crowdsourcing a viable method for determining value in the field.

34 Leave a comment on paragraph 34 5 We intend to continue on with the current iteration of DHThis for a calendar year, and try and encourage more people to join and participate in the community. The experiment might never be able to show sufficient results if we do not generate enough user mass. But if it does, what happens will be able to tell us a great deal about the changing nature of fields, and the different structures of gatekeeping.

35 Leave a comment on paragraph 35 0  

36 Leave a comment on paragraph 36 0 Footnotes

37 Leave a comment on paragraph 37 0 [1] Collaborators in this project were myself, Martin Eve, Jesse Stommel, Roopika Risam and Alex Gil.

38 Leave a comment on paragraph 38 0 [2] The most comparable outlet to DHThis is Digital Humanities Now, or DHNow (http://digitalhumanitiesnow.org/), published by the Center for History and New Media at George Mason University. DHNow aggregates content from digital humanities feeds, but highlighted content (the “Editor’s Choice” content that appears on main site) are chosen by an editorial group.

39 Leave a comment on paragraph 39 0 [3] See for example: Alexis Lothian and Amanda Phillips, “Can the Digital Humanities Mean Transformative Critique?” Journal of E-Media Studies, 3.1 (2013): n. page. Web. DOI:10.1349/PS1.1938-6060.A.425 <http://journals.dartmouth.edu/cgi-bin/WebObjects/Journals.woa/1/xmlpage/4/article/425>; Adeline Koh and Roopika Risam, “Open Thread: The Digital Humanities as a Historical ‘Refuge’ from Race/Class/Gender/Sexuality/Disability?” Postcolonial Digital Humanities,  10 May 2013, Web. <http://dhpoco.org/blog/2013/05/10/open-thread-the-digital-humanities-as-a-historical-refuge-from-raceclassgendersexualitydisability/>

40 Leave a comment on paragraph 40 0 [4] Some of these critiques and counterarguments took place on Twitter,  but most were expressed to the team orally and were not published in any way.

41 Leave a comment on paragraph 41 0  

42 Leave a comment on paragraph 42 0  

Page 78

Source: http://adareview.fembotcollective.org/ada-issue-4-publication-and-its-discontents/dhthis-an-experiment-in-crowdsourcing-review-in-the-digital-humanities/