Why I think the Plos Commenting System kinda sucks
I like PLOS. I really do. Look I’ve published two articles with them. I fully believe that they represent a big stepping stone to the future of science publishing.
Yet, oh yet, I feel like their attempt at building community is kinda shoddy at the moment. The heart of any social media website (reddit, slashdot, 4chan etc.) is the commenting and rating system. The commenting and rating system of the PLOS website is really poor, and I can’t see how it will gain traction given the usability failings.
- Log-in is too complicated. You actually have to go to a separate page. The rest of the site uses javascript, why can’t the log-in? Most major sites just have some fields on that you fill in. From usability studies, the biggest obstacles are the simplest ones at the beginning, such as extra clicks for log-in.
- The ranking of articles, though admirable, is unusable. I know PLOS wants to be the Amazon of science. But until you have Amazon’s user base, don’t try to have a rating system even more complicated than Amazon. For usability, the rating system should be considerably easier that Amazon, and less like a D&D character sheet. One three star rating, comes to mind, with no categories. This helps you avoid choice paralysis. But really, if you want to really exploit social media, you should have only two choices: like/dislike, and let the aggregation of the users make an article significant or not. That would be truly embracing social media.
- The comments page is horrible. The way the comments are separated from the article suggests that the editors have no real faith in social media, separating the anarchy of commenting from the sober peer-reviewed content. Well the way they’ve got it set up, it’s unlikely they’ll ever get enough people commenting to have any problems whatsoever.
The comments are not integral to the article. There is a hermetic seal between the comments from the article by the fact you need to make two clicks from the article page before you see even one comment. No one will ever discover comments by accident. Discoverability is inversely related to the number of links to the information.
The comment page is a mess. Whoever designed it must have used an off-the-shelf forum commenting system. All sorts of metadata are included making the comment itself the smallest and faintest font on the page. The comments are not actually shown on the first comment page, just the comment headers. If you’re pumping out hundreds of comments a day, this might be acceptable, but since it’s a dribble, it’s ridiculous that there is more screen-space devoted to comment headers and comment meta-data than the comments. In top commenting websites, such as Reddit, you are shown all the comments, and there is no need for headers, which are not only a waste of time but another usability obstacle.
Even worse, if you want to post a message, you need to click again! That’s 3 clicks from the article, virtually burying the feature in web 2.0 terms. Most community websites have a text-entry html box at the bottom of the page, often using javascript to give it a nice interactive experience. Since PLOS uses some nice javascript in other parts of the site, it’s not the technology that’s the failing, but the lack of vision.
In short, it is prohibitively expensive to add comments, and when the comments are posted, they are displayed out of context and difficult to read.
If I may as be bold as to make some suggestions, then
- Add login and password fields to every page. Make it ridiculously easy to enter the system as a participant.
- Change the rating system to either just three stars or like/dislike
- Add a popular article page where articles are actively ranked by a combination of the rating and page views.
- Make an abstract page where the comments are shown directly below the abstract.
- Show the comments like reddit.com – no comment heading, all comments threaded below the abstract, minimal meta-data, so that all you see is beautiful comments.
- Stick a text-entry box at the end of the comments to invite new ones.
So, please PLOS, I think you’re great, but you could be greater still, you might even be the first open-access site that truly embraces social media.
They don’t even embrace the right file formats, requiring submissions in Word.
You can submit in LaTex.
Bosco – how likely do you think it is that at some point the job of managing secondary journal content (ratings, comments) will be done by third parties and not journals themselves?
I’m thinking about services like DISQUS and StackOverflow. They both are showing the power of aggregating and organizing discussions separately from their source.
Even if Plos manages to fix some aspects of its comments/ratings system, the problem remains that Plos is but one of many information resources demanding valuable attention and time. My guess is scientists would much rather have one site they go to for secondary discussion/rating of the literature.
Nobody’s found the right approach yet, but that doesn’t mean it isn’t out there.
I think Rich has a good point, especially about Disqus (which, incidentally, I use for my site). They have more or less “solved” the commenting problem and made it very modular and even made it easy to create forums/comments using your favourite scripting language via their API. They also have a simple up/down rating widget next to each comment.
Given a weekend or three, I think a sufficiently motivated person could set up a PLoS-discuss site that created a page with a title, abstract and Disqus discussion for each paper in PLoS. This would require very little in terms of hosting since each PLoS paper page could be static while Disqus looks after all the dynamic comments.
I did something similar to this (but with DokuWiki instead of Disqus) for ICML 2008.
@Rich & @Mark, all excellent suggestions. I have a feeling that the web guy at PLoS was told to add comments and just picked something convenient but not well thought out. It totally makes sense that PLoS should hook their comment system to a third party. As far as I know, most web-masters just hook in a black-box commenting module to the CMS. So there is hardly any difference in hooking in Disqus as opposed to a module.
It would be awesome if they did what @Mark did with ICML 2008 and slap Disqus right onto their official article page. They would solve a lot of their issues in one fell swoop.
@Rich, stackoverflow.com is pretty awesome!
What do you think of their notes system? Where you just highlight in the middle of an article and can comment on that specific part. I completely agree with you on their general commenting system being a huge burden to contribute to, but I think there might be something in the idea of context-sensitive comments. This idea has been applied elsewhere (ie Django Book ), and I believe that it would work well for scientific articles, but the PLOS implementation still has many of the issues you pointed out with their general commenting mechanism (and more as I find it difficult to discover these notes inside the text).
In terms of the actual software, their using something called Ambra on top of something called Topaz. They invented both of them in house. The main thing I noticed in my short investigation is their attempt at a user story , which doesn’t match at all how I use any journal website. It’s quite a weird story.
Really good post, Bosco. I especially like your suggestion for like/dislike: make it as simple as possible. Makes me think of FriendFeed. Might I amend your first suggestion by allowing OpenID login? That way people don’t even need to create a separate account either.
BTW: I like your Manneken Pis. At least something we Belgians can be good at: inciting nonsense on the web :-)
@David, I’m quite impressed with the notes system. However, I would never login and use the service if it’s just to add notes to PLoS articles. I’d just rather not use a service if it’s going to be a fragmentary experience. Would it mean that I need to a different account in each journal to keep notes?
That user story that you dug was quite a surreal experience. It was a very optimistic painting of what a user might do. Unrealistically so. What kind of person logs on to a website, fills out a lengthy survey and uploads a photo before finding something useful on a unknown website? Answer: the staff of the journal. This person just automatically knows how to annotate the article without having it to explain to him. Talk about wishful thinking.
@Jan, Ik spreken nederlands. Neen. Only joking. I worked in Brussels for 2 years. Absolutely loved it. OpenID is a great idea. But I don’t think it’s reached critical mass.
Just want to add one more suggestion to PLoS: there’s no point in adding ratings if it doesn’t have any consequences. If you truly believed in the rating, you would stick it next to the article title in every link, somewhat like how BMC journals have a “highly accessed” tag. Notice how Youtube started sticking the rating in every embedded video. That’s the kind of thing you need to make the ratings both important and discoverable.
@Mark, I was thinking that citeulike.org could easily add a discussion board for every article and serve as the place for scientific discussion. I haven’t created an account, but I do end up there often in article searches. I haven’t used it much because it’s web design is such an eye-sore.
@Bosco, it’s strange they haven’t. They have user blogs, group forums and public notes and reviews for articles. Surely it can’t be much of a leap for them to do it.
I agree about their web interface. It’s not great. It’s a shame they haven’t got an API for their data either. It would make it much easier to mashup the articles in their DB and something like Disqus.
I think the main problem why people do not comment is that comments there are not anonymous. All the rest technical things are important but not crucial.