reviewers discrepancies (sp*)

Discuss our reviews or just talk about any old album.

reviewers discrepancies (sp*)

Postby groundrounder » Mon Jun 09, 2003 9:14 pm

since chris is encouraging discussion of reviews, i just read one that kinda got me intrigued. while in understand how reviews are people opinions, i assume that the reviewers are infact reviewers because of their experiences in a cappella, hence they know their stuff. so how can there be such large discrepancies in the reviews. for instance, the most recent instance is in the bostonians of boston college review of "in the office."
where dave trendler writes "Strong soloists could make up for the lack of energy. Unfortunately, soloists are not a strength of this album. The overall effect is that each song is on pitch, on tempo, well mixed and … well, not terribly exciting. Most soloists are uninspiring. Many of the female voices are sugarcoated and pretty, but most are run-of-the-mill in voice quality and classically trained to the point of stereotype." however, in the previous review by j. sears "Female soloists ... wow. Karleen Green, Stephanie Wolfe, Meaghan Mullholland, and Katie Anson-Chapman are all as solid as they come. I even liked Diane Macedo's lead on My Love Is Your Love in some weird dance-your-butt-off kind of way."
now where as arrangements and mixing can be very felt about very differently due to opinions, shouldn't something as obvious as solo voices be something that are not at extremes. i mean i think every one who has heard the 'bubs can say greg b. is amazing, or that morgan from "off the beat" is headed for fame. how do respected judges of a cappella feel so differently about something so obvious.
as a whole, sears loved the album, trendler didn't.

the same can be said for the reviews on the soloists of Northwestern's purple haze. once again trendler praises them (we love you dave), giving a 5, meanwhile t.czerwinki awards a 2. these are two that stick out in my mind.
not criticizing reviewers. i would love the free cds and getting to listen to so much a cappella, but would hate all the writing and thought behind it. and i do believe that you/they all do generally a fantastic job.

any other feelings?
groundrounder
 
Posts: 62
Joined: Mon Jun 09, 2003 8:55 pm
Location: Chestnut Hill, MA

re:

Postby jonathansears » Tue Jun 10, 2003 7:07 am

Unfortunately, like everything, reviews are <gasp> subjective. After reading through the three Bostonians reviews, here is my take on it:

Regarding soloists, we scored the album like this:

Rebecca: 4
Jonathan: 5
Dave: 3

As you eluded to in your post, it seemed that I enjoyed this album more than Dave did for many various reasons. I think its human nature to lean toward seeing the best in things when you really enjoy it, and the worst in things when you don't care so much for it. With Rebecca scoring the soloists a 4, my reading of this would be that I probably scored the soloists a little higher because I enjoyed the album and Dave probably scored them a little lower because he didnt. All in all, they still come out to a 4.

It's just one of those subjective moments in life. This is why I love having three separate opinions on each album instead of just one. It allows you to help balance out the subjective side of reviewing as much as possible.

One other factor could be that I've reviewed Bostonians albums in the past whereas Dave hasn't. Right away this has us coming at the album from different reference points.

Dave could completely disagree with me on this, and I don't want to speak for him. This is just my opinion.
Jonathan Sears
RARB Reviewer (2000-2004)
Certified ICCA Adjudicator
NCHSA Adjudicator
Lunatic Groove (2002 - 2005)
Elizabethtown Phalanx (1996-2000)
jonathansears
RARB
RARB
 
Posts: 183
Joined: Wed Oct 30, 2002 12:44 pm
Location: Philadelphia, PA

Postby elocomotive » Tue Jun 10, 2003 7:29 am

Music is subjective. What one reviewer finds to be a great quality in a soloist could be annoying or distracting to another. What one reviewer finds to be a voice with character, another could view as poor diction or mechanics. Obviously Bob Dylan was a lousy singer by the standards of a music professor, but a lot of people love his voice. Mechanics are more objective (though still left to interpretation), but passion, style, and character in a solo are all very subjective things, and critical to a good performance.

One of the things I enjoy about RARB IS the difference of opinion. Its interesting to see as a reader and gives you a perspective on some different reactions a buyer might have to the material, though admittedly, few members of your audience are examining it so closely.
elocomotive
 
Posts: 451
Joined: Fri Feb 21, 2003 2:32 pm
Location: Cleveland, OH

Postby mikey » Tue Jun 10, 2003 8:10 am

Elocomotive wrote:One of the things I enjoy about RARB IS the difference of opinion. Its interesting to see as a reader and gives you a perspective on some different reactions a buyer might have to the material, though admittedly, few members of your audience are examining it so closely.


agreed. i understand the reasoning for it, but i was disappointed when they dropped the reviewer number from five to three. the multiple viewpoints make me think about things i wouldn't normally think about when i listen to the reviewed disc.
mikey vu
johns hopkins octopodes
http://www.octopodes.org
mikey
 
Posts: 29
Joined: Tue May 20, 2003 9:29 pm
Location: Baltimore, MD

Postby nosugrefneb » Tue Jun 10, 2003 12:35 pm

I STRONGLY agree with Mikey regarding the number of reviewers for each album. Why was it changed in the first place? Is there a lack of bodies? Does it become too inefficient and difficult organize and 'compile' 5 reviews?

Something that also irks me is that it's all too easy, esp. recently, to score 4s and 5s. It seems almost as if a bad album scores a 3, an average one scores a 4, and anything above average scores a 5. Why is this? Not that I have seen every review that's been posted ever, but I have never seen a 1 and have seen very few 2s. What's the point of having a 5-point scale in the first place? Perhaps it should be a 3-point scale in reality.

This brings me to another point: Since it is in fact a *5*-point scale, I'd like to encourage ALL the reviewers to make use of every point available to them. A 1 should denote piss poorness. A 5 should denote the best possible work. A 3 should be average. In no case should a 5 denote something simply above average, nor should a 3 denote something intolerable. Is that not logical?

I'd actually much prefer 5 (or more) reviewers employing 10-point scale, as it has been in the past. The discrepancy would be reduced with an increased number of reviewers, and perhaps they would feel more liberal in their opinions.

For that matter, why should the group's reputation have ANY influence on the scores? It seems that all too often, the scores are inflated or deflated according to how the reviewer perceives the group or what he/she expects the quality to be. I think a better system would be one in which the reviewer has just a blank CD with track titles to listen to and goes only off that. Sure, there would be no opportunity to specifically praise or scorn individuals, mix engineers, etc. etc., but obviously that's something that could be worked out after the review is finished (maybe by an individual or group of third-party researchers who insert names of soloists, VPists, mix engineers and so on into the completed reviews?).

Anyone agree? Anyone disagree?

If anyone has any insight as to why things involving the scale or number of reviewers have changed from the above (or ever change, for that matter), please let me know.

Ben
nosugrefneb.com
nosugrefneb
 
Posts: 242
Joined: Mon Mar 10, 2003 8:35 am
Location: Chicago IL

Postby mikey » Tue Jun 10, 2003 1:20 pm

Ben Ferguson wrote:I'd actually much prefer 5 (or more) reviewers employing 10-point scale, as it has been in the past. The discrepancy would be reduced with an increased number of reviewers, and perhaps they would feel more liberal in their opinions.


yeah, not that i think the 5 point system is necessarily faulty, but there just seems to be too much quantization. with the older reviews, it looks as if tens and even nines were reserved for over the top songs. i have a deep respect for reviewers' expert opinions, but it does seem that score inflation exists here. maybe this is just a factor of evolving studio production upping the ante overall, but during the 10 point system, even decently good songs on even better cds were getting rather average marks. perhaps giving the option of one decimal point or even half scores would distinguish between that "so close to being perfect, but not quite, 4" and that "very solid effort, deserving better of an average, 4." or would that just make things more confusing? or maybe i should quit analyzing rarb methodology because they've gone over this thousands of times?

damn score inflation... damn grade inflation... damn harvard kids... ;) (just kidding to harvard peoples... kind of)

Ben Ferguson wrote:For that matter, why should the group's reputation have ANY influence on the scores? It seems that all too often, the scores are inflated or deflated according to how the reviewer perceives the group or what he/she expects the quality to be.


a comment i've noticed in recent reviews is to the effect of, "i reviewed [group name's] last cd and this time around, i had expected great things." should a group be expected to innovate and progress? or should each disc be considered a seperate entity regardless, based on an absolute scale? (perhaps it already is and i'm just blowing smoke.)
mikey vu
johns hopkins octopodes
http://www.octopodes.org
mikey
 
Posts: 29
Joined: Tue May 20, 2003 9:29 pm
Location: Baltimore, MD

Postby Tom C » Tue Jun 10, 2003 2:20 pm

Well on the whole I would have to say I tend to reserve 4s and 5s for groups I think are great or exceptional. But I think much of the problem lies in this:

Most of the albums we get are from the good groups. For every album we review, there are probably four or five (or more) at the collegiate level released that we do not review. We do very few high school. The groups that submit are the groups that know about RARB, which usually corresponds with groups that have the know-how to make a good-to-great album. Hence why we end up with a lot of the higher scores.

Other reasons? Who knows. To be honest, it's really hard to give a one, but I do it when things are that bad. It takes guts to knock an album, and its easier to sugarcoat your comments. I have gotten more than a dozen emails from people disgruntled over what I have written, some bordering on hate mail. But I think a much of it is that we get a lot of better-than-average albums.
Thomas Czerwinski
RARB Reviewer
http://www.rarb.org/people/tom-czerwinski.html
Tom C
RARB
RARB
 
Posts: 166
Joined: Tue Nov 12, 2002 7:59 pm
Location: Austin, TX

Postby seth » Tue Jun 10, 2003 3:52 pm

We went from 5 reviewers per album to 3 so we could speed up the process. The limited pool of reviewers means we can only write reviews so fast, and we'd rather write them for more albums than increase the time spent on each one. Adding 40% more reviewers is infeasible for a number of reasons.

The move from a 10 point scale to a 5 point scale is to reflect the realities of scoring. Reviewers want a 10 point scale because they want to be able to express themselves more precisely, and readers want a 10 point scale because they want more information, but the truth is that a 10 point scale in this domain is more misleading than useful. Reviews are too subjective and reviewers are too inconsistent. The only way to change that would be to make the process more formal, and that wouldn't benefit anyone.

A 10 point scale gives more precision, but there's not enough accuracy to support it. It'd be like having a bathroom scale that reported your weight in miligrams but was only accurate to within 1 kilogram. You'd get the extra numbers, but they wouldn't be as meaningful as they seemed. People would focus on slight differences in score that didn't really mean much. Scores exist to give you a rough summary. They'll never tell the whole story. For that you need to read the review.
seth
Site Admin
Site Admin
 
Posts: 667
Joined: Wed Oct 30, 2002 1:56 am
Location: San Francisco, CA

Postby Chris » Tue Jun 10, 2003 4:27 pm

Seth wrote:We went from 5 reviewers per album to 3 so we could speed up the process. The limited pool of reviewers means we can only write reviews so fast, and we'd rather write them for more albums than increase the time spent on each one. Adding 40% more reviewers is infeasible for a number of reasons.


The increased number of albums we can review is substantial, as Seth said, and I don't think we really gain THAT much more to have two more opinions. Wouldn't you rather have our output going into reviewing more albums, rather than having more reviewers review the same album?

When RARB started, we had fewer albums coming in for review, so having 5 on each one didn't slow us down too much. But when all of our reviewers were constantly booked for several months, we thought this would be a good change to make to allow us to get reviews out more timely.

Seth wrote:The move from a 10 point scale to a 5 point scale is to reflect the realities of scoring. <snip> but the truth is that a 10 point scale in this domain is more misleading than useful. Reviews are too subjective and reviewers are too inconsistent. The only way to change that would be to make the process more formal, and that wouldn't benefit anyone.

A 10 point scale gives more precision, but there's not enough accuracy to support it. <snip> Scores exist to give you a rough summary. They'll never tell the whole story. For that you need to read the review.


Seth's reasoning above was our main one; it's the same as the importance of using significant figures in science.

There were a couple of other reasons I wanted the change. One is that, when dealing with base 10 scoring, people come with preconceptions. They equate a 7 with a 70%. They have a feeling for what "70%" means, since they dealt with percentages in school. They often see a 60% as failing, since that would give you a failing grade in high school.

A 5-point scale allows us to define each point, and to define 3 as average in comparison to "the general body of a cappella recordings available". While we tried to define 5 as average on the 10 point scale, it was too easy for people to see this as a 50% failing grade.

And because reviewers also have these preconceptions of a 5 out of 10 being failing, it was challenging to get them to see that as average.

Hope that clears up those two changes a bit. We did think long and hard about them before making them, since it's always easier just to go with the status quo than to make waves.

On the News page (http://www.rarb.org/news.html), there is an old entry about these and other changes that were made at that time.

-Chris
Chris
Site Admin
Site Admin
 
Posts: 345
Joined: Wed Oct 30, 2002 3:44 am
Location: Boston, MA

Postby elocomotive » Thu Jun 12, 2003 8:14 am

I agree with Chris here. I'd much rather see more reviews and faster reviews than increase the number of reviews per album. RARB already gives you more reviews (3) than most review outlets out there (1).

I have one question that I'd love for a RARBer to jump in on. What determines the innovation/creativity score? It seems like this is a category that often scores lower than others. Is it arranging? Writing? Originality? The only review I've seen break that trend this year is the Moosebutter review, which acknowledged the group's creativity, even if the reviewers appreciated other aspects of the CD less so.

Here is what I don't get: I've heard a few tracks off of Code Red, and the production is incredible and its very enjoyable to listen to. For the 100s upon 100s of groups doing contemporary college a cappella, it is certainly among the best albums I've ever heard. Its production should get a 6.0, but why does innovation get a 5.0? I've seen this many times. Groups doing covers (and even common covers) exceling in the innovation category. It may be interesting arranagements making this score, but can that alone make a 5? And why does a group like Duwende, perhaps the only semi-pro group around to record all originals on their CD, get only a 4.3 on innovation/creativity (despite a 4.7 overall) by comparison? Are authors of original material being fairly compared to the levels of creativity among their a cappella peers, or is their original work judged on its content in ways that groups doing covers are not?

So, I'm not asking if someone can explain Duwende or the 'Bubs CD scores (and not trying to exalt or put down those groups), just wanted to know what makes the innovation/creativity score and provide an example of why its confusing to me. Cause I don't get it. Seems like groups pushing the envelope with their own material should score a little better in this category than groups doing what 100s of other groups have done before them. If RARB provides buying advice and I am a buyer who values innovation and creativity, I feel I might be led astray by the scores in this category.
elocomotive
 
Posts: 451
Joined: Fri Feb 21, 2003 2:32 pm
Location: Cleveland, OH

the same pick a reviewer....

Postby tekay » Thu Jun 12, 2003 8:56 am

Elocomotive wrote: If RARB provides buying advice and I am a buyer who values innovation and creativity, I feel I might be led astray by the scores in this category.


I can only answer for this question without sounding like an A$$ and hopefully, not with this one as well. Elo, hahah, I'd say that you'd need to pick a reviewer that you like and solely follow his or her single scores rather than the average. No reviewer is going to approach a CD exactly the same each time. Oftimes, I do appreciate a better constructed and executed cover over some schlocky original. Other times, I do give the benefit to strong originals. Criticism is not an exact science, and especially since the reviewers don't collaborate on the scoring, we're sometimes just as shocked at the final scoring as you may be.

[color=#8040BF]http://www.rarb.org/people/thomas-king.html http://www.deltacappella.com CASA Dir. of Ambassador Program SoJam Producer & Concert Mgr Sing Producer CAL jd All Things A Cappella FOTS #1 ICCA Producer Emeritus "the

tekay
RARB
RARB
 
Posts: 358
Joined: Mon Nov 04, 2002 8:34 pm
Location: Memphis, TN

Postby Tom C » Thu Jun 12, 2003 12:05 pm

To be perfectly honest, I've never given much though to original covers in the innovation/creativity. I look at it this way (and maybe its wrong, but what I've been doing)

If a group is doing originals, I compare it to the originals out there in a cappella: is it good writing? How is the production? Is it borrowed or copied from other groups that do originals?

When a group does covers, I compare it to other groups that do covers: how did you imitate the original song? How is the production? Do they do anything new that caught my ear?

If a group does originals, it doesn't make their innovation/creativity score any higher than if they do covers: you can be tremendously original and creative in the way that you cover a song (just see the UAlbany Earth Tones' version of "Smells Like Teen Spirit")

I would have given the Bubs a 5 in creativity/innovation because I think they way they capture the originals in exactness is done in a very innovative way.
Thomas Czerwinski
RARB Reviewer
http://www.rarb.org/people/tom-czerwinski.html
Tom C
RARB
RARB
 
Posts: 166
Joined: Tue Nov 12, 2002 7:59 pm
Location: Austin, TX

Thanks

Postby elocomotive » Fri Jun 13, 2003 9:41 am

Thanks for the responses, guys. Tom, your clarification on the questions you ask yourself was nice to see. I guess I was kind of wondering if RARB itself provides a sort of RUBRIC to its reviewers for what factors into the categories. The RUBRIC might say:

Creativity/Innovation
1-Songs that have been covered repeatedly before, little to no interpretation in the arrangement
(skip ahead)
5-Covers chosen that have not been done often or at all, significant interpretation in arrangements, creative original material appears on the album

I understand TeKay's point about subjectivity, but from reading reviews over the last couple years, I don't really understand the parameters which that subjectivity exists within. If you have ever arranged or written material, you know straight away the latter is much harder. Its like a paint-by-number compared to a blank canvas. You can still paint the duck blue if you want or go outside the lines, but somebody's already laid out those lines for you, which is much easier than drawing them yourself.

I completely agree with Tekay that hearing a really well done cover you've heard done by five other groups can still be more enjoyable than hearing an original song you just don't care for. But it should still get props for creativity/innovation (maybe not for repeat listenability or overall score) over the large body of covers that RARB reviews. The Moosebutter review is a great example of this point, but its seems the exception rather than the rule.
elocomotive
 
Posts: 451
Joined: Fri Feb 21, 2003 2:32 pm
Location: Cleveland, OH

Re: Thanks

Postby Chris » Fri Jun 13, 2003 1:36 pm

Elocomotive wrote:I guess I was kind of wondering if RARB itself provides a sort of RUBRIC to its reviewers for what factors into the categories. The RUBRIC might say:

Creativity/Innovation
1-Songs that have been covered repeatedly before, little to no interpretation in the arrangement
(skip ahead)
5-Covers chosen that have not been done often or at all, significant interpretation in arrangements, creative original material appears on the album
.


Thanks for the thoughtful question. The scoring guidelines we've had for the reviewers ever since switching to the current format is posted at:
http://www.rarb.org/scores.html

It give a general idea of what we're looking for in innovation/creativity, and gives a general idea of what each score means for songs and categories in general, but doesn't nail down what each score means for each category.

I'm betting that, if the reviewers were polled, they would have different opinions about what they consider innovative and creative, and would have a hard time scoring according to rules that don't fit their descriptions. Hopefully, you get a flavor for those traits of an album when reading the text of the review.

Keep in mind that the scores are an imperfect quick guide. They don't really hold up well when scrutinized, but I think they are helpful enough as a quick guide to keep them around.

-Chris
Chris
Site Admin
Site Admin
 
Posts: 345
Joined: Wed Oct 30, 2002 3:44 am
Location: Boston, MA

Postby elocomotive » Mon Jun 16, 2003 6:02 am

Cool, thanks for the information, Chris.

You make an excellent point about not letting the scores trump the reviews themselves.
elocomotive
 
Posts: 451
Joined: Fri Feb 21, 2003 2:32 pm
Location: Cleveland, OH

Next

Return to zzCommunity Reviews

Who is online

Users browsing this forum: No registered users and 1 guest

cron