Average overall scores and more

All things Recorded A Cappella Review Board.

Average overall scores and more

Postby RnBMrE » Fri Sep 26, 2008 8:39 am

H.F. wrote:At the least, you help bring the medium score closer to 3, which statistically should be a goal I guess (um, Emery? we're waiting...).


Ha, thanks for the prompt! A week or two ago, I took some stats from all reviews implemented since the post-1998-all-database-collected trend began. Here's what I came up with:

Current

Reviewer: Total Reviews (1 star, 2 stars, 3 stars, 4 stars, 5 stars), Total Raw, Average Rating

Haverkate: 18 (0, 0, 3, 11, 4), 73, 4.06
Christie: 228 (2, 18, 51, 79, 78), 897, 3.93
Whitley: 107 (0, 5, 32, 44, 26), 412, 3.85
Landau: 160 (0, 11, 45, 66, 38), 611, 3.82
Stanley: 45 (0, 4, 13, 16, 12), 171, 3.80
Minkoff: 199 (2, 19, 65, 48, 65), 752, 3.78
Dietz: 27 (0, 4, 5, 12, 6), 101, 3.74
King: 111 (2, 8, 34, 47, 20), 408, 3.68
Sailor: 74 (1, 7, 24, 25, 17), 272, 3.68
Trendler: 106 (0, 8, 34, 53, 11), 385, 3.63
Colton: 8 (0, 1, 2, 4, 1), 29, 3.63
DiMartino: 5 (0, 0, 2, 3, 0), 18, 3.60
Kolko: 110 (1, 14, 38, 39, 18), 389, 3.54
Sawyer: 17 (1, 2, 6, 3, 5) 60, 3.53
Soo: 134 (3, 12, 52, 49, 18), 469, 3.50
Czerwinski: 119 (3, 21, 46, 31, 18), 397, 3.34
Bates: 29 (2, 4, 14, 6, 3), 91, 3.14
Danziger: 13 (0, 2, 9, 2, 0), 39, 3.00

Total: 1510 (17, 140, 475, 538, 340), 5574, 3.69

Past

Reviewer: Total Reviews (1 star, 2 stars, 3 stars, 4 stars, 5 stars), Total Raw, Average Rating

Harris: 20 (1, 1, 3, 8, 7), 79, 3.95
Levin: 68 (0, 7, 12, 31, 18), 264, 3.88
Stevens: 123 (1, 11, 33, 46, 32), 466, 3.79
Stotland: 82 (0, 10, 27, 27, 18), 299, 3.65
Dargie: 29 (1, 0, 12, 12, 4), 105, 3.62
Cohen: 79 (1, 9, 31, 21, 17), 281, 3.56
Harris: 54 (0, 6, 21, 20, 7), 190, 3.52
Sears: 73 (1, 10, 23, 30, 9), 255, 3.49
Diamant: 45 (2, 5, 15, 15, 8), 157, 3.49
Skalinder: 31 (0, 6, 13, 4, 8), 107, 3.45
Joyce: 17 (0, 1, 9, 6, 1), 58, 3.41
Manley: 42 (2, 3, 18, 17, 2), 140, 3.33
Ballard: 9 (1, 2, 2, 2, 2), 29, 3.22
Slutsky: 26 (3, 7, 6, 6, 4), 79, 3.04

Total: 698 (13, 78, 225, 245, 137), 2509, 3.59

-------

No agenda here from me really. Certainly lots to talk about, though! Discuss....

Matt Emery CASA Director of Communications Three-time Recipient of RARB "Post of the Year" Title

RnBMrE
 
Posts: 712
Joined: Mon Mar 06, 2006 8:14 pm
Location: Washington, DC

Postby jazzydanziger » Fri Sep 26, 2008 9:07 am

Look at me and my perfect 3!
jazzydanziger
RARB
RARB
 
Posts: 149
Joined: Tue Dec 20, 2005 5:51 pm
Location: St. Louis, MO

Postby nosugrefneb » Fri Sep 26, 2008 9:55 am

Can I get a p-value, please?? :)
nosugrefneb
 
Posts: 242
Joined: Mon Mar 10, 2003 8:35 am
Location: Chicago IL

Postby seth » Fri Sep 26, 2008 10:27 am

I wouldn't expect the average to be exactly 3 due to the selection bias in our sample. (Most people wouldn't submit a really bad album.)

Some years back, I had a funny idea. If you pretend each album has a true score and that the scores given by reviewers are biased estimations of it, and if you assume some simple model for their estimation (say, each reported score is drawn from a normal distribution where each reviewer uses a particular and unchanging bias and variance for all their reviews), then because more than one reviewer is on each album, you could use the EM algorithm to simultaneously determine the model parameters for all the reviewers and the "true" scores for all the albums.

We probably don't have enough data for that many parameters though. You might have to simplify the estimators and use the same variance for everyone.

The assumptions are mostly hogwash, so the results would have little value, but it's still kind of fun to think about if you're into that sort of thing.

Another interesting (and much simpler) thing would be to track the moving average (and variance) for each reviewer. And then you have to ask yourself how much of the drift is due to changes in the reviewer vs changes in the sample population vs measurement error due to the small sample.
seth
Site Admin
Site Admin
 
Posts: 667
Joined: Wed Oct 30, 2002 1:56 am
Location: San Francisco, CA

Postby RnBMrE » Fri Sep 26, 2008 10:32 am

Seth wrote:Some years back, I had a funny idea. If you pretend each album has a true score and that the scores given by reviewers are biased estimations of it, and if you assume some simple model for their estimation (say, each reported score is drawn from a normal distribution where each reviewer uses a particular and unchanging bias and variance for all their reviews), then because more than one reviewer is on each album, you could use the EM algorithm to simultaneously determine the model parameters for all the reviewers and the "true" scores for all the albums.

We probably don't have enough data for that many parameters though. You might have to simplify the estimators and use the same variance for everyone.


PICS OR IT DIDN'T HAPPEN

Matt Emery CASA Director of Communications Three-time Recipient of RARB "Post of the Year" Title

RnBMrE
 
Posts: 712
Joined: Mon Mar 06, 2006 8:14 pm
Location: Washington, DC

Postby vocalmark » Fri Sep 26, 2008 10:56 am

RnBMrE wrote:
Seth wrote:Some years back, I had a funny idea. If you pretend each album has a true score and that the scores given by reviewers are biased estimations of it, and if you assume some simple model for their estimation (say, each reported score is drawn from a normal distribution where each reviewer uses a particular and unchanging bias and variance for all their reviews), then because more than one reviewer is on each album, you could use the EM algorithm to simultaneously determine the model parameters for all the reviewers and the "true" scores for all the albums.

We probably don't have enough data for that many parameters though. You might have to simplify the estimators and use the same variance for everyone.


PICS OR IT DIDN'T HAPPEN


ROFL

Mark Hines The Vocal Company - www.thevocalcompany.com SoJam, 2011 Executive Producer CASA, 2010 Board of Directors

vocalmark
 
Posts: 417
Joined: Sun Apr 03, 2005 9:13 pm
Location: Raleigh-Durham, NC

Postby clyxz » Fri Sep 26, 2008 1:26 pm

Seth wrote:Some years back, I had a funny idea. If you pretend each album has a true score and that the scores given by reviewers are biased estimations of it, and if you assume some simple model for their estimation (say, each reported score is drawn from a normal distribution where each reviewer uses a particular and unchanging bias and variance for all their reviews), then because more than one reviewer is on each album, you could use the EM algorithm to simultaneously determine the model parameters for all the reviewers and the "true" scores for all the albums.


Maybe I've been spending too long in my physics lab, but reading that post made me feel all warm and fuzzy inside.

Oh, cobblers... talking of 'physics lab', I think I left the laser on...
clyxz
 
Posts: 159
Joined: Tue Dec 18, 2007 6:10 am
Location: Oxford, UK

Postby streetnix04 » Fri Sep 26, 2008 2:10 pm

That was very cool, thanks for that! Not sure if I'm bummed or thrilled that my average score was second lowest ever. But wow, with only 26 reviews, I'm tied for the most "1" scores ever?!? I've turned in 10% of the "1" scores?!? Wow. I'm not a cranky old man, I swear! :-)

Of course, it has to be reminded that these numbers have to be taken with a grain of salt, as the actual quality of the albums each reviewer received isn't factored in, as well as if they're pro or collegiate. It would be cool to see those numbers for college vs pro groups.

Very cool nonetheless. Thanks!
-Corey Slutsky
Voices Only Founder
Former Collegiate and Professional A cappella Performer '97-'05
www.voicesonlyacappella.com
streetnix04
RARB
RARB
 
Posts: 377
Joined: Sat Dec 27, 2003 12:12 pm
Location: Murrieta, CA

Postby colton » Fri Sep 26, 2008 7:07 pm

CLyxZ wrote:
Seth wrote:Some years back, I had a funny idea. If you pretend each album has a true score and that the scores given by reviewers are biased estimations of it, and if you assume some simple model for their estimation (say, each reported score is drawn from a normal distribution where each reviewer uses a particular and unchanging bias and variance for all their reviews), then because more than one reviewer is on each album, you could use the EM algorithm to simultaneously determine the model parameters for all the reviewers and the "true" scores for all the albums.


Maybe I've been spending too long in my physics lab, but reading that post made me feel all warm and fuzzy inside.

Oh, cobblers... talking of 'physics lab', I think I left the laser on...


Mmm. Physics lab. :-)
colton
RARB
RARB
 
Posts: 543
Joined: Mon Mar 03, 2008 1:45 pm
Location: Orem, UT

Postby brianhaverkate » Sat Sep 27, 2008 8:12 am

Guess I'm the nicest reviewer of ALL TIME! Peace and love folks....peace and love.
brianhaverkate
RARB
RARB
 
Posts: 419
Joined: Wed Feb 12, 2003 7:56 am
Location: Detroit, Michigan

Postby bstevens » Sat Sep 27, 2008 12:04 pm

I'm glad to see -- in this thread and others -- such an energetic and articulate discussion of the review process! That's precisely what the Forum is for. (In addition to announcing a cappella events, w00ting about things, and boasting about one's number of posts.)

But. Ahem. Let me take this moment to remind all of our energetic, articulate readers that RARB accepts applications for reviewers on a rolling basis. If you'd be interested in putting your wagging tongue to regular use, drop me a line and I'll let you know how to apply.

Benjamin Stevens

CASA Director of Education

Educational Officer for Festivals and Events

bstevens
Site Admin
Site Admin
 
Posts: 636
Joined: Wed Oct 30, 2002 3:19 pm
Location: Mid-Hudson Valley, NY

Postby billhare » Sun Sep 28, 2008 9:07 am

bstevens wrote:That's precisely what the Forum is for. (In addition to announcing a cappella events, w00ting about things, and boasting about one's number of posts.)


Really? Is that something to boast about?? I would think that the person with the most posts on this board should get a life.

My name is Bill Hare, and I approve of this post (which happens to be #1,430)

-B

Bill Hare Some dude who records and mixes people who can't play instruments. http://www.dyz.com

billhare
 
Posts: 2002
Joined: Thu Feb 13, 2003 11:14 am
Location: Silicon Valley, CA

Postby clyxz » Mon Sep 29, 2008 1:01 am

I used to post on this other discussion board, and one day one of the more prolific posters persuaded a moderater to change his post count to 10,000. Of course we all thought it was hilarious.

Fast forward a couple of years, and a friend did reach 10,000 posts. And they were actual genuine posts.

So be careful what you joke about; it's a slippery slope.
clyxz
 
Posts: 159
Joined: Tue Dec 18, 2007 6:10 am
Location: Oxford, UK


Return to RARB

Who is online

Users browsing this forum: No registered users and 1 guest

cron