Voting, Karma and Vaults - Oh my!

[Legacy_Rolo Kipp]

               <getting a little...>

I do have something like that in mind, but it will be transparent to the user, I.e. they can make all three votes bam bam bam, but the latency diminishes vote weight. Likewise, one component of vote weight also degrades with time, making the votes of active members more valuable.

I'd really like to discuss this in more nuts & bolts depth on the private developer forum I'm setting up tomorrow.

For this thread, I'd like to stick to concepts and suggestions/concerns/questions.

I'm really digging it so far :-)

One other possibly controversial point: I really like the idea of incorporating AME reviews for mods. Perhaps providing the equivalent of "editor rating/ user rating" dual system.

<...on the side>
               
               

               
            

[Legacy_FunkySwerve]

               

Rolo Kipp wrote...

<getting a little...>

I do have something like that in mind, but it will be transparent to the user, I.e. they can make all three votes bam bam bam, but the latency diminishes vote weight. Likewise, one component of vote weight also degrades with time, making the votes of active members more valuable.

I'd really like to discuss this in more nuts & bolts depth on the private developer forum I'm setting up tomorrow.

For this thread, I'd like to stick to concepts and suggestions/concerns/questions.

I'm really digging it so far :-)

One other possibly controversial point: I really like the idea of incorporating AME reviews for mods. Perhaps providing the equivalent of "editor rating/ user rating" dual system.

<...on the side>

That sounds excellent on all counts. The transparency, along with the mechanism, makes it clear to the voter what they have to do to vote most effectively - but adds a cost in time and care. Done properly, this should prevent a lot of casual trolls and fanbois from skewing votes in undesirable ways.

One thing I think should be left on these boards, however, are the stated goals of the voting system, even down to somewhat technical stuff - for example, valuing current members more highly in a portion of the weighting. That speaks to a desire for currency, as opposed to being weighted down by the dead hand of the past (or, to put it less prosaicly, a strong respect for precedence). Setting out those goals as you figure them out will promote overall transparency, without bogging the discussion down in technicalities.

As for AME reviews, I don't like the notion of discarding potentially useful information on a piece of content, so I would suggest including them in some way. If I recall correctly, however, they review the work of their own members, so I would be leery of lending them the same 'official' credence as the user voting system, which should be pretty darn accurate by the way it sounds to be shaping up. Maybe just mark content with their reviews in some way, and provide the review or a link to it with the rating and comments, and let the user decide? Self-dealing and evenhanded comparisons mix about as well as oil and water. '> Further, it would be difficult to time-scale those reviews in the same way, if currency is in fact one of the goals of the system, but I'll save elaboration on the technicalities of that for the dev forum.

Funky
               
               

               
            

[Legacy_Pstemarie]

               The AME does review the work of its own members, however, the member whose work is being considered is not part of the review and voting process for their own work. Although not 100% proof against potential bias, this policy is consistent with policies adopted by other awards associations. Just because you're a member of the association shouldn't preclude your work from being reviewed by it.
               
               

               

                     Modifié par Pstemarie, 01 mai 2013 - 09:55 .
                     
                  

            

[Legacy_FunkySwerve]

               

Pstemarie wrote...
. Just because you're a member of the association shouldn't preclude your work from being reviewed by it.

Actually, it should. We've had this discussion before, though, AME and I. I understand that it's not practical for you guys to not look at your own work, as you are among the community's better builders. That doesn't change that fact that you inevitably introduce bias by doing so. The voting system for the new Vault, by contrast, should hopefully set a gold standard for evenhandedness, and it doesn't seem appropriate to me to set your reviews on an equal footing with it.

Please understand that I'm not calling the AME out for doing anything scurrulous. A similar conflict of interest exists for me - or anyone - making suggestions that affect voting outcomes, if they have any of the content being voted on. I'd recuse myself from the discusssion altogether if not for my experience in the field, and the widespread nature of the conflict. As with the AME, practical realities conflict with ideal situations. That's part of the reason I've been focusing on suggestions that will likely harm my content's overall ratings. And, happily, I will not have final say on any of it, as this is Rolo's ball of yarn In theory, though, we should be able to set up a system that is relatively in line with our goals, and absent statistically significant bias.

Funky
               
               

               
            

[Legacy_Rolo Kipp]

               <ducking and dodging...>

Avoiding *that* particular discussion as a consideration taken into account with implementation, My idea with AME ratings is as a side-bar type rating (actually, full on review). That is, bias or not, the AME reviews give a great deal of considered text relevant to the mod in question.

I have good hopes we can work out something multi-faceted enough to provide a(several!)  good sorting mechanisms.

But all that is moot until/if AME participates. Right now it's in my "Druthers" file.

<...weaving and *ouch*!>
               
               

               
            

[Legacy_ffbj]

               I think things could be different.  Certain hard limits like 25 votes, just to me make it a popularity contest.  Not to say those in the Hall of Fame don't deserve to be there, but there maybe some question of how they got there, in that just have numbers doesn't guarantee quality.
Maybe there should be like a candidate list, as in the certain sports Halls of Fame, wherein each month, or quarter, a number of entries would be inducted into the Hall of Fame.  Maybe like Ame has where there are certain categories and a number from each category are eligible for induction within a certain time period.
               
               

               
            

[Legacy_Gruftlord]

               

Rolo Kipp wrote...

<icing his feet surreptitiously...>

Just a statement here: I am still following this thread with interest and still "staying loose" on final design.

However, I am pretty solid now of a multi-metric voting/rating system that will (I hope) incorporate the best of the ideas I'm seeing. A hefty portion of that will be subrosa to reduce "gaming the system" but will include:

• Useage stats for content that is accessed (with a 24hr granularity to reduce spam-updating)
• Download stats
• Comment activity (Hot Projects)
• A karma system with a floating threshold that weights votes according to various inputs.(Edit: influenced by the slashdot moderator system)
• Seed values taken from the V1 (with minimum weight). I'll discuss that if needed, but one of my goals is that no project will be harmed during this migration.

@ Gruftlord: I already have some pretty good search options. I may expand them later with SOLR search, but give it a try :-) (the VPP in my sig)

<...before walking the coals>

• 
  
• ime dependence of voting wight sounds like an interesting approach.sounds good.
  
• 
  
• but what does VPP stand for? i.e.: which of these links should i click to see what you refer to regarding the search features?
  
• and adding to the current discussion: rather than automatically adding a certain community supported review to the side bar, why not give the mod authors the ability to add review banners to the side bar on their own with links to whichever review they like. thereby the new vault wouldn't have to pick specific reviewers as the "official" ones, but rather allow everyone to review mods (in the forum sections, or maybe by introducing a article section) and modauthors would be free to use these as promotion (or not).
  
• of course they could hide bad votes that way, but that just means the reviewers would have to gain credibility on their own; i.e.: people would look for their reviews to see what they have to say about a mod, rather than reliying on the vault to present them with every vote automatically. just like gaming companies only add nice scores to their advertisings, but players know to search their trusted gaming reviewers for the review they trust.
  
• and yes, i realize that i sound more and more like an american liberal :-D
  
• btw: i'm sorry, but i couldn't get rid of these dots

               
               

               

                     Modifié par Gruftlord, 01 mai 2013 - 06:15 .
                     
                  

            

[Legacy_Rolo Kipp]

               <tossing t.l.a.s around...>

LOL, sometimes I forget not everyone lives in my head <a *good* thing>
Eh? Why? <there's too many of us as it is :->
Heh.

VPP is the Vault Preservation Project which is the New Vault or V2 (last link in my sig).  http://neverwintervault.org

<...like pigeon food>
               
               

               
            

[Legacy_Gruftlord]

               thanks for the link. btw: i just edited to above post. i guess i should either increase my writing speed or stop editing my posts :-D

edit: oh i see, an articles section is already there. sweet.
               
               

               

                     Modifié par Gruftlord, 01 mai 2013 - 06:18 .
                     
                  

            

[Legacy_FunkySwerve]

               To remove the list dots

• , click on bbcode and delete them that way.
  
  Funky

               
               

               
            

[Legacy_Pstemarie]

               

FunkySwerve wrote...

Pstemarie wrote...
. Just because you're a member of the association shouldn't preclude your work from being reviewed by it.

Actually, it should. We've had this discussion before, though, AME and I. I understand that it's not practical for you guys to not look at your own work, as you are among the community's better builders. That doesn't change that fact that you inevitably introduce bias by doing so. The voting system for the new Vault, by contrast, should hopefully set a gold standard for evenhandedness, and it doesn't seem appropriate to me to set your reviews on an equal footing with it.

Please understand that I'm not calling the AME out for doing anything scurrulous. A similar conflict of interest exists for me - or anyone - making suggestions that affect voting outcomes, if they have any of the content being voted on. I'd recuse myself from the discusssion altogether if not for my experience in the field, and the widespread nature of the conflict. As with the AME, practical realities conflict with ideal situations. That's part of the reason I've been focusing on suggestions that will likely harm my content's overall ratings. And, happily, I will not have final say on any of it, as this is Rolo's ball of yarn In theory, though, we should be able to set up a system that is relatively in line with our goals, and absent statistically significant bias.

Funky

I'm not a member of the AME and you do make many valid points. I just don't agree that the AME notation has to be absent from the new Vault. I do however feel that the AME should have no weight in terms of voting. Although maybe by having that tag, voting might be swayed...My 'ead 'urts. 
               
               

               
            

[Legacy_FunkySwerve]

               

Pstemarie wrote...

I'm not a member of the AME and you do make many valid points. I just don't agree that the AME notation has to be absent from the new Vault. I do however feel that the AME should have no weight in terms of voting. Although maybe by having that tag, voting might be swayed...My 'ead 'urts. 

Then it seems we're in complete agreement. '> Gruft, not I, was arguing for total exclusion. I think it should definitely be at LEAST linked to on the same page, at a minimum, if not displayed in full. Anything less would break valuable associations between related pieces of information. I just don't want to see an AME rating posted in a way that makes it look equal in importance to the vault2's rating, because it's my hope that the official ratings we wind up with will be as objective as possible.

That said, it seems like any given AME review is going to be more useful in determining whether you want to play a mod than a comment or a vote average. Taking that kind of information off-page seems like a bad idea.

Funky
               
               

               
            

[Legacy_Gruftlord]

               maybe i just expressed myself badly, but truth be told, i'm in complete agreement with your last post (except for the part that says i disagree :-D).

let there be different ratings on the page, but don't make it look like the "official vault2" rating
               
               

               
            

[Legacy_wkbuilder]

               I have only read to page 3 and a half, but this is all very interesting. It will be a sad day to see the OV go the way of the do-do, and I hope it isn't soon. But I'd like to go ahead and leave an opinion as a new-ish HoF author, and long time down loader.

As an author, HoF status did mean "something' to me. Not really sure what it meant, perhaps validation from current users of NWN. It wouldn't matter to me if it was done away with, however I do hope that status is grandfathered over to the new sight. If it's just a gold star or a blue ribbon icon on the module page, that would be good enough. Something that says "this content achieved HoF status on the OV with a score of X". What I would really like to have is the return of Module of the Year, and these type of rewards. Or maybe a system like that could be used to replace Hall of fame. Instead of voting on the individual mod, users could vote on "favorite" mod, and you could do a "mod of the month" featuring the mod that got the most votes during that month on the front page. At the end of the calender year you could do a top ten. This way people like me who want to work towards the goal of a status symbol have something to shoot for.

The problem with the OV scoring system as I see it is that people either vote too high, or they don't vote at all. I feel like or dislike would  be sufficient, or maybe add an "it was OK" option. Kind of like how rotten tomatoes does with movies. "78% of people like this" and so on. As for why people don't vote, my theory is the fact that the votes are public. People are going to shy away from giving negative scores if the person can see their name attached to it, and a flame war can start from there. Or sometimes people may just be too lazy to comment with their vote. Anonymous voting may remedy that problem. Heck it happened to me on the OV I gave a mod a 9/10 and was condescended to and accused of trolling for "down-voting" Because the score was sitting at a high 9.75 or something. Then they went on to accuse me of self promoting so people would play my own mod. I thought a 9 was a good score, and this person acted kind of snotty toward me. I took the higher ground and simply didn't comment further. So yea, anonymous voting is what I would recommend.
               
               

               
            

[Legacy_wkbuilder]

               Additionally: The question came up what should be on the front page?  I think the newest content should always be in the fore front. The newest content is by the current CC makers who deserve the most visibility, and given the incentive to stick around and build more.