book

Building Web Reputation Systems

by Bryce Glass, F. Randall Farmer

March 2010

Intermediate to advanced

335 pages

10h 56m

English

O'Reilly Media, Inc.

Read now

Unlock full access

What Is This Book About?Why Write a Book About Reputation?Who Should Read This BookOrganization of This BookPart I: Reputation Defined and IllustratedPart II: Extended Elements and Applied ExamplesPart III: Building Web Reputation SystemsRole-Based Reading (for Those in a Hurry)Conventions Used in This BookSafari® Books OnlineHow to Contact UsAcknowledgmentsFrom RandyFrom Bryce
An Opinionated ConversationPeople Have Reputations, but So Do ThingsReputation Takes Place Within a ContextWe Use Reputation to Make Better DecisionsThe Reputation StatementExplicit: Talk the TalkImplicit: Walk the WalkThe Minimum Reputation StatementReputation Systems Bring Structure to ChaosReputation Systems Deeply Affect Our LivesLocal Reputation: It Takes a VillageGlobal Reputation: Collective IntelligenceFICO: A Study in Global Reputation and Its ChallengesWeb FICO?Reputation on the WebAttention Doesn’t ScaleThere’s a Whole Lotta Crap Out TherePeople Are Good. Basically.Know thy userHonor creators, synthesizers, and consumersThrow the bums outThe Reputation Virtuous CircleWho’s Using Reputation Systems?Challenges in Building Reputation SystemsRelated SubjectsConceptualizing Reputation Systems
The Reputation Statement and Its ComponentsReputation Sources: Who or What Is Making a Claim?Reputation Claims: What Is the Target’s Value to the Source? On What Scale?Reputation Targets: What (or Who) Is the Focus of a Claim?Molecules: Constructing Reputation Models Using Messages and ProcessesMessages and ProcessesReputation Model Explained: Vote to PromoteBuilding on the Simplest ModelComplex Behavior: Containers and Reputation Statements As TargetsSolutions: Mixing Models to Make SystemsFrom Reputation Grammar to…
Extending the Grammar: Building BlocksThe Data: Claim TypesQualitative claim typesText commentsMedia uploadsRelevant external objectsQuantitative claim typesNormalized valueRank valueScalar valueProcesses: Computing ReputationRoll-ups: Counters, accumulators, averages, mixers, and ratiosSimple CounterReversible CounterSimple AccumulatorReversible AccumulatorSimple AverageReversible AverageMixerSimple RatioReversible RatioTransformers: Data normalizationSimple normalization (and weighted transform)Scalar denormalizationExternal data transformRouters: Messages, Decisions, and TerminationCommon decision process patternsSimple TerminatorSimple EvaluatorTerminating EvaluatorMessage SplitterConjoint Message DeliveryInputTypical inputsReputation statements as inputPeriodic inputsOutputReturn valuesSignals: Breaking out of the reputation frameworkLoggingPractitioner’s Tips: Reputation Is TrickyThe Power and Costs of NormalizationLiquidity: You Won’t Get Enough InputBias, Freshness, and DecayRatings bias effectsFirst-mover effectsFreshness and decayImplementer’s NotesMaking Buildings from Blocks
Simple ModelsFavorites and FlagsVote to promoteFavoritesReport abuseThis-or-That VotingRatingsReviewsPointsKarmaParticipation karmaQuality karmaRobust karmaCombining the Simple ModelsUser Reviews with KarmaeBay Seller Feedback KarmaFlickr Interestingness Scores for Content QualityWhen and Why Simple Models FailParty CrashersKeep Your Barn Door Closed (but Expect Peeking)Decay and delayProvide a moving targetReputation from Theory to Practice
Asking the Right QuestionsWhat Are Your Goals?User engagementEstablishing loyaltyCoaxing out shy advertisersImproving content qualityContent Control PatternsWeb 1.0: Staff creates, evaluates, and removesBug report: Staff creates and evaluates, users removeReviews: Staff creates and removes, users evaluateSurveys: Staff creates, users evaluate and removeSubmit-publish: Users create, staff evaluates and removesAgents: Users create and remove, staff evaluatesBasic social media: Users create and evaluate, staff removesThe Full Monty: Users create, evaluate, and removeIncentives for User Participation, Quality, and ModerationPredictably irrationalIncentives and reputationAltruistic or sharing incentivesTit-for-tat and pay-it-forward incentivesFriendship incentivesCrusader, opinionated incentives, and know-it-allCommercial incentivesDirect revenue incentivesIncentives through branding: Professional promotionEgocentric incentivesFulfillment incentivesRecognition incentivesPersonal or private incentives: The quest for masteryConsider Your CommunityWhat are people there to do?Is this a new community? Or an established one?The competitive spectrumBetter Questions
The Objects in Your SystemArchitect, Understand ThyselfSo…what does your application do?Perform an application auditWhat Makes for a Good Reputable Entity?People are interested in itThe decision investment is highThe entity has some intrinsic value worth enhancingThe entity should persist for some length of timeDetermining InputsUser Actions Make Good InputsExplicit claimsImplicit claimsBut Other Types of Inputs Are Important, TooGood InputsEmphasize quality, not simple activityRate the thing, not the personReward firsts, but not repetitionUse the right scale for the jobMatch user expectationsCommon Explicit InputsThe ratings life cycleThe interface design of reputation inputsStars, bars, and letter gradesThe schizophrenic nature of starsDo I like you, or do I “like” like youTwo-state votes (thumb ratings)Vote to promote: Digging, liking, and endorsingUser reviewsCommon Implicit InputsFavorites, forwarding, and adding to a collectionFavoritesForwardingAdding to a collectionGreater disclosureReactions: Comments, photos, and mediaConstraining ScopeContext Is KingLimit Scope: The Rule of EmailApplying Scope to Yahoo! EuroSport Message Board ReputationGenerating Reputation: Selecting the Right MechanismsThe Heart of the Machine: Reputation Does Not Stand AloneCommon Reputation Generation Mechanisms and PatternsGenerating personalization reputationGenerating aggregated community ratingsRanking large target sets (preference orders)Generating participation pointsPoints as currencyGenerating compound community claimsGenerating inferred karmaPractitioner’s Tips: Negative Public KarmaDraw Your Diagram

How to Use a Reputation: Three QuestionsWho Will See a Reputation?To Show or Not to Show?Personal Reputations: For the Owner’s Eyes OnlyPersonal and Public Reputations CombinedPublic Reputations: Widely VisibleCorporate Reputations Are Internal Use Only: Keep Them Hush-hushHow Will You Use Reputation to Modify Your Site’s Output?Reputation FilteringReputation Ranking and SortingReputation DecisionsContent Reputation Is Very Different from KarmaContent ReputationKarmaKarma is complex, built of indirect inputsKarma calculations are often opaqueDisplay karma sparinglyKarma caveatsReputation Display FormatsReputation Display PatternsNormalized Score to PercentagePoints and AccumulatorsStatistical EvidenceLevelsNumbered levelsNamed levelsRanked ListsLeaderboard rankingTop-X rankingPractitioner’s TipsLeaderboards Considered HarmfulWhat do you measure?Whatever you do measure will be taken way too seriouslyIf it looks like a leaderboard and quacks like a leaderboard…Leaderboards are powerful and capriciousWho benefits?Going Beyond Displaying Reputation
Up with the GoodRank-Order Items in Lists and Search ResultsContent ShowcasesThe human touchDown with the BadConfigurable Quality ThresholdsExpressing DissatisfactionOut with the UglyReporting AbuseWho watches the watchers?Teach Your Users How to FishInferred Reputation for Content SubmissionsJust-in-time reputation calculationA Private ConversationCourse-Correcting FeedbackReputation Is IdentityOn the User ProfileMy AffiliationsMy HistoryMy AchievementsAt the Point of AttributionTo Differentiate Within ListingsPutting It All Together
Integrating with Your ApplicationImplementing Your Reputation ModelRigging InputsApplied OutputsBeware Feedback Loops!Plan for ChangeTesting Your SystemBench Testing Reputation ModelsEnvironmental (Alpha) Testing Reputation ModelsPredeployment (Beta) Testing Reputation ModelsPerformance: Testing scaleConfidence: Testing computation accuracyApplication optimization: Measuring use patternsFeedback: Evaluating customer’s satisfactionValue: Measuring ROITuning Your SystemTuning for ROI: MetricsModel tuningApplication tuningTuning for BehaviorEmergent effects and emergent defectsDefending against emergent defectsKeep great reputations scarceTuning for the FutureLearning by Example
What Is Yahoo! Answers?A Marketplace for Questions and Yahoo! AnswersAttack of the TrollsTime was a factorLocation, location, locationBuilt with ReputationAvengers Assemble!Initial Project PlanningSetting GoalsCutting costsCleaning up the neighborhoodWho Controls the Content?IncentivesThe High-Level Project ModelObjects, Inputs, Scope, and MechanismThe ObjectsLimiting ScopeAn Evolving ModelIteration 1: Abuse reportingInputsMechanism and diagramAnalysisIteration 2: Karma for abuse reportersInputsMechanism and diagramAnalysisIteration 3: Karma for authorsInputsMechanism and diagramAnalysisFinal design: Adding inferred karmaInputsMechanism and diagramAnalysisDisplaying ReputationWho Will See the Reputation?How Will the Reputation Be Used to Modify Your Site’s Output?Is This Reputation for a Content Item or a Person?Using Reputation: The…UglyApplication Integration, Testing, and TuningApplication IntegrationTesting Is Harder Than You ThinkLessons in Tuning: Users Protecting Their PowerDeployment and ResultsOperational and Community AdjustmentsAdieu
Reputation Framework RequirementsCalculations: Static Versus DynamicStatic: Performance, performance, performanceDynamic: Reputation within social networksScale: Large Versus SmallReliability: Transactional Versus Best-EffortModel Complexity: Complex Versus SimpleData Portability: Shared Versus IntegratedOptimistic Messaging Versus Request-ReplyPerformance at scaleFramework DesignsThe Invisible Reputation Framework: Fast, Cheap, and Out of ControlRequirementsImplementation detailsLessons learnedThe Yahoo! Reputation Platform: Shared, Reliable Reputation at ScaleYahoo! requirementsYahoo! implementation detailsHigh-level architectureMessaging dispatcherModel execution engineExternal signaling interfaceReputation repositoryReputation query interfaceYahoo! lessons learnedYour Mileage May VaryRecommendations for All Reputation Frameworks
Further ReadingRecommender SystemsSocial IncentivesPatents

Content preview from Building Web Reputation Systems

Preface

What Is This Book About?

Today’s Web is the product of over a billion hands and minds. Around the clock and around the globe, people are pumping out contributions small and large: full-length features on Vimeo, video shorts on YouTube, comments on Blogger, discussions on Yahoo! Groups, and tagged-and-titled Del.icio.us bookmarks. User-generated content and robust crowd participation have become the hallmarks of Web 2.0.

But the booming popularity of social media has brought with it a whole new set of challenges for those who create websites and online communities, as well as for the users who consume those sites:

Problems of scale (how to manage—and present—an overwhelming inflow of user contributions)
Problems of quality (how to tell the good stuff from the bad)
Problems of engagement (how to reward contributors in a way that keeps them coming back)
Problems of moderation (how to “stamp out” the worst stuff quickly and efficiently)

Reputation systems can provide an effective solution to all of these problems.

What is reputation in an online community? In its broadest sense, reputation is information used to make a value judgment about an object or a person. There are potentially many components to reputation. For example, karma is a reputation score for a user in a community, and it may be an aggregation of answers to the following: ...