View PDF ↗
PDF Viewer
Open Full PDF

BUILDER'S SANDBOX

Build This Paper

Use an AI coding agent to implement this research.

OpenAI Codex
OpenAI CodexAI Agent

Lightweight coding agent in your terminal.

Claude Code
Claude CodeAI Agent

Agentic coding tool for terminal workflows.

AntiGravity IDE
AntiGravity IDEScaffolding

AI agent mindset installer and workflow scaffolder.

Cursor
CursorIDE

AI-first code editor built on VS Code.

VS Code
VS CodeIDE

Free, open-source editor by Microsoft.

Estimated $9K - $13K over 6-10 weeks.

See exactly what it costs to build this -- with 3 comparable funded startups.

7-day free trial. Cancel anytime.

Discover the researchers behind this paper and find similar experts.

7-day free trial. Cancel anytime.

References (21)

[1]
Unmasking survey fraud: investigating data quality issues in an MTurk sample
2025Marissa N. Maline, Andrea N. Polonijo
[2]
"All that Glitters": Techniques for Evaluations with Unreliable Model and Human Annotations
2025Michael Hardy
[3]
Using Linkage Sets to Improve Connectedness in Rater Response Model Estimation
2023J. Casabianca, John R. Donoghue et al.
[4]
Best Practices for Constructed‐Response Scoring
2022D. McCaffrey, J. Casabianca et al.
[5]
“Technique for the Measurement of Attitudes, A”
2022R. Likert
[6]
Detecting Rater Biases in Sparse Rater-Mediated Assessment Networks
2021Stefanie A. Wind, Yuan Ge
[7]
Measuring Rater Centrality Effects in Writing Assessment: A Bayesian Facets Modeling Approach
2021T. Eckes, K. Jin
[8]
Learning to summarize from human feedback
2020Nisan Stiennon, Long Ouyang et al.
[9]
SummEval: Re-evaluating Summarization Evaluation
2020A. R. Fabbri, Wojciech Kryscinski et al.
[10]
Accounting for Rater Effects With the Hierarchical Rater Model Framework When Scoring Simple Structured Constructed Response Tests
2019Ricardo Nieto, J. Casabianca
[11]
The Effects of Incomplete Rating Designs in Combination With Rater Effects
2019Stefanie A. Wind, E. Jones
[12]
A New Facets Model for Rater's Centrality/Extremity Response Style
2018K. Jin, Wen Chung 王文中 Wang
[13]
The Stabilizing Influences of Linking Set Size and Model–Data Fit in Sparse Rater-Mediated Assessment Networks
2018Stefanie A. Wind, E. Jones
[14]
Item Response Models for Human Ratings: Overview, Estimation Methods, and Implementation in R
2018A. Robitzsch, J. Steinfeld
[15]
Detecting Rater Effects under Rating Designs with Varying Levels of Missingness.
2018Rose E. Stafford, E. Wolfe et al.
[16]
Exploring the Effects of Rater Linking Designs and Rater Fit on Achievement Estimates Within the Context of Music Performance Assessments
2016Stefanie A. Wind, G. Engelhard et al.
[17]
Application of Latent Trait Models to Identifying Substantively Interesting Raters
2012E. Wolfe, A. McVay
[18]
Detecting and measuring rater effects using many-facet Rasch measurement: Part II.
2004C. Myford, E. Wolfe
[19]
Detecting and measuring rater effects using many-facet Rasch measurement: part I.
2003C. Myford, E. Wolfe
[20]
Applications of Item Response Theory To Practical Testing Problems
1980F. Lord

Showing 20 of 21 references

Founder's Pitch

"Develop a tool using psychometric models to enhance the reliability of human evaluations in AI systems by correcting rater effects."

AI EvaluationScore: 5View PDF ↗

Commercial Viability Breakdown

0-10 scale

High Potential

1/4 signals

2.5

Quick Build

1/4 signals

2.5

Series A Potential

1/4 signals

2.5

Sources used for this analysis

arXiv Paper

Full-text PDF analysis of the research paper

GitHub Repository

Code availability, stars, and contributor activity

Citation Network

Semantic Scholar citations and co-citation patterns

Community Predictions

Crowd-sourced unicorn probability assessments

Analysis model: GPT-4o · Last scored: 2/26/2026

Explore the full citation network and related research.

7-day free trial. Cancel anytime.

Understand the commercial significance and market impact.

7-day free trial. Cancel anytime.

Get detailed profiles of the research team.

7-day free trial. Cancel anytime.