Goal: Qualify submissions in terms of quality and veracity.
MetricsDAO provides a 6 step process for ​​Organized, On-Demand Analytics Delivery, or as we like to call it, OODAD.
QA for Solutions is the 5th step, which we experimented with for the first time with OlympusDAO. Here’s how we did it, what we learned and how we plan to improve!
Timeline: We gave ourselves a week to review and grade the 11 OlympusDAO submissions based on a predefined rubric.
Reviewers: After putting the word out on Discord, we found 2 community members to review the 11 Olympus DAO submissions. Reviewers had to know basic SQL and have experience with crypto analytics. Our third reviewer was a member of the core team.
Incentive: Reviewers will earn a reviewer-specific POAP.
Peer-Review Process: The Core Team member was in charge of reviewing the other reviewers’ work.
QA for OlympusDAO solutions went very smoothly. The reviewers broadly agreed with each other, and it was easy to identify winning submissions. We were able to use average scores to validate submissions, and we finished grading in a week.
Quality of submissions varied widely. The worst quality submissions were directly tied to users doing the bare minimum. The best quality ones were done by experienced users. Experience matters but it is not the only factor determining quality – better incentives should be put in place.
Incentivizing better quality submissions
Dispute & resolution mechanism:
We want to implement a dispute & resolution mechanism in case one reviewer disagrees with everyone else. A few experienced graders should carry somewhat more weight.
Attracting more reviewers:
As the number of submissions increases, we will need more reviewers. Being a Reviewer is an excellent way for talented analysts with SQL and crypto skills to get involved at a low time commitment (2-3 hours a week, maximum).
For upcoming programs, like Convex, we plan to offer a Reviewer POAP to track reviewer contributions. Additionally - from experience, Reviewers have indicated that seeing the submissions of others is an excellent way to improve their own work by getting exposure to a range of different techniques.
If you have any interest in being a Reviewer for upcoming programs, please contact Snowslinger.ust!
We’d love to hear your thoughts and feedback so please let us know in Discord!
MetricsDAO is uniting the best analytical minds in the space to build the future of crypto analytics.