Skip to content
Technical University of Munich

Framework

QualityCrowd is a framework that allows codec independent quality assessment with a simple web interface, usable with common web browsers.

Note that QualityCrowd is not a finished software product, but rather a framework that will need further customizations to fit all your requirements. If you use the framework, please cite the publication provided below.

QualityCrowd 2

Since the publication of the original QualityCrowd paper, we have rewritten the framework completly in order to allow for more flexibility and a wider range of supported plattforms.

QualityCrowd 2 differs in many points from the previous QualityCrowd software. Instead of providing a web interface for designing and defining a test batch, QualityCrowd 2 introduces a text based definition of tests. Quality tests are now defined through little QualiyCrowd-Scripts (short: QC-Scripts). The QC-Scripting language is easy to learn and gives the the operator even more control over his test than before.

In addition QualityCrowd 2 does no longer require a database and the complete rewrite of all source code allows much more flexibility and a much easier extensibility.

To keep it simple all the connections to services like Mechanical Turk, Crowdflower and Amazon S3 have been removed. Instead a system of worker ids and tokens is used:

  • the worker browses a URL constructed like this:

    "http://qualitycrowdserver.example/<batchid>/<workerid>"

     

  • after finishing the task the worker is told a token which he can use to prove that he has completed this task

This method is quite common on platforms like microWorkers.com and can also equally be used on e.g. Mechanical Turk.

QualityCrowd 2 is hosted on Github:

https://github.com/ldvpublic/QualityCrowd2

QualityCrowd 2 Screenshots

Some screenshots of a QualityCrowd 2 installation.

Different batches
Individual batch
QCScript descriping an individual batch
Results
Online documentation
Video as seen by the test subjects

QualityCrowd 1

The original version of the QualityCrowd framework is still availabale and also hosted on GitHub:

https://github.com/ldvpublic/QualityCrowd

We recommend, however, to use the current version QualityCrowd 2. 

More Information and Reference

More information about the framework can be found in the corresponding publication. If you use the framework, please cite this publication:

More information on how to generate lossless encoded H.264/AVC video files that can be used with the Adobe Flash plug-in are available here.

For general questions about QualityCrowd, you can contact us at quality(at)ldv.ei.tum.de.

To top