Frequently Asked Questions

Can I run TurkPrime studies if I do not have a Mechanical Turk (MTurk) account?

Yes. Although TurkPrime is designed for Amazon Mechanical Turk Requesters with an MTurk account, we also provide a service so that you can use the TurkPrime.com MTurk account from where you can run your studies. This is particularly useful for non-United States based researchers who are not eligible for an MTurk account.

Contact us for details and pricing.

How do I set up my TurkPrime account?

Connecting your TurkPrime to your Mechanical Turk account is easy and should not take more than a few minutes.

Now that you have signed up and logged into your TurkPrime account, you will need to connect your TurkPrime account to your MTurk account. If you do not have a Mechanical Turk account you can set one up here.

TurkPrime needs your permission to access your MTurk account and to take control over your HITs. This includes being able to launch, stop, and make changes to your HITs. This is accomplished by associating your TurkPrime account with your MTurk account.

Associating your accounts is fast and easy.
Instructions:
  1. Go to Amazon Mechanical Turk Developer page and follow the instuctions (Step 1-Step 3) to link your MTurk account with an AWS account (Step 4 may be omitted).
  2. After successfully linking your AWS account to your MTurk account, create an IAM (Identity and Access Management) user to access the MTurk API. (MTurk documentation)
    1. Go to https://console.aws.amazon.com/iam/home#/users and sign in using the same username and password as you used for MTurk
    2. Choose Add user
    3. By User name, fill out a username. Select Add another user to add multiple users.
    4. By Access type, select Programmatic access
    5. Select Next: Permissions
  3. Set up MTurk permissions for IAM user (More at Overview of IAM Policies)
    1. Select Attach existing policies directly
    2. Check off AmazonMechanicalTurkFullAccess. (You can use the search bar to find it)
    3. Select Next: Review
    4. On this page, review your information, and select Create user
  4. On this page, save your credentials:
    1. Choose Download .csv
    2. Your browser automatically downloads the private key file (credentials.csv). Save the private key file in a safe place.
      Important : This is the only chance for you to save the private key file. You will need these credentials to use the Amazon Mechanical Turk API.
    3. Choose Show to view your Secret Access Key.
    4. Copy the values for your Access Key ID and Secret Access Key into the form on the bottom of this page.
Please Note: Mechanical Turk may be used in either LIVE or SANDBOX mode.
  1. LIVE mode will run your surveys and cost actual money. This is the option to use when running REAL (LIVE) surveys.
  2. SANDBOX mode may be used to test creating HITs. Your HITS will not be answered by workers. Your HITs are viewable on the MTurk site at http://requestersandbox.mturk.com. Sandbox mode sets your account balance to a fixed $10,000.

For your security, once entered, the Secret Access Key will not be visible to you. Store the credentials.csv file (which contains your secret keys) that you previously downloaded in a safe place.

How do I set up and launch a study?

To set up a study, click on Design Survey on top of the TurkPrime home page.
The Design Linked Survey page is designed to give you the same features as you would find on MTurk. For example, you will find fields for naming your study, indicating the price, selecting the quality of workers etc. Additionally, there are other fields that are not found on MTurk, which provide added functionality. Below you will find a description of each field

  • Project Name – This is the name of your project that is not visible to Workers. It will be visible on the dashboard, and will allow you to easily differentiate between multiple HITs with the same name.
  • Title – This is the name of your HIT as it will appear to Workers on Mechanical Turk.
  • Description – A brief description of the Hit as it will appear to Workers on Mechanical Turk.
  • Keywords – Including keywords will make it easier for Workers to find your HIT.
  • Workers payment per survey – How much each participant will be paid per HIT.
  • Estimated time to complete HIT – This estimated time lets potential workers know approximately how long the HIT will take. This estimated time to complete the HIT will be appended to the Description field, described above.
    For example, if the HIT Description is “Short personality survey”, and the Estimated Time to Complete HIT is “7 minutes”, then the Description will appear on Mechanical Turk as “Short personality survey, (~ 7 minutes)”
  • Total Number of Survey Participants – The total number of participants that you want to take your HIT.
  • Time allotted per assignment – This is the maximum amount of time that a worker can spend on a HIT. If a worker does not submit a HIT within the time allotted per assignment they cannot complete the HIT and cannot get paid.
    Note that participants differ as to how fast they work. You should thus provide workers with ample time to complete their HIT. However do not make the time allotted per assignment overly long, since this will deter workers from taking your HIT, as they will assume that the HIT will take a long time.
  • HIT expired in – The HIT will expire and will no longer be visible to workers after this time.
  • Survey Hyperlink – Copy and paste the link to your study in this field. Click on Preview Link in New Tab button to view your study.
  • Secret code – This is an optional feature, which allows you to automatically approve and reject workers based on whether or not they correctly enter the Secret Code.

    • Why do I need to secret code?
      There is nothing stopping a Worker from accepting your HIT on MTurk and clicking Submit without ever actually having looked at your HIT. For this reason requesters routinely include quality assurance checks. The Secret Code is one such quality assurance test. At some point in the study, for example at the very end, you can show the Worker your Secret Code (for example HGTR4EFG) and indicate to the Worker that they should copy this code and paste it on MTurk Prior to submitting the HIT. Here is an example text: “The study Secret Code is HGTR4EFG. Please enter this code on Amazon Turk before submitting this HIT.” This will guarantee that, at the very least, the Worker reached the end of your survey and saw the last question.
    • How does the Secret Code work?
      When the worker completes the HIT having entered the correct Secret Code, TurkPrime will automatically approve the HIT. This will save you the time to go through each workers and manually check whether they entered the Secret Code. However, if a worker does not enter the Secret Code correctly, they will be automatically rejected by TurkPrime. Being rejected means that a Worker will not get paid for the HIT, and that the rejection will now be part of that Worker’s permanent MTurk record.
    • Can I reverse a rejection for a rejected a worker?
      Yes. After a study is launched you can check the progress by clicking on Results at the top of the TurkPrime homepage. On the DashBoard page you will be able to see the status of each study. A number of buttons also provide options for further actions that can be taken. To reverse a rejection for a specific Worker click on Reverse Rejections. In the Worker ID field paste the Worker ID of the workers whose rejections you want to reverse. You can include a message that the Worker will receive in their email notifying them that their HIT had been approved, with the rejection reversed.
    • What if I don’t want to automatically approve and reject workers?
      The use of the Secret Code is optional. To deactivate the Secret Code feature simply check off the box that appears below the Secret Code field. You will then be able to manually approve and reject Workers.
  • Auto pay workers in – Time after which all Workers will be automatically approved. This field is relevant only when the Secret Code option is not selected.
  • What is the Exclude feature?
    The Exclude feature can be extremely useful for requesters who wish to exclude Workers who have participated in previous HITs. Launching a HIT while excluding participants from previous HITs will open your launched HIT to all Mechanical Turk Workers, but will not allow the excluded Workers to take your HIT.
  • How do I use the Exclude and the Include features?
    When setting up your study in the Design a Linked Survey page, go to Advanced. All of the HITs that you have previously launched will appear in the Exclude Workers Who Completed These Surveys field. Click on a study that you want to exclude. To exclude multiple studies, click on each study while holding the Ctrl key.

    • Can I exclude only some of the Workers from a study?
      Yes. Simply paste all of the Worker IDs in the Exclude Workers with These WorkerIds field.
  • What is the Include feature?
    The Include feature can be extremely useful for requesters who wish to run a HIT with only the Workers who have taken a previous HIT, or multiple previous HITS. Launching a HIT while including participants from previous HITs will open your launched HIT to only the Workers who have taken those HITs in the past, but will not allow any other Workers to take that HIT.
  • How do I use the Include features?
    When setting up your study in the Design a Linked Survey page, go to Advanced. All of the HITs that you have previously launched will appear in the Allow Workers Who Completed These Surveys field. Click on a study that you want to include. To include multiple studies, click on each study while holding the Ctrl key. When including multiple studies use the All option to include Workers who have taken all of these studies, and use the Any option to include Workers who have taken any of the studies.

    • Can I include only some of the Workers from a study?
      Yes. Simply paste all of the Worker IDs in the Include Workers with These WorkerIds field.
  • Worker Level – Mechanical Turk provides the option of using “experts”. Selecting Any Level will make your study visible to all Mechanical Turk Workers. Selecting the Master options will only make your HIT visible to Workers that MTurk preselects based on their criteria. Note that MTurk charges more for Master workers.
  • HIT approval rate – Each time a participant completes a HIT the requester has to either approve or reject that HIT. Rejections become part of a Worker’s permanent record. A high rejection rate indicates that Requesters consistently reject a Worker, and may be used as an index of overall work quality. Use the right and left sliders to select the range of approval rates for your Hits.

    Below, the left slider was used to select Workers whose approval rating is above 90%. Having selected these Workers, only Workers with approval ratings above 90% will be eligible to take your HITs.
  • Number of HITs approved – When a Worker completes a HIT and a requester approves that HIT, the approved HIT becomes part of that Worker’s permanent record. MTurk keeps track of the number of approved HITs. Use the right and left sliders to select the range of approved HITs rates for your HIT.

    The number of approved HIT is an indicator of a Workers experience. Thus you may want to use only Workers who have completed many HITs if you want experienced Workers, or you may want to use only Workers who have a few or no HITs if you are looking for novice Workers.

    Note – The HIT approve rate does not become operational, until the number of approved HITs is at least 100, for survey HITs on MTurk.