The project warnings report allows you access to survey-level errors and counts of other potential data collection issues. Email notifications are sent to subscribed users when errors or data collections issues occur. Now you can track survey quality and react quickly to issues to ensure data quality after your survey goes live.
Note: Only staff and supervisors have access to this report.
Accessing the Project Warnings Report
For a Company
In the research hub, open your company page by clicking the "Company Page" link located under the company name/logo.
Then click the "Project Warnings" link found along the left panel of the page.
The project warnings report displays projects for your company based on the selected filter options.
1. Filter menu: Click to open the filter menu. You can select to filter by date, company and survey state.
2. Selected Filter options: Applied filters from the filter menu.
3. Search: Enter a project name to filter the project list by a specific project.
4. Company name(s): The name(s) of the companies included in the report.
5. Project list: A listing of the projects with project path that meet the filter or search criteria.
6. Exceptions: The number of loading or execution errors reported for a project.
7. Missing Resources: The number of missing static resources such as images, scripts, or other media.
8. Quota Misses: The number of respondents that do not qualify for any quota marker in a quota table and are consequently terminated.
9: Blocked Respondents: The number of respondents being blocked from entering the survey.
10. Drill down: Click a link to drill down on the exception or warning for detailed information for a project.
Color Coded Thresholds
Additionally, exceptions and warnings shown on the report are color coded according to severity. Thresholds have been preset for your convenience.
- [ ] - Any count (>0) for exceptions, quota misses, and missing resources
- [ ] - Count for blocked respondents greater than the threshold
- [ ] - Count for blocked respondents or quota misses within the threshold
The threshold for exceptions, quota misses and missing resources (shown in [ ]) generally indicate a problem with the survey and these need immediate attention. Blocked respondents (shown in [ ]) may indicate that there's a problem with the survey setup.
For information about the thresholds levels, click here.
For a Project
You can also access this report for a project when you select "Project Warnings" from the research hub.
Note: Only projects that have been launched or closed will contain project warning data.
Filtering a Company's Project Warnings Report
You can filter the project list on this report when you click "+ Filter". You can filter by date, company, and by survey state.
Note: For a project, you can only filter by date.
Filtering by Date
By default, the project list is filtered by "All Time." This displays all the events for project. You can filter the project list to display the project warnings for a project by a specific time period.
Filtering by Company
Note: Only staff and supervisors can filter by company.
By default, the project list is filtered by the current company but you can add to the companies displayed. To see other available companies, click in the box. Select/deselect companies you want to display.
Filtering by Survey State
By default, the project list displays live surveys. You can can filter the project list to display closed surveys or both (live and closed surveys) as displayed below.
Notification emails are sent to users subscribed to a LIVE project. The emails include errors and warnings that have exceeded the preset thresholds:
- Missing static resources
- Not qualifying for a quota (respondents that do not qualify for any quota marker in a quota table and are consequently terminated)
- Blocked respondent
- Drop-outs (although totals are not included on this report). The percentage of drop-outs reported is a "snapshot" in time and will not match the overall project drop-out rate shown on the field report. For performance reasons, here, the dropout formula is based on the number of respondents active in the last 60 minutes.
Notification begins when a threshold is exceeded. Afterwards, one email is sent out when there is a change in the count for aggregated events. The server checks and sends out emails approximately every 5 minutes for event counts that changed to each subscriber.
You will never receive repeat notifications within 60 minutes of each other for the same event exceeding its threshold; however, if a different type of event exceeds its threshold, the notification email will still include all new activity (including counts for events of which you were recently notified). The exception to this is for survey errors which should be corrected as soon as possible to prevent further loss of data. As long as the survey is getting errors, the subscribers will receive the summary email as well as the individual error emails.
An survey error could be a fatal error such as one that causes the survey to abort or a survey error can occur when attempting to load the survey (e.g. when the survey file is malformed). Every unique respondent that encounters that same error will count toward the next summary email received detailing the number of errors during the time interval. Summary emails are delivered until either the subscriber unsubscribes from the survey or the error is fixed.
How Notification Thresholds are Set
The below details how the threshold levels are set for notifications:
- Exceptions - 1+
- Missing resources - 1+
- Blocked respondents - >1 per minute, with a minimum of 5 blocked respondents in a notification period (5 per min)
- Drop-outs - 30% dropouts / active with a minimum of 10 respondents active during the 1 hour timeframe leading up to the 5-minute system review period
- Missed quotas - 1+
Subscribing to a Project
You can subscribe to a project in the following ways:
- automatically when you launch a project
- actions menu when testing (Show Survey with Tools)
For users with shell access, there are additional ways to subscribe/manage subscriptions. See Managing Subscriptions for a Project for details.
Note: Users with shell access, you cannot use the hooks subscription system for project warnings email notifications. Click here for information about the hooks system.
Drilling Down On the Report
From a company's project warnings report, you can click the project link to monitor the errors and warnings for a project displayed on the following tabs.
- missing resources
- quota misses
- blocked respondents
On the exceptions tab, you can see details about the errors generated from a live project so you can take immediate action to resolve the exception.
Click the error code link to get a complete description of the error text.
First, on the exceptions tab, look at the line that begins with "Error," as this usually contains a brief summary of the problem. If you need more information, the trace back, program context snippet, and/or variables (displayed when you click the error code link), will usually clarify the origin of the problem that needs correcting.
Monitoring Missing Resources
On the missing resources tab, you can see any resources that are referenced in the survey but not found on the server. You may need to upload a missing file to the survey. For more information, see Uploading System Files.
Note: The project warning system does not monitor images that are protected [protected image.png] and [image image.png]. Make sure you thoroughly test your project before launching.
Monitoring Quota Misses
On the quota misses tab, you can can see the number of respondents who do not qualify for any quota marker in the given quota table. This is likely caused by a programming error and should be addressed immediately. See Non-qualified Respondents and Unspecified Terminates.
Create a quota cell for these "missed" respondents. If they should be terminated from the survey, create a term point prior to the quotas. This step ensures understandable survey completion statistics and helps to identify real problems with programming and/or sampling.
Monitoring Blocked Respondents
On the blocked respondents tab, you can see the reason that respondents have been blocked from the survey. Respondents may be blocked for various reasons and some of which do not indicate a problem with the survey setup and/or sampling. For example, copying/pasting or otherwise entering a URL without it's required appended variables will block a the respondent from the survey.
The following are reasons why a respondent may be blocked:
- invalid URL variables (See Locking Down a Survey by Unique Variables)
- invalid CNAME (See Setting Up a Domain Alias (CNAME))
- duplicate URL variables (respondent tries to enter the survey again)
- unallowed device (See Device Settings)
- unallowed country/region (See Digital Fingerprinting)
- improper language
For blocked respondents, complete the following checks:
- For an invalid or missing variable or CNAME, verify that the sample source setup and the survey link is correct.
- For multiple counts of duplicate variables, verify that the URLs being sent to respondents are being properly populated with unique variables.
- For multiple counts of wrong type of device or answering from an unallowed country, check the survey setup.