Partial data can be useful to researchers for a variety of reasons, and is captured within every Decipher survey. Our system defines a partial record as a respondent who has not fully completed a survey before dropping out or closing their browser window.
However, while partial data is included in every survey, it is not accessible using any of the reporting tools in the project portal. The partial recovery system was created to allow you access to partial records within Crosstabs.
Note: Please refer to our Data Retention policy for how long this data is available for your project.
1: Enabling Partial Recovery System
By default, all partial data is stored in the survey ‘state.db’ file and is used by the system to save each respondent's state between survey pages.
To enable the partial recovery system and access your partial data through Crosstabs, select the “Auto Recover Data” setting within the survey Field Settings menu:
Once enabled and your project is no longer in the testing state, survey data will begin recovering automatically (within 15 minutes) to Crosstabs.
2: How It All Works
The partial recovery system tracks partials by the system-generated "
uuid" variable and stores all partial data in a separate database, so you must enable this system for each project individually.
Every 5 minutes, our system will scan your survey for potential partial records. To be considered a partial record, the respondent must have not submitted a survey page in the last 120 seconds for that run. Please see Project-Specific Overrides to configure the 120-second default.
The system will check each
uuidagainst all previously recovered
uuidsin its memory and either append or replace the data.
If the partially recovered respondent completes the survey, their latest partial record is converted to a standard complete record.
The default settings mean that a partial record will take a maximum of 7 minutes to be recovered (5-minutes + the 120 second delay). You will also never have duplicate
uuid entries for partial data. All partial data in the survey ‘state.db’ file can become recovered partial data in the main data file once the partial recovery system is enabled.
3: Project-Specific Overrides
If needed, you can further tune the system to get partial data quicker. An XML attribute called “autoRecoverDelay” controls the delay timer, and can be set to something other than the default 120 seconds.
To ensure the quickest recovery of partial data, you can set this attribute to “0” via the XML Editor by adding autoRecoverDelay=“0” to your survey tag:
Regardless of the value you set, keep in mind that:
Constant recovery of partials will decrease the performance of any hidden virtual data and campaign sends, as the partial recovery system resets certain cached data. This is more visible in larger surveys.
Even at 0, the lowest possible threshold for partial data recovery is 5 minutes.
The default settings require that a partial be recovered after a maximum of 7 minutes since their last activity (5-minutes + the 120 second delay).
4: Finding/Interpreting Partial Data in Decipher
Partial data respondents will have a distinct “recovered” marker set and these records are excluded from what our system normally considers “completes”. There are some limitations as to where you will find partial data, and how each of our tools will treat it:
Crosstabs: Within Crosstabs, partial data is included in respondent bases only when selecting "all with partials" or "partials only" in the Respondents filter:
The partial recovery system will also add a virtual question to Crosstabs called “vdropout” that will record the last-seen question by that respondent.
Field Report: The ‘Click-through’ count within the Completions tab will include all partials, along with a partial-only count listed directly below it:
Partials are also shown in the Drop Outs tab in the Field Report, as all clicks per question are shown there by default.
Data Downloads: All outputted data will follow the report settings from the survey report/Crosstabs - in our system, partial records will always have a status of “4”. It will also add 2 markers to your respondent’s markers data field*: “autorecover” and “last_XX”, where ‘XX’ is the label of the last question seen.
Note: *The vdropout virtual uses this information.
Email Campaigns: In email sends, partials will always receive a reminder (our email system will not consider partial records as completes).
Command Line ( Requires Decipher Cloud ): When using the generate command, the config status of “all” does not include partials; use “partials” to output partials only or “everything” to get all complete and partial data combined.
5: Editing Partial Data
Respondents’ recovered partial data can be edited in the same way that you edit complete data, using the View/Edit Responses tool:
To learn about editing data, click here.
Note: If the respondent returns and completes the survey at any time while the partial recovery system is still enabled, their data and status will be updated; this process will overwrite any previous changes made via the View/Edit Responses tool.
6: Disabling Partial Recovery System
To disable the partial recovery system, deselect the “Auto Recover Data” setting within the survey Field Settings menu. This disables the system from running every 5 minutes. Once disabled, you should wait 5-10 minutes for any pending partial data runs to complete before pulling down your partial data.
Note: This does not delete/remove any already recovered partial data from your Crosstabs. If you wish to remove your partial data from Crosstabs, you can clear them out using our data editor.
7: Partial Recovery System Logs
Requires Decipher Cloud
7.1: Project Level Logging
All partial events are logged on our server for each enabled survey. This is located in the root folder of the project in the survey.log file.
In the survey.log file, you will find timestamps of when each partial
uuid was restored and when that existing partial
uuid record was updated.
7.2: Global Logging
Global logging is available for “staff” level users and can be found in the following location:
All partial events for each project are logged here. This includes when the project began restoring data (and its delay), when a partial record was restored, to where it was restored, and a small summary of how many records were restored/converted to completes.
8: Limitations & Unsupported Features
Here is a list of some frequently asked questions regarding our partial recovery system:
Q: If a user drops out and then re-enters the survey, will they have the same
A: No, the partial recovery system will assign a new
uuideach time a user enters the survey, unless you have Autosave enabled. See more about survey resuming here.
Q: If there are multiple partial records, how do I match a user to their assigned
A: It is recommended that you use an additional unique identifier for each user if you would like to match partial data back to specific records (for example, a user’s “source” value will remain the same for all of their partial records).
Q: If a
uuideventually completes the survey, does the partial system delete that partial record from the recovery process?
A: Yes, it will overwrite the most recent partial record with the respondent’s complete data. Before recovering a
uuid, the system checks whether it has completed the survey, so there is also minimal chance of duplicate records.
Q: What if the system picked up that ID in a prior run?
A: On a survey with the partial recovery system enabled, subsequent recovery runs or completes will update each
uuid'splace in the data file and will not save a new record.
Q: Does the 5-minute recovery process run at 8:00/8:05/8:10, etc. or some other 5-minute interval?
A: It will run at 12:00am and then again every 5 minutes after that.
Q: Can I change the 5-minute timer?
A: No, this is a server setting and such an override function does not exist.
Q: On which data variable is autorecover/autoRecoverDelay based?
A: Both of these processes will always be based on each user’s unique identifier or
Q: Can I enable the partial recovery system after the survey has fielded?
A: No, for a closed survey, you must submit a ticket to support to enable this data. Also note that the data must be available per our Data Retention policy.
Q: For how long is partial data available?
A: By default, partial data is stored in our system for 3 weeks from the date the last respondent finished the survey. After 3 weeks, all non-recovered partial data is erased completely. You should recover any partial data you wish to save past this point.
Q: Can I extend the allowed storage time for my partial data?
A: There is no way to extend the storage. It is recommended you enable the system from the start of fielding so that all partial data is available for you in Crosstabs.
Q: Is there a limit to how much partial data I can store on a survey?
A: Currently, there is no limit to this. Note that additional storage costs may be incurred due to partial data storage. See your Contract or Account Manager for details.