Scroll

Data exports: Methods, Pros & Cons

Follow
Summary This document provides examples and explanations of four different export methods and the pros and cons of each
Applies to Askia Analyse, Askia Tools, Askia field, Fast Export Tool
Written for Survey Scripters, Data Processors and Analysts.
Keywords Supervisor, Fast Export, Analyse, Tools, Export, CCA, Synchronise, CmdQesExporter, 2GB, Limit, SQL, Inverted, Command line, Additional filter.

 

Table of contents:

There are a number of data download methods available to clients and this article briefly summarises the main four along with their pros and cons.

    1. Right click on task in Supervisor and export.
    2. Fast export.
    3. Data Inversion method.
    4. Synchronise from SQL into directly into .qes file.

 


1. Right click on task in Supervisor and export.

 

Export interviews to qes file - Pros & Cons:

PROS:

  • Easy to understand and action
  • Stores exports by date in task for fast access later

CONS:

  • Large exports put strain on CCA. Many large exports running concurrently can cause unresponsiveness
  • Also stores data on server side
  • Relatively long download time.
  • Cannot filter download.
  • Limit of 2GB on exported .qes file.

 


2. Fast export.

For particularly large/ complex surveys the export to QES via Supervisor can take a long time. As an alternative we have developed a tool that you can run at the command line that runs an export in a fraction of the time. (Tests have indicated that this is more than 10 times faster than using Supervisor). You can find instructions for use of this method in the feature ticket: https://dev.askia.com/issues/5692 

Or download January 2015 CmdQesExporter.rar here

To summarise briefly:

  • Download the latest CmdQesExporter.rar
  • Read the Instructions.txt file.
  • In case not already installed then install the access runtime: http://www.microsoft.com/en-us/download/details.aspx?id=4438
  • Open the "run" command dialog by pressing Winkey + R then type "cmd" then press enter to enter the command line interface
  • Go to the directory where the CmdQesExporter.exe is, e.g. type the following in:

          cd C:\Users\Seyf\Downloads\CmdQesExporter\CmdQesExporter

  • You can now launch the executable "CmdQesExporter.exe" followed by its command line arguments.

Format:

CmdQesExporter.exe [QesSource] [SqlFormat] [ConnectString] [SurveyID] [QesTarget] [ProgressReportMode] [ProgressPipeName] [CompletedFilter] [StartDateFilter] [EndBehaviour]

Example:

CmdQesExporter.exe "c:\users\matt\desktop\other.qes" "NEW" "Provider=sqloledb;Data Source=(local);Initial Catalog=CCA_SURVEYS;User Id=name;Password=pass;" "123" "c:\data\news_extracted.qes" "CONSOLE" "" "COMPLETED" "24/06/2012" "UNPAUSE"

The parameters are explained in more detail in the Instructions.txt file. There is also a separate article on this here:

https://support.askia.com/hc/en-us/articles/200432011-Fast-export-to-QES-from-command-line

 

Fast export - Pros & Cons:

PROS:

  • Does not put strain on CCA
  • Data not stored on server side
  • Relatively quick download time 

CONS:

  • Cannot filter download
  • Limit of 2GB on exported .qes file

 


3. Data Inversion method.

  • Create a .qes file that matches exactly the current structure of your task. You can do this by getting a working copy, putting this somewhere locally and saving as a .qes file. This will not, however, keep weightings, sub-populations or created variables so if you want to do this you will need to import them later in Analyse or take a copy of the .qes file which contains these, if necessary, delete all interviews and use this locally.
  • It is advised the routing from this local .qes is removed so that the imported data is not verified against the final routings.
  • Open a fresh Analyse session (no .qes file selected). Then, File > Open field task > enter the task id, surveys connection string and point towards the .qes file you have stored locally.
  • From the Tools menu in Analyse > Re-invert database
  • Once this is done you will note the .dat folder in the same place as your .qes file. This stores the inverted data and the data will be picked up so long as the name of the folder matches the name of your .qes file.
  • In Analyse, you can now close the field task (this is only required to invert the data). Now, File > Open in the normal way in Analyse to access inverted data.
  • In order to access the inverted data in Tools, open a fresh Tools session (no qes file selected). Then, File > Open an inverted survey > Navigate to your .qes file and use Tools in the normal way. Most, if not, all actions are considerably quicker, including exports to .sav / triple s etc.

 

Data inversion method - Pros & Cons:

PROS:

  • Does not put strain on CCA
  • Data not stored on server side
  • Relatively quick download time
  • Final download size not limited to 2GB because data stored in inverted format and not .qes file

CONS:

  • Cannot filter download.
  • Some functionality not available with inverted data e.g. created variable by interview information in Anlayse

 


4. Synchronise from SQL into directly into .qes file. 

  • Get a working copy of the task which you want to download the data (Image A)

  • Delete all the routing from the .qex file and save it as a NO_ROUTING.qes file.
  • Take a copy of the qes file and add _IMPORT to the name.
  • Open Askia Tools and File > Open >  navigate to ‘exporttest_NO_ROUTING_IMPORT.qes
  • Merge > Interviews
  • For Askia file, navigate to ‘exporttest_NO_ROUTING.qes
  • Select: ‘Interviews are read from SQL server (new format)’ in this case. (This info can be seen in Image B 'Interview storage').
  • Enter the Surveys database connection string. This can be found in Supervisor > CCA Settings > Tasks > Surveys > Database connection string.  (Image C)
  • Enter the task ID: 1004 in this case. (Image B)
  • Enter any sub-population (Additional filter) that you need to download a selection only. e.g. ??q1?? Has {1} and ??q2?? Has {2}

 

 

  •  Hit OK and the data will start downloading directly from SQL into the empty .qes file locally.

 

Synchronise from SQL into directly into .qes file.  - Pros & Cons:

PROS:

  • Does not put strain on CCA
  • Data not stored on server side
  • Can filter download
  • Taking the idea further, you can use the different merge types to download data for only a selection of questions

CONS:

  • Final download size limited to 2GB
  • Relatively slow download time
Have more questions? Submit a request

Comments