Improve speed of large Surf set-ups

Improve speed of large Surf set-ups

Summary This article sets out the steps needed for using askia Analyse & Surf to improve the (metadata) speed of Surf set-ups with a large number of .qes (wave) files.
Applies to Askia Analyse, Askia Surf, Askia Vista.
Written for Data Processors, Analysts, Researchers.
Keywords Analyse; Surf; Speed; Metadata; Qes; Qex; Size; Access; Table; Data; Inverted; Invert; dat Folder; Waves; Qew; Vista 6.

For general tips on speed improvement, please read the relevant section in our Analyse FAQ article.

We know that inverting your data in Analyse will create a .dat folder in the same directory as your .qes or .qew and store your data in such a way that it is read faster by Analyse.

Analyse reads the data in conjunction with the question information in the .qes file.

For a stand alone data-set where the file is a single .qes, this is fine. 

However, for complicated Surf set-ups when you have a large number of .qes (wave) files, we can begin to see a reduction in speed because the question information in all these .qes files needs to be accessed repeatedly from different databases. Opening and closing these tables in the .qes back-end starts to become a cumbersome process when there are many.

To combat this there is a feature in Surf which can help when your Surf set-ups get to this point. The 'Improve Metadata Speed' option can be found in the Edit menu of Surf.

The first thing to do is ensure your .qew is inverted i.e. when your .qew is open in Analyse you see the INV sign in the bottom right-hand corner of Analyse:

The folder should have .dat folders for all of the individual .qes files and the .qew:

You can then open the .qew file up in Surf and go to Edit > Improve Metadata Speed:

What does this do? 

It opens all the source files in the .qew and for each of them, removes screens, routings, screen groups, question classes, resources, versions, languages, revisions and all captions. 
They're then saved to .qex file with "-meta" and the end of the file name.

These can now be used along with all the .dat folders and the .qew to access the data in Analyse - No need for the .qes files anymore.

Why does this improve speed?

The way the required question information is stored in the .qex file is quicker to access than it is in a .qes file. The observed speed improvement is more noticeable if you are using 64 bit Analyse.

Using the .qew from the folder in the screenshot above allows you to access the data in Analyse in exactly the same way as before with the added benefit of seeing an improvement in speed for large set-ups.

A key point is that you can take advantage of this functionality in Surf set-ups that are placed on Vista - You will need Vista versions and higher (using AskiaAPI.dll 14.11.2017 onward).

Have more questions? Submit a request