Egressing Data

All data egressed from the Trusted Research Environment is inspected and approved by SDF Data Operations.

Your Request

In your request please include the following information:

• the full path of where the data you wish to be egressed is located in the TRE including the full file(s) name(s)
• the email addresses of those who will receive the egressed data.  Include your own email address if you are also to receive the egressed data.
• Which dataset is the data from and the nature of the data (e.g. a plot or a table).
• Who the data will be shared with (for example: Smart Data Foundry Employees only, Approved Project Researcher, Data Providers, General Public) or how it will be shared (for Example SDF Website, SDF Portal, Data Partner Report or Dashboard).

Request Approval

• The request is received and reviewed by the SDF Team, who will review the data you have requested to be egressed and either approve the request or ask for additional information / an adjustment to your request.
• Please note that the time to approve requests can vary depending on complexity, sensitivity and the quality of the information provided.  
• Once the request is approved and data is cleared you will receive a link and a password to retrieve the compressed file and the md5 file (if provided) from the Serv-U portal.

Guidance for Egressing Data.

In order to ensure approval is as efficient as possible the guidance below should be followed:

• keeping all export requests organised into a single folder inside your project folder.
  
  • ensure each file has a data dictionary (col name, and description)
  • ensure each relevant file has a group count column and the group is defined in the dictionary
  • each row is censored (convert to NA) for counts or proportions of 1-9 individuals (note that if this step is not followed the request will be denied).
• Compress files into a single archive (e.g zip or tar.gz).
• Many researchers also produce an MD5 file at this stage, and compare this to a new MD5 check once the data is saved in it's eventual destination, so they can ensure the data is not changed in transit.

Tips and Tricks.

• Do not include an index column. When writing a .csv, from R for example, an index column is often automatically included. This should be avoided as these indices usually contain numbers <10, which need to be manually checked by Data ops, creating unnecessary overhead and potential delay.
• Columns that contain many different strings need manual checking, so avoid them where possible.