Enterprise Docs
Search…
Download with Python
This page demonstrates how you can access and download the raw data files the Data Connector exposes. This is useful if you want to import these files into your own data lake or want to manually process and analyze the data locally.
If you want to easily access and query the Data Connector data, directly from Python, DataCamp provides the dcdcpy Python package. Installation details and examples of data import are provided in the linked GitHub README.
With the script below you can easily list or download all files from your Data Connector S3 bucket. All you need to do is follow the instructions below.
Step 1: Install the required packages
This example script requires only standard Python 3 installation and the boto3 library which can be installed by executing the following command:
1
$ pip3 install boto3
Copied!
Step 2: Save the script
Save the script below in a file named
data_connector.py data_connector.py
1
import boto3
2
import os
3
import sys
4
import datetime
5
​
6
# Your Data Connector credentials
7
aws_access_key = "ACCESS_KEY"
8
aws_secret_key = "SECRET_KEY"
9
aws_bucket = "BUCKET_NAME"
10
​
11
s3 = boto3.resource('s3', aws_access_key_id=aws_access_key, aws_secret_access_key=aws_secret_key)
12
​
13
def list_s3_files(bucket_name=aws_bucket):
14
file_list = []
15
bucket = s3.Bucket(bucket_name)
16
for bucket_object in bucket.objects.all():
17
file_list.append(bucket_object.key)
18
print(bucket_object.key)
19
return file_list
20
​
21
def download_from_s3(dirname, bucket_name=aws_bucket):
22
bucket = s3.Bucket(bucket_name)
23
current_directory = os.getcwd()
24
target_directory = os.path.join(current_directory, dirname)
25
os.makedirs(target_directory, exist_ok=True)
26
​
27
for object in bucket.objects.filter(Prefix=dirname):
28
bucket.download_file(object.key, object.key)
29
print('Downloading', object.key, '...')
30
print('All downloads finished!')
31
return True
32
​
33
if len(sys.argv) == 1:
34
list_s3_files()
35
else:
36
download_from_s3(sys.argv[1])
Copied!
Step 3: Add credentials
Update the script with your organization's credentials.
In the script update the three variables on lines 7 to 9 with your Data Connector credentials.
The above example script requires you to enter your credentials in the script itself, this is generally considered an insecure practice. Learn more about how you can safely store your access credentials in our Storing your Credentialsdocument
Step 4: Run the script
Now the script is ready, below are a couple of examples of how you can use the script.
Calling the script without any extra arguments will show a list of all files and directories in the bucket as demonstrated with the example below.
This could potentially output a very long list of items!
1
python3 data_connector.py
2
​
3
2021-12-15/assessment_dim.csv
4
2021-12-15/chapter_dim.csv
5
2021-12-15/course_dim.csv
6
2021-12-15/date_dim.csv
7
2021-12-15/docs.csv
8
2021-12-15/exercise_dim.csv
9
2021-12-15/learning_assessment_fact.csv
10
2021-12-15/learning_chapter_fact.csv
11
2021-12-15/learning_course_fact.csv
12
2021-12-15/learning_exercise_fact.csv
13
2021-12-15/learning_practice_fact.csv
14
2021-12-15/learning_project_fact.csv
15
2021-12-15/learning_track_content_fact.csv
16
2021-12-15/learning_track_fact.csv
17
2021-12-15/practice_dim.csv
18
2021-12-15/project_dim.csv
19
2021-12-15/team_dim.csv
20
2021-12-15/track_content_dim.csv
21
2021-12-15/track_dim.csv
22
2021-12-15/user_dim.csv
23
2021-12-15/user_team_bridge.csv
24
2021-12-15/xp_fact.csv
25
...
Copied!
​PreviousAWS CLI (Linux)If you specify a directory name (eg:
2021-12-12) this script will download all files from that directory to your local machine.1
python3 data_connector.py 2021-12-12
2
​
3
Downloading 2021-12-12/assessment_dim.csv ...
4
Downloading 2021-12-12/chapter_dim.csv ...
5
Downloading 2021-12-12/course_dim.csv ...
6
Downloading 2021-12-12/date_dim.csv ...
7
Downloading 2021-12-12/docs.csv ...
8
Downloading 2021-12-12/exercise_dim.csv ...
9
Downloading 2021-12-12/learning_assessment_fact.csv ...
10
Downloading 2021-12-12/learning_chapter_fact.csv ...
11
Downloading 2021-12-12/learning_course_fact.csv ...
12
Downloading 2021-12-12/learning_course_fact_v2.csv ...
13
Downloading 2021-12-12/learning_exercise_fact.csv ...
14
Downloading 2021-12-12/learning_exercise_fact_v2.csv ...
15
Downloading 2021-12-12/learning_practice_fact.csv ...
16
Downloading 2021-12-12/learning_project_fact.csv ...
17
Downloading 2021-12-12/learning_track_content_fact.csv ...
18
Downloading 2021-12-12/learning_track_fact.csv ...
19
Downloading 2021-12-12/practice_dim.csv ...
20
Downloading 2021-12-12/project_dim.csv ...
21
Downloading 2021-12-12/team_dim.csv ...
22
Downloading 2021-12-12/track_content_dim.csv ...
23
Downloading 2021-12-12/track_dim.csv ...
24
Downloading 2021-12-12/user_dim.csv ...
25
Downloading 2021-12-12/user_team_bridge.csv ...
26
Downloading 2021-12-12/xp_fact.csv ...
27
All downloads finished!
Copied!
Time saver! There is a "smart" directory called "latest" which always contains the latest export. You don't need to dynamically calculate or specify today's date if you need the latest data. Just calling the script with
python3 data_connector.py latest will download the most recent export to your local machine!Last modified 4mo ago