Export GraphQL data to CSV
This tutorial shows how to create a Python script to query the GraphQL API for Network Analytics data and convert the response to comma-separated values (CSV). Produced CSV could be easily ingested by tools like Splunk for further visualization and usage.
Therefore, this example queries the ipFlows1mAttacksGroups dataset,
which contains minutely aggregates of Network Analytics attack activity.
Prerequisites
The tutorial requires a valid Cloudflare API Token with Account Analytics:read
permission. It also expects that account you are interested in is entitled to
access Network Analytics.
Scripts in this tutorial requires Python version 3.6 or higher.
If you are looking to configure a Cloudflare API Token for a specific account, please refer to Configure an Analytics API token. Make sure you have access to the account.
Set up a script with authentication
The first step is to set up the script and define the variables for further authentication with the GraphQL API using a Cloudflare API token. The script also provides variables to set the range of data to export.
This example queries for a seven-day period that ended yesterday.
Calculate the date n days ago
The get_past_date() function takes a number of days (num_days), subtracts
that value from today’s date, and returns the date num_days ago.
The script uses get_past_date() with the offset_days and historical_days
variables to calculate the appropriate date range (min_date and max_date)
when it queries the GraphQL API.
Query the GraphQL API
The get_cf_graphql() function assembles and sends a request to the GraphQL
API. The headers will include the data for authentication.
The payload contains the GraphQL query. In this query, we would like to get a list of the next fields for a given account and time range:
- attack ID
- attack type
- start time
- end time
- mitigation type
- avg, max rate of packets per second
To get started with GraphQL queries, please refer to Querying basics.
The braces used in the GraphQL query are doubled to escape them in Python’s f-string.
GraphQL requires a query to be a single-line text, therefore we should remove all newline symbols before sending it.
Convert the data to CSV
Use a tool such as the open-source
pandas library (pd) to convert a
response from the GraphQL API (JSON) to CSV.
In this example, the convert_to_csv() function does some JSON processing
before conversion — normalizing the data, selecting only the desired data, and
renaming the columns so that they are user-friendly. The function also checks
whether the API responded successfully or we got an error.
The result is output to file in the directory specified by file_dir.