A1666715: Created page with "==Basic Description== The data captured from AEMO is stored in a database built on the HDF5 format. A more detailed description of the structure can be found in section 2.4. T..."

2017-11-27T03:31:16Z

Created page with "==Basic Description== The data captured from AEMO is stored in a database built on the HDF5 format. A more detailed description of the structure can be found in section 2.4. T..."

New page

==Basic Description==
The data captured from AEMO is stored in a database built on the
HDF5 format. A more detailed description of the structure can be
found in section 2.4. The basic premise of our solution was to use
a web scraper to download the data files from AEMO, then extract
the pertinent data from these files and rearrange them into
a more easy to use and logical format. The AEMO data comes
in spreadsheet form, with data from one five minute sample per
spreadsheet or one days data per spreadsheet depending on the
dataset.

===Tracking Time===
Keeping track of the sample times was very important for this
project. As the samples are recorded by AEMO in discrete five
minute (or thirty minute) intervals, we chose to simply use a reference
point and data indices to track the passage of time. This
allowed us to save significant memory space as storing time data
separately would have meant two values per data point rather
than just one. We chose a reference time of 1st January 2015
at 00:00:00 (the start of the new year). This means that midday
on January 1st 2015 has an index of 144. The functions Sample-
ToDatetime and DatetimeToSample perform the conversion between
sample index and time strings.

==Database Modules==
ausp_archivedownloader: Downloads files from the AEMO
archive that have not been previously downloaded.

ausp_configuration: This file is where all user defined settings
are stored. This includes websites, spreadsheet indices,
database structure and attributes.

ausp_databasesetup: Creates a database structure based on
the user defined lists of structure data in the configuration.

ausp_datetimetosample: Converts date-time strings to sample
numbers.

ausp_dbupdater: Searches the AEMO archive files for files that
have not been downloaded. Then downloads and processes
the files into the database.

ausp_filelogger: Logs downloaded file names into the file log
dataset.

ausp_firstrun: This is the first module to run if there is no
database in place. It will setup a database, fill data and set
attributes.

ausp_generaldownloader: Downloads a file that from a list
of filenames

ausp_importdispatchisdata: Performs the processing to transfer
data from AEMO Dispatch spreadsheet into the database

ausp_importpvactualdata: Performs the processing to transfer
data from AEMO PV spreadsheet into the database

ausp_importscadadata: Performs the processing to transfer
data from AEMO SCADA spreadsheet into the database

ausp_parse: Parses the AEMO website for the spreadsheet or
zipfile file names

ausp_rawdatafileprocessor: Processes spreadsheets downloaded
from AEMO, extracting and placing the data into
the database.

ausp_sampletodatetime: Converts a sample index to a datetime
string based on the reference time listed in the configuration
file

ausp_scrapelistoffilesarchive: Creates lists of files that
are on AEMO data dashboard (Archive) for each of the
three datasets used: SCADA, Reports and rooftop PV.

ausp_scrapelistoffilescurrent: Creates lists of files that
are on AEMO data dashboard (Current) for each of the
three datasets used: SCADA, Reports and rooftop PV.

ausp_setgeneratorattributes: Sets generator attributes from
the NEM Registration list spreadsheet provided by AEMO

ausp_setinterconnectorattributes: Sets interconnector attributes
based on the user defined attributes in the configuration
module

ausp_setregionattribute: Sets region attributes based on the
user defined attributes in the configuration module

==GA Module==
The GA module contains all the functions required to run the GA
as we have for this project. The functions are:

generate_parent: Generates a list of specified length containing
randomly generated values between 0 and 1. This is
a real valued parent that can be used in a fitness function
to determine a fitness as required. It is a simple matter to
instead make discrete parents if that is what is required.

generate_population: Generates a population of parents using
the generate_parent function. This population can be
any integer value.

mutate: This function traverses a parent and mutates genes based
on a mutation probability. A mutation involves replacing
the gene with a randomly generated new gene. In this case,
the genes are real valued in the range [0, 1], but could be
modified to discrete as required.

crossover: The crossover function takes a population as its main
input and outputs a new population of the same size that
has undergone the crossover procedure. Crossover requires
pairs of parents and each parent is only considered once
per generation and so the population size must be divisible
by two to ensure all parents are considered. An error will
be raised if the population is not divisible by two.

tournament_selection: This function takes a population then
pairs the parents up at random and compares the fitness
of each pair. The parent with the better fitness is kept
while the other parent is discarded. This function outputs
a population that is half the size of the input population
and so should generally be run twice in each generation to
ensure population size is consistent throughout the simulation.
As the function pairs up parents, the population size
must again be divisible by two.

fitness: This is the key user determined function for the GA.
This function will change depending on the aims, constraints
and requirements of each simulation. The function must
utilise a parent to determine its output or else the GA will
essentially be a fancy random number generator.

main: This function puts all the other functions together to allow
simple operation of the GA. The steps of the main function
are described generally in figure 4.

==Useful Code Fragments==
===Using matplotlib on a Mac===
I ran into an issue using python to plot using matplotlib on macOS.
Importing matplotlib in the following way solved the problem.

<code>
1

2 import matplotlib as mpl

3 mpl.use(’TkAgg’)

4 import matplotlib.pyplot as plt

5 import any_other_packages_as_required
</code>

===Retrieving data from a .csv===
Without using any other packages, the data in a column of a .csv
file can be extracted to a python list using the following:

<code>
1 data = []

2 column_index = 0

3 with open (’file.csv’,’r’) as file:

4      for lines in file:

5           try:

6                x = lines.split(’,’)

7                data.append(float(x[column_index]))

8           except:

9                pass
</code>

===Create a .csv file from a list of lists===
It was often useful to export data from the database to a .csv file
for later retrieval. There are a few ways to do this. The first
method uses the pandas package:

<code>
1 import pandas as pd

2

3 column_data = [colA, colB, colC]

4 labels = [’A’, ’B’, ’C’]

5

6 df = pd.DataFrame(column_data)

7 df = df.transpose() # Remove this line if you want rows of data

8 df.columns = labels

9 pd.DataFrame.to_csv(df, ’filename.csv’, index = False)
</code>

To achieve the same result with default packages:

<code>
1 labels = [’A’, ’B’]

2 colAdata = [some_data]

3 colBdata = [some_other_data]

4 RowsAsStrings = [ ]

5

6 # Convert data to strings

7 for i in range (0,len (longest_column)):

8      RowsAsStrings.append(str(colAdata[i]) + ’,’ + str(colBdata[i]))

9

10 # Write strings to a file

11 with open (’file.csv’, ’w+’) as file:

12      file.writelines(’labels’)

13      for i in data_text:

14           file.writelines(i + ’\n’)
</code>

===Extract data from database===
Extracting data from the database is simple. For example this
code will retrieve all the demand data for the SA1 region:

<code>
1 import h5py

2

3 with h5py.File(’STORPROJ_Database.h5’, ’r’) as f:

4      demand_data = f[’Regions’][’SA1’][’Demand’][...,0]
</code>

Extracting data over a specified time frame:

<code>
1 import h5py

2 from AUSP_DatetimeToSample import DatetimeToSample as d2s

3

4 start_date = d2s (’2016/09/01 00:00:00’)

5 end_date = d2s (’2017/09/01 00:00:00’)

6

7 with h5py.File(’STORPROJ_Database.h5’, ’r’) as f:

8      demand_data = f[’Regions’][’SA1’][’Demand’][start_date:end_date][...,0]
</code>

Extracting and summing several known datasets.
The following code will extract all the data from the generators
HDWF1 and HDWF2 and sum across each sample resulting in an
array of summed generator outputs.

<code>
1 import h5py

2 import numpy as np

3

4 with h5py.File(’STORPROJ_Database.h5’, ’r’) as f:

5      data = np.nansum(zip(f[’DUIDs’][’HDWF1’][...,0],f[’DUIDs’][’HDWF2’][...,0]),1)
</code>

Finally, extracting all data from a range of datasets that have an
attribute that is the same. This example retrieves all generators
that are in the SA1 region with a primary fuel source of "wind".
It also skips past any generators that have no attributes.

<code>
1 import h5py

2 import numpy as np

3

4 with h5py.File(’STORPROJ_Database.h5’,’r’) as f:

5      for generator in f[’DUIDs’]:

6           if f[’DUIDs’][generator].attrs.keys( ) != [ ]:

7                if f[’DUIDs’][generator].attrs[’Fuel Source - Primary’ ] == "Wind":

8                     if f[’DUIDs’][generator].attrs[’Region’] == "SA1":

9                          sa_wind = [np.nansum(x) for x in zip(sa_wind,f[’DUIDs’][generator][...,0])]
</code>

===Plotting and Animating===
Plotting data is simple using matplotlib and the documentation
is full of information and examples on how to fully customise
your plots. The following examples shows how simple it is to get
a quick plot:

<code>
1 import matplotlib as mpl

2 mpl.use(’TkAgg’) # macOS only!

3 import matplotlib.pyplot as plt

4

5 data = [some_data]

6

7 plt.plot(data)

8 plt.show()
</code>

Animating is a bit trickier and I found the examples confusing
at first so here is an example of how to animate some data:

<code>
1 import ma tplot l ib as mpl

2 mpl.use(’TkAgg’)

3 import matplotlib.pyplot as plt

4 import matplotlib.animation as animation

5

6 data = [some_data]

7 fig = plt.figure()

8 axis = fig.add_axes([0.1, 0.1, 0.8, 0.7])

9

10 def animate (frames, kwargs):

11      plt.plot(data[frames])

12

13 ani = animation.FuncAnimation(fig, animate, interval = 1000, frames = len(data), kwargs = None)

14 ani.save(’filename.mp4’)

15 plt.show()
</code>

Basically, the FuncAnimation function creates the animation
using the user written function "animate" to create all the frames
of the animation. The frames parameter is a generator (similar
to a list of a range, e.g. frames = 10 is similar to, but not the
same as frames = range(0, 10)). The interval between frames in
this example is 1000ms, or 1s. The animate function can be as
complex as necessary and as long as the frames input changes
the plot, there will be an animation video at the end. I found
that the video file that is created works better if the frames are
short in duration (20ms or so). This means that an animation will
usually need a large number of frames.

Code Guide - Revision history

A1666715: Created page with "==Basic Description== The data captured from AEMO is stored in a database built on the HDF5 format. A more detailed description of the structure can be found in section 2.4. T..."