Editing Code Guide (section)

==Useful Code Fragments==
===Using matplotlib on a Mac===
I ran into an issue using python to plot using matplotlib on macOS.
Importing matplotlib in the following way solved the problem.


<code>
1

2 import matplotlib as mpl

3 mpl.use(’TkAgg’)

4 import matplotlib.pyplot as plt

5 import any_other_packages_as_required
</code>


===Retrieving data from a .csv===
Without using any other packages, the data in a column of a .csv
file can be extracted to a python list using the following:


<code>
1 data = []

2 column_index = 0

3 with open (’file.csv’,’r’) as file:

4 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for lines in file:

5 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;try:

6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;x = lines.split(’,’)

7 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data.append(float(x[column_index]))

8 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;except:

9 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pass
</code>


===Create a .csv file from a list of lists===
It was often useful to export data from the database to a .csv file
for later retrieval. There are a few ways to do this. The first
method uses the pandas package:


<code>
1 import pandas as pd

2

3 column_data = [colA, colB, colC]

4 labels = [’A’, ’B’, ’C’]

5

6 df = pd.DataFrame(column_data)

7 df = df.transpose() # Remove this line if you want rows of data

8 df.columns = labels

9 pd.DataFrame.to_csv(df, ’filename.csv’, index = False)
</code>


To achieve the same result with default packages:


<code>
1 labels = [’A’, ’B’]

2 colAdata = [some_data]

3 colBdata = [some_other_data]

4 RowsAsStrings = [ ]

5

6 # Convert data to strings

7 for i in range (0,len (longest_column)):

8 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;RowsAsStrings.append(str(colAdata[i]) + ’,’ + str(colBdata[i]))

9

10 # Write strings to a file

11 with open (’file.csv’, ’w+’) as file:

12 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file.writelines(’labels’)

13 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for i in data_text:

14 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;file.writelines(i + ’\n’)
</code>


===Extract data from database===
Extracting data from the database is simple. For example this
code will retrieve all the demand data for the SA1 region:


<code>
1 import h5py

2

3 with h5py.File(’STORPROJ_Database.h5’, ’r’) as f:

4 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;demand_data = f[’Regions’][’SA1’][’Demand’][...,0]
</code>


Extracting data over a specified time frame:


<code>
1 import h5py

2 from AUSP_DatetimeToSample import DatetimeToSample as d2s

3

4 start_date = d2s (’2016/09/01 00:00:00’)

5 end_date = d2s (’2017/09/01 00:00:00’)

6

7 with h5py.File(’STORPROJ_Database.h5’, ’r’) as f:

8 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;demand_data = f[’Regions’][’SA1’][’Demand’][start_date:end_date][...,0]
</code>


Extracting and summing several known datasets.
The following code will extract all the data from the generators
HDWF1 and HDWF2 and sum across each sample resulting in an
array of summed generator outputs.


<code>
1 import h5py

2 import numpy as np

3

4 with h5py.File(’STORPROJ_Database.h5’, ’r’) as f:

5 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;data = np.nansum(zip(f[’DUIDs’][’HDWF1’][...,0],f[’DUIDs’][’HDWF2’][...,0]),1)
</code>


Finally, extracting all data from a range of datasets that have an
attribute that is the same. This example retrieves all generators
that are in the SA1 region with a primary fuel source of "wind".
It also skips past any generators that have no attributes.


<code>
1 import h5py

2 import numpy as np

3

4 with h5py.File(’STORPROJ_Database.h5’,’r’) as f:

5 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;for generator in f[’DUIDs’]:

6 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if f[’DUIDs’][generator].attrs.keys( ) != [ ]:

7 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if f[’DUIDs’][generator].attrs[’Fuel Source - Primary’ ] == "Wind":

8 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if f[’DUIDs’][generator].attrs[’Region’] == "SA1":

9 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sa_wind = [np.nansum(x) for x in zip(sa_wind,f[’DUIDs’][generator][...,0])]
</code>


===Plotting and Animating===
Plotting data is simple using matplotlib and the documentation
is full of information and examples on how to fully customise
your plots. The following examples shows how simple it is to get
a quick plot:


<code>
1 import matplotlib as mpl

2 mpl.use(’TkAgg’) # macOS only!

3 import matplotlib.pyplot as plt

4

5 data = [some_data]

6

7 plt.plot(data)

8 plt.show()
</code>


Animating is a bit trickier and I found the examples confusing
at first so here is an example of how to animate some data:


<code>
1 import ma tplot l ib as mpl

2 mpl.use(’TkAgg’)

3 import matplotlib.pyplot as plt

4 import matplotlib.animation as animation

5

6 data = [some_data]

7 fig = plt.figure()

8 axis = fig.add_axes([0.1, 0.1, 0.8, 0.7])

9

10 def animate (frames, kwargs):

11 &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;plt.plot(data[frames])

12

13 ani = animation.FuncAnimation(fig, animate, interval = 1000, frames = len(data), kwargs = None)

14 ani.save(’filename.mp4’)

15 plt.show()
</code>


Basically, the FuncAnimation function creates the animation
using the user written function "animate" to create all the frames
of the animation. The frames parameter is a generator (similar
to a list of a range, e.g. frames = 10 is similar to, but not the
same as frames = range(0, 10)). The interval between frames in
this example is 1000ms, or 1s. The animate function can be as
complex as necessary and as long as the frames input changes
the plot, there will be an animation video at the end. I found
that the video file that is created works better if the frames are
short in duration (20ms or so). This means that an animation will
usually need a large number of frames.