Python/Grafana Coding

Korishan

Administrator
Joined
Jan 7, 2017
Messages
7,525
I now have InfluxDB and Grafana running on my laptop with my powermonitor connected to it. I use python to do the polling and import into InfluxDB.

I have a questions here.

1) I'm having an issue with receiving full lines from the serial monitor. I've coded up the python script to recognize the errors and keep going. I get no errors sent to the DB now.
Code:
print "Connecting to", host, "using", database, "&", measurement, "with user:", username

port = '/dev/ttyUSB0'
#baud = 115200
baud = 128000

ser = serial.Serial(port, baud, timeout=0)

client = InfluxDBClient(host=host, username=username, password=password, database=database, use_udp=True, udp_port=8089, )

def send_udp(payload):
  return client.write_points(payload)

cntr = 1
max = 200000
timersleep = float(.125)

refreshUpdate = 125

while cntr <= max:
  lines = ser.readlines()
  if cntr % 5 == 0:
    print "Interation: ", cntr, " of ", max

  starttime = time.time() * 1000
  previous = float(0)

  for line in lines:
    line = line.rstrip("\n\r")
    line = line.split(", ")

    if len(line) == 7:
     millis, volt, current1, watt1, current2, watt2, totalwatt = line
     if previous == 0:
       previous = float(millis)

     newtime = starttime + float(millis) - float(previous)
     previous = float(millis)

     payload = [{ "measurement": measurement,
           "time": int(newtime),
           "fields": {
            "volt": float(volt),
            "current1": float(current1),
            "watt1": float(watt1),
            "current2": float(current2),
            "watt2": float(watt2),
            "totalwatt": float(totalwatt)
            }
          }]
     # print payload
     send_udp(payload)
    else:
     print("=============================================")
     print("Iteration:", cntr)
     print(" - Line not proper length of elements:", len(line))
     print(" - line:", line)
     print("=============================================")

  # We sleep to allow the nano to dump data to the serial buffer
  time.sleep(timersleep)
  cntr += 1

ser.close()

I will randomly get these results:
Code:
Interation: 105365 of 200000
Interation: 105370 of 200000
Interation: 105375 of 200000
Interation: 105380 of 200000
=============================================
('Iteration:', 105383)
(' - Line not proper length of elements:', 5)
(' - line:', ['13411972', '118.21', '7.37', '870.75', ''])
=============================================
=============================================
('Iteration:', 105384)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.85', '1991.74', '2862.49'])
=============================================
Interation: 105385 of 200000
Interation: 105390 of 200000
=============================================
('Iteration:', 105390)
(' - Line not proper length of elements:', 5)
(' - line:', ['13412865', '118.14', '7.27', '858.34', ''])
=============================================
=============================================
('Iteration:', 105391)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.87', '1993.34', '2851.68'])
=============================================
It's like the arduino nano isn't sending all the data, but not sure why. I was at a baud of 115200, and changed to 128000 to see if that made any difference. It didn't decrease or increase the occurrences, that I could tell. I'm currently not counting the errors or figuring how often they occur

Code:
void setup()
{
 Serial.begin(128000); //115200);
.......
}
void loop()
{
 String buf;
 unsigned long currentTimer = millis();
 static unsigned long lastUpdate;
 static unsigned long lastSerialUpdate;
 static unsigned long counter;

 float currentLine1 = emon1.Irms;   //extract Irms into Variable
 float voltageLine1 = emon1.Vrms;  //extract Vrms into Variable
 float currentLine2 = emon2.Irms;   //extract Irms into Variable
 float voltageLine2 = emon2.Vrms;  //extract Vrms into Variable
 float totalWatts;
 totalWatts = voltageLine1 * currentLine1;
 totalWatts += voltageLine2 * currentLine2;

 if (currentTimer - lastSerialUpdate > refreshSerial)
 {
  buf = String(currentTimer) + ", " +
     String(voltageLine1) + ", " +
     String(currentLine1) + ", " +
     String(currentLine1 * voltageLine1) + ", " +
     String(currentLine2) + ", " +
     String(currentLine2 * voltageLine1) + ", " +
     String((currentLine1 * voltageLine1) + (currentLine2 * voltageLine1));

//     String(voltageLine2) + ", " +

  Serial.println(buf);
lastSerialUpdate = currentTimer;
 }
....
}
So what needs to be changed to make it run with less errors? I'm assuming the nano is sending "full" lines of text, as I don't see why it wouldn't. It just seems the serial read in the script isn't getting all the data for some reason.
2) In Grafana, I have some graphs for Volts/Current/Watts. What I would like to do is have a few queries that display the kWh over time. The time I would like to be static [1h,12h, 24h, 1month, 6month, 1yr] for example, or possibly event he ability to change that on the fly using only 1 panel. And also include the costs of power over time.
My current results for the past day look like this:

image_rfrsfj.jpg

I'd like to do something like this guy did:https://fablab.dyn.nerd2nerd.org/grafana/d/000000001/fablab-power?refresh=1m&orgId=1
 
Sean said:
Why not ask the originators, you received some helpful advise with the last question you asked them ?

https://community.openenergymonitor.org

If you've got an instance of emon running correctly and reporting, why not use the OEM EmonCMS for visualisation ? - it might not be as pretty as Grafana but you'll have it working in minutes.

https://emoncms.org/site/home

There's nothing wrong with the emon libs or it's reporting. This is beyond that portion of it. If there was a problem with the emon lib, I "would" go ask them. But it's working as it should. This is a python/serial issue on my laptop.
I want to fix "my" code and design. That's why I'm asking here. Yes I did get some helpful info over there. Asking here for help will also yield me some good results as I know there are really good python/grafana users here. Plus I'm far more active on this forum than emons forums
 
Sean the error is not about Emon or influx. Its about getting the data in first place.



Korishan:
1. Dont have time to read it through now but il take a look later unless someone solved it
But it looks like you are not receiving the bginning of the data? Are you sending it to fast? Lower the speed of transmit and also the interval (I spent 10 seconds looking)


2. You need to group and calculate the value. as you can see he have set static intervals of 1day and 365 days and so forth. That is one way to do it and calculate it in grafana itself reading data from influx. This though take some cpu. (Check the raspberry solar sis thread from me with examples)
You can also have influx calulate this in the backround with set number of minutes between and just store the value that you then read ou. More examples in my thread about that.
 
Korishan said:
Code:
=============================================
('Iteration:', 105383)
(' - Line not proper length of elements:', 5)
(' - line:', ['13411972', '118.21', '7.37', '870.75', ''])
=============================================
=============================================
('Iteration:', 105384)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.85', '1991.74', '2862.49'])
=============================================
Interation: 105385 of 200000
Interation: 105390 of 200000
=============================================
('Iteration:', 105390)
(' - Line not proper length of elements:', 5)
(' - line:', ['13412865', '118.14', '7.27', '858.34', ''])
=============================================
=============================================
('Iteration:', 105391)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.87', '1993.34', '2851.68'])
=============================================

This happens to me in many other languages where you work with serial ports, when the serial port data_received event triggers doesn't means that all the data is received, you must control by your own if all the data is received, with a parser, or a endline. In your case you don't event work with events, you only check constantly in a loop if there is data.


If I'm not wrong reading your code you do a while 200000 times checking if some data from arduino is received, trying to parse the line as 7 fields of data, these times you get errors is because not the whole line is received so it process some data in one cycle and the rest in the next 105383-105384 and 105390-105391.


I see you receive newline and return at the end of line, or at least you clean it
Code:
line = line.rstrip("\n\r")


You must cycle and store all data received in a buffer until you get \r\n and then split the data but not before you are sure the full line is received.


Regards
 
Are you using TTL to USB? If so you're probably picking up interference. How long is the cable and is it properly grounded? I am having the same issue with my BMS and for now I'm just ignoring some of the data. RS232 and RS485 serial are much more reliable.

Just print out the raw data to the console and you can see if the data is corrupt.

*edit* Also you get better results when you lower the baud rates. Remember the old days of modem dialups...
 
not2bme said:
Are you using TTL to USB? If so you're probably picking up interference. How long is the cable and is it properly grounded? I am having the same issue with my BMS and for now I'm just ignoring some of the data. RS232 and RS485 serial are much more reliable.

Just print out the raw data to the console and you can see if the data is corrupt.

No interferences, just fragmented data

Code:
('Iteration:', 105383)
(' - Line not proper length of elements:', 5)
(' - line:', ['13411972', '118.21', '7.37', '870.75', ''])
=============================================
=============================================
('Iteration:', 105384)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.85', '1991.74', '2862.49'])
=============================================

First data received,
13411972,118.21,7.37,870.75,
Spliting with , char gets 5 strings, the last empty

Second data
16.85,1991.74,2862.49

All seems to be OK with expected data

millis, volt, current1, watt1, current2, watt2, totalwatt
 
Thanks guys for looking it over.

not2bme: I'm connecting using USB cable directly between the laptop and the arduino nano. The cable is about 1 meter. So as jesusangel says, there isn't any interference. The data just gets chopped off some how.
jesusangel: yes, I'm reading each set of "lines()" that have been stored in the buffer. Each line ends with a \n\r. The data is stored in the "lines" variable as an array of strings. Then I read each "line" of "lines". Each "line" is stripped of the '\n\r' so that it'll get stored into the variables properly to be sent to the influxdb
There is supposed to be 7 entries in each line string (6 commas). In the 2 entries you quoted, there's only 5 and 3, respectively.

I hadn't thought of slowing down the baud rate. I'll give that a shot and see what happens.
 
daromer said:
Korishan:
1. Dont have time to read it through now but il take a look later unless someone solved it
But it looks like you are not receiving the bginning of the data? Are you sending it to fast? Lower the speed of transmit and also the interval (I spent 10 seconds looking)

2. You need to group and calculate the value. as you can see he have set static intervals of 1day and 365 days and so forth. That is one way to do it and calculate it in grafana itself reading data from influx. This though take some cpu. (Check the raspberry solar sis thread from me with examples)
You can also have influx calulate this in the backround with set number of minutes between and just store the value that you then read ou. More examples in my thread about that.

The totalwatts is stored in the DB, so that I don't have to worry about adding watt1 and watt2 fields in the displays. I knew I'd need to calculate the kWh over time, so that's why I did that so I didn't have to worry about it later ;)
I looked closer at the guys json code for the panel, and it looks like he's not using a db some how. Which is partly why it's confusing. I'm not sure how he's grabbing the data. InfluxDB calls are a bit different from the SQL code I learned when dealing with MySQL and MS Access. So the tricks I'd do there don't work here.
 
Ok, I was watching the "Errors" really closely this time. I noticed a breakthrough.

In all of the errors, they seem to always comes in pairs. Because I also include the 'Iteration' counter, I've noticed they are always successive.

What appears to be happening, is the line is being split in half some how during serial.read.

for example:
Code:
Iteration: 21859
 - Line not proper length of elements: 5
 - line: ['2781924', '118.84', '7.86', '933.66', '1']
=============================================
=============================================
Iteration: 21860
 - Line not proper length of elements: 3
 - line: ['2.18', '1446.90', '2380.56']
=============================================
=============================================
Iteration: 21916
 - Line not proper length of elements: 5
 - line: ['2789173', '120.19', '7.86', '944.63', '1']
=============================================
=============================================
Iteration: 21917
 - Line not proper length of elements: 3
 - line: ['2.39', '1488.74', '2433.36']
=============================================

21859/21850, 21916/21917. They are in pairs. So I took a closer look. If I take 59 & 60 'line' data, and read them together, I get:
['2781924', '118.84', '7.86', '933.66', '1'] ['2.18', '1446.90', '2380.56']

Here, we can see there are 8 entries. But are there really? If I put them together in a way I know the data is valid (but reading the data before/after the errors), I get this:
['2781924', '118.84', '7.86', '933.66', '12.18', '1446.90', '2380.56']

There's now 7 entries. The 5th entry is getting cut in half somehow. I noticed all the other errors were doing the same thing.

This is really odd. And I even slowed the baud down to 57600, and it seems there's actually more errors occurring now
 
The output you get is already a parsed list. So you're not seeing everything. Print the raw output from serial.readline() and see what you get. It looks like
4 out of 10 is bad so it's easy to look at the raw output to see what's wrong.

You can do the same with the arduino and send to the serial console the data. See the raw data.

TTL is very finicky, since it goes from 0 to +3V? for the arduino. 0V is just noisy. RS232 swings -12V to +12V for a PC. There's no mistaking what's low and what's high.
 
Korishan said:
Ok, I was watching the "Errors" really closely this time. I noticed a breakthrough.

In all of the errors, they seem to always comes in pairs. Because I also include the 'Iteration' counter, I've noticed they are always successive.

What appears to be happening, is the line is being split in half some how during serial.read.

Well, it seems like you notice what I told you in my previous explanation

jesusangel said:
If I'm not wrong reading your code you do a while 200000 times checking if some data from arduino is received, trying to parse the line as 7 fields of data, these times you get errors is because not the whole line is received so it process some data in one cycle and the rest in the next 105383-105384 and 105390-105391.

, you have a loop that is executed without any control just a delay of 0.125 seconds, so you try to read the line when is wasn't completely received, you read half of the line in one cycle y and the rest in the next, that is what you call errors in pairs, readline should wait for a \n but it doesn't do it, and it is a very common error. No matter if you set a lower baudrate.

Print the raw output bytes from serial asnot2bme said and you will see in first error there is no \n (0A byte)at the end but it is in the second error.

This error is not so aleatory as you think, in you case readline triggers in all the errors that you shown when there are 28 byte in serial buffer.

I don't usually code in python but you can use serial.read_until('\n',maxbytes) maybe you would get betters results, or you candefinitely use serial.read instead of serial.readline and be you who parse the bytes, store every byte in a buffer until you get \r\n, (that's a weird thing in you code, usualy end of line is \r\n it seems you use \n\r, change it in arduino code too).
 
Jesusangel is right. You need to handle the Read bettet and thats what you get those results
 
Hmm, interesting. Thanks.
I figured since the arduino sends data as a whole line, and the pyserial is reading in a whole line, there'd be no issue with the loops.
serial.readlines() is supposed to read all lines including the \r\n. That's where it's supposed to stop at. The arduino code dumps the lines and attaches the \r\n to each serial print. So that's why I was a bit confused on the results.
I don't recall seeing those errors when did a straight dump when I was testing. I will have to revisit that.

Now, for the Grafana portion. What is the syntax for getting the consumed power over time? I get I need to override TimeRange. But what about the rest of it? I put in the kWh for 1Hr, and I think that one worked. Then I put in the syntax for 24h, or 1d, and it tells me the value "mega-watts" ?!?!!?
Code:
{
  "type": "singlestat",
  "title": "Consumption (1D)",
  "gridPos": {
    "x": 6,
    "y": 0,
    "w": 6,
    "h": 8
  },
  "id": 9,
  "datasource": "Power Monitor",
  "targets": [
    {
      "policy": "default",
      "resultFormat": "time_series",
      "orderByTime": "ASC",
      "tags": [],
      "groupBy": [
        {
          "type": "time",
          "params": [
            "$__interval"
          ]
        },
        {
          "type": "fill",
          "params": [
            "null"
          ]
        }
      ],
      "select": [
        [
          {
            "type": "field",
            "params": [
              "totalwatt"
            ]
          },
          {
            "type": "sum",
            "params": []
          }
        ]
      ],
      "refId": "A",
      "measurement": "powermon",
      "query": "SELECT sum(\"totalwatt\") FROM \"powermon\" WHERE $timeFilter GROUP BY time($__interval) fill(null)",
      "rawQuery": false
    }
  ],
  "links": [],
  "maxDataPoints": 100,
  "interval": null,
  "cacheTimeout": null,
  "format": "watt",
  "prefix": "",
  "postfix": "",
  "nullText": null,
  "valueMaps": [
    {
      "value": "null",
      "op": "=",
      "text": "N/A"
    }
  ],
  "mappingTypes": [
    {
      "name": "value to text",
      "value": 1
    },
    {
      "name": "range to text",
      "value": 2
    }
  ],
  "rangeMaps": [
    {
      "from": "null",
      "to": "null",
      "text": "N/A"
    }
  ],
  "mappingType": 1,
  "nullPointMode": "connected",
  "valueName": "avg",
  "prefixFontSize": "50%",
  "valueFontSize": "100%",
  "postfixFontSize": "50%",
  "thresholds": "",
  "colorBackground": false,
  "colorValue": false,
  "colors": [
    "#299c46",
    "rgba(237, 129, 40, 0.89)",
    "#d44a3a"
  ],
  "sparkline": {
    "show": true,
    "full": true,
    "lineColor": "rgb(31, 120, 193)",
    "fillColor": "rgba(31, 118, 189, 0.18)"
  },
  "gauge": {
    "show": false,
    "minValue": 0,
    "maxValue": 100,
    "thresholdMarkers": true,
    "thresholdLabels": false
  },
  "tableColumn": "",
  "timeFrom": "1d"
}
Was going to show just the query syntax, but that doesn't show "everything". So there's the json code.
When I load the data into the influxdb, I automatically add fields 'watt1' and 'watt2' to get the total. This way I don't have to add extra calculations/work-load to the laptop afterwards.

*just got up and haven't had coffee yet, please excuse if the above if it seems a little confusing/off*
 
Look at daromer's post on his rasberry pi in the build section. Others have just posted how to calculate the kwh. I used to use the integral function but it seems others found another way with a nested select statement. It should be in the most recent posts.
 
Yeah in that thread you have 2 Good example.

Egen reading Serial port you need to Read untill new Line. Not based on a timefrme...


IF data then save in buffer untill new Line found
Hej. Newline then parse data
 
I went through Daromer's Raspi thread, and I didn't find where there was a discussion on the kWh conundrum I'm needing. There was a link, but it didn't help me any (unless I missed something in there). It just discussed adding two fields together.
 
Korishan:

One way: https://secondlifestorage.com/t-Esperyd´s-Raspberry-PI-project?pid=37129#pid37129
Another: https://secondlifestorage.com/t-Esperyd´s-Raspberry-PI-project?pid=37043#pid37043

Basically what you need to do is group every wattage calculation. If you know you have an average of 1kw during 1 hour you also know its 1kWh during that hour. So basically for each minute values you know that if you have a use of 1,2kw you know that the minute then produced 1,2/60 = 0,02kWh during that time. You do that for every minute and then you have the total kWh during that hour.

Im not infront of the correct computer so cannot give you the code I used.
 
I used the query that not2bme posted, the first link, and it's basically what I figured out already:

SELECT mean("totalwatt") FROM "powermon" WHERE $timeFilter GROUP BY time(1h)

That works for 1h. But if I change it to 24h, the kWh doesn't change much. For example, it goes from 1.256kWh to 1.245kWh. If I also add in the time override and set the GROUP BY to 1h, and override to 24h, I get 1.601kWh. If I GROUP BY 24h with override of 24h: 1.617kWh.

I'm obviously missing something here. Is the above only for the average for the time specified and something different for outputting 24hrs?

With oldserg's code:
Code:
5. cq_kWh_batt_discharg
CREATE CONTINUOUS QUERY cq_kWh_batt_discharg ON powerwall RESAMPLE FOR 1h BEGIN SELECT integral(kwh_bco, 1h) / 1000 AS kWh_batt_discharg INTO powerwall.autogen.kWh_ALL FROM (SELECT mean(battery_discharge_current) * mean(battery_voltage) AS kwh_bco FROM powerwall.autogen.pip_query_general_status GROUP BY time(2m) fill(0)) GROUP BY time(20m) TZ('Europe/Rome') END

6. cq_kWh_PV_batt_charg
CREATE CONTINUOUS QUERY cq_kWh_PV_batt_charg ON powerwall RESAMPLE FOR 1h BEGIN SELECT integral(kwh_PV_bat, 1h) / 1000 AS kWh_PV_batt_charg INTO powerwall.autogen.kWh_ALL FROM (SELECT mean(battery_charging_current) * mean(battery_voltage_from_scc) AS kwh_PV_bat FROM powerwall.autogen.pip_query_general_status WHERE pv_input_current_for_battery > 0 GROUP BY time(2m) fill(0)) GROUP BY time(20m) TZ('Europe/Rome') END

I'm a little lost here. Maybe just an explanation of what's going on here would help. He mentions needing to create a new table to use the those.
 
Hmmm, the new code for the python script is not working as intended.

I changed:
Code:
 lines = ser.readlines()
  for line in lines:
    line = ser.readline()
    line = line.rstrip("\r\n")
    line = line.split(", ")
and now I just do:
Code:
        line = ser.readline()
        line = line.split(", ")

But, I have ran into another situation. It seemed like the code was running really slowly. So I added in another timer to see how long each iteration was taking.
The results are:
Code:
StartTime: 1.53832745413e+12 Endtime: 1.53832745441e+12 Different: 279.645996094
StartTime: 1.53832745441e+12 Endtime: 1.5383274546e+12 Different: 196.064941406
StartTime: 1.5383274546e+12 Endtime: 1.5383274548e+12 Different: 194.843261719
StartTime: 1.5383274548e+12 Endtime: 1.53832745507e+12 Different: 275.984863281
StartTime: 1.53832745507e+12 Endtime: 1.53832745527e+12 Different: 195.929931641
Interation: 985 of 1000
StartTime: 1.53832745527e+12 Endtime: 1.53832745547e+12 Different: 196.938964844
StartTime: 1.53832745547e+12 Endtime: 1.53832745597e+12 Different: 502.886962891
StartTime: 1.53832745597e+12 Endtime: 1.53832745625e+12 Different: 278.920166016
StartTime: 1.53832745625e+12 Endtime: 1.53832745645e+12 Different: 196.957763672
StartTime: 1.53832745645e+12 Endtime: 1.53832745663e+12 Different: 187.982177734
Interation: 990 of 1000
StartTime: 1.53832745663e+12 Endtime: 1.53832745714e+12 Different: 509.819091797
StartTime: 1.53832745714e+12 Endtime: 1.53832745742e+12 Different: 280.929931641
StartTime: 1.53832745743e+12 Endtime: 1.53832745762e+12 Different: 198.078857422
StartTime: 1.53832745762e+12 Endtime: 1.53832745782e+12 Different: 192.843994141
StartTime: 1.53832745782e+12 Endtime: 1.53832745811e+12 Different: 294.964111328
Each line is an iteration. It's taking far longer to run this code, than it was the previous code. serial.readline() reads till it finds a newline feed. But apparently I'm missing something here as I'm not getting all the data. The previous code, even with it's errors, I was getting almost 90 successful reads a second.
Code:
StartTime: 1.53832794366e+12 Endtime: 1.53832794379e+12 Different: 125.501708984
StartTime: 1.53832794379e+12 Endtime: 1.53832794391e+12 Different: 128.621826172
StartTime: 1.53832794391e+12 Endtime: 1.53832794404e+12 Different: 125.321777344
StartTime: 1.53832794404e+12 Endtime: 1.53832794417e+12 Different: 126.493896484
StartTime: 1.53832794417e+12 Endtime: 1.53832794429e+12 Different: 125.306152344
Interation:  35  of  200000
StartTime: 1.53832794429e+12 Endtime: 1.53832794442e+12 Different: 125.291015625
StartTime: 1.53832794442e+12 Endtime: 1.53832794454e+12 Different: 125.329833984
StartTime: 1.53832794454e+12 Endtime: 1.53832794467e+12 Different: 126.826904297
StartTime: 1.53832794467e+12 Endtime: 1.53832794479e+12 Different: 125.281005859
StartTime: 1.53832794479e+12 Endtime: 1.53832794492e+12 Different: 126.439208984
Interation:  40  of  200000
StartTime: 1.53832794492e+12 Endtime: 1.53832794505e+12 Different: 125.260253906
StartTime: 1.53832794505e+12 Endtime: 1.53832794517e+12 Different: 127.439941406
StartTime: 1.53832794517e+12 Endtime: 1.5383279453e+12 Different: 128.583740234
StartTime: 1.5383279453e+12 Endtime: 1.53832794543e+12 Different: 125.509033203
StartTime: 1.53832794543e+12 Endtime: 1.53832794556e+12 Different: 128.573974609
Interation:  45  of  200000
StartTime: 1.53832794556e+12 Endtime: 1.53832794568e+12 Different: 125.311035156
StartTime: 1.53832794568e+12 Endtime: 1.53832794581e+12 Different: 128.396972656
StartTime: 1.53832794581e+12 Endtime: 1.53832794594e+12 Different: 125.503173828
StartTime: 1.53832794594e+12 Endtime: 1.53832794606e+12 Different: 128.58203125
1/8 of a second compared to 1/4 of a second, or 1/2 as some of the reads were.
 
Back
Top