Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python/Grafana Coding
#1
I now have InfluxDB and Grafana running on my laptop with my powermonitor connected to it. I use python to do the polling and import into InfluxDB.

I have a questions here.

1) I'm having an issue with receiving full lines from the serial monitor. I've coded up the python script to recognize the errors and keep going. I get no errors sent to the DB now.
Code:
print "Connecting to", host, "using", database, "&", measurement, "with user:", username

port = '/dev/ttyUSB0'
#baud = 115200
baud = 128000

ser = serial.Serial(port, baud, timeout=0)

client = InfluxDBClient(host=host, username=username, password=password, database=database, use_udp=True, udp_port=8089, )

def send_udp(payload):
    return client.write_points(payload)

cntr = 1
max = 200000
timersleep = float(.125)

refreshUpdate = 125

while cntr <= max:
    lines = ser.readlines()
    if cntr % 5 == 0:
        print "Interation: ", cntr, " of ", max

    starttime = time.time() * 1000
    previous = float(0)

    for line in lines:
        line = line.rstrip("\n\r")
        line = line.split(", ")

        if len(line) == 7:
           millis, volt, current1, watt1, current2, watt2, totalwatt = line
           if previous == 0:
               previous = float(millis)

           newtime = starttime + float(millis) - float(previous)
           previous = float(millis)

           payload = [{ "measurement": measurement,
                       "time": int(newtime),
                       "fields": {
                        "volt": float(volt),
                        "current1": float(current1),
                        "watt1": float(watt1),
                        "current2": float(current2),
                        "watt2": float(watt2),
                        "totalwatt": float(totalwatt)
                        }
                    }]
          # print payload
           send_udp(payload)
        else:
          print("=============================================")
          print("Iteration:", cntr)
          print(" - Line not proper length of elements:", len(line))
          print(" - line:", line)
          print("=============================================")

    # We sleep to allow the nano to dump data to the serial buffer
    time.sleep(timersleep)
    cntr += 1

ser.close()

I will randomly get these results:
Code:
Interation:  105365  of  200000
Interation:  105370  of  200000
Interation:  105375  of  200000
Interation:  105380  of  200000
=============================================
('Iteration:', 105383)
(' - Line not proper length of elements:', 5)
(' - line:', ['13411972', '118.21', '7.37', '870.75', ''])
=============================================
=============================================
('Iteration:', 105384)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.85', '1991.74', '2862.49'])
=============================================
Interation:  105385  of  200000
Interation:  105390  of  200000
=============================================
('Iteration:', 105390)
(' - Line not proper length of elements:', 5)
(' - line:', ['13412865', '118.14', '7.27', '858.34', ''])
=============================================
=============================================
('Iteration:', 105391)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.87', '1993.34', '2851.68'])
=============================================
It's like the arduino nano isn't sending all the data, but not sure why. I was at a baud of 115200, and changed to 128000 to see if that made any difference. It didn't decrease or increase the occurrences, that I could tell. I'm currently not counting the errors or figuring how often they occur

Code:
void setup()
{
  Serial.begin(128000); //115200);
.......
}
void loop()
{
  String buf;
  unsigned long currentTimer = millis();
  static unsigned long lastUpdate;
  static unsigned long lastSerialUpdate;
  static unsigned long counter;

  float currentLine1 = emon1.Irms;      //extract Irms into Variable
  float voltageLine1 = emon1.Vrms;    //extract Vrms into Variable
  float currentLine2 = emon2.Irms;      //extract Irms into Variable
  float voltageLine2 = emon2.Vrms;    //extract Vrms into Variable
  float totalWatts;
  totalWatts = voltageLine1 * currentLine1;
  totalWatts += voltageLine2 * currentLine2;

  if (currentTimer - lastSerialUpdate > refreshSerial)
  {
    buf = String(currentTimer) + ", " +
          String(voltageLine1) + ", " +
          String(currentLine1) + ", " +
          String(currentLine1 * voltageLine1) + ", " +
          String(currentLine2) + ", " +
          String(currentLine2 * voltageLine1) + ", " +
          String((currentLine1 * voltageLine1) + (currentLine2 * voltageLine1));

//          String(voltageLine2) + ", " +

    Serial.println(buf);
lastSerialUpdate = currentTimer;
  }
....
}
So what needs to be changed to make it run with less errors? I'm assuming the nano is sending "full" lines of text, as I don't see why it wouldn't. It just seems the serial read in the script isn't getting all the data for some reason.
2) In Grafana, I have some graphs for Volts/Current/Watts. What I would like to do is have a few queries that display the kWh over time. The time I would like to be static [1h, 12h, 24h, 1month, 6month, 1yr] for example, or possibly event he ability to change that on the fly using only 1 panel. And also include the costs of power over time.
My current results for the past day look like this:

I'd like to do something like this guy did: https://fablab.dyn.nerd2nerd.org/grafana...1m&orgId=1
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage; VALID email req'd)
Reply
#2
Why not ask the originators, you received some helpful advise with the last question you asked them ?

https://community.openenergymonitor.org

If you've got an instance of emon running correctly and reporting, why not use the OEM EmonCMS for visualisation ? - it might not be as pretty as Grafana but you'll have it working in minutes.

https://emoncms.org/site/home
Reply
#3
(09-28-2018, 04:02 PM)Sean Wrote: Why not ask the originators, you received some helpful advise with the last question you asked them ?

https://community.openenergymonitor.org

If you've got an instance of emon running correctly and reporting, why not use the OEM EmonCMS for visualisation ? - it might not be as pretty as Grafana but you'll have it working in minutes.

https://emoncms.org/site/home

There's nothing wrong with the emon libs or it's reporting. This is beyond that portion of it. If there was a problem with the emon lib, I "would" go ask them. But it's working as it should. This is a python/serial issue on my laptop.
I want to fix "my" code and design. That's why I'm asking here. Yes I did get some helpful info over there. Asking here for help will also yield me some good results as I know there are really good python/grafana users here. Plus I'm far more active on this forum than emons forums
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage; VALID email req'd)
Reply
#4
Sean the error is not about Emon or influx. Its about getting the data in first place.



Korishan:
1. Dont have time to read it through now but il take a look later unless someone solved it
But it looks like you are not receiving the bginning of the data? Are you sending it to fast? Lower the speed of transmit and also the interval (I spent 10 seconds looking)


2. You need to group and calculate the value. as you can see he have set static intervals of 1day and 365 days and so forth. That is one way to do it and calculate it in grafana itself reading data from influx. This though take some cpu. (Check the raspberry solar sis thread from me with examples)
You can also have influx calulate this in the backround with set number of minutes between and just store the value that you then read ou. More examples in my thread about that.
Korishan likes this post
The Ultimate DIY Solar and build place
YouTube / Forum system setup / My webpage  Diy Tech & Repairs

Current: 10kW Mpp Hybrid | 4kW PIP4048 | 2x PCM60x | 83kWh LiFePo4 | 10kWh 14s 18650 |  66*260W Poly
Upcoming: 14S 18650~30kWh | Automatic trip breakers, and alot more
Reply
#5
(09-28-2018, 03:48 PM)Korishan Wrote:
Code:
=============================================
('Iteration:', 105383)
(' - Line not proper length of elements:', 5)
(' - line:', ['13411972', '118.21', '7.37', '870.75', ''])
=============================================
=============================================
('Iteration:', 105384)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.85', '1991.74', '2862.49'])
=============================================
Interation:  105385  of  200000
Interation:  105390  of  200000
=============================================
('Iteration:', 105390)
(' - Line not proper length of elements:', 5)
(' - line:', ['13412865', '118.14', '7.27', '858.34', ''])
=============================================
=============================================
('Iteration:', 105391)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.87', '1993.34', '2851.68'])
=============================================

This happens to me in many other languages where you work with serial ports,  when the serial port data_received event triggers doesn't means that all the data is received, you must control by your own if all the data is received, with a parser, or a endline. In your case you don't event work with events, you only check constantly in a loop if there is data.


If I'm not wrong reading your code you do a while 200000 times checking if some data from arduino is received, trying to parse the line as 7 fields of data, these times you get errors is because not the whole line is received so it process some data in one cycle and the rest in the next 105383-105384  and 105390-105391.


I see you receive newline and return at the end of line, or at least you clean it 
Code:
line = line.rstrip("\n\r")


You must cycle and store all data received in a buffer until you get \r\n and then split the data but not before you are sure the full line is received.


Regards
completelycharged likes this post
Reply
#6
Are you using TTL to USB? If so you're probably picking up interference. How long is the cable and is it properly grounded? I am having the same issue with my BMS and for now I'm just ignoring some of the data. RS232 and RS485 serial are much more reliable.

Just print out the raw data to the console and you can see if the data is corrupt.

*edit* Also you get better results when you lower the baud rates. Remember the old days of modem dialups...
Reply
#7
(09-28-2018, 08:21 PM)not2bme Wrote: Are you using TTL to USB? If so you're probably picking up interference. How long is the cable and is it properly grounded? I am having the same issue with my BMS and for now I'm just ignoring some of the data. RS232 and RS485 serial are much more reliable.

Just print out the raw data to the console and you can see if the data is corrupt.

No interferences, just fragmented data

Code:
('Iteration:', 105383)
(' - Line not proper length of elements:', 5)
(' - line:', ['13411972', '118.21', '7.37', '870.75', ''])
=============================================
=============================================
('Iteration:', 105384)
(' - Line not proper length of elements:', 3)
(' - line:', ['16.85', '1991.74', '2862.49'])
=============================================

First data received, 
13411972,118.21,7.37,870.75,
Spliting with , char gets 5 strings, the last empty

Second data
16.85,1991.74,2862.49

All seems to be OK with expected data

millis, volt, current1, watt1, current2, watt2, totalwatt
Reply
#8
Thanks guys for looking it over.

not2bme: I'm connecting using USB cable directly between the laptop and the arduino nano. The cable is about 1 meter. So as jesusangel says, there isn't any interference. The data just gets chopped off some how.
jesusangel: yes, I'm reading each set of "lines()" that have been stored in the buffer. Each line ends with a \n\r. The data is stored in the "lines" variable as an array of strings. Then I read each "line" of "lines". Each "line" is stripped of the '\n\r' so that it'll get stored into the variables properly to be sent to the influxdb
There is supposed to be 7 entries in each line string (6 commas). In the 2 entries you quoted, there's only 5 and 3, respectively.

I hadn't thought of slowing down the baud rate. I'll give that a shot and see what happens.
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage; VALID email req'd)
Reply
#9
(09-28-2018, 04:50 PM)daromer Wrote: Korishan:
1. Dont have time to read it through now but il take a look later unless someone solved it
But it looks like you are not receiving the bginning of the data? Are you sending it to fast? Lower the speed of transmit and also the interval (I spent 10 seconds looking)

2. You need to group and calculate the value. as you can see he have set static intervals of 1day and 365 days and so forth. That is one way to do it and calculate it in grafana itself reading data from influx. This though take some cpu. (Check the raspberry solar sis thread from me with examples)
You can also have influx calulate this in the backround with set number of minutes between and just store the value that you then read ou. More examples in my thread about that.

The totalwatts is stored in the DB, so that I don't have to worry about adding watt1 and watt2 fields in the displays. I knew I'd need to calculate the kWh over time, so that's why I did that so I didn't have to worry about it later Wink
I looked closer at the guys json code for the panel, and it looks like he's not using a db some how. Which is partly why it's confusing. I'm not sure how he's grabbing the data. InfluxDB calls are a bit different from the SQL code I learned when dealing with MySQL and MS Access. So the tricks I'd do there don't work here.
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage; VALID email req'd)
Reply
#10
Ok, I was watching the "Errors" really closely this time. I noticed a breakthrough.

In all of the errors, they seem to always comes in pairs. Because I also include the 'Iteration' counter, I've noticed they are always successive.

What appears to be happening, is the line is being split in half some how during serial.read.

for example:
Code:
Iteration: 21859
- Line not proper length of elements: 5
- line: ['2781924', '118.84', '7.86', '933.66', '1']
=============================================
=============================================
Iteration: 21860
- Line not proper length of elements: 3
- line: ['2.18', '1446.90', '2380.56']
=============================================
=============================================
Iteration: 21916
- Line not proper length of elements: 5
- line: ['2789173', '120.19', '7.86', '944.63', '1']
=============================================
=============================================
Iteration: 21917
- Line not proper length of elements: 3
- line: ['2.39', '1488.74', '2433.36']
=============================================

21859/21850, 21916/21917. They are in pairs. So I took a closer look. If I take 59 & 60 'line' data, and read them together, I get:
['2781924', '118.84', '7.86', '933.66', '1'] ['2.18', '1446.90', '2380.56']

Here, we can see there are 8 entries. But are there really? If I put them together in a way I know the data is valid (but reading the data before/after the errors), I get this:
['2781924', '118.84', '7.86', '933.66', '12.18', '1446.90', '2380.56']

There's now 7 entries. The 5th entry is getting cut in half somehow. I noticed all the other errors were doing the same thing.

This is really odd. And I even slowed the baud down to 57600, and it seems there's actually more errors occurring now
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage; VALID email req'd)
Reply


Who read this thread?
48 User(s) read this thread:
raccooon (11-28-2018, 11:28 PM), Riplash (10-14-2018, 06:11 PM), Walde (10-07-2018, 05:32 PM), Bubba (10-14-2018, 06:50 PM), duwdu (11-10-2018, 05:29 AM), SilverNodashi (09-30-2018, 10:35 AM), iomagico (10-13-2018, 06:41 PM), Ibiza (10-04-2018, 06:34 PM), completelycharged (10-08-2018, 09:16 PM), Redpacket (10-13-2018, 01:12 PM), gpn (09-28-2018, 06:32 PM), w0067814 (09-30-2018, 03:08 PM), michaell (10-02-2018, 06:40 PM), cmmurphyj (11-08-2018, 02:52 PM), 5buBZMKeJZgapTGsbGzKf (10-08-2018, 01:58 PM), talaldki (09-30-2018, 06:01 PM), Wolf (11-21-2018, 04:21 PM), Carl (12-03-2018, 12:39 AM), Beholder (10-08-2018, 07:18 PM), lordxenu (10-12-2018, 04:53 PM), emuland-metroman (10-07-2018, 11:17 AM), chuckp (10-07-2018, 04:28 PM), yoeri_w (09-29-2018, 10:04 AM), rtgunner (10-07-2018, 04:32 PM), brwainer (10-14-2018, 10:06 AM), wim (10-14-2018, 06:14 PM), jdeadman (10-15-2018, 12:10 AM), Mike C (10-01-2018, 09:31 PM), Frnandu Martiński (10-04-2018, 01:38 PM), Franky Beuselinck (10-01-2018, 01:12 PM), PAF (10-03-2018, 01:41 PM), Korishan (10-14-2018, 06:21 PM), Majorphill (10-12-2018, 10:13 PM), mike (10-15-2018, 07:38 PM), Sean (10-14-2018, 04:54 PM), frnandu (09-29-2018, 07:54 PM), watts-on (10-14-2018, 06:27 PM), CarelHassink (10-06-2018, 09:51 PM), not2bme (09-29-2018, 12:40 PM), ChrisD5710 (10-12-2018, 04:31 PM), floydR (10-07-2018, 01:02 AM), Geek (10-12-2018, 03:27 AM), Mazlem (09-30-2018, 01:41 AM), tremors (10-09-2018, 09:48 PM), tamkov (11-04-2018, 09:01 PM), Hanssing (10-07-2018, 03:28 PM), jesusangel (10-14-2018, 05:26 PM), daromer (10-23-2018, 07:03 PM)

Forum Jump:


Users browsing this thread: 2 Guest(s)