Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python/Grafana Coding
#11
The output you get is already a parsed list. So you're not seeing everything. Print the raw output from serial.readline() and see what you get. It looks like
4 out of 10 is bad so it's easy to look at the raw output to see what's wrong.

You can do the same with the arduino and send to the serial console the data. See the raw data.

TTL is very finicky, since it goes from 0 to +3V? for the arduino. 0V is just noisy. RS232 swings -12V to +12V for a PC. There's no mistaking what's low and what's high.
Reply
#12
(09-28-2018, 11:25 PM)Korishan Wrote: Ok, I was watching the "Errors" really closely this time. I noticed a breakthrough.

In all of the errors, they seem to always comes in pairs.  Because I also include the 'Iteration' counter, I've noticed they are always successive.

What appears to be happening, is the line is being split in half some how during serial.read.

Well, it seems like you notice what I told you in my previous explanation

(09-28-2018, 07:51 PM)jesusangel Wrote: If I'm not wrong reading your code you do a while 200000 times checking if some data from arduino is received, trying to parse the line as 7 fields of data, these times you get errors is because not the whole line is received so it process some data in one cycle and the rest in the next 105383-105384  and 105390-105391.

, you have a loop that is executed without any control just a delay of 0.125 seconds, so you try to read the line when is wasn't completely received, you read half of the line in one cycle y and the rest in the next, that is what you call errors in pairs, readline should wait for a \n but it doesn't do it, and it is a very common error. No matter if you set a lower baudrate.

Print the raw output bytes from serial as not2bme said and you will see in first error there is no \n  (0A byte) at the end but it is in the second error.
 
This error is not so aleatory as you think, in you case readline triggers in all the errors that you shown when there are 28 byte in serial buffer.

I don't usually code in python but you can use serial.read_until('\n',maxbytes) maybe you would get betters results, or you can definitely use serial.read instead of serial.readline and be you who parse the bytes, store every byte in a buffer until you get \r\n, (that's a weird thing in you code, usualy end of line is \r\n it seems you use \n\r, change it in arduino code too).
completelycharged likes this post
Reply
#13
Jesusangel is right. You need to handle the Read bettet and thats what you get those results
The Ultimate DIY Solar and build place
YouTube / Forum system setup / My webpage  Diy Tech & Repairs

Current: 10kW Mpp Hybrid | 4kW PIP4048 | 2x PCM60x | 83kWh LiFePo4 | 10kWh 14s 18650 |  66*260W Poly
Upcoming: 14S 18650~30kWh | Automatic trip breakers, and alot more
Reply
#14
Hmm, interesting. Thanks.
I figured since the arduino sends data as a whole line, and the pyserial is reading in a whole line, there'd be no issue with the loops.
serial.readlines() is supposed to read all lines including the \r\n. That's where it's supposed to stop at. The arduino code dumps the lines and attaches the \r\n to each serial print. So that's why I was a bit confused on the results.
I don't recall seeing those errors when did a straight dump when I was testing. I will have to revisit that.

Now, for the Grafana portion. What is the syntax for getting the consumed power over time? I get I need to override TimeRange. But what about the rest of it? I put in the kWh for 1Hr, and I think that one worked. Then I put in the syntax for 24h, or 1d, and it tells me the value "mega-watts" ?!?!!?
Code:
{
  "type": "singlestat",
  "title": "Consumption (1D)",
  "gridPos": {
    "x": 6,
    "y": 0,
    "w": 6,
    "h": 8
  },
  "id": 9,
  "datasource": "Power Monitor",
  "targets": [
    {
      "policy": "default",
      "resultFormat": "time_series",
      "orderByTime": "ASC",
      "tags": [],
      "groupBy": [
        {
          "type": "time",
          "params": [
            "$__interval"
          ]
        },
        {
          "type": "fill",
          "params": [
            "null"
          ]
        }
      ],
      "select": [
        [
          {
            "type": "field",
            "params": [
              "totalwatt"
            ]
          },
          {
            "type": "sum",
            "params": []
          }
        ]
      ],
      "refId": "A",
      "measurement": "powermon",
      "query": "SELECT sum(\"totalwatt\") FROM \"powermon\" WHERE $timeFilter GROUP BY time($__interval) fill(null)",
      "rawQuery": false
    }
  ],
  "links": [],
  "maxDataPoints": 100,
  "interval": null,
  "cacheTimeout": null,
  "format": "watt",
  "prefix": "",
  "postfix": "",
  "nullText": null,
  "valueMaps": [
    {
      "value": "null",
      "op": "=",
      "text": "N/A"
    }
  ],
  "mappingTypes": [
    {
      "name": "value to text",
      "value": 1
    },
    {
      "name": "range to text",
      "value": 2
    }
  ],
  "rangeMaps": [
    {
      "from": "null",
      "to": "null",
      "text": "N/A"
    }
  ],
  "mappingType": 1,
  "nullPointMode": "connected",
  "valueName": "avg",
  "prefixFontSize": "50%",
  "valueFontSize": "100%",
  "postfixFontSize": "50%",
  "thresholds": "",
  "colorBackground": false,
  "colorValue": false,
  "colors": [
    "#299c46",
    "rgba(237, 129, 40, 0.89)",
    "#d44a3a"
  ],
  "sparkline": {
    "show": true,
    "full": true,
    "lineColor": "rgb(31, 120, 193)",
    "fillColor": "rgba(31, 118, 189, 0.18)"
  },
  "gauge": {
    "show": false,
    "minValue": 0,
    "maxValue": 100,
    "thresholdMarkers": true,
    "thresholdLabels": false
  },
  "tableColumn": "",
  "timeFrom": "1d"
}
Was going to show just the query syntax, but that doesn't show "everything". So there's the json code.
When I load the data into the influxdb, I automatically add fields 'watt1' and 'watt2' to get the total. This way I don't have to add extra calculations/work-load to the laptop afterwards.

*just got up and haven't had coffee yet, please excuse if the above if it seems a little confusing/off*
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage)
Reply
#15
Look at daromer's post on his rasberry pi in the build section. Others have just posted how to calculate the kwh. I used to use the integral function but it seems others found another way with a nested select statement. It should be in the most recent posts.
Korishan likes this post
Reply
#16
Yeah in that thread you have 2 Good example.

Egen reading Serial port you need to Read untill new Line. Not based on a timefrme...


IF data then save in buffer untill new Line found
Hej. Newline then parse data
Korishan likes this post
The Ultimate DIY Solar and build place
YouTube / Forum system setup / My webpage  Diy Tech & Repairs

Current: 10kW Mpp Hybrid | 4kW PIP4048 | 2x PCM60x | 83kWh LiFePo4 | 10kWh 14s 18650 |  66*260W Poly
Upcoming: 14S 18650~30kWh | Automatic trip breakers, and alot more
Reply
#17
I went through Daromer's Raspi thread, and I didn't find where there was a discussion on the kWh conundrum I'm needing. There was a link, but it didn't help me any (unless I missed something in there). It just discussed adding two fields together.
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage)
Reply
#18
Korishan:

One way: https://secondlifestorage.com/t-Esperyd%...9#pid37129
Another: https://secondlifestorage.com/t-Esperyd%...3#pid37043

Basically what you need to do is group every wattage calculation. If you know you have an average of 1kw during 1 hour you also know its 1kWh during that hour. So basically for each minute values you know that if you have a use of 1,2kw you know that the minute then produced 1,2/60 = 0,02kWh during that time. You do that for every minute and then you have the total kWh during that hour.

Im not infront of the correct computer so cannot give you the code I used.
The Ultimate DIY Solar and build place
YouTube / Forum system setup / My webpage  Diy Tech & Repairs

Current: 10kW Mpp Hybrid | 4kW PIP4048 | 2x PCM60x | 83kWh LiFePo4 | 10kWh 14s 18650 |  66*260W Poly
Upcoming: 14S 18650~30kWh | Automatic trip breakers, and alot more
Reply
#19
I used the query that not2bme posted, the first link, and it's basically what I figured out already:

SELECT mean("totalwatt") FROM "powermon" WHERE $timeFilter GROUP BY time(1h)

That works for 1h. But if I change it to 24h, the kWh doesn't change much. For example, it goes from 1.256kWh to 1.245kWh. If I also add in the time override and set the GROUP BY to 1h, and override to 24h, I get 1.601kWh. If I GROUP BY 24h with override of 24h: 1.617kWh.

I'm obviously missing something here. Is the above only for the average for the time specified and something different for outputting 24hrs?

With oldserg's code:
Code:
5. cq_kWh_batt_discharg
CREATE CONTINUOUS QUERY cq_kWh_batt_discharg ON powerwall RESAMPLE FOR 1h BEGIN SELECT integral(kwh_bco, 1h) / 1000 AS kWh_batt_discharg INTO powerwall.autogen.kWh_ALL FROM (SELECT mean(battery_discharge_current) * mean(battery_voltage) AS kwh_bco FROM powerwall.autogen.pip_query_general_status GROUP BY time(2m) fill(0)) GROUP BY time(20m) TZ('Europe/Rome') END

6. cq_kWh_PV_batt_charg
CREATE CONTINUOUS QUERY cq_kWh_PV_batt_charg ON powerwall RESAMPLE FOR 1h BEGIN SELECT integral(kwh_PV_bat, 1h) / 1000 AS kWh_PV_batt_charg INTO powerwall.autogen.kWh_ALL FROM (SELECT mean(battery_charging_current) * mean(battery_voltage_from_scc) AS kwh_PV_bat FROM powerwall.autogen.pip_query_general_status WHERE pv_input_current_for_battery > 0 GROUP BY time(2m) fill(0)) GROUP BY time(20m) TZ('Europe/Rome') END

I'm a little lost here. Maybe just an explanation of what's going on here would help. He mentions needing to create a new table to use the those.
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage)
Reply
#20
Hmmm, the new code for the python script is not working as intended.

I changed:
Code:
lines = ser.readlines()
  for line in lines:
    line = ser.readline()
    line = line.rstrip("\r\n")
    line = line.split(", ")
and now I just do:
Code:
line = ser.readline()
        line = line.split(", ")

But, I have ran into another situation. It seemed like the code was running really slowly. So I added in another timer to see how long each iteration was taking.
The results are:
Code:
StartTime: 1.53832745413e+12 Endtime: 1.53832745441e+12 Different: 279.645996094
StartTime: 1.53832745441e+12 Endtime: 1.5383274546e+12 Different: 196.064941406
StartTime: 1.5383274546e+12 Endtime: 1.5383274548e+12 Different: 194.843261719
StartTime: 1.5383274548e+12 Endtime: 1.53832745507e+12 Different: 275.984863281
StartTime: 1.53832745507e+12 Endtime: 1.53832745527e+12 Different: 195.929931641
Interation: 985 of 1000
StartTime: 1.53832745527e+12 Endtime: 1.53832745547e+12 Different: 196.938964844
StartTime: 1.53832745547e+12 Endtime: 1.53832745597e+12 Different: 502.886962891
StartTime: 1.53832745597e+12 Endtime: 1.53832745625e+12 Different: 278.920166016
StartTime: 1.53832745625e+12 Endtime: 1.53832745645e+12 Different: 196.957763672
StartTime: 1.53832745645e+12 Endtime: 1.53832745663e+12 Different: 187.982177734
Interation: 990 of 1000
StartTime: 1.53832745663e+12 Endtime: 1.53832745714e+12 Different: 509.819091797
StartTime: 1.53832745714e+12 Endtime: 1.53832745742e+12 Different: 280.929931641
StartTime: 1.53832745743e+12 Endtime: 1.53832745762e+12 Different: 198.078857422
StartTime: 1.53832745762e+12 Endtime: 1.53832745782e+12 Different: 192.843994141
StartTime: 1.53832745782e+12 Endtime: 1.53832745811e+12 Different: 294.964111328
Each line is an iteration. It's taking far longer to run this code, than it was the previous code. serial.readline() reads till it finds a newline feed. But apparently I'm missing something here as I'm not getting all the data. The previous code, even with it's errors, I was getting almost 90 successful reads a second.
Code:
StartTime: 1.53832794366e+12 Endtime: 1.53832794379e+12 Different: 125.501708984
StartTime: 1.53832794379e+12 Endtime: 1.53832794391e+12 Different: 128.621826172
StartTime: 1.53832794391e+12 Endtime: 1.53832794404e+12 Different: 125.321777344
StartTime: 1.53832794404e+12 Endtime: 1.53832794417e+12 Different: 126.493896484
StartTime: 1.53832794417e+12 Endtime: 1.53832794429e+12 Different: 125.306152344
Interation:  35  of  200000
StartTime: 1.53832794429e+12 Endtime: 1.53832794442e+12 Different: 125.291015625
StartTime: 1.53832794442e+12 Endtime: 1.53832794454e+12 Different: 125.329833984
StartTime: 1.53832794454e+12 Endtime: 1.53832794467e+12 Different: 126.826904297
StartTime: 1.53832794467e+12 Endtime: 1.53832794479e+12 Different: 125.281005859
StartTime: 1.53832794479e+12 Endtime: 1.53832794492e+12 Different: 126.439208984
Interation:  40  of  200000
StartTime: 1.53832794492e+12 Endtime: 1.53832794505e+12 Different: 125.260253906
StartTime: 1.53832794505e+12 Endtime: 1.53832794517e+12 Different: 127.439941406
StartTime: 1.53832794517e+12 Endtime: 1.5383279453e+12 Different: 128.583740234
StartTime: 1.5383279453e+12 Endtime: 1.53832794543e+12 Different: 125.509033203
StartTime: 1.53832794543e+12 Endtime: 1.53832794556e+12 Different: 128.573974609
Interation:  45  of  200000
StartTime: 1.53832794556e+12 Endtime: 1.53832794568e+12 Different: 125.311035156
StartTime: 1.53832794568e+12 Endtime: 1.53832794581e+12 Different: 128.396972656
StartTime: 1.53832794581e+12 Endtime: 1.53832794594e+12 Different: 125.503173828
StartTime: 1.53832794594e+12 Endtime: 1.53832794606e+12 Different: 128.58203125
1/8 of a second compared to 1/4 of a second, or 1/2 as some of the reads were.
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at https://discord.gg/c7gJ5uA
(this chat is not directly affiliated with SecondLifeStorage)
Reply


Who read this thread?
48 User(s) read this thread:
raccooon (11-28-2018, 11:28 PM), Riplash (10-14-2018, 06:11 PM), Walde (10-07-2018, 05:32 PM), Bubba (10-14-2018, 06:50 PM), duwdu (11-10-2018, 05:29 AM), SilverNodashi (09-30-2018, 10:35 AM), iomagico (10-13-2018, 06:41 PM), Ibiza (10-04-2018, 06:34 PM), completelycharged (10-08-2018, 09:16 PM), Redpacket (10-13-2018, 01:12 PM), gpn (09-28-2018, 06:32 PM), w0067814 (09-30-2018, 03:08 PM), michaell (10-02-2018, 06:40 PM), cmmurphyj (11-08-2018, 02:52 PM), 5buBZMKeJZgapTGsbGzKf (10-08-2018, 01:58 PM), talaldki (09-30-2018, 06:01 PM), Wolf (11-21-2018, 04:21 PM), Carl (12-03-2018, 12:39 AM), Beholder (10-08-2018, 07:18 PM), lordxenu (10-12-2018, 04:53 PM), emuland-metroman (10-07-2018, 11:17 AM), chuckp (10-07-2018, 04:28 PM), yoeri_w (09-29-2018, 10:04 AM), rtgunner (10-07-2018, 04:32 PM), brwainer (10-14-2018, 10:06 AM), wim (10-14-2018, 06:14 PM), jdeadman (10-15-2018, 12:10 AM), Mike C (10-01-2018, 09:31 PM), Frnandu Martiński (10-04-2018, 01:38 PM), Franky Beuselinck (10-01-2018, 01:12 PM), PAF (10-03-2018, 01:41 PM), Korishan (10-14-2018, 06:21 PM), Majorphill (10-12-2018, 10:13 PM), mike (10-15-2018, 07:38 PM), Sean (10-14-2018, 04:54 PM), frnandu (09-29-2018, 07:54 PM), watts-on (10-14-2018, 06:27 PM), CarelHassink (10-06-2018, 09:51 PM), not2bme (09-29-2018, 12:40 PM), ChrisD5710 (10-12-2018, 04:31 PM), floydR (10-07-2018, 01:02 AM), Geek (10-12-2018, 03:27 AM), Mazlem (09-30-2018, 01:41 AM), tremors (10-09-2018, 09:48 PM), tamkov (11-04-2018, 09:01 PM), Hanssing (10-07-2018, 03:28 PM), jesusangel (10-14-2018, 05:26 PM), daromer (10-23-2018, 07:03 PM)

Forum Jump:


Users browsing this thread: 1 Guest(s)