Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Python/Grafana Coding
#41
Just looks like standard serial handling behaviour to me, with 128000 baud rate the time it takes to send 32 bytes is around 2ms and your loop timing will be in that ballpark. One loop misses, next loop 32 bytes - the interrupt flag internally for the serial may have someething to do with the approx 32 byte regular split, unless the short run sample was just coincidentally time syncronised.

Sometimes when you run the code for longer the regular pattern is broken and it is these breaks that actually start to show and help with understanding what is going on.
Reply
#42
Im just a bit confused since I have never needed to sync loop times on serial work when using any high level code like in python and such with a proper serial adaptor.
I have only needed to do that on systems where you bitbang the serial system on a IO port or had bad libraries. For its all about either using the built in interrupt or just take 1 byte at a time and wait for the EOL. I would for sure investigate that.

But yeah the split that happens could be the buffer not handling the data. Im not sure where that limit is since I never sent that much data at high speed. I have always sent it in binary form instead or just short messages.

Im looking forward to the solution here since this is an area I feel i lack knowledge in Tongue
NOTE! My links supplied in this message may be affiliated with Ebay and by clicking on them you agree on the terms.
YouTube / Forum system setup / My webpage  Diy Tech & Repairs

Current: 10kW Mpp Hybrid | 4kW PIP4048 | 2x PCM60x | 100kWh LiFePo4 | 20kWh 14s 18650 |  66*260W Poly | ABB S3 and S5 Trip breakers
Upcoming: 14S 18650~30kWh
Reply
#43
I also lack the knowledge here Tongue We all shall learn together

The odd thing is when i do byte reads of the buffer, it is so slow. But when I read "lines" of the buffer, it's a lot faster.

When I use the python terminal, i can read the lines without issues, and in bulk. When I execute serial.readlines(), it outputs all the lines in the buffer with "no" breaks in them.
It seems that the break occurs "after" the data has been read from the serial. Not sure why.

I'm still doing different things with the code. Today I was working on the millisecond reading/incrementing/sending_to_db. That's an oddity in itself.

On that note, I'm only getting about 4 or 5 distinct values for each "second". So I'm loosing data somewhere in the database itself.
I got the original udp code from Phisicks, and he's ran tests and shows that he can dump far more data with udp than with tcp. That's fine and good, but udp can't handle non-int's for the "time" field in the write_points() method. Why on earth would a "time based" database not accept milliseconds through a faster communications protocol. I don't get that.
I'm sure I can get this working right, but just gonna take, well, time Wink
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at Discord Chat (channels: general, 3d-printing, linux&coding, 18650, humor, ...)
(this chat is not directly affiliated with SecondLifeStorage; VALID email req'd)
Reply
#44
have you set the time precision correctly? You need to have it in ms to save more than 1 value per second.

Do you buffer the data before sending to the database? If you care about the data do not use UDP at all. There is no need to not use TCP on influx. You can send several 100 000 values per second without issues if done correctly.

When you get the data read correctly and you can parse through them without issues then its time to store them. What you then do is that you collect data to a buffer and every 1-10 seconds flush that buffer to influx. You cannot send 1 value per message in any pace faster than 1 message per 40ms on a normal server and on a slower one dont count on more than 3 per seconds in worst case.

Edit: I will hook up an ESP here and test the read code and se how it works for me. i just parse random data at set intervall and we see
NOTE! My links supplied in this message may be affiliated with Ebay and by clicking on them you agree on the terms.
YouTube / Forum system setup / My webpage  Diy Tech & Repairs

Current: 10kW Mpp Hybrid | 4kW PIP4048 | 2x PCM60x | 100kWh LiFePo4 | 20kWh 14s 18650 |  66*260W Poly | ABB S3 and S5 Trip breakers
Upcoming: 14S 18650~30kWh
Reply
#45
I have now done several pure serial send/recieve tests to see if I get the data properly

ESP8266
Serial speed 230400

Windows computer running python 2.7


Code:
void loop() {
 i++;

 Serial.print(i);
 Serial.println(", 1874, 120.61, 4.04, 486.81, 10.39, 1252.79, 1739.60");
 delay(2);        // delay in between reads for stability

 if ((i % 500) == 0 ) {
   i = 0;
   delay(500);
 }
}


And the python code

Code:
#! python
import sys
import serial


sys.stdout.write("hello from Python %s\n" % (sys.version,))
serialPort = serial.Serial(port = "COM6", baudrate=230400,
                          bytesize=8, timeout=2, stopbits=serial.STOPBITS_ONE)
                         
                         
serialString = ""                           # Used to hold data coming over UART
while(1):

   # Wait until there is data waiting in the serial buffer
   if(serialPort.in_waiting > 0):
       serialString = serialPort.readline()
       print(serialString.decode('Ascii'))


I have tested at several speeds and delays between. Problem is when printing to a screen that tend to delay and trash alot of data comming in. But above is 2ms delay and then run it so thats alot of data sent. Added some delay and printout to have a visual of when i run through every 500 sends. 

I ran this for many minutes getting many thousands of values without ANY problem what so ever.



Let me know if I missed something or should test something else for you. 

Note that above is just to show that the serial transmit of data should not be any big issue.
NOTE! My links supplied in this message may be affiliated with Ebay and by clicking on them you agree on the terms.
YouTube / Forum system setup / My webpage  Diy Tech & Repairs

Current: 10kW Mpp Hybrid | 4kW PIP4048 | 2x PCM60x | 100kWh LiFePo4 | 20kWh 14s 18650 |  66*260W Poly | ABB S3 and S5 Trip breakers
Upcoming: 14S 18650~30kWh
Reply
#46
The important difference here is Korishan is using an "arduino nano" and Daromer is running an ESP. The hardware implementation of the serial, processing clock speed and memory is vastly different. The issues Korishan is seeing is mainly due to the way the nano copes with the limited clock speed for processing and task switching. Run the code on a PC and it will have no issue whatever, nice multithreaded GHz of resource.

This is what you see with limited single hardware threaded low MHz processing :
"The odd thing is when i do byte reads of the buffer, it is so slow." It is how you handle the byte you have just read off the buffer and also could be that the python serial buffer handling is also poor at byte handling.
Your code:
while bRun:
char = ser.read()
...
buf += char

the buf += char will incrementally take more processing time each new byte that is added and also leave more for the garbage collection process.

the buf += is creating new memory allocation 1 byte larger than the existing buf memory space, copying over the existing value of buf into the new memory space and then copying over the new char byte value at the end. Each time this runs an extra byte is added and you then have more data that is copied in the loop, progressively getting slower.
If the python library is handling it in reverse and copy removing then you end up with double the amount of meory being shuffled around. The readline is taking a chunk of bytes in one go, so a single copy and then clearing the serial buffer memory (deallocating the existing memory and leaving it for the garbage collector).

Programming microcontrollers can be an interesting challenge to work around issues that don't occur with the same data volumes on other platforms. This is they main challenge, to find out what is impacted and to work with or around it.

UPD is always faster because it is not establishing a full connection with handshake to an end point and just throwing data out of the network port. TCP delays can make the timing issue on the arduino worse as you may have seen with the wifi, which would make timing issues even worse..

With TCP connections they are normally kept open and persisted in your scenario so you would have more of a delay on creation compared to UDP but that is just a one off at startup. TCP also confirms packet delivery, so ensuring the listner gets the data, UDP does not. The issue with TCP is that if your connection is not 100% then your data would end up lost at the TCP calls may end up taking more time or blocking, allowing the buffer to overflow. The case may be that UDP is better for the arduino purely because it requires fewer clock cycles to send data and the clock cycles are needed if running at higher data rates.

"I have always sent it in binary form instead or just short messages." - at higher data rates to gain efficiency this is the only route....

Arduino Nano clock speed : 16MHz (8bit)
ESP8266 clock speed : 80MHz (32bit)
For some operations the ESP will be more than 10x faster... that's just on memory copy.
Reply
#47
Thanks for the information Smile

One thing to add though, is that the networking portion is not handled on the andruino at all. It's on the laptop. Which, btw, is also running the influxdb on linux Mint.
I understand the shuffling around of memory with the buf += char. But this is, again, happening on the dual core processor of the laptop, not the arduino. If I was doing all these things on the arduino and had an ethernet shield, I could really see the logic behind why things are different. But all the data is being moved over.

So the arduino sends the data to the serial port. Where is the buffer allocated at? On the arduino, or the host of the usb port, in this case the laptop. If it's the laptop, I shouldn't be getting any weird issues, like I've posted above.


Taking a little bit on working on some of the things you guys suggested as I'm also working on other things as well. I need multiples of me, like Daromer showed he has in a few of his videos Tongue
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at Discord Chat (channels: general, 3d-printing, linux&coding, 18650, humor, ...)
(this chat is not directly affiliated with SecondLifeStorage; VALID email req'd)
Reply
#48
Ah, I thought you were trying to run the upload on the arduino and the database was internet/elsewhere, that changes a lot....
If your able to get the data no issue on the terminal the sending side is fine, forget the arduino, it works at the packet rate you have. The byte structure being sent is consistent.
With the data comming through on the terminal fine, the serial handling on the hardware side is fine... because no data is lost and the packets are only being split (no buffer overflows) in code later.

Is there any particular reason yourt not using readStringUntil() ?

Separate note : reading serial on a byte by byte basis should not be noticably slow on the laptop... the python loop should be able to run fast enough to be waiting a long time (many loops) for each byte.

readline() will read whatever is in the buffer at the time of the call and return.... if there is no delimeter defined within the data available to be read. Its a non blocking call.

This is the code and explanation for the byte approach for python in general.
https://hackingmajenkoblog.wordpress.com...e-arduino/
Reply
#49
Yeah, that's makes a huge difference on the understanding where the data is coming/going from Tongue

Well, firstly, I didn't know about readStringUntil(). I didn't see it in the pySerial docs (not saying I've read everything, that is).

From my understanding, in pySerial, readline() and readlines() will pull a line(s) and uses the '\n' as the delimiter. 1 line or all lines, respectively. That's really the reason why I've stayed with that method as it reads the data exactly in the chunks that the arduino is sending.
When serial.println() is used in arduino, it automatically adds a trailing '\n' to the end of the buffer. Actually, it adds both, '\n' and '\r'. That's why the one line of code in the python script removes the '\r' as it's not needed and causes other issues with the rest of the code. Leaving or removing the '\n' seems to have no effect. Granted, that's just for outputting to the terminal. I suppose the DB wouldn't care either way, so I could probably remove that line in production.

Now, granted, if I was "reading" serial on the arduino, that'd be a totally different situation Tongue
Actually, I do want to incorporate reading so I can send configuration adjustments. but that's the next task I want to tackle after I get the output working correctly


Reading over the linked info, thanks. Looks a bit more detailed than some of the other docs I've seen

Definitely a good explanation. Thanks for link. That helps understand more how the serial is handles. This is something that could also be achieved in the python code too. Albeit, the python will run far faster than the mcu's Tongue
I do have a serial.cpp in my other project that I will probably incorporate these changes into it. It's a bit clunky in it's job, but it does work. But clunky means it adds unnecessary byte size to the final compiled program; and we all know how precious that memory is for those mcu's
Proceed with caution. Knowledge is Power! Literally! Cool 
Knowledge is Power; Absolute Knowledge is Absolutely Shocking!
Certified 18650 Cell Reclamation Technician

Please come join in general chit-chat and randomness at Discord Chat (channels: general, 3d-printing, linux&coding, 18650, humor, ...)
(this chat is not directly affiliated with SecondLifeStorage; VALID email req'd)
Reply
#50
Completelycharged yes im kind of simulating the same as Korishan did. I can hook up a nano for fun and repeat the test but im pretty sure that above code will run exactly same and i would not have any problem.  Since the problem as I see it is not on the sending part but just on the reader side. 

UDP on Influx should be skipped and as said its not from the arduino but from the computer itself.

Korishan: Most of the devices out there have a buffer for serial reading in some way or another. In this case you are sending to the computer and that one got a huge buffer! You should not have to do any delay or anything on that computer reading that amount of data. Its all limited by the speed of Arduino in my world!


EDIT: I have tested now and it works the same on the Nano here. 328p. No dropout and have had it running for some time now without issues.
So im back to stating that on reader side on a normal computer you should not have to do much fiddling around with delays and such. Just make sure you read the buffer fast enough so you dont get dropouts Smile

I stand to be corrected on my saying but above is something i have done in a long time but i would not say I know what im doing  Smile
NOTE! My links supplied in this message may be affiliated with Ebay and by clicking on them you agree on the terms.
YouTube / Forum system setup / My webpage  Diy Tech & Repairs

Current: 10kW Mpp Hybrid | 4kW PIP4048 | 2x PCM60x | 100kWh LiFePo4 | 20kWh 14s 18650 |  66*260W Poly | ABB S3 and S5 Trip breakers
Upcoming: 14S 18650~30kWh
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)