Home

Welcome! Log in or Join us

Forums Brultech How do I fix and/or prevent bogus data spikes?

Subscribe to How do I fix and/or prevent bogus data spikes? 11 posts, 3 voices

Reply to topic
 

 
W_medium
Jun 28, 2013 11:53pm
11 posts

I have 3 ECM-1240's and am running btmon.  I occasionally see heatmaps that are blank.  Further investigation leads me to suspect that heatmaps don't like negative values?  When I looked at some other graphs, I saw occasional huge negative spikes.  I exported one and got this:

2013-06-24T21:29:59Z,3236.18
2013-06-24T20:59:59Z,3939.37
2013-06-24T20:29:59Z,3966.79
2013-06-24T19:59:59Z,2345.87
2013-06-24T19:29:59Z,-2377980.0
2013-06-24T18:59:59Z,4041.37
2013-06-24T18:29:59Z,4053.49
2013-06-24T17:59:59Z,3550.49
2013-06-24T17:29:59Z,2690.48
2013-06-24T16:59:59Z,3798.61

Am I doing something wrong on btmon or in my SEG setup? What can be done to prevent this?

Can I fix this by editing that huge negative number to "0.00" and uploading it? I would I need a positive 2377980.0 to offset the negative?

Is there a way to prevent a feed from saving insane negative numbers?
 
Samotage_300_medium
Jun 30, 2013 1:20am
555 posts

Hey Waynew,

There are a couple of things you can do  (FWIW when I was using Zigbee its message meshing feature would do this kind of thing all the time...)

In the stream you are getting crud,

- uncheck the allow negatives box, this will stop the negative numbers (unless they are valid)

- adjust the ludiucrous value down to the highest you are likely to be recording

These little rules will help ignore this kind of crud.

Sam, @samotage

 
W_medium
Jun 30, 2013 3:21am
11 posts

Thanks Sam.  I didn't know about the checkbox to prevent negatives, so I have checked that now.

Is there a way to fix the bad data already in the SEG system?

Would there be any merit to asking Matt for an enhancement to btmon to prevent negatives and insanely large numbers at the source?  Maybe those features are already in btmon and I don't know how to confuigure them.

 

 

 
Samotage_300_medium
Jun 30, 2013 3:24am
555 posts

Get the crud early on is my default approach. 

How many streams have you got funky data for?  I can run a re-calibrate which should re-apply the rules, or you can go to the stream yourself and then  search for points ordered by lowest to hightest or vise versa and delete them.

Sam, @samotage

 
W_medium
Jun 30, 2013 3:39am
11 posts

How do I delete bad data?  Do I upload and it will overwrite?

 

 

 
Samotage_300_medium
Jun 30, 2013 3:40am
555 posts

Go into the stream, there will be a tab called data.

There is a little search for, search away and press the red X to delete the point.  All recalculations will then take place!

Sam, @samotage

 
W_medium
Jun 30, 2013 4:15am
11 posts

Very Cool, thank you very much.  I learn more every day, or more things get added everyday :)

I fixed my streams, but I have a calculated stream also.  It seems that a calculated stream doesn't have the "data" tab, so I cannot fix that myself.  Does a calculated stream ever get "re calculated", like once a day?  Otherwise, if you can force a re-calibrate for my "total HVAC" energy & power streams, it would be appreciated.

 
Samotage_300_medium
Jun 30, 2013 4:46am
555 posts

There is a lot in SEG, much of it gets discoverable as the service is used more and more without overwhealming newcomers!

There is no data for calculated streams, however its possible that you can press the reset calculations on the Calculations tab, then this will trigger a re-calc.

Calculated streams are updated every 15 minutes or so.

Sam, @samotage

 
Icon_missing_medium
Jan 2, 2014 5:51pm
9 posts

I have a similar setup to the original post, and had the same problem described:  occasionally scrambled data with obviously bad information in the Brultech/Engine-G database, as well as in Google Power Meter (while it existed).  What finally, completely fixed it for me was replacing the "multiplex" serial cable with three separate serial-to-Ethernet adapters.  I was fortunate to find some surplus adapters at low cost, though it appears there one device available from online sellers in the $25 to $30 range - still cheaper than Brultech's at $74 plus mux cable.

For a while I was using the Brultech mux-3 cable with btmon for upload to SEG instead of running Engine-G locally.  While that also eliminated the bad data, there were occasional (every day or two) events where one of the ECM-1240s would go offline for a couple hours, and then recover by itself.  Separate Ethernet adapters also corrected this problem.

It has only been a few weeks with the new configuration, it has been rock solid for data collection to SEG.  Long enough, I hope, to declare victory.

 
Samotage_300_medium
Jan 3, 2014 2:30am
555 posts

Victory, always fleeting!  

Looks like this is all hard work to me...  ;)

I'm about to roll a few of mods into dev,

1.  Recalculate data - which will recalculate all data in your stream from the raw data

2.  Re-calibrate data, which will apply adjusted calibrations, i.e. adjusted Ludicrous values and also perform a re-calculation

3.  As you suggested Paul, a setting on the dial gadgets to set a fixed max and min as an option to the the automatic rolling weekly calculations

Sam, @samotage

 
Samotage_300_medium
Jan 3, 2014 7:38am
555 posts

Hi all,

This is all now done.

For the re-munge of data, go to the Stream->Edit tab.  The new functions are available there.  Note there are no-warranties express or implied iwith this ;)  Though all care has been taken, and the functionality has been in use and tested for some time.  

For the gauge, this has a definable max and min.  You need to tell your gauge that you would like to override the max.  Don't do silly things like make the max less than the minimum, and the gauge will spin out if your current values are greater than the max you have defined!

Let me know here how you get on.

Sam, @samotage

 

Log in to reply to this topic

Forums Brultech How do I fix and/or prevent bogus data spikes?

copyright © 2008 Smart Energy Groups pty ltd