Industrial use Gameduino2/FT800 SPI corruption?

[diesel]

  I have built a common rail Diesel injector tester, using an FT800 5" screen and the Gameduino2 code (Thanks James!). I have an occasional issue, and by the way this isn't my first attempt, this is the 3rd version named Accu-Rate CR. I have 2 versions, one running the Gameduino2 and code, and this one. I also have another machine that tests common rail Diesel high pressure pumps, using the Gameduino2 and code.  

  These machines are somewhat complex with contactors, relays, 240 volts, and some are 3 phase, with a variable frequency drive. I am a fabricator/machinist by trade. I kind of was led into electronics, kicking and screaming the entire way! I get by, thanks to all the "real experts" who have solutions to issues I can read about online.

 Occasionally I encounter a frozen FT800/Gameduino2 screen. Only a reset will bring it back to life. I believe it is EMF, from the contactor when the motor is started. I have put ferrite cores everywhere, then removed them selectively, trying to isolate the issue. I have changed the SPI div, I have added shielded wires, twisted pairs, relocated the microcontroller. I have also rewritten some of the code, that incorporates delay between high voltage/high amperage starting loads, and the feedback associated with it. I have had to build in safety measures, in realization of a freeze up.

  I have looked over this by Paul Stoffgren www.dorkbotpdx.org/blog/paul/better_spi_bus_design_in_3_steps, which really relates to multiple SPI shared devices.

  Suggestions, advice, and ideas are greatly appreciated. I have tried many many different things, and read and re-read a lot of information.

  Link to machine operating on youtube youtu.be/hhlQPpgRDtE

[jamesbowman]

Aha, OK. So just to be clear both the Gameduino2 and the FTDI modules can become "hung"?

The electrical environment looks "challenging".  It sounds like the setups I have seen used for *inducing* lock-ups during IC testing!

The FTDI modules have a reset pin called "PD" - is this hooked up to the microcontroller? As a first line of defense the microcontroller could recover from the crash by pulsing PD then re-initializing the graphics system starting with "GD.begin()". Maybe you are doing this already?

[diesel]

Yes, I have had both to lock up. I wasn't aware of the PD pins function, so I don't have a connection there.

[tftlcdcyg]

In order to avoid the calibration on startup and get the original parameters of calibration of the gameduino2, you can modified the GD2.ccp file:

 GDTR.wr(REG_PCLK_POL, 1);
 GDTR.wr(REG_PCLK, 5);
 GDTR.wr(REG_ROTATE, 1);   //add this line

#if PROTO == 1
 GDTR.wr(REG_ROTATE, 1);
 GDTR.wr(REG_SWIZZLE, 3);
#endif

 GDTR.wr(REG_GPIO_DIR, 0x83);
 GDTR.wr(REG_GPIO, 0x80);
The wire between FT800-TFT and arduino board must be in the 12 and 20 cms range.

[diesel]

Thank you, yes the wires are short (170 mm). I have ran them up to 3 feet with the same results.

[jamesbowman]

Pulling PD down (see the FT800 datasheet) and restarting the GD library should unhang the FT800.

Of course reliably detecting a hung FT800 is another problem. Perhaps use the Arduino watchdog to reset the whole system?

[diesel]

Yes that is my concern in how to detect a disruption. If the SPI or FT800, could set a flag of some sort, it could be handled in a simple if statement. But that would be too easy.
Any ideas are greatly appreciated.

[jamesbowman]

OK, here is what I have done when building similar systems.

My favorite book on circuits is this one. Im the section on watchdogs it makes a good point: watchdogs should make a system *more* reliable. So a watchdog system should be very simple!

So this led me to a system design like this:

• the bootup sequence resets every peripheral
  
• as soon as anything goes wrong, reboot
  

That's it. The first part is to reset the FT800 by pulling PD low at boot. Something like this:

PinMode(5, OUTPUT);
digitalWrite(5, 0);
delay(10);
digitalWrite(5, 1);
delay(10);
GD.begin();


The second part is "anything goes wrong". Very often going wrong means that the CPU will get stuck in a loop, and the hardware watchdog can help. This article forum.arduino.cc/index.php?topic=63651.0 explains how to set it up with some sample code.  Since we can guarantee to kick the watchdog about every 16ms, the watchdog timer should be configured to reset after about 125ms.

But of course the CPU may not be stuck in a loop - perhaps it is still sending data to the FT800 but the FT800 is dead. So as a consistency check, the code can read the ID register to confirm that the FT800 is still alive.

Putting these together, the sequence after GD.swap() might look like:


void loop()
{
 ... drawing ...
 GD.swap();

 // kick the watchdog
 wdt_reset();

 // if the FT800 isn't reading registers, reboot
 if (GD.rd(REG_ID) != 0x7c) {
   asm volatile ("  jmp 0");  // software reset
 }
}

[diesel]

Thank you very much for the help James. I will try to implement the code and see if it is in fact the MC or the display that is locking up. In my experiences when the MC "arduino" has an issue, it will reset. At least that has been my experience with a 4 line LCD screen, that uses digital pins to write to the display.

A question, when the SPI data is interrupted, or corrupted, will it lock up both the MC and the Gameduino2/FT800?

I guess my question is more centered to the MC being the master, and the Gameduino2/FT800 being the slave. Or is it the opposite, or both, send and recieve?

If the MC is the master, and the Gameduino2/FT800 is the slave/receiver. Then would the loop be compromised, and the display would just be inoperable. I feel like they work together, the MC expects to see something. Well, receive certain bits from the Gameduino2/FT800 I suppose.

I am in no way questioning your intelligence or experience, just trying to understand exactly what is being transferred, and or expected on both ends of the transmitted data.

Is it possible (forgive me if this is impossible, or just ignorant) to resend the lost data, without resetting the display/MC. Resetting both would be fine as it would relieve the machine from generating pressure 30,000 psi, or 2000 bar when it hangs up. But the better solution would be to re-establish the lost communication. Could data outside of a certain parameter be rejected, I have no idea what the corrupted data even looks like.

I don't mind doing the research and trial if it is at all possible to do this, and I trust your judgement James.

[jamesbowman]

No problem! Ask away!

Right so the MC (master) is constantly talking to the FT800 (slave), getting replies, and if the FT800 is hosed, two things can happen to that conversation:

(1) The MC can get something completely illegal. In this case the GD library *should* show a blue-screen-of-death and hang.

(2) The MC can enter an infinite loop waiting for the FT800 to be ready, but the FT800 has stopped making forward progress so nothing can happen.

In general, implementing a "recovery" path is hard. The problem is that resetting the FT800 causes it to lose track of where it was in the command stream. So after resetting the FT800 (by pulling PD low) the only way to recover is to go through the FT800 initialization sequence.

[diesel]

I think the best solution has to be, no lock up at all. I found 2 issues. Songle relay board, and overrated relay capacity, now I use them to trigger 12 volt logic Bosch 30 amp relays. This seems to take a good bit feedback off the MC/CPU. Second, I was using an older version of the Arduino IDE, I read that SPI has changed for version 1.6.0, so that is what I tried today. I torture tested the machine, and it never gave any issues. Not that it is fixed, but so far so good. I think the MC was the lock up, not the screen.

Thanks for the help, and the book recommendation, as soon as I get a chance I will order it.

[diesel]

Well it appears that the contactor I was using to control the motor operation, was causing too much EMF to be introduced into the system. I replaced it with an SSR (solid state relay) and it seems to have eliminated the interference, time will tell for sure. One thing for certain even when I wasn't experiencing screen lock up, I experienced some screen blurring quite often, essentially every time the contactor was enabled. The SSR has totally eliminated this.