SB70LC Flash Memory card problem

[BillC]

Hi all, my SB70LC netburner project has been running unattended for the last 6 months, its purpose is to control and monitor a home A/C thermostat and read a serial external temperaure and humidity sensor, it also talks to an external X10 controller to switch on/off mains appliances. It reports current status every minute to a host website and can be remotly controlled via its internal web server. It updates daily log files stored on its Flash memory card every minute, these files can also be accesses via its FTP controller. This netburner system is in the USA and I contol it from the UK.

All has been running fine until about a month ago when I was unable to see Flash files via the netburner FTP server , my FTP client could logon OK but timed out getting the initial directory listing. I have remotly reset and restarted the system several times but it has made no difference (the external website updates and the internal web server continue to work normally). The problem has persisted since then ( I try again every few days).

Suddenly today it has started working again, I can see no reason why, but have noted that it has not written any of its log file to the Flash card during this time, but is writing them again now. It appears that the Flash card has somehow been unavailable despite several resets and spells when the house has had power outages.

Has anyone else seen this type of problem.

Thanks, Bill

[khoney]

Weird problem, Bill.

Did you do any type of file manipulation with the FTP server before it started working again? Is it possible there was another FTP session open during this time? I'm currently seeing a problem related to SD card updates while I have an FTP session open (this is on a MOD5270). You might want to keep an eye on my recent thread to see if anything related to your problem pops up. I'll certainly be keeping an eye on yours

How are you handling the log file updates - open, append, close for each record written?

[BillC]

Hi Khoney,thanks for your reply.

No I wasnt online to it when it started working agiain, I think that it started working again after a house power outage, as the first entry in the log (after it started working again) is an entry noting time and date that the program restarted. I only noticed it was working when I tried to access it via FTP about a day later.

I think that the house had other power outages whilst it was down but cant be sure as the netburner occasionaly fails to update the external web host every minute due to internet problems(also normally logged), I might have though that these were due to power outages. It would make more sense that this was the first power outage and the first time that power was completly removed from the netburner, I can only reset it remotly, not remove power. Maybe I need a second netburner that I can command to switch power off/on to the main one !!

One change I shall be implementing is for the netburner to notify the webhost after a restart. The logfile updates by using a Critical section and do open/write/close.

I have just read your post regarding file writes not seeming to be saved, I have had similar problems, I tried writing a log file entry whenever the netburner received a command to restart, but they didnt appear in the file after the restart, I even tried delaying the resart a second or so after the write, but never got it to work, I gave up and moved on to more pressing matters, also had similar problems with log entry when remotely changing the date & time.

[khoney]

So far I've only been using a direct connection to my PC to test the web server, FTP, etc. Sure hope I don't run into lots of problems when going over the internet.

[lgitlitz]

It would be helpful if you could insert some logging to determine where the error is coming from. If no serial ports are available then you can use syslog and send UDP broadcasts. Just make sure to report if any return values are failures for the FTP and EFFS functions you call.

Do you have any other devices sharing QSPI with the SD Card or any other extra hardware that can be related? Is it possible that a power outage triggered this issue? If you were writing a log file when you lost power then it is very possible you could have also corrupted the FAT tables or even damaged the physical flash. Can you access and read the same files from the SD Card directly on a PC?

What version of the NNDK are you using? Try to rebuild with the latest tools, you will also need to replace any files taken from the EFFS examples and used locally in your application... ftp_f.cpp, filesystemutils.cpp... ext.
I suspect your problems are from the EFFS side of the FTP code. This code is found in the ftp_f.cpp that you are using from one of the example applications. There have also been quite a few changes in ftp_f.cpp to make it more robust and correct a few problems where errors did not report correctly. This file had directory size limitations that were fixed, I believe a directory over 32 chars triggered an error in older versions. The core code of the receive and send functions was completely rewritten. These functions were changed to correct issues with reporting transfer failures to clients. File system access and network transfers are is now re-attempted multiple times if a failure occurs. This corrects issues that were occurring with large file transfers and also slow network connections. Some performance optimizations were also added.

[BillC]

Hi Igitlitz, I am using Rel24_RC2, no other hardware, not using QSPI, just using the flash card, 3 onboard uart ports and ethernet.

I think that FTP is only a symptom of the problem, not the cause as it just times out getting the initial directory listing, I am pretty certain that this is due to the flash card going "ofline" somehow, the rest of the program continues to function perfectly but all write/read to the Flash file system fails, this includes FTP access and adding lines to the system log file and the daily stats files that are updated every minute., I dont have any error checking on this as its not critical to the main operation of the system. I dont think that the flash card is corrupted as it eventually started working again and allowed access to all existing files. I now suspect that somehow the flash locked up and went offline, it stayed offline for over a month despite resetting the Netburner card remotly over a dozen times. I guess that it only recovered when the power was removed and reapplied.

I am unwilling to make changes to the code at present as the system is still performing its main function and I found in the past that updating (over the internet) caused the unit to hang, I beleive that this was a problem with earlier NNDK versions but dont want to risk it.

I shall be going over to the USA in about a month for a few weeeks so can investigate further and make changes whilst I am there. It gives me a good excuse to have a vacation.


Thanks, Bill

[BillC]

I had this problem last year on a SB70LC dev board,and didnt manage to get the problem resolved, it has not re-occured on this project so far.

I now have a different project that is using an unattended SB70LC board, and have come across the same problem, the symptoms are

1 Data and log files open/append operations to the Flash memory card are failing, I have no error detection on these as its not an essential part of the system operation, the main purpose is to update a website with current conditions. The Flashcard is just a backup for times when the internet/website are down

2 FTP clients can connect to the the board but no files are listed and files cannot be retrevied or saved.

3 A software or hardware reset does not resolve the problem, the hardware reset button does not solve it.

The only way to get the Flash Memory card working again is to remove the DC power and then reapply it.

At the present time I am not able to propose the Netburner board for future projects until we can overcome this problem.

Any help would be appreciated.

Thanks, Bill

[Ridgeglider]

Bill:
3 responses:

1) How are the data files failing? I have found that the process of open, read (or write), then close is slow for SD card files. Therefore, I often open the files and log for either a preset duration, or filesize. To ensure critical data is written to files (rather than simply being buffered and ready to write, or instead of calling f_close() ) you can call f_flush(filehandle), although that slows down access too.

To ensure data is not lost when the system gets shutdown (eg during an update, or a software initiated reset process), I also set up the callback:
update_shutdown_func = CloseAnyOpenFiles; // where CloseAnyOpenFiles() is my function.

2) If you are connecting to the NB via FTP with a Windows IE browser, I have seen the browser cache settings affect what is being displayed, particularly with older WinXP installations. Specifically, it will show an earlier (possibly erroneous) snapshot of what was on the FTP-accessed SD card. An NB reset would re-establish the connection to the browser so the cache was updated and what is displayed becomes current. Of course, if the reset NB had been logging to the SD without an f_flush() or f_close(), it will look like the card access had failed.

3) Have you tried to reformat the SD card, or use a different card? I've found that you want FAT32 on cards where longfilename support is enabled.

[BillC]

Hi, thanks for your reply.

I dont think timing is a problem, the main thread just polls 2 serial ports for data every minute, sends updates to a website and then adds a line to a CSV file on the flash card so is idle most of the time, everyting works fine for months then suddenly the flash upates fail (I dont have any checks for errors), once in this condition all further flash access fail until power is removed, so it looks like the flash circuitry has locked up. Once I reapply power everything works fine again with no errors on the flsah card, just that files are missing for the time period of the lockup, there is a web server running but its behind a firewall and and is only accessed by myself for occasional testing, Firewall logs show no access on the day that the flash problem started.

System shutdown is not a problem, all startups are are notified to the website and none occured when the problem started.

I am using CuteFTP for the FTP accesses and all works oK until the flash locks up, FTP is not the cause of the problem just a symptom of the flash not working.

It is strange that a power up solves it but a reset dosent, makes me think that the hardware logic get screwed up.

Thanks, Bill

[lgitlitz]

Is this running off of the NetBurner dev board? Do you know if the processor reset or trapped when the lock-up occured? Another possibility is a quick power brown out, I have seen SD Cards lock-ups from this in the past. Have you always used the same model SD Card?