MD5 checksum for os802.img ???

Discuss Sugar, upgrades and developer perspectives.

MD5 checksum for os802.img ???

Postby medoomi » Thu Jun 11, 2009 8:10 pm

So... I've tried to download os802.img about three times, each time resulting in a different md5 checksum. I've tried two different checksum utilities...
Any suggestions where the problem is occuring?
I'm on XP, and downloading with Firefox.
Thanks
medoomi
 
Posts: 4
Joined: Thu Jun 11, 2009 6:04 pm

[SOLVED]: MD5 checksum for os802.img ???

Postby medoomi » Thu Jun 11, 2009 9:21 pm

Wow, so I don't know what's causing it, but the laptop I'm using to download the files just can't get a right md5sum.

Don't know why that's happening. I've tried it six or so times, using firefow, safari & IE. I've also tried three hash verification utilities. All of them report wrong numbers & each time I try to obtian a number, it reports a different number again (?). I've even re-imaged the hard drive to a version I'm sure is OK.

If anyone knows what causes this, please let me know. Apart from that, using a different computer, I have the right download os802.img & the OLPC has re-installed just fine.
medoomi
 
Posts: 4
Joined: Thu Jun 11, 2009 6:04 pm

Re: MD5 checksum for os802.img ???

Postby kmag » Sat Feb 20, 2010 6:17 am

In short, HTTP and all other protocols that rely on the TCP checksum for error detection are a poor choices for transferring large files.

I'd run memtest86 or memtest86+ just to make sure you don't have a problem with corrupted RAM. However, the dirtly little secret about TCP is that its error checking is terrible. Its 16-bit additive checksum was chosen not for its ability to detect errors, but because it's dead simple to calculate on 1970s-erra hardware, in a platform-independent (endian-agnostic) way. Better error detection at lower levels usually hides the problems with the TCP error detection, but these checksums (such as the ethernet CRC-32) aren't end-to-end, so you're pretty well protected from bits getting scrambled on the wire/fibre, but much less so from bits getting scrambled as routers copy them from one wire/fibre to another along the way.

The solution is to use a higher level protocol with a better checksum whenever possible. HTTPS uses TLS/SSL, which has a very strong end-to-end integrity check, so use HTTPS instead of HTTP for file downloads whenever possible. Bittorrent, Gnutella, and many other P2P protocols also use strong checksums along with efficient retransmission of corrupted blocks. (Note that Kaaza/FastTrack is a notable exception, using a terrible terrible checksum that protects a smaller and smaller percentage of the bytes from corruption as file size increases.)

I should really write a Python script that will download a file and check its md5sum, downloading a second time if the md5sum is wrong, and seeing where the two downloads differ, then getting a third download of just the corrupted bytes to figure out which values are correct. I wrote a similar script for the Windows 7 Beta DVD because I was getting such a high corruption rate, but that read checksums out of a .torrent file and used plain HTTP to download all of the actual data from Microsoft. I also have a prototype Apache plugin lying around somewhere that adds makes Gnutella TigerTree checksums available to Gnutella-aware HTTP clients.
kmag
 
Posts: 2
Joined: Sat Feb 20, 2010 5:47 am

Re: MD5 checksum for os802.img ???

Postby kmag » Sat Feb 20, 2010 8:19 am

It's also possible that you have a bad block on your HD that isn't being detected as such by the drive firmware and/or the drive has run out of spare blocks for remapping. To avoid this problem, when you get a corrupted download, move the file to the trash but don't empty the trash before downloading a new copy of the file. That way, the bad block is still used by the file that's in the trash, and not available for re-use by the filesystem.

If you empty the trash (or use rm or del) before the second copy is finished downloading, you may very well end up re-using the bad physical block on disk for the new download, also corrupting it.

Also, some home routers have a "gaming" mode that will look for what appears to be your IPv4 address in the incomming packets, and replaces those 4 bytes with your local NAT'd IP address (and the reverse for outgoing traffic). However, this is a terrible terrible feature and I'm not aware of any routers that use the feature on ports 80 or 443 (or 21-23, for that matter). It's only an ugly ugly hack for poorly written games, and those routers that use it have enough sense to avoid the hack for the ports for commonly used protocols. This almost certainly isn't your problem, but I mention it for completeness in case someone else finds this page while searching for a solution to some other download corruption problem. Here's hoping IPv6 rolls out faster, and that IPv6 routers will have real firewalls rather than pretending NAT is security feature.
kmag
 
Posts: 2
Joined: Sat Feb 20, 2010 5:47 am


Return to Sugar, Linux and System Software

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron