Author Topic: Be prepared for a flood of non-Tosec ADFs  (Read 7209 times)

Offline Kodoichi

  • Full Member
  • ***
  • Posts: 162



Offline Cassiel

  • Administrator
  • Hero Member
  • *****
  • Posts: 1574
    • Email
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #1 on: April 14, 2012, 03:10:15 PM »
5000 geek points to this guy. That is fucking genius!

Only sad part is the LEGO solution didn't work. That would have made it PERFECT. LEGO (obviously) rules...   :)

Offline Crashdisk

  • TOSEC Member
  • Sr. Member
  • ****
  • Posts: 264
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #2 on: April 14, 2012, 10:13:27 PM »
In terms of the result with dumping to the blind, I fear the worst ...

Offline Dweller

  • Newbie
  • *
  • Posts: 5
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #3 on: April 15, 2012, 07:42:25 AM »
Hi =)

Yes, I'm dumping blind, but I'm capturing the raw stream format, and ADF, using a Kryoflux. The kryoflux does a pretty decent job of attempting track rereads (with a head reposition) when the track looks like a valid Amiga track, but has bad checksums when read etc. With the raw mfm stream, I can go back and have another go at any disk, if I'm willing to attempt an alternative interpretation of the stream data. It's pretty much the best option I had for reading via automation, barring running the disks through multiple brands of disk drive etc.

The vast majority of the disks are Amiga, and are data, (a lot of sound samples, older public domain disks, source code, midi files, soundtracker/octamed, etc). Then there are the inevitable stack of 'yet another cracked version of blah', some of which have been written over originals, in addition, there are some original games/apps in there, a LOT of coverdisks, and a decent smattering of MSDOS disks, since if you are going to read disks.. might as well read all of them =)

As for quantities.. once I'd finally finished digging disks out of cupboards, boxes, etc.. I read just over 4000 disks, the 5000 figure being passed around being an estimate based on how many disks were in each crate etc.. that's still way too many to read by hand =) Of those, around 2000 were my own disks, 500 or so belonged to a friend who loaned me a catweasel to try to read them (and made me realise doing it by hand would be a bad plan...), and the remainder were 'free' when I picked up a 2nd hand A1200 at a carboot sale.

I've now got a Java app that scans the dirs, lets me navigate by the disk image, and view information about the disk. The app runs ADFINFO against each adf, and associates the output to the jpeg, this gives me the volume name for any proper amigados disks. Also gives basic info on checksum failures, limited virus detection, filesystem type, and a crc32. In addition the java app calculates its own crc32 for the image, which over 4k disks has disagreed with the AFDINFO crc32 once.. which is odd.. someone has an error in how to build a crc32.

The app loads all the TOSEC amiga dats, and builds an internal map of crc32 to disk(s) (allowing for clashes). The map is then used to attempt to identify the adfs, where present in the dirs. This approach gives a hit rate of about 1 in 5, that leaves the other 4 in 5 as being not known to the dats, requiring manual handling.

1 in 5 is a good sign, enough to know the rig is properly reading disks.. the 4 in 5 includes non Amiga disks, which are not expected to match, and includes many many disks that likely are in TOSEC already, but my version is altered, either through virus damage, bitrot, icon timestamp updates due to being used write-enabled, 'protective' bootblock installation, different intro/group, etc.

Once I'm done, I'll export a dat from the app containing checksums & info for the disks I have, that other people may have. I doubt you guys will want many (if any) of the entries from it, but it may be of use to anyone following in my footsteps in years to come.

I may make select subsets of the disks available online afterward, especially if they are content not known to TOSEC, but before I can do that, I've got to tag the collection to remove the ones with my old files on =)

Offline Crashdisk

  • TOSEC Member
  • Sr. Member
  • ****
  • Posts: 264
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #4 on: April 15, 2012, 10:00:25 AM »
You thought of everything and I congratulate you for this approach very professional! ^^
Good luck for "the 4 in 5"

Offline Cassiel

  • Administrator
  • Hero Member
  • *****
  • Posts: 1574
    • Email
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #5 on: April 15, 2012, 02:49:24 PM »
I stand by my original statement, genuinely impressed at this.

Looking forward to following your progress, especially if we can work together to get any of your unique images/softwares into the project.

Oh, and welcome to the forum  :)

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1332
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #6 on: April 15, 2012, 03:38:59 PM »
Hi Dweller,

First welcome here. Congratulations for the solution and the detailed description of it in your posts, it is really interesting to read.
Good luck with it and hope you can find some rarities there. There are a few more dats if you might want to compare (there is a huge UnRenamed Commodore i think) but as you said, many of the disks might not mach for the known reasons.

Offline Dweller

  • Newbie
  • *
  • Posts: 5
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #7 on: April 15, 2012, 06:56:48 PM »
At the mo I'm using all the dats starting "Commodore Amiga" from the download at http://www.tosecdev.org/index.php/downloads/category/15-2012-02-29

If you know where I can get more.. I can just add them to the folder, I tag any matches with the version / author / name of the dat file the match came from, and can handle multiple matches from different dats fine. (Although my dat parser is a bit .. umm .. hacky.. it works ;p) Please point the way to where the UnRenamed one can be found =)

At this stage just getting the matches at all is a headstart =)

I did this mainly to regain the physical space taken by the disks.. Figured a few people would be interested, but the number of hits I've had has amazed me.. (>140k views of the video! which yes, isn't the most stunningly high adrenaline pounding rush that it could have been.. next time I'll let Michael Bay direct it ;p expect slow motion eject sequences with disk platters rotating overhead while the flash guns for the camera go off like large explosions.. It was originally just meant to illustrate the blog post, but try explaining that in hungarian, estonian, etc!)

It would seem no-one enjoys the prospect of having to read data back from disks they own, regardless of how few they may have, and the idea of avoiding the task has somewhat widespread appeal ;p

If anyone here fancies building something similar, I'm happy to answer questions.






Offline Dweller

  • Newbie
  • *
  • Posts: 5
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #8 on: April 19, 2012, 12:39:54 PM »
A lot of hunting.. and I've found the unrenamed commodore dat =)

Found yoshi's webpage at http://www.freewebs.com/yoshizuki/ which has a link to skydrive with the latest ones in..

Going need to tweak my code to integrate this, as the structuring isn't quite aligned with the way I'm using the tosec dats (if I use the unrenamed as-is, i'll think I've found many matches for disks of the 'amiga adfs [1]' game ;p)

Still a very nice resource of filename/adf/crc32 mappings.. thanks for the heads up.

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1332
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #9 on: April 19, 2012, 05:09:11 PM »
A bit late since you already found it. Yoshi's main site is: http://yoshizuki.webs.com/
There are a few dats around with sets that are not currently renamed in TOSEC or other projects. Many of them might be just garbage and alternates due to simple use of non copy protected disks and all that. Still they are a good source of information for you to know what's available somewhere out there and identify possible valuable sets that nobody has.

Offline Cassiel

  • Administrator
  • Hero Member
  • *****
  • Posts: 1574
    • Email
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #10 on: April 19, 2012, 08:46:27 PM »
RE Yoshi

They may be some use to you, but remember they very much live up the title 'Unrenamed'....  ;)

Offline Dweller

  • Newbie
  • *
  • Posts: 5
Progress update on processing adf images..
« Reply #11 on: April 20, 2012, 02:05:18 PM »
Bit of tweaking of the parser complete, processing the unrenamed dats added an additional 15652 crcs to the pool. (now at 79594 total)..

My hit rate is now up at slightly better than 1 in 4..

The unrenamed dats has mainly given good hit responses on..
 - Amiga Computing coverdisks.  (another 6 or so)
 - Amiga Format coverdisks (23 or so)
 - Amiga Format subscriber disks. (7 hits)
 - Amiga Format 'Collection' disks. (4 hits)
 - Amiga User International coverdisks. (3 hits)

As well as quite a few apps / games..  and less useful hits like "DISK11.ADF" ;p   (update, checked the photo for DISK11.ADF.. "Monkey Island 2 - Disk 11" is scrawled on it)

I'll go through & verify the matches brought from unrenamed.. none of the titles matched sound like titles that I wouldn't have had.. so overall a good result..
« Last Edit: April 20, 2012, 02:11:13 PM by Dweller »

Offline Crashdisk

  • TOSEC Member
  • Sr. Member
  • ****
  • Posts: 264
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #12 on: April 21, 2012, 12:09:10 AM »
In the next issue of TOSEC, DAT coverdisk will have a good update.
That will probably help a little for other unnamed ;D

Offline Dweller

  • Newbie
  • *
  • Posts: 5
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #13 on: April 21, 2012, 08:55:22 AM »
Cool.. I think I'm now at around 1800 'unique' adf images in my set..

That's already quite a nice drop from my start point, although it's mainly achieved through not considering any data on 720k/1.44mb pc disks, and removal of dupes.. specifically a large number of adfs coming back with the same crc, and browseable as empty.

 (I need to check, as the crc in question IS in tosec, in the coverdisk dat, for an amiga power disk.. I'll update with the one in question.. it's extremely unlikely I have hundreds of copies of that coverdisk, it's slightly more plausible that coverdisk shares a crc with an empty disk, but not by much).

I also need to check if some of the non standard formatted amiga disks made it as adfs, or if they are only stored as raw stream data, although theres only a hundred or so of those.. (unless you count all those hdd backup disk sets, which don't seem to be a normal amigados format.. )


Offline Cassiel

  • Administrator
  • Hero Member
  • *****
  • Posts: 1574
    • Email
Re: Be prepared for a flood of non-Tosec ADFs
« Reply #14 on: April 23, 2012, 10:30:49 PM »
You might want to grab the latest TOSEC release too (v2012-04-23) since Crashdisk has updated the Coverdisk DAT...