Author Topic: Special DAT for bad dump  (Read 8464 times)

Offline Crashdisk

  • TOSEC Member
  • Full Member
  • ***
  • Posts: 248
Special DAT for bad dump
« on: September 25, 2012, 04:55:43 PM »
Hi,

I need to create a special file DAT with special specifications to clean the AMIGA database (but no only). Why?
 - Underdump
 - Overdump (unless track 80, 81,...)
 - Duplicate track (merged DMS. ex: DEMOa.dms with track 00-40 + DEMOb.dms with track 40-79)
 - Auto modification of bootvirus (counter, mutation as Lamer Exterminator)
 - Some mutilation caused by viruses (saddam disk validator) which are reversible.
 - bad dumping
 - Read error when transferring in ADF (bad sector)
 - Read error during dms dump (classic errdms tag...)
 ...
This is a real pollution of unnecessary files and unusable.
Many mistakes can be corrected, or sometimes identical files exist but without errors.
I've created a dat file but compliance with the TNC is complicated because we must not waste time on maintenance.
I mean that we should not change the name of a bad file because the good has changed its name.
At the moment I use this model:
3F76E4F50150DCBA5B57ED18101D1EA2 [73D36888][b saddam damage].adf
3F76E4F50150DCBA5B57ED18101D1EA2 [CAEA6070][b saddam damage].adf
...

the MD5 hash of the good version => 3F76E4F50150DCBA5B57ED18101D1EA2
the CRC32 of the bad version => [73D36888]
the cause of the bad version => [b saddam damage]

Or more long, a new tag "ZZZ-BAD-" :
ZZZ-BAD-3F76E4F50150DCBA5B57ED18101D1EA2 [73D36888][b saddam damage].adf
ZZZ-BAD-3F76E4F50150DCBA5B57ED18101D1EA2 [CAEA6070][b saddam damage].adf

Suggestions?



Offline TKaos

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 533
Re: Special DAT for bad dump
« Reply #1 on: September 26, 2012, 05:30:47 PM »
I dont really see the need to create special datfiles for all the other dumps, it only increases the amount of DATs we have.
It just creates extra work on DATs which I dont see a point of, anyway you didnt say if you need the DAT for yourself or if u want it for the project standard, I guess second.
But in the end it has been talked about lots of time, if you want perfect files only then you simply dont use TOSEC DATs and rather get a gamebase or DATs of projects that collect good dumps only.

Offline Crashdisk

  • TOSEC Member
  • Full Member
  • ***
  • Posts: 248
Re: Special DAT for bad dump
« Reply #2 on: September 26, 2012, 06:32:12 PM »
GameBase? I'm not talking about games, but dumps in general. We integrate sometimes damaged files by mistake or want of better. When a version is integrated without error, the former becomes annoying because you have to maintain it like any other fileset. Again we lose time to brew waste. I repeat, do not exclude the hack, crack, programming error or even viruses but the damage post production. The alternative would be to simply remove them from TOSEC db but it is the worst solution ....

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1316
Re: Special DAT for bad dump
« Reply #3 on: September 26, 2012, 11:24:03 PM »
Hey guys,

The aim of the project is to identify all these sets. Without them in the dats, renamers will not be able to identify them as something already checked/datted and will repeat work forever. That's why they are kept and renamed in the dats and there is no point in dumping all this information.

Still, i do think we should provide some way / tool to filter an existent dat, removing the sets a user might not want to be included in the datfile (based on our flags). Hopefully i will manage to do that someday :P

Offline Crashdisk

  • TOSEC Member
  • Full Member
  • ***
  • Posts: 248
Re: Special DAT for bad dump
« Reply #4 on: September 27, 2012, 01:11:05 PM »
Could we arrange a private space on the server to share dat files for internal use? early dat WiP, very bad files DB, ....

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1316
Re: Special DAT for bad dump
« Reply #5 on: September 28, 2012, 02:51:49 PM »
Sure, you can put them anywhere you like. Will pm you for details. :P

Offline mai

  • TOSEC Member
  • Full Member
  • ***
  • Posts: 175
Re: Special DAT for bad dump
« Reply #6 on: October 02, 2012, 07:33:54 PM »
Said it many times before, i dont like those overdump and underdump and also this unique [b errdms] flag, should not be included in TOSEC, all this bad dumps are result of incorrect dumping procedure.
In most cases, we have good dump from the same image.
« Last Edit: October 02, 2012, 07:35:26 PM by mai »

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1316
Re: Special DAT for bad dump
« Reply #7 on: October 02, 2012, 08:10:06 PM »
Well i understand the situation considering Amiga but i don't see any optimal solution, all have different problems.

Offline mai

  • TOSEC Member
  • Full Member
  • ***
  • Posts: 175
Re: Special DAT for bad dump
« Reply #8 on: October 02, 2012, 08:40:23 PM »
Well i understand the situation considering Amiga but i don't see any optimal solution, all have different problems.
I am aware, that i would never change anything with my opinion, in any case my job is rather (scene)software preserving, instead of collecting and including any garbage.
« Last Edit: October 02, 2012, 08:44:18 PM by mai »

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1316
Re: Special DAT for bad dump
« Reply #9 on: October 02, 2012, 08:49:32 PM »
Well, since this is a collaborative project, changes start with brainstorming and proposal of solutions. We shouldn't change things easily and without care because that was one of the causes of TNC complexity. Still, i do consider that in this case the problem is relevant / interesting and something so it will indeed be considered.

Offline Maddog

  • Global Moderator
  • Full Member
  • *****
  • Posts: 160
Re: Special DAT for bad dump
« Reply #10 on: October 02, 2012, 09:39:49 PM »
I support the idea of a "clean-up" of TOSEC dats.
I think we could add one dat per system (no need to have separate dats for Demos, Games etc since we already have tons of folder in a complete TOSEC tree) that includes all bad dumps from that system.

This way you get the best of both worlds.
-Remove junk files from regular dats.
-Reduce size of complete downloads for anyone that only wants a complete usable collection.
-Keep hashes of known bad dumps in a specific place and still be able to recognize them easily.
-Obsessive collectors of roms that want every piece of crap out there can still download the bad files if they wish. All others will just ignore the "bad dump" dat.

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1316
Re: Special DAT for bad dump
« Reply #11 on: October 02, 2012, 11:14:36 PM »
I got the idea.

Based on what Crashdisk and mai explained, there are some systems (namely amiga and c64) where a lot of dumps float. Many (most?) of those dumps are bad dumps but there are also a lot of variations of original sets created by dumping modified disks (unfortunately not write protected). Our goal is to catalog images and not to create a purely original, best dumps collection, however the situation in these systems is indeed ridiculous.

By adding so much of these sets in the last years, the project in part even helped preserving them. I recall that idoru was already avoiding the addition of new alts and bad dumps for amiga due to the same reasons and i do support the idea, since following this path will end in 50000 new repeated or unused sets.


Now the other side... for good and bad those unrenamed sets will not go away since they are datted and collected by some as we know. If these sets don't get cataloged, the current renamers will keep receiving the same sets to verify again and again. This probably already happens today, with mai and others testing sets (which don't get renamed) that were tested by idoru and others before (and also not renamed).

I do support the addition of (part of) these unrenamed sets, still if we add all these to the current dats the number of sets/dupes will skyrocket with many being even unusable. The creation of different dats might be the best (or less bad?) solution and should be done to help the renamers even if they take some time to be introduced in the catalog.

Still there are many questions. What really is a bad dump? What should be done with alternates and the many different types that exist (i imagine this may be a problem with the same dimension than the bad dumps' one)? What about sets that where only bad dumps exist? Over and under dumps?
Even more, is the fact that many of the current bad dumps and alts might after all be badly renamed (as mai is discovering :)). Plus, using a single dat per system might create an huge, unusable dat with "various" types of software but creating various is in many cases plain stupid. Here, the best solution would probably depend from system to system and its dimensions, just as it happens to the other dats.

Finally, i think (albeit i don't have knowledge about every other system) that this problems are common in a few systems.
Anyone has better ideas or opinions against the solution?


... and now as recreation, some random stats:
83,26% of the alternate images are in 5 systems (C64, Amiga, Spectrum, Atari ST and Atari 8bit), with 60.737 of a total of 72.494 alts.
69,51% of the bad dumps are in 5 systems (NES, Amiga, N64, Megadrive, C64) with 7.649 in 11.004.
If we look at percentages, we have Robotron Z1013 with 43% alts and NEC SuperGrafx where 83,3% of the sets are bad dumps (however the dat is just 6 dumps of the same software with 5 bad dumps), Game boy comes second with 54,65% of the sets being bad dumps but the dat is really small too.
All these numbers would actually change a lot by adding these dumps left out until now. :P

Offline mai

  • TOSEC Member
  • Full Member
  • ***
  • Posts: 175
Re: Special DAT for bad dump
« Reply #12 on: October 03, 2012, 09:39:52 AM »
I like to use examples to show my opinion:
Charlie J Cool (1996)(NRC)(Disk 1 of 2)
Charlie J Cool (1996)(NRC)(Disk 1 of 2)bad
Charlie J Cool (1996)(NRC)(Disk 2 of 2)
Charlie J Cool (1996)(NRC)(Disk 2 of 2)bad
This are unnecessary overdumps.
First 880kb(standard Amiga image file size) data are byte per byte exactly the same, where is the reason to collect and catalog such stuff.
All those images <880kb and >880kb are result of faulty dumping.

Offline Maddog

  • Global Moderator
  • Full Member
  • *****
  • Posts: 160
Re: Special DAT for bad dump
« Reply #13 on: October 03, 2012, 10:19:24 AM »
Still there are many questions. What really is a bad dump? What should be done with alternates and the many different types that exist (i imagine this may be a problem with the same dimension than the bad dumps' one)? What about sets that where only bad dumps exist? Over and under dumps?
Even more, is the fact that many of the current bad dumps and alts might after all be badly renamed (as mai is discovering :)). Plus, using a single dat per system might create an huge, unusable dat with "various" types of software but creating various is in many cases plain stupid. Here, the best solution would probably depend from system to system and its dimensions, just as it happens to the other dats.

Finally, i think (albeit i don't have knowledge about every other system) that this problems are common in a few systems.
Anyone has better ideas or opinions against the solution?

I don't look to go the NoIntro way of 1 dump per game. Sometimes it might be hard to say which (alt) is best and it's even possible that spending energy on that isn't really needed. So, alts that are not CLEARLY bad should just stay in their existing dats. On the other hand, if something is absolutely, positively bad/over/under (like what Mai illustrates above...), then it should be removed from the normal dat and go to a clearly marked "Bad" dat.
This helps all ways:
-Less clutter for normal people
-One more dat to complete for the Pokemon "wanna have everything" guys
-Availability of the file in case it's misidentified as bad (nothing prevents moving something back from the "Bad" to the "Regular" dat if required)
-Fewer unidentified files to be checked and re-checked ad infinitum for the renamers

On the other hand, if something only exists as a "bad" dump, then it should stay in the regular dat, since that is the file people will want to collect (at least until a better option is available one day). These cases should be relatively few. Will need hand picking, ie no automatic creation of the "Bad" dat just by looking at the ["b"] flag, unless we try the MAME way and have a new flag along the lines of "Best available/No good dump known" for those few files, in which case the "Bad" dat can still be created automatically.

I supported the idea of a single "Bad" dat per system, regarding it as a pool of shit. Most people would want to avoid it completely, but some might still want to have it in their back yard and some others might even want to dive in to search for a single missing gem. But there's not any truly compelling reason to have multiple shitpools around. :P
If the renamers actually feel it's better to have multiple "Bad" dats for Demos, Games etc, it's fine by me. Suppose each one will know the needs of the system they are working with better.
Hope you get my idea...  ;)

Offline Crashdisk

  • TOSEC Member
  • Full Member
  • ***
  • Posts: 248
Re: Special DAT for bad dump
« Reply #14 on: October 03, 2012, 01:18:34 PM »
I updated my program to detect a "new" type of corruption and the result is afflicting. 19 new bad files (need mai confirmation) for "Commodore Amiga - Games - [ADF]" starting from 0 to A.This is just the beginning of a flood of [b useless] ...
http://eab.abime.net/showpost.php?p=843000&postcount=526
« Last Edit: October 03, 2012, 01:33:21 PM by Crashdisk »