TOSECdev Forum

TOSEC Project => Database / Datfiles => Topic started by: Cassiel on September 05, 2010, 03:35:19 PM

Title: DAT Fields in WIP
Post by: Cassiel on September 05, 2010, 03:35:19 PM
Guys,

Have just noticed some of the WIP DATs incorrectly include sha1 hash checksums.

Remember that the only fields to be included in DATs should be: name, description, rom, size, crc and md5 (or that's how its always been anyway!).

Whilst I don't think there's anything wrong with including sha1 as well as md5 hashes (in fact you could easily argue its a positive change!), we should at least be consistent.

(arww..... 300th post and I sound like an overly anal douche!  ::)  )
Title: Re: DAT Fields in WIP
Post by: PandMonium on September 06, 2010, 02:17:21 AM
well spotted! Nothing against sha1 obviously but all or no sets should use it indeed :)
Title: Re: DAT Fields in WIP
Post by: Diaboł on September 06, 2010, 07:44:09 AM
Looks like I have to fix my DATs  :P
Title: Re: DAT Fields in WIP
Post by: Aral on September 06, 2010, 10:32:10 AM
All the PIX dats have sha1 enabled i think.
Title: Re: DAT Fields in WIP
Post by: PandMonium on September 06, 2010, 02:24:02 PM
Well, all ISO dats also use SHA1 instead of md5 i think so i guess we are talking about the disk branch currently.
As said, i'm not opposed to it, was just saying that we should use all or none, instead of having some dats with / without sha1 and others with / without md5.
Title: Re: DAT Fields in WIP
Post by: Cassiel on September 06, 2010, 09:07:22 PM
No, its a mix of all 3 branches. It's definitely all new DATs though (I guess since TIM didn't ever accept sha1).

For example:

Main:
Texas Instruments CC-40
Texas Instruments TI-XX
Visual Technology Visual 1050

ISO:
is a real mix, some do some don't.

PIX:
most (all?) PIX include sha1
Title: Re: DAT Fields in WIP
Post by: PandMonium on September 06, 2010, 10:46:06 PM
Unfortunately i don't have all that info available for ISO and PIX currently and Main is based on older dats (from June), anyway you're right about it and there are sha1 hashs in the following dats too:

Code: [Select]
Amstrad GX4000 - Games - [BIN]
Amstrad GX4000 - Games - [CPR]
Coleco ColecoVision - Applications
Coleco ColecoVision - Educational
Coleco ColecoVision - Games
Coleco ColecoVision - Public Domain
Funtech Super A'can - Games

like i said, not against it, but we should make it a rule and apply to all dats if it is to be used. I can't remember well but i think TIM didn't use sha1, not really sure.
Title: Re: DAT Fields in WIP
Post by: Cassiel on September 08, 2010, 12:57:09 AM
AFAIR, a 'roms_sha1' column existed in TIM.DB but it was never populated (nor enabled in the actual TIM application). No doubt another casualty of TIM's over reaching ambitions...

In fact, the only reason the DATs include today what they do is due to the historic lineage of what TIM could handle/wanted.

That doesn't mean we can't re-visit requirements now though...   :)

Maybe we should discuss weather to now include SHA-1, DAT versus XML, or any other beneficial info/fields that could be included as standard (things have moved forward somewhat in the last 5+ years!).

TKaos and [idoru]'s thoughts on this would be most beneficial...
Title: Re: DAT Fields in WIP
Post by: PandMonium on September 08, 2010, 01:35:21 PM
For me, there is no problem with it. Having both or going just from md5 to sha1, i only think a new release should be done first obviously (it could include a few dats already with the sha1).
Adding sha1 would mean generating all dats again so all would be modified, even if the only thing new would be the hashs, no change to the sets, so, should the version of them change too? I guess i'm a bit ahead, one step at a time. :P

As for XML dats, afaik they were not really accepted generally (at least around here) due to their complexity (compared to the previous format) when editing them manually, some renamers like to do that too. And there are always chars you can't use in XML (like & = & and other entities).

As a final note, adding sha1 or xml would increase the datsize too :P (I'm more interested in sha1 first btw)
Title: Re: DAT Fields in WIP
Post by: TKaos on September 09, 2010, 07:29:20 PM
The reason I dislike it is that I won't be able to use TIM anymore to make DATs, I really hate using clrmamepro for DATs.

Also would like to know what's the deal with having sha1, I'm not that familiar to it, so can someone explain the benefits of chaning from md5 to sha1?
Title: Re: DAT Fields in WIP
Post by: Cassiel on September 10, 2010, 01:10:15 PM
It wouldn't be changing from MD5 to SHA-1, just including the option for people that want to use it (i.e. you could use crc32, md5 OR sha1 to scan/rebuild files).

Both MD5 and SHA-1 are just cryptographic hash functions, and are fairly comparable (though technical SHA-1 is ‘better’). Both are obviously much more reliable than CRC32.

SHA-1 is very popular in the MAME/Arcade world.
Title: Re: DAT Fields in WIP
Post by: VG8020 on November 09, 2010, 10:40:31 PM
Hello,

Very interesting topic. In the MSX scene SHA1 is the rule so in order to be consistent I would opt for default crc32+sha1 or "maniac version" crc32+md5+sha1, in spite of the redundancy that would add.

I've used datutil to generate a new .dat filled with SHA1s. Dat Workshop Pro does not seem to be able to update to SHA1 all .dats at once. I tried, though. Now, is there any way to get all my MSX .dats updated consistently and recursively to SHA1? Only TOSEC MSX .dats ammount to about 18 files, apart from TOSEC MSX2 .dats... Doing it one by one every time, there's a new update can be a real pain... Any ideas?

Greetings,