TOSECdev Forum

TOSEC Project => Database / Datfiles => Topic started by: PandMonium on January 20, 2011, 05:49:42 PM

Title: TOSEC Release (2011-01-19)
Post by: PandMonium on January 20, 2011, 05:49:42 PM
For those who haven't noticed, we finally released a new pack of dats yesterday.

Today someone asked (in IRC) how much space the entire collection of sets would take. I have no idea of the real size (compressed), still i will provide a few numbers on what changed between the last 2 releases (2009-12-27 to 2011-01-19), sizes are raw/not compressed.

Feel free to use this topic or create a new one if you have any question about the release.

TOSEC (main) statistics:

Release       |  2011-01-19  |  2009-12-27  |  difference
----------------------------------------------------------
Datfiles      |       1,002  |         883  |      + 119
Sets          |     372,706  |     334,165  |   + 38,541
Roms          |     439,110  |     384,841  |   + 54,269
Size          |    169.09GB  |    156.56GB  |    + 12.53
ZZZ-UNK       |       3,874  |       3,969  |       - 95
Invalid Sets  |          14  |          14  |          0
---
Images        |     369,418  |     330,796  |   + 38,622
Alts          |      55,881  |      53,279  |    + 2,602
Cracked       |      44,762  |      45,691  |      - 929
Fixed         |       1,129  |       1,251  |      - 122
Hacked        |      24,616  |      19,340  |    + 5,276
Modified      |       3,032  |       2,987  |       + 45
Pirated       |       1,698  |       1,684  |       + 14
Trained       |      13,914  |      13,757  |      + 157
Translated    |       2,949  |       2,869  |       + 80
Virus         |         997  |         999  |        - 2
Bad Dumps     |      10,064  |       9,090  |      + 974
Over Dumps    |       1,597  |       1,767  |      - 170
Under Dumps   |         965  |         993  |       - 28
Good Dumps    |       2,414  |       2,413  |        + 1


TOSEC-ISO statistics:

Release       |  2011-01-19  |  2009-12-27  |  difference
----------------------------------------------------------
Datfiles      |         100  |         100  |          0
Sets          |       5,239  |       5,202  |       + 37
Roms          |      33,296  |      33,032  |      + 264
Size (GB)     |    2,634.06  |    2,604.01  |    + 30.05
ZZZ-UNK       |           0  |           0  |          0
Invalid Sets  |           0  |           0  |          0
---
Images        |       5,239  |       5,202  |       + 37
Alts          |          11  |          11  |          0
Cracked       |          11  |          11  |          0
Fixed         |           0  |           0  |          0
Hacked        |           3  |           3  |          0
Modified      |          94  |          94  |          0
Pirated       |           0  |           0  |          0
Trained       |           5  |           5  |          0
Translated    |           0  |           0  |          0
Virus         |           0  |           0  |          0
Bad Dumps     |           9  |           9  |          0
Over Dumps    |           1  |           1  |          0
Under Dumps   |           0  |           0  |          0
Good Dumps    |       3,471  |       3,437  |       + 34


TOSEC-PIX statistics:

Release       |  2011-01-19  |  2009-12-27  |  difference
----------------------------------------------------------
Datfiles      |         274  |         128  |        146
Sets          |      11,283  |       4,392  |    + 6,891
Roms          |      11,776  |       4,393  |    + 7,383
Size (GB)     |      219.20  |      120.78  |    + 98.42
ZZZ-UNK       |           0  |           0  |          0
Invalid Sets  |           2  |           0  |          0
---
Images        |      11,281  |       4,392  |    + 6,889
Alts          |         349  |          98  |      + 251
Cracked       |           0  |           0  |          0
Fixed         |           0  |           0  |          0
Hacked        |           0  |           0  |          0
Modified      |           0  |           0  |          0
Pirated       |           0  |           0  |          0
Trained       |           0  |           0  |          0
Translated    |           0  |           0  |          0
Virus         |           0  |           0  |          0
Bad Dumps     |           0  |           0  |          0
Over Dumps    |           0  |           0  |          0
Under Dumps   |           0  |           0  |          0
Good Dumps    |           0  |           0  |          0
Title: Re: TOSEC Release (2011-01-19)
Post by: Cassiel on January 20, 2011, 06:12:14 PM
TOSEC (main) is around 95GB when RAR'd at max compression....
Title: Re: TOSEC Release (2011-01-19)
Post by: BlueArrow on January 21, 2011, 02:32:49 AM
Great work dudes!!

:D
Title: Re: TOSEC Release (2011-01-19)
Post by: Gruby on January 22, 2011, 06:41:11 AM
A little bug. ???
Commodore C64 - Collections - Blast from the Past (TOSEC-v2010-10-16)
All these files is part of the Commodore C64 - Compilations - Games - [D64] (TOSEC-v2006-03-30)  ::)

and a lots of collection discs (Like C64CD'96, High Voltage) is placed in other C64 dats (Games or stg..).
Maybe need a crc duplicates search ?
Title: Re: TOSEC Release (2011-01-19)
Post by: Kodoichi on January 22, 2011, 08:18:26 AM
Fuck yeah \o/

Now what ROM manager do I use? I deinstalled Romcenter recently, Clrmame is out of question anyway and others still seem to be beta versions. Should be able to read 7Zip-packed files.
Title: Re: TOSEC Release (2011-01-19)
Post by: Duncan Twain on January 22, 2011, 09:35:45 AM
Hi Gruby,

You're absolutely right, I'm aware of it. The idea is to keep collections as a whole, and this could create duplicates over folders.

Duncan

A little bug. ???
Commodore C64 - Collections - Blast from the Past (TOSEC-v2010-10-16)
All these files is part of the Commodore C64 - Compilations - Games - [D64] (TOSEC-v2006-03-30)  ::)

and a lots of collection discs (Like C64CD'96, High Voltage) is placed in other C64 dats (Games or stg..).
Maybe need a crc duplicates search ?
Title: Re: TOSEC Release (2011-01-19)
Post by: Diaboł on January 22, 2011, 10:30:24 AM
There are several DATs with SHA1 checksums: Funtech - Super Acan, IBM PC Compatibles, Sega Mark III & Master System + a few more. I didn't scan everything yet so I can't list them all.
Title: Re: TOSEC Release (2011-01-19)
Post by: TKaos on January 22, 2011, 11:17:51 AM
@Duncan Twain
Well it's confusing and it's something that PandMonium had to know so he could write it in the release text, else people might question more why it is like that.
I personally dislike the idea a bit of having 1 file in 2 DATs but seems to be usefull in those collection cases.
Title: Re: TOSEC Release (2011-01-19)
Post by: Aral on January 22, 2011, 12:04:53 PM
Agreed Tkaos, If they are part of a collection they need to be removed from the original dat.
Title: Re: TOSEC Release (2011-01-19)
Post by: Duncan Twain on January 22, 2011, 12:33:49 PM
sure, still lot a work to be done in the C64 DATs; it's on my list.
Title: Re: TOSEC Release (2011-01-19)
Post by: TKaos on January 23, 2011, 05:29:37 PM
Did I miss something?
What happened to the Acorn 8bit DATfile?
Title: Re: TOSEC Release (2011-01-19)
Post by: PandMonium on January 23, 2011, 07:44:25 PM
What do you mean?
Title: Re: TOSEC Release (2011-01-19)
Post by: TKaos on January 23, 2011, 07:52:24 PM
I mean where is the DAT for it, can't see it in current release pack.
Have they been moved to somewhere and I just don't know about it?
"H:\Acorn\8bit\Utilities"
619 files
Title: Re: TOSEC Release (2011-01-19)
Post by: PandMonium on January 23, 2011, 08:52:49 PM
Hey TKaos, the dat you talk about is probably "Acorn 8bit - Utilities (TOSEC-v2006-03-16)".
It was in the 2007 release but was removed in the previous release already. I think the files were moved to several dats by Cassiel, but don't remember exactly how they were sepparated. The first 3 sets for example are now in the dat "Acorn BBC - Applications - [BIN]".
Title: Re: TOSEC Release (2011-01-19)
Post by: TKaos on January 23, 2011, 09:02:16 PM
Well I didn't scan the whole collection since a while so it's really outdated. :P
Title: Re: TOSEC Release (2011-01-19)
Post by: Diaboł on January 24, 2011, 05:39:59 AM
TKaos, check the following files:

Atarian Attack (19xx)(Münzenloher)(DE)[req OSb].zip
Basketball (2004)(Miller, Alan - Lüdtke, Matthias)(DE)(en)

from the Atari 8bit - Games - [XEX] (TOSEC-v2010-11-14). Because of the Germanic umlaut it's impossible to rebuild the set (using cmpro). I was trying to rename the files by hand but the ROM manager is still complaining about missing and unneeded files.
Title: Re: TOSEC Release (2011-01-19)
Post by: TKaos on January 24, 2011, 05:28:34 PM
Thanks for the info, I usually use "ue" instead of "ü".
Fixed DATfile is uploaded to our FTP, maybe PandMonium can create a newsitem later with it.
Title: Re: TOSEC Release (2011-01-19)
Post by: PandMonium on January 25, 2011, 03:53:16 PM
@Diaboł, can i ask you please just out of curiosity to open the datfile in a text editor, search for the first set you listed and paste it here? :)
Title: Re: TOSEC Release (2011-01-19)
Post by: Diaboł on January 25, 2011, 04:38:45 PM
Sure (from the Windows Notepad):

game (
   name "Atarian Attack (19xx)(Münzenloher)(DE)[req OSb]"
   description "Atarian Attack (19xx)(Münzenloher)(DE)[req OSb]"
   rom ( name "Atarian Attack (19xx)(Münzenloher)(DE)[req OSb].xex" size 4115 crc 04e7180d md5 46038bb763f3e80be64d037d21c8dacf )
)

If you try  to rebuild this set it will get funny characters instead of ü.
Title: Re: TOSEC Release (2011-01-19)
Post by: Gruby on January 26, 2011, 03:50:43 AM
Zipping with correct name works if you disable "OEM / ANSI Conversion" from clrmamepro zipper for that dat set.
Title: Re: TOSEC Release (2011-01-19)
Post by: PandMonium on January 27, 2011, 02:58:51 AM
Thanks for the hint Gruby!
Title: Re: TOSEC Release (2011-01-19)
Post by: sadman on February 19, 2011, 03:12:03 AM
i've made a script that checks for duplicate romsets, the results are attached
could help to weed out some unnecessary ones
Title: Re: TOSEC Release (2011-01-19)
Post by: Crashdisk on February 20, 2011, 08:52:25 AM
Already fixed for Amiga set.
apart Collection of course  ;)
Title: Re: TOSEC Release (2011-01-19)
Post by: PandMonium on February 22, 2011, 03:00:22 PM
Hi sadman, thanks for that! We noticed that too a bit after the release, hope the renamers of each system will handle them but i bet some will pass. There are some funny cases of files that were marked as bad dumps or unknown sets and after all belonged already to other system. In C64 as you see there was a massive misunderstanding but the problem will hopefully be solved in the next releases.

Thanks again! ;)