vg8020 WIP

[VG8020]

Hello,

First WIP topic! This one deals with TOSEC PIX. I've made use of previous MSX PIX dat files:

Code: [Select]

25 files @ MSX - Magazines - MSX Micro (BR) (TOSEC-v2010-01-30_CM)
2 files @ MSX - Magazines - MSX Top Class (BR) (TOSEC-v2010-01-30_CM)
and merged them into my new MSX publication dat files:

Code: [Select]

109 files @ MSX - Fanzines (TOSEC-v2011-11-15_CM)
251 files @ MSX - Magazines (TOSEC-v2011-11-15_CM)
17 files @ MSX -Misc (TOSEC-v2011-11-15_CM)
So easily typed, so much work behind, just as every TOSEC-minded datter knows indeed  

The 3 dat files can be located in my TOSEC uploads folder. Yes, splitting dats on a publication basis seems like a good idea I'll have to deal with at some point. Currently the most relevant issue is not to miss forthcoming TOSEC PIX update, whenever that happens in the near future.

Greetings,

[VG8020]

Back in May I was busy with dat chores and much editing was done following the latest TOSEC MSX release up to date (by then):

TOSEC - DAT Pack - Complete (1528) (TOSEC-v2011-04-07)

Just for everyone to keep on track, the following and lastest TOSEC MSX dat set so far (released only a few days ago):

TOSEC - DAT Pack - Complete (1638) (TOSEC-v2011-11-11)

does not include any update. Therefore, is the same as as the previous one (v2011-04-07).

I'll go into detail below. However, there is further level of detail in the log files from the dat comparisons.

command: mamediff old.dat new.dat

Code: [Select]

MSX MSX - Compilations - [CAS] (TOSEC-v2009-09-01_CM) versus MSX MSX - Compilations - [CAS] (TOSEC-v2011-04-23_CM)
240 files renamed

MSX MSX - Compilations - [DSK] (TOSEC-v2007-01-01_CM) versus MSX MSX - Compilations - [DSK] (TOSEC-v2011-04-23_CM)
27 files renamed

MSX MSX - Firmware (TOSEC-v2009-09-01_CM) versus MSX MSX - Firmware (TOSEC-v2011-04-23_CM)
105 files renamed

MSX MSX - Magazines - [CAS] (TOSEC-v2007-01-01_CM) versus MSX MSX - Magazines - [CAS] (TOSEC-v2011-04-23_CM)
133 files renamed

MSX MSX - Magazines - [DSK] (TOSEC-v2009-09-01_CM) versus MSX MSX - Magazines - [DSK] (TOSEC-v2011-04-23_CM)
234 files renamed

MSX MSX - Various - [CAS] (TOSEC-v2009-09-01_CM) versus MSX MSX - Various - [CAS] (TOSEC-v2011-04-23_CM)
3 files removed
1014 files renamed

MSX MSX - Various - [DSK] (TOSEC-v2009-09-01_CM) versus MSX MSX - Various - [DSK] (TOSEC-v2011-04-23_CM)
1 file removed
1720 files renamed

MSX MSX - Various - [ROM] (TOSEC-v2009-09-01_CM) versus MSX MSX - Various - [ROM] (TOSEC-v2011-04-23_CM)
1063 files renamed

MSX MSX2 - Compilations - [CAS] (TOSEC-v2007-01-01_CM) versus MSX MSX2 - Compilations - [CAS] (TOSEC-v2011-04-23_CM)
4 files renamed

MSX MSX2 - Compilations - [DSK] (TOSEC-v2007-01-01_CM) versus MSX MSX2 - Compilations - [DSK] (TOSEC-v2011-04-23_CM)
42 files renamed

MSX MSX2 - Various - [CAS] (TOSEC-v2009-09-01_CM) versus MSX MSX2 - Various - [CAS] (TOSEC-v2011-04-23_CM)
5 files renamed

MSX MSX2 - Various - [DSK] (TOSEC-v2009-09-01_CM) versus MSX MSX2 - Various - [DSK] (TOSEC-v2011-04-23_CM)
2138 files renamed

MSX MSX2 - Various - [ROM] (TOSEC-v2009-09-01_CM) versus MSX MSX2 - Various - [ROM] (TOSEC-v2011-04-23_CM)
302 files renamed

MSX MSX2+ - Various (TOSEC-v2009-09-01_CM) versus MSX MSX2+ - Various (TOSEC-v2011-04-23_CM)
47 files renamed

MSX Turbo-R - Applications (TOSEC-v2009-09-27_CM) versus MSX Turbo-R - Applications (TOSEC-v2011-04-23_CM)
8 files renamed

MSX Turbo-R - Demos (TOSEC-v2009-09-27_CM) versus MSX Turbo-R - Demos (TOSEC-v2011-04-23_CM)
no file edited

MSX Turbo-R - Games (TOSEC-v2009-09-27_CM) versus MSX Turbo-R - Games (TOSEC-v2011-04-23_CM)
76 files renamed

MSX Turbo-R - Operating Systems (TOSEC-v2009-09-27_CM) versus MSX Turbo-R - Operating Systems (TOSEC-v2011-04-23_CM)
no file edited

Total files renamed: 7158
Total files removed: 4

The files that have been temporarily removed from these unofficial WIP dat files...

Code: [Select]

MSX MSX - Various - [CAS] (TOSEC-v2009-09-01_CM) versus MSX MSX - Various - [CAS] (TOSEC-v2011-04-23_CM)
Bob 007 - Infiltrado (1986)(Grupo de Trabajo Software)(es)[RUN'CAS-'].cas
Bob 007 - Infiltrado (1986)(Grupo de Trabajo Software)(es)[a][RUN'CAS-'].cas
Pinguino, El (1986)(Grupo de Trabajo Software)(es)[BLOAD'CAS-',R].cas

MSX MSX - Various - [DSK] (TOSEC-v2009-09-01_CM) versus MSX MSX - Various - [DSK] (TOSEC-v2011-04-23_CM)
Bob 007 - Infiltrado (1986)(Grupo de Trabajo Software)(es).dsk
have been properly renamed and moved to their respective Magazines folder as actually they are tape magazines:

Code: [Select]

MSX Games - Issue 02 - Bob 007 - Infiltrado (1986)(Grupo de Trabajo Software)(ES)[a][RUN'CAS-'][Martos].cas
MSX Games - Issue 02 - Bob 007 - Infiltrado (1986)(Grupo de Trabajo Software)(ES)[RUN'CAS-'].cas
MSX Games - Issue 02 - Bob 007 - Infiltrado (1986)(Grupo de Trabajo Software)(ES).dsk
MSX Games - Issue 01 - Pinguino, El (1986)(Grupo de Trabajo Software)(ES)[BLOAD'CAS-',R].cas
The file

Code: [Select]

Bob 007 - Infiltrado (1986)(Grupo de Trabajo Software)(es).dsk
added no vale whatsoever since it consisted of the file in the .cas inserted into a disc image.

TOSEC-v2011-04-23 WIP dat files as well as dat diffs (v2009-09-01 versus v2011-04-23) can be located in my TOSEC uploads folder.

Greetings,

[VG8020]

This dat file release notably enlarges TOSEC-MSX catalogue by adding 252 previously uncatalogued dumps.

Details of the TOSEC-MSX WIP v2011-11-19 dat comparisons follow; further details in the log files from the dat comparisons (ftp-uploaded).

Code: [Select]

MSX MSX - Compilations - [CAS] (TOSEC-v2011-04-23_CM) versus MSX MSX - Compilations - [CAS] (TOSEC-v2011-11-19_CM)
34 additions

MSX MSX - Compilations - [DSK] (TOSEC-v2011-04-23_CM) versus MSX MSX - Compilations - [DSK] (TOSEC-v2011-11-19_CM)
no changes

MSX MSX - Firmware (TOSEC-v2011-04-23_CM) versus MSX MSX - Firmware (TOSEC-v2011-11-19_CM)
no changes

MSX MSX - Magazines - [CAS] (TOSEC-v2011-04-23_CM) versus MSX MSX - Magazines - [CAS] (TOSEC-v2011-11-19_CM)
52 additions

MSX MSX - Magazines - [DSK] (TOSEC-v2011-04-23_CM) versus MSX MSX - Magazines - [DSK] (TOSEC-v2011-11-19_CM)
no changes

MSX MSX - Various - [CAS] (TOSEC-v2011-04-23_CM) versus MSX MSX - Various - [CAS] (TOSEC-v2011-11-19_CM)
164 additions

MSX MSX - Various - [DSK] (TOSEC-v2011-04-23_CM) versus MSX MSX - Various - [DSK] (TOSEC-v2011-11-19_CM)
1 addition

MSX MSX - Various - [ROM] (TOSEC-v2011-04-23_CM) versus MSX MSX - Various - [ROM] (TOSEC-v2011-11-19_CM)
1 addition

MSX MSX2 - Compilations - [CAS] (TOSEC-v2011-04-23_CM) versus MSX MSX2 - Compilations - [CAS] (TOSEC-v2011-11-19_CM)
no changes

MSX MSX2 - Compilations - [DSK] (TOSEC-v2011-04-23_CM) versus MSX MSX2 - Compilations - [DSK] (TOSEC-v2011-04-23_CM) v2
no changes

MSX MSX2 - Various - [CAS] (TOSEC-v2011-04-23_CM) versus MSX MSX2 - Various - [CAS] (TOSEC-v2011-11-19_CM)
no changes

MSX MSX2 - Various - [DSK] (TOSEC-v2011-04-23_CM) versus MSX MSX2 - Various - [DSK] (TOSEC-v2011-11-19_CM)
no changes

MSX MSX2 - Various - [ROM] (TOSEC-v2011-04-23_CM) versus MSX MSX2 - Various - [ROM] (TOSEC-v2011-11-19_CM)
no changes

MSX MSX2+ - Various (TOSEC-v2011-04-23_CM) versus MSX MSX2+ - Various (TOSEC-v2011-11-19_CM)
no changes

MSX Turbo-R - Applications (TOSEC-v2011-04-23_CM) versus MSX Turbo-R - Applications (TOSEC-v2011-11-19_CM)
no changes

MSX Turbo-R - Demos (TOSEC-v2011-04-23_CM) versus MSX Turbo-R - Demos (TOSEC-v2011-11-19_CM)
no changes

MSX Turbo-R - Games (TOSEC-v2011-04-23_CM) versus MSX Turbo-R - Games (TOSEC-v2011-11-19_CM)
no changes

MSX Turbo-R - Operating Systems (TOSEC-v2011-04-23_CM) versus MSX Turbo-R - Operating Systems (TOSEC-v2011-11-19_CM)
no changes

Total additions: 252 additions

[VG8020]

This dat file release aims for further coherence by following TNC.

Details of the TOSEC-MSX WIP v2011-11-22 dat comparisons follow; further details in the log files from the dat comparisons (ftp-uploaded).

Code: [Select]

MSX MSX - Compilations - [CAS] (TOSEC-v2011-11-19_CM) versus MSX MSX - Compilations - [CAS] (TOSEC-v2011-11-22_CM)
35 renames

MSX MSX - Compilations - [DSK] (TOSEC-v2011-11-19_CM) versus MSX MSX - Compilations - [DSK] (TOSEC-v2011-11-21_CM)
1 rename

MSX MSX - Firmware (TOSEC-v2011-11-19_CM) versus MSX MSX - Firmware (TOSEC-v2011-11-22_CM)
15 renames

MSX MSX - Magazines - [CAS] (TOSEC-v2011-11-19_CM) versus MSX MSX - Magazines - [CAS] (TOSEC-v2011-11-22_CM)
4 renames

MSX MSX - Magazines - [DSK] (TOSEC-v2011-11-19_CM) versus MSX MSX - Magazines - [DSK] (TOSEC-v2011-11-22_CM)
2 renames

MSX MSX - Various - [CAS] (TOSEC-v2011-11-19_CM) versus MSX MSX - Various - [CAS] (TOSEC-v2011-11-22_CM)
13 renames

MSX MSX - Various - [DSK] (TOSEC-v2011-11-19_CM) versus MSX MSX - Various - [DSK] (TOSEC-v2011-11-22_CM)
109 renames

MSX MSX - Various - [ROM] (TOSEC-v2011-11-19_CM) versus MSX MSX - Various - [ROM] (TOSEC-v2011-11-22_CM)
107 renames

MSX MSX2 - Compilations - [CAS] (TOSEC-v2011-11-19_CM) versus MSX MSX2 - Compilations - [CAS] (TOSEC-v2011-11-22_CM)
no changes

MSX MSX2 - Compilations - [DSK] (TOSEC-v2011-11-19_CM) versus MSX MSX2 - Compilations - [DSK] (TOSEC-v2011-11-22_CM)
no changes

MSX MSX2 - Various - [CAS] (TOSEC-v2011-11-19_CM) versus MSX MSX2 - Various - [CAS] (TOSEC-v2011-11-22_CM)
no changes

MSX MSX2 - Various - [DSK] (TOSEC-v2011-11-19_CM) versus MSX MSX2 - Various - [DSK] (TOSEC-v2011-11-19_CM)
31 renames

MSX MSX2 - Various - [ROM] (TOSEC-v2011-11-19_CM) versus MSX MSX2 - Various - [ROM] (TOSEC-v2011-11-22_CM)
15 renames

MSX MSX2+ - Various (TOSEC-v2011-11-19_CM) versus MSX MSX2+ - Various (TOSEC-v2011-11-22_CM)
no changes

MSX Turbo-R - Applications (TOSEC-v2011-11-19_CM) versus MSX Turbo-R - Applications (TOSEC-v2011-11-22_CM)
no changes

MSX Turbo-R - Demos (TOSEC-v2011-11-19_CM) versus MSX Turbo-R - Demos (TOSEC-v2011-11-22_CM)
no changes

MSX Turbo-R - Games (TOSEC-v2011-11-19_CM) versus MSX Turbo-R - Games (TOSEC-v2011-11-22_CM)
no changes

MSX Turbo-R - Operating Systems (TOSEC-v2011-11-19_CM) versus MSX Turbo-R - Operating Systems (TOSEC-v2011-11-22_CM)
no changes

[Cassiel]

Thanks VG8020.

I've spotted a few problems with your MSX (excluding PIX) DATs. I've corrected these issues and have uploaded to your submitted DAT area on our FTP (not the big gorski FTP, the official TOSEC one for WIP DATs).

For ref:

Headers - I've corrected all the headers, they had extra fields not needed. You might find it easier (I know I do!) to copy and paste the headers direct from previous DAT and simply update the date counter. This is easier than entering into ClrMamePro when creating.

File extensions - with images like these, the extensions of the files should always be lower case. I've corrected any that weren't.

TNC Errors - in some of the DATs you had a few cases of the file extension being included in the 'description' and 'rom' fields (they shouldn't), a few incorrectly ordered flags, invalid language flags, smaller things like that... Additionally, please note that if a country flag is used, we assume that the software language is the official country language, so there is no need to use "(JP)(ja) ", "(ES)(es)" or "(NL)(nl)" etc only the country code. Conversely, software released in Japan but using English language should still use "(JP)(en)" though (for example). Make sense?   

Like I say, I've corrected these issues and added your MSX DATs to the planned release.

On a side note, don't feel compelled to keep the categories as they are. This may be a big more further down the road, but you'd make a lot of people very happy by re-organizing the DATs into more modern categories we are currently using (Applications/Compilations/Demos/Educational/Firmware/Games/Multimedia/Sampler etc). For example, 'Various' is now a redundant 'catch all' category we no longer use. Any remaining DATs that us this are generally very old and no longer worked on. And from checking the DATs, even I could tell a lot of images are clearly mis-categorised.

Anyhow, see how comfortable you feel with taking that on.... 

[Cassiel]

Regarding your PIX DATs, I have not included these yet.... I think these need splitting out properly rather than just including all Magazines in one DAT for example. They should mirror how Aral already has the DATs split in TOSEC-PIX.

I'm thinking of the Misc one in particular too.... REALLY want to avoid any Misc/Various/Other DATs unless nothing else can be done...

[VG8020]

@Cassiel

Thank you for your post and 'mending' work As every careful datter knows, every effort is made to make data consistent. Some errors tend to crop up at some points. Glad that TOSEC is a supportive team; glad about your report too.

Headers - right. I resourced to CMP to create the dats.

File extensons - right.

TNC errors - right. It should be fairly obvious. It does make sense.

Categories. I've been giving much thought to it. My outline consists of releasing one more TOSEC-MSX update with a major software update before sorting files into new categories. As suggested, 'modern' categories are more representative and useful and consequently, TOSEC-MSX will go in that direction.

PIX - The dats are much WIP. So better just let them 'sleep' till I've added further scans and they I'll have it properly sorted out.

[Aral]

Please do not modify, create or release any PIX related dat without consulting with me first.

[VG8020]

Easy Aral... My PIX-WIP is in my folder. You can check it out there.

[Cassiel]

Aral - what are your plans re PIX? I know on RS you've mentioned your 2011-11-11 DATs may be the last, but I've not seen anything mentioned further (or here).

[VG8020]

@Aral

TOSEC-PIX latest update includes NEW:

MSX - Magazines - MSX Magazine (FR) (TOSEC-v2012-02-26_CM)
MSX - Magazines - MSX World (ES) (TOSEC-v2012-02-26_CM)
MSX - Magazines - MSX-FAN (JP) (TOSEC-v2012-02-26_CM)

There's a whole lot more scans uploaded and yet not listed in this brand new TOSEC-PIX release - TOSEC Release (2012-02-29). So the question is easy. Any forthcoming plans for these scans to be released? I'd like to hear about from you instead of learning about it from a TOSEC post, if that's not too much asking.

Great work, all the best!

[VG8020]

TOSEC-MSX WIP (TOSEC-v2012-04-15)

Text from http://msxcassettes.wordpress.com/2012/04/16/tosec-msx-wip-tosec-v2012-04-15/
Easier said than done, a major update to TOSEC-MSX has been performed. The too-generic Various category in MSX, MSX2 and MSX2+ dat files has been split into a number of alredy-existing or newly-created fully-meaningful software categories:

Applications
Compilations
Demos
Educational
Firmware
Games
Magazines

In addition to this, every file has been been reasigned to its corresponding category; the colossal task of sorting out the 8,777 files being carried out by knowledgeable XakMSX.

Further TOSEC-MSX updates will bring more filename fixing (when ascertained) and new software being added to the most detailed MSX software collection up to date.

Credit from previous contributors is kept in every brand new .dat file.

[Cassiel]

This is excellent, really good work. The whole MSX tree has been crying out for some (knowledgeable) attention for years.

Unfortunately, I'm also the bearer of bad news since there are some issues:

#1
The DATs haven't been built 100% correct. You need to disable the extension being added for both 'name' and 'description', it should ONLY appear within 'rom' field. Also, you need to include CRC32, MD5 and SHA-1 hashes.

I assume you're using ClrMamePro, if so I have some Dat2Dir templates here that you can use:
http://www.redsplinter.com/tosec/TOSEC%20(multi%20files).d2d
http://www.redsplinter.com/tosec/TOSEC%20(single%20files).d2d
though you will still need to manually copy&paste (then update) the header.

#2
The new categories are great, the old deprecated 'Various' category should no longer be used so this is perfect. Unfortunately, what's also depreciated is having 'mixed' file types per category.

So, for 'MSX MSX - Applications' for example you currently have a mix of '.dsk' and '.cas' files. These need to be separated into two DATs (or more if there are other file types), i.e. 'MSX MSX - Applications - [DSK]' and 'MSX MSX - Applications - [CAS]'. This split/DAT naming is only applicable if there are mixed file types. For example 'MSX MSX - Firmware' only includes '.rom' files (as far as I could see) so it should remain 'MSX MSX - Firmware' rather than 'MSX MSX - Firmware - [ROM]'. Make sense?

#3
Because of issue number 1, when I ran your new DATs through our TOSEC 'DAT validation' tool it flagged all images as having issues so I wasn't able to check for any specific TNC errors. It suddenly occurred to me you may not have been given/have had explained this tool. The latest version (as well as historic) can be found on our FTP under '/tools/tde', otherwise I've also mirrored the latest here: http://www.redsplinter.com/tosec/TOSEC%20Dat%20Explorer%20v20120225.rar.

You can load individual DATs to display, or it can bulk check an entire folder of DATs for a multitude of errors/issues (File->Batch Job->Check Folder). I would recommend using this tool to check your DATs once #1 and #2 are sorted since it can flag up any issues that need correcting.

Hope that's not too disheartening, I can see already that a huge amount of work has one into this all ready (well done all round BTW!). Once these couple of issues are sorted, we'll be golden....   

[VG8020]

Appreciate the feedback. Here's my views on it:

#1 (& #3, by extension): Oh, yes. I used datutil which was rather cumbersome (I had to create a .bat in order to speed things up). New version (same naming) already available. Thank you for linking the .d2d files as well as the TOSEC Dat Explorer (great tool, by the way). I noticed there was 1 filename with a duplicate extension .cas.cas so it's solved now.

#2: I gave a lot of thought to this issue. I had realised about it being deprecated by looking at ZX Spectrum dat files. However, there's strong reasoning for the present choice. I'll try to make it short. In the MSX days back in the 80's and early 90's (MSX is alive and kicking today; new releases every year!) you would either own an original tape or disc or catridge (.cas, .dsk and .rom are the related file extensions). However, much software was converted by amateurs/fans... to more comfortable formats e.g. a tape would be converted to a .dsk format (easier loading, no tape data read errors, faster loading...) Or different versions would be created out of cartridges (e.g. a catridge would be converted to a .dsk for different purposes). So what it should be a cut and clear format distinction (tapes should be .cas, discs should be .dsk ...) turns to be a blurry field where say an original tape would be converted to .dsk at some point and to .rom at a different time (there's a great deal of examples to choose from). So keeping the different formats under the same category .dat provides not only a whole lot of possibilities for tracking and tracing different versions of a same piece of sotware but keeps things together by reducing the dispesion ratio. Otherwise, the same title should be checked in different dat files (little productive procedure). So, in general terms, that's the rationale.

One more issue: even though there are new categories now, I anticipate more categories might be added in forthcoming releases (e.g. type-in category for listings from MSX printed magazines. Just stay tuned

[Cassiel]

#1 (& #3, by extension): Oh, yes. I used datutil which was rather cumbersome (I had to create a .bat in order to speed things up). New version (same naming) already available. Thank you for linking the .d2d files as well as the TOSEC Dat Explorer (great tool, by the way). I noticed there was 1 filename with a duplicate extension .cas.cas so it's solved now.

Good stuff...   

#2: I gave a lot of thought to this issue. I had realised about it being deprecated by looking at ZX Spectrum dat files. However, there's strong reasoning for the present choice. I'll try to make it short. In the MSX days back in the 80's and early 90's (MSX is alive and kicking today; new releases every year!) you would either own an original tape or disc or catridge (.cas, .dsk and .rom are the related file extensions). However, much software was converted by amateurs/fans... to more comfortable formats e.g. a tape would be converted to a .dsk format (easier loading, no tape data read errors, faster loading...) Or different versions would be created out of cartridges (e.g. a catridge would be converted to a .dsk for different purposes). So what it should be a cut and clear format distinction (tapes should be .cas, discs should be .dsk ...) turns to be a blurry field where say an original tape would be converted to .dsk at some point and to .rom at a different time (there's a great deal of examples to choose from). So keeping the different formats under the same category .dat provides not only a whole lot of possibilities for tracking and tracing different versions of a same piece of sotware but keeps things together by reducing the dispesion ratio. Otherwise, the same title should be checked in different dat files (little productive procedure). So, in general terms, that's the rationale.

Appreciate what you saying, but splitting DATs by category/file format is the TOSEC convention we use. I understand that software on these platforms was often modified to work on other media etc, but remember the DAT is being split by the file format the image is in, not the original media format necessarily. Also, if for example you have a '.cas' and '.dsk' image with the same name, in the same DAT (not an unlikely scenario given what you saying) then CMP (and similar) won't be able to scan/rebuild those sets correctly.

One more issue: even though there are new categories now, I anticipate more categories might be added in forthcoming releases (e.g. type-in category for listings from MSX printed magazines. Just stay tuned

Not a problem at all as long as it's a sensible new category. Obviously we want to keep categories as standardised as possible across all sets/DATs.