TOSECdev Forum

TOSEC Project => Database / Datfiles => Work In Progress => Topic started by: newpain on April 01, 2020, 05:38:40 PM

Title: Newpain WIP
Post by: newpain on April 01, 2020, 05:38:40 PM
I have send links by mail on the contact @ tosec mail.

(translate with google translator)
I am an old french TOSEC contributor known by the pseudonym 'NEWPAIN'.
It has been a long time since I had contributed to the TOSEC project, for some time I have seen database projects related to retrogaming which are of importance for cultural and digital preservation in the world of which TOSEC with the renaming convention is the most understandable and clearest should be in the public interest.
This is why I work there, not all the time but at least I try to contribute to it. So below the link below which will slightly increase the TOSEC databases:


1- To Update (to update actuals BDD)

Amstrad CPC - Games - [DSK].tar.xz --- 1.4mo
Atari ST - Educational - [ST].tar.xz --- 269.3ko
Atari ST - Games - [ST].tar.xz --- 55.3mo
Atari ST - Operating Systems - [IMG].tar.xz --- 1.5mo
Bandai WonderSwan - Games.tar.xz --- 236.5ko
Commodore Amiga - Applications - Public Domain.tar.xz --- 3.9mo
Commodore Amiga - Applications - [ADF].tar.xz --- 3.7mo
Commodore Amiga - Collections - Amiga Amateur Radio User Group.tar.xz --- 4.7mo
Commodore Amiga - Collections - Assassins, The.tar.xz --- 4mo
Commodore Amiga - Collections - Bavarian.tar.xz --- 5mo
Commodore Amiga - Collections - MegaDisc.tar.xz --- 524.2ko
Commodore Amiga - Collections - TOGA.tar.xz --- 4.4mo
Commodore Amiga - Compilations - Applications.tar.xz --- 30mo
Commodore Amiga - Compilations - Games.tar.xz --- 3.3mo
Commodore Amiga - Compilations - Various.tar.xz --- 654.4ko
Commodore Amiga - Coverdisks - [ADF].tar.xz --- 1.8mo
Commodore Amiga - Demos - Animations and Videos.tar.xz --- 4.4mo
Commodore Amiga - Demos - Music.tar.xz --- 4.8mo
Commodore Amiga - Demos - Packs.tar.xz --- 34mo
Commodore Amiga - Demos - Slideshows.tar.xz --- 3.8mo
Commodore Amiga - Demos - Various - [ADF].tar.xz --- 10.7mo
Commodore Amiga - Demos - Various - [EXE].tar.xz --- 1.3mo
Commodore Amiga - Diskmags.tar.xz --- 17mo
Commodore Amiga - Docs.tar.xz --- 14.3mo
Commodore Amiga - Educational - [ADF].tar.xz --- 291.3ko
Commodore Amiga - Firmware.tar.xz --- 515.3ko
Commodore Amiga - Games - Emerald Mine.tar.xz --- 153.3ko
Commodore Amiga - Games - Public Domain - [ADF].tar.xz --- 12.3mo
Commodore Amiga - Games - Save Disks.tar.xz --- 37.5ko
Commodore Amiga - Games - Unofficial Addons & Patches.tar.xz --- 526.1ko
Commodore Amiga - Games - [ADF].tar.xz --- 96.2mo
Commodore Amiga - Kickstart-Disks.tar.xz --- 233.5ko
Commodore Amiga - Packmags.tar.xz --- 864.2ko
Fujitsu FM Towns - Games - [BIN].tar.xz --- 8.7mo
IBM PC Compatibles - Applications - [IMG].7z --- 32.6mo
Nintendo 64 - Games.tar.xz --- 608mo
Nintendo Famicom & Entertainment System - Games - [NES].tar.xz --- 1.4mo
Nintendo Game Boy Advance - Games.tar.xz --- 113.7mo
Nintendo Game Boy Color - Games.tar.xz --- 2.2mo
SNK Neo-Geo Pocket Color - Games.7z ---1.9mo
Sega 32X - Games.tar.xz ---15.5mo
Sega Mega Drive & Genesis - Games - [BIN].tar.xz --- 22.6mo


2 - To Add (new database)

Commodore Amiga - Collections - Amiga Magazin PD.tar.xz --- 103mo
Commodore Amiga - Collections - Amigan PD.tar.xz --- 4.8mo
Commodore Amiga - Collections - Time Disk Magazin.tar.xz --- 6.2mo
IBM PC Compatibles - Games - Addons & Patches [IMG].7z ---55.8mo
NEC SuperGrafx - Samplers.7z --- 72.5ko


3 - To Replace (all the roms wich are in the oldest TOSEC databases are in this archives)

IBM PC Compatibles - Compilations - Games - [IMG].7z --- 26.2mo
IBM PC Compatibles - Educational - [IMG].7z ---23.9mo
IBM PC Compatibles - Games - [IMG].7z --- 1.9go
IBM PC Compatibles - Operating Systems - [IMG].7z --- 575.7mo
NEC PC-Engine & TurboGrafx-16 - Games.tar.xz --- 274.2mo
NEC PC-Engine & TurboGrafx-16 - Samplers.tar.xz --- 3.3mo
NEC SuperGrafx - Games.7z --- 2.5mo
Nintendo Game Boy - Games.7z --- 135.6mo
Nintendo Super Famicom & Super Entertainment System - Games.tar.xz --- 3.3Go


Alert !!! On the Ibm PC Compatible, the formats :
IMG, IMA, DSK, VFD are the same "MFM RAW Format" I used on linux many of floppy disk applications like SamDisk, Disk-Utilities, Hfe disk Tools and others libdsk to manipulate IBM pc floppy images.
The IMZ is a zip format, example : you can change extension to zip and extract the image file as IMA format.
It's why, it is necessarly to goup the IMA, IMG, DSK, and VFD format in one database, i choose the IMG format wich is actually very usefull and used on the web.
I also have merged 3.5 and 5.25 format in one only database.
I work on virtual Box or Pcem all IBM PC disk and install them to test them, maybe, i have include some errors in databases but not very hards errors, littles errors ;)
For this time, it must be interessant to delete these databases :
IBM PC Compatibles - Diskimages - 3.5'' (TOSEC-v2014-06-30_CM)
IBM PC Compatibles - Diskimages - 5.25'' (TOSEC-v2014-06-30_CM)
IBM PC Compatibles - Games - [DSK] (TOSEC-v2018-07-01_CM)
IBM PC Compatibles - Games - [IMA] (TOSEC-v2018-11-15_CM)
To include all of the floppy disk images in one database as i have make my archive :
IBM PC Compatibles - Games - [IMG]

Like the IB PC Games, the OS database who need to be deleted :
IBM PC Compatibles - Operating Systems - [IMA] (TOSEC-v2018-11-15_CM)
IBM PC Compatibles - Operating Systems - [VFD] (TOSEC-v2018-11-15_CM)
To be include as i have make my os img archive
IBM PC Compatibles - Operating Systems - [IMG]

I am working on others database but more to come in next months ;)


Cordially, from Newpain ;)
Title: Re: Newpain WIP
Post by: yori yoshizuki on May 05, 2020, 10:28:49 PM
maybe u should rename

normal floppy images in .IMA  [pc compatbiles]

only boot floppy images in .IMG [pc compatbiles]

as usually used into the web


DSK (mainly unix xenix aix sun etc)
Title: Re: Newpain WIP
Post by: newpain on May 06, 2020, 08:35:30 AM
Either we chose the IMG format or the IMA format, but not both. The .IMA format has been democratized with the use of Winimage but the file structure remains IMG and VFD compatible.
All floppy images with the DSK extension found in the TOSEC databases have been scanned and validated with a file structure belonging to the IMG structure with SamDISK.
The DSK format on IBM PC does not seem to exist.
Out of the 15,000 files, I sorted according to their structure: C2P, IMD, IMG, PSI, SCP, TC and TD0.

All files found on the internet should not be classified according to their extensions but by their structure. You have to detect the structure and classify the files accordingly.
For example, structure detection with Samdisk:
samdisk info 'Blades of Steel (1990) (Konami) .dsk'
[Blades of Steel (1990) (Konami) .dsk]
 Type: BPB
 Format: 250Kbps MFM, 80 cyls, 2 heads, 9 sectors, 512 bytes / sector
BPB = FAT12 / 16 BIOS Parameter Block (MS-DOS, Atari ST)
Should be renamed: 'Blades of Steel (1990) (Konami) .img'
Title: Re: Newpain WIP
Post by: mictlantecuhtle on July 27, 2020, 10:25:15 PM
Hey there, just finished working now to put a lot of the hard work you've done in for the next release. First want to say thank you very much for this, there is a lot of work here which is really appreciated.

I haven't had time to include the excellent work you've done under "To Replace" e.g. with IBM PC. Because in this case we are replacing older DATs and merging content from some others I want to make sure I have taken time to double-check everything and make sure that we don't end up accidentally losing files or information (I have made this mistake in the past and it is a lot of trouble to sort out!). This is still being looked at and will definitely be included, it's just that I want to make sure it's done right rather than rushed in.

We've also made the decision not to include the NEC Supergrafx "Samplers" dat for now, as the person who has been maintaining these dats feels that this is not the right way to categorise these files and has plans of how best to include them in a future release.

We really appreciate the huge amount of work that has been put into these dats and would love to see more contributions coming from you soon!