Author Topic: Newpain WIP  (Read 1583 times)

Offline newpain

  • Newbie
  • *
  • Posts: 4
    • My website
    • Email
Newpain WIP
« on: April 01, 2020, 05:38:40 PM »
I have send links by mail on the contact @ tosec mail.

(translate with google translator)
I am an old french TOSEC contributor known by the pseudonym 'NEWPAIN'.
It has been a long time since I had contributed to the TOSEC project, for some time I have seen database projects related to retrogaming which are of importance for cultural and digital preservation in the world of which TOSEC with the renaming convention is the most understandable and clearest should be in the public interest.
This is why I work there, not all the time but at least I try to contribute to it. So below the link below which will slightly increase the TOSEC databases:


1- To Update (to update actuals BDD)

Amstrad CPC - Games - [DSK].tar.xz --- 1.4mo
Atari ST - Educational - [ST].tar.xz --- 269.3ko
Atari ST - Games - [ST].tar.xz --- 55.3mo
Atari ST - Operating Systems - [IMG].tar.xz --- 1.5mo
Bandai WonderSwan - Games.tar.xz --- 236.5ko
Commodore Amiga - Applications - Public Domain.tar.xz --- 3.9mo
Commodore Amiga - Applications - [ADF].tar.xz --- 3.7mo
Commodore Amiga - Collections - Amiga Amateur Radio User Group.tar.xz --- 4.7mo
Commodore Amiga - Collections - Assassins, The.tar.xz --- 4mo
Commodore Amiga - Collections - Bavarian.tar.xz --- 5mo
Commodore Amiga - Collections - MegaDisc.tar.xz --- 524.2ko
Commodore Amiga - Collections - TOGA.tar.xz --- 4.4mo
Commodore Amiga - Compilations - Applications.tar.xz --- 30mo
Commodore Amiga - Compilations - Games.tar.xz --- 3.3mo
Commodore Amiga - Compilations - Various.tar.xz --- 654.4ko
Commodore Amiga - Coverdisks - [ADF].tar.xz --- 1.8mo
Commodore Amiga - Demos - Animations and Videos.tar.xz --- 4.4mo
Commodore Amiga - Demos - Music.tar.xz --- 4.8mo
Commodore Amiga - Demos - Packs.tar.xz --- 34mo
Commodore Amiga - Demos - Slideshows.tar.xz --- 3.8mo
Commodore Amiga - Demos - Various - [ADF].tar.xz --- 10.7mo
Commodore Amiga - Demos - Various - [EXE].tar.xz --- 1.3mo
Commodore Amiga - Diskmags.tar.xz --- 17mo
Commodore Amiga - Docs.tar.xz --- 14.3mo
Commodore Amiga - Educational - [ADF].tar.xz --- 291.3ko
Commodore Amiga - Firmware.tar.xz --- 515.3ko
Commodore Amiga - Games - Emerald Mine.tar.xz --- 153.3ko
Commodore Amiga - Games - Public Domain - [ADF].tar.xz --- 12.3mo
Commodore Amiga - Games - Save Disks.tar.xz --- 37.5ko
Commodore Amiga - Games - Unofficial Addons & Patches.tar.xz --- 526.1ko
Commodore Amiga - Games - [ADF].tar.xz --- 96.2mo
Commodore Amiga - Kickstart-Disks.tar.xz --- 233.5ko
Commodore Amiga - Packmags.tar.xz --- 864.2ko
Fujitsu FM Towns - Games - [BIN].tar.xz --- 8.7mo
IBM PC Compatibles - Applications - [IMG].7z --- 32.6mo
Nintendo 64 - Games.tar.xz --- 608mo
Nintendo Famicom & Entertainment System - Games - [NES].tar.xz --- 1.4mo
Nintendo Game Boy Advance - Games.tar.xz --- 113.7mo
Nintendo Game Boy Color - Games.tar.xz --- 2.2mo
SNK Neo-Geo Pocket Color - Games.7z ---1.9mo
Sega 32X - Games.tar.xz ---15.5mo
Sega Mega Drive & Genesis - Games - [BIN].tar.xz --- 22.6mo


2 - To Add (new database)

Commodore Amiga - Collections - Amiga Magazin PD.tar.xz --- 103mo
Commodore Amiga - Collections - Amigan PD.tar.xz --- 4.8mo
Commodore Amiga - Collections - Time Disk Magazin.tar.xz --- 6.2mo
IBM PC Compatibles - Games - Addons & Patches [IMG].7z ---55.8mo
NEC SuperGrafx - Samplers.7z --- 72.5ko


3 - To Replace (all the roms wich are in the oldest TOSEC databases are in this archives)

IBM PC Compatibles - Compilations - Games - [IMG].7z --- 26.2mo
IBM PC Compatibles - Educational - [IMG].7z ---23.9mo
IBM PC Compatibles - Games - [IMG].7z --- 1.9go
IBM PC Compatibles - Operating Systems - [IMG].7z --- 575.7mo
NEC PC-Engine & TurboGrafx-16 - Games.tar.xz --- 274.2mo
NEC PC-Engine & TurboGrafx-16 - Samplers.tar.xz --- 3.3mo
NEC SuperGrafx - Games.7z --- 2.5mo
Nintendo Game Boy - Games.7z --- 135.6mo
Nintendo Super Famicom & Super Entertainment System - Games.tar.xz --- 3.3Go


Alert !!! On the Ibm PC Compatible, the formats :
IMG, IMA, DSK, VFD are the same "MFM RAW Format" I used on linux many of floppy disk applications like SamDisk, Disk-Utilities, Hfe disk Tools and others libdsk to manipulate IBM pc floppy images.
The IMZ is a zip format, example : you can change extension to zip and extract the image file as IMA format.
It's why, it is necessarly to goup the IMA, IMG, DSK, and VFD format in one database, i choose the IMG format wich is actually very usefull and used on the web.
I also have merged 3.5 and 5.25 format in one only database.
I work on virtual Box or Pcem all IBM PC disk and install them to test them, maybe, i have include some errors in databases but not very hards errors, littles errors ;)
For this time, it must be interessant to delete these databases :
IBM PC Compatibles - Diskimages - 3.5'' (TOSEC-v2014-06-30_CM)
IBM PC Compatibles - Diskimages - 5.25'' (TOSEC-v2014-06-30_CM)
IBM PC Compatibles - Games - [DSK] (TOSEC-v2018-07-01_CM)
IBM PC Compatibles - Games - [IMA] (TOSEC-v2018-11-15_CM)
To include all of the floppy disk images in one database as i have make my archive :
IBM PC Compatibles - Games - [IMG]

Like the IB PC Games, the OS database who need to be deleted :
IBM PC Compatibles - Operating Systems - [IMA] (TOSEC-v2018-11-15_CM)
IBM PC Compatibles - Operating Systems - [VFD] (TOSEC-v2018-11-15_CM)
To be include as i have make my os img archive
IBM PC Compatibles - Operating Systems - [IMG]

I am working on others database but more to come in next months ;)


Cordially, from Newpain ;)
« Last Edit: April 01, 2020, 09:53:49 PM by newpain »

Offline yori yoshizuki

  • Newbie
  • *
  • Posts: 19
Re: Newpain WIP
« Reply #1 on: May 05, 2020, 10:28:49 PM »
maybe u should rename

normal floppy images in .IMA  [pc compatbiles]

only boot floppy images in .IMG [pc compatbiles]

as usually used into the web


DSK (mainly unix xenix aix sun etc)

Offline newpain

  • Newbie
  • *
  • Posts: 4
    • My website
    • Email
Re: Newpain WIP
« Reply #2 on: May 06, 2020, 08:35:30 AM »
Either we chose the IMG format or the IMA format, but not both. The .IMA format has been democratized with the use of Winimage but the file structure remains IMG and VFD compatible.
All floppy images with the DSK extension found in the TOSEC databases have been scanned and validated with a file structure belonging to the IMG structure with SamDISK.
The DSK format on IBM PC does not seem to exist.
Out of the 15,000 files, I sorted according to their structure: C2P, IMD, IMG, PSI, SCP, TC and TD0.

All files found on the internet should not be classified according to their extensions but by their structure. You have to detect the structure and classify the files accordingly.
For example, structure detection with Samdisk:
samdisk info 'Blades of Steel (1990) (Konami) .dsk'
[Blades of Steel (1990) (Konami) .dsk]
 Type: BPB
 Format: 250Kbps MFM, 80 cyls, 2 heads, 9 sectors, 512 bytes / sector
BPB = FAT12 / 16 BIOS Parameter Block (MS-DOS, Atari ST)
Should be renamed: 'Blades of Steel (1990) (Konami) .img'

Offline mictlantecuhtle

  • Global Moderator
  • Full Member
  • *****
  • Posts: 143
Re: Newpain WIP
« Reply #3 on: July 27, 2020, 10:25:15 PM »
Hey there, just finished working now to put a lot of the hard work you've done in for the next release. First want to say thank you very much for this, there is a lot of work here which is really appreciated.

I haven't had time to include the excellent work you've done under "To Replace" e.g. with IBM PC. Because in this case we are replacing older DATs and merging content from some others I want to make sure I have taken time to double-check everything and make sure that we don't end up accidentally losing files or information (I have made this mistake in the past and it is a lot of trouble to sort out!). This is still being looked at and will definitely be included, it's just that I want to make sure it's done right rather than rushed in.

We've also made the decision not to include the NEC Supergrafx "Samplers" dat for now, as the person who has been maintaining these dats feels that this is not the right way to categorise these files and has plans of how best to include them in a future release.

We really appreciate the huge amount of work that has been put into these dats and would love to see more contributions coming from you soon!

Offline newpain

  • Newbie
  • *
  • Posts: 4
    • My website
    • Email
Re: Newpain WIP
« Reply #4 on: October 31, 2020, 12:59:32 AM »
Oops, I did it again :)

After some tweaking on the IBM PC GAmes IMG version DATs - after the last official IMG dat with 2839 entries, today I pushed to 4011 entries. I must have more than double of them to rename but I take a little break on the IBM PC DATs to focus on the DATs which have been at a standstill for a long time.
I will ask some of you for help !!! But it takes me some time to upload the folders of files to be renamed. 90kb / sec for the upload on a total of 12Giga of files. : ((

I have a Mega Big DAT FM Towns from Fujistu CD Games in BIN Format of 440 renamed entries that I should split into Magazine / Public Domain, Applications, and Operating Systems but I am not able to upload them because of my Extremely slow internet speed :(, I keep warm while waiting. Also because there is a new Linux compatible emulator and I can test the isos with still compatibility problems making renaming difficult and I do not understand the Japanese which brakes me on some files.
I also added 370 entries in the Atarai ST Firmware IMG dat mostly concerning EMUTOS and about 40 entries in the DAT Games of the Ataris ST 'St' games. I have 7giga of Atari ST files left.

I am working quite intensively on the "Nintendo Super Famicom & Super Entertainment System - Games" dat which it would be interesting to split into the 4 existing SFC - SMC - FIG and SF3 formats because of their different structures.
The latest official Tosec DAT shows around 1600 Entries which I just increased to over 8000 entries. It's not even half of what I have in stock, so still a lot of work.
I have good surprises and annoyances concerning certain games which are extremely Hacked like Super Mario World or Metroid 3 whose hacked versions are counted in the thousands.

I have also prepared some Working File for the future DAT Game Boy Colors and GBA which weighs very heavy but half of the files are from emulators on this platform, I have about if I'm not mistaken 8000 entries in GBA Games and 4000 for GBC games.
I also have some entries for possible NDS or 3DS dat but for as these are systems still in commercial use I am leaving them out for now.
I have a lot of DAT Tosec project but also a lot of projects outside, with the Covid19 containment in France, we can say that I am giving myself more to the TOSEC project.

Dat O Matic guys !! Are you dumped? : p

Offline Cassiel

  • Administrator
  • Hero Member
  • *****
  • Posts: 1573
    • Email
Re: Newpain WIP
« Reply #5 on: November 08, 2020, 10:09:22 PM »
I am working quite intensively on the "Nintendo Super Famicom & Super Entertainment System - Games" dat which it would be interesting to split into the 4 existing SFC - SMC - FIG and SF3 formats because of their different structures.

I would stick with 'clean' ROM images for the console ones. Adding all the weird formats with junk headers really a waste of time (and messy).

I have a lot of DAT Tosec project but also a lot of projects outside, with the Covid19 containment in France, we can say that I am giving myself more to the TOSEC project.

Stay safe in these Crazy Days!