Author Topic: Dumps that are not in tosec yet: May be this could be useful for the teem.  (Read 12756 times)

Offline Xttx

  • Newbie
  • *
  • Posts: 10
First of all I'm sorry for my poor English: I'm Russian.

I would like to contribute to project but not yet realize how could I do it, so, I just list here what I have and you decide if it can help you in your work.

Some notes about Elektronika BK-0010* dats:
For now we have:
Elektronika BK-0010-0011M - Applications, Compilations, Demos, Games, Operating Systems.
Elektronika BK-0011-411 - Games.
so,
Elektronika BK-0011-411 have never existed. There are 3 versions of BK: BK0010, BK0010-01 (almost same hardware, fully compatible) and BK0011(M) (a little faster, 128kb ram insted of 16kb+16kb video, 16 indexed colors (+mixed colors) insted of 4, integrated FDD. Compatible with previous versions of BK but sometimes colors are wrong).
All the games from "Elektronika BK-0011-411 - Games.dat" are actually BK0010 - BK0010-01 games.

1). I have A LOT of games and applications, in form of original tapes, tape images, original floppys and floppy images for BK 0010-(01) that are not in any BK dat. Of cause there are a lot of dupes, but theese are 'different' dupes, that have different CRC, size and filenames. I really don't know how to separate original and modified versions of games, because when playing I can not see any difference between 2 (or sometimes more) versions. I can upload all this collection somewhere if somewone could take this pain.
2). A lot of software, a dozen of games and one OS (SMSQ/E, delivered with later versions of QL) for SINCLAIR QL that are not in the dat.
3). Some games for SHARP X68K + open source unix like os (BSD) - the only English OS for this computer.
4). Some CD images for SegaCD / MegaCD, marked as "not dumped" in TOSEC-ISO project.
5). Original and not original PC CD for MS-DOS....

Now the main question:
How can I help you with the project with all this???
« Last Edit: December 02, 2009, 03:08:01 PM by Xttx »



Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1332
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #1 on: December 02, 2009, 05:28:37 PM »
Hi Xttx,

welcome aboard :P
I'm not currently at home and can't write a long post explaining it all + i'm not into any systems to talk about details of what you said (like Elektronika), anyway i'm here to say that any help is always welcome and your work is appreciated.

1) Once i get back home i will contact you back so we can talk about it, i'm sure someone will pick those files and see whats new, don't promise that to be really soon because we are few but i know some of us are interested in those kind of systems.
2) The same as before, new sets (especially good ones [i mean, the good, new, not alternate ones are always funnier to add :P]) are always wanted, gorski maintains Sinclair (specially Spectrum) branch currently but has been busy lately, when he gets back i hope he can keep up working on some stuff.
3) About this i can only say that mek1 was doing some work on them and also has new stuff, TKaos might know something more.
4 & 5) If you have new ISOs and time, you can dump them and submit the hashes to our database, we appreciate that, www.toseciso.org for more detail, here you will also find a guide on how to dump them, i suggest contacting idoru or one of the guys that work most with ISO part but if that isn't possible i will also try to help you out :)

That's it, if you have the time and will, you can really help our project and if you're really interested and stick around you can even start renaming / adding stuff to our dats, i'm sure TKaos or someone else might help you out too :)

Offline TKaos

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 539
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #2 on: December 02, 2009, 09:56:11 PM »
I would really like to help with the Elektronika BK but at the moment I'm too busy with the other DATs I'm working on, if you maybe want to create DATs for that system I could help you with any questions you might have.

Offline Xttx

  • Newbie
  • *
  • Posts: 10
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #3 on: December 03, 2009, 11:20:30 AM »
Thank you all for the repply!
So,
2). For Sinclair QL i'm waiting for somebody contact me.
3). For Sharp X68K i'm waiting the release of mek1 to compare with what i have and see what is still missing.
4). For Mega-CD games, I have read the CD-ripping User Guide and i have tryed to rip the game that is allready dumped to check if I can actually rip the games and add them to database. I've got the same CRC for audio files, but wrong CRC for ISO (and 2kb more in size). I will try some other methods on other games. If I understand why it happens, i will be able to rip the games easyly, and it should not take a big time to rip all titles I have that are not already dumped.

1). For Electronika BK xxxx I will try to make the dat myself, using TOSEC Naming Convention.

Now, the main question:
There are a lot of dupes.
1.1). Some of them have a visible changes (for example title screen, or notes like "Cracked by xxxx" or "Modified version for KUVC" [KUVC is a some numbers of modified BK0010-01 units with network support + PC based server. KUVC was used in schools]), so, I suppose, i should note this in rom-name. But how exactly???
1.2). The others have no visible difference at all! Even after playing some hours. But they are different in size, crc and in filenames. What should I do with this ones???

Thank in advance.

PS. I'm really sorry for my English :(.
« Last Edit: December 03, 2009, 11:24:43 AM by Xttx »

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1332
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #4 on: December 03, 2009, 01:38:09 PM »
You english is really not bad, i can understand it perfectly and i'm not great at it too :)

4) About the different dumps in ISOs i can't really help you out because i've never ripped anything myself, afaik you should be obtaining equal CRCs if the sets are the same, in some cases you could have different revisions or something but i guess you're doing right, trying to dump something and match the existent data so you can learn the theory :)
The ISOs work +- like this (i bet you already know it but i will be short): There is a list of isos (in toseciso.org for example) where dumps can have at least 3 different status: 1) no dump (nobody has dumped it yet and submitted valid information) ; 2) dumped once ; 3) verified good dump (it was dumped at least twice by different persons and the hashes matched. Of course i suppose there are a lot of sets not yet inserted there even on status 1 (that you might have).
So in case of you having cds listed there with status 1 or 2 or not appearing at all, those are wanted. Following the guide you can learn how to dump them and start submitting information / being a contributor, for this you may need to talk with idoru or others from toseciso and i can't really help you (have no knowledge in this).

2) I'm here today like i promised and will try to give you some help in that, at least some place where you can put all those sets until someone has the time to work on them, have you scanned these unrenamed sets with "unrenamed" dats? (a collection of dats with files that exist but are still unrenamed, TKaos uses those to find new sets :P, you may find them at romshepherd.com for example)

3) I'm not sure when they will be done/released (i wanted it to be asap :P), anyway you can wait, upload those new sets you have so mek1 can look at them or wait for him, maybe he appears around here and gives some feedback :)


1) Ok, i guess TKaos, I and some others may help you out without a problem in this (even if by this i mean giving hints, indications and pointing the first errors you will produce :P), the TNC doc is a bit extensive but it covers most of the stuff you need to know, it may suffer an update soon (i should have done it already), the main things i will add there are new country/language codes (they changed, it is discussed somewhere around here), the " + " used to separate groups in dump flags was dropped and only " - " is used now and some more fixes.
I've never manually renamed anything myself (just did some fixes on the dats) but it isn't hard, the difficult part is getting to know all flags in naming convention, a tool to help out with renaming could be useful to people starting but after that all renamers just prefer to do it manually. Unfortunately in our guide section we don't have a guide covering it and would be really helpful (how to create a dat, how to rename a set, how to check for new sets and avoid creating new dupes :P and so on).

The tips i can give you are:
- Read TNC and keep asking questions on everything you don't understand, i and others will be here to help you out
- Learn how to use clmpro (i suppose you already know :)) and rebuild / scan all your sets with the last dats to make sure you really have new sets to add. *
- Get some way to test the sets (some good emulators, hexeditors, ...)
- Start inspecting the new sets, the basic flags you should try to always find is Title, Year (date) and Publisher(s), anything more is always good too.
- Make sure the set didn't change after you launched them (crc for example), some systems/sets change when running, highscores are saved, data is stored for many reasons for some sets, can't say much more but you should be careful with it, this is one of the many reasons so many alts exist with few visible changes and why getting data from original disks is great :P
- After you got all information you can just rename the setfile (setfile is a container, like the zipfile that olds the roms), the important part is to rename the setfile, if the sets you are renaming are disk images or some other type of stuff where you can also rename the romname to match the setname + ext, you can also do it, please be careful with this, some sets will need the original filenames to work :)
- Move it to the new renamed folder(s) and later we will talk about adding them to the existent datfiles or newer ones (if you got stuff that don't fit in any of the existent dats).


* 4 of the 6 existent dats for that system were updated since the last release (2007) but there wasn't anything added, just a really few changed correcting TNC errors i've found. You should start renaming a few sets and adding them to a new folder, ideally you would scan all your sets with the latest dats and then start renaming the new sets and adding them to the right folders, later you would just generate new datfiles from those folder using clrmamepro. Other way would be editing the dats manually and adding the stuff (with a text editor), that is a bit more risky because you can get the hashes wrong or do something wrong and the latest datfiles aren't out yet, thats why i talked about putting the new renamed sets in some new sepparated folders for now :)


1.1 After you start getting inside TNC you will find it's easy to answer many of those questions, if a set is cracked you add the flag [cr Cracker], trying always to get correct information about the groupname :P, [m] flag is for modified stuff, there is also a systemflag that may be useful for that KUVC thing, but you have more knowledge about the system and maybe TKaos, Cassiel or others might help you with it.
1.2 No idea, it can be a different version, revision or something but most of the times they are just alternate versions created by something like what i described earlier, or have really small differences, some done on purpose to spread more garbage around, others can be a bad dump case and so on, i would say you could compare 2 sets with diff tools and hex editors but most of the time it will be too much work to add just another garbage set, my advice is to let those sets there and take care of the others first, adding new stuff is better than adding dupes and if they are just garbage or hacked with something just saying "dumped by xpto/xpto rocks!" and so on i would say the best thing is just let them disappear instead of spreading them :)


Lunch time now!

Offline Xttx

  • Newbie
  • *
  • Posts: 10
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #5 on: December 03, 2009, 07:25:08 PM »
1.1 Understood. By the way, it would be nice for me to get the latest version of TNC :)
1.2 I did not explain well. My fault.... I already eliminate all the titles, and all the dupes of the titles that are already in the dats. But I have a realy big amount of titles that are not in any BK dat, and in most cases 3, 4 or even more versions of them, apparently completely the same. Which one should I choose? That is a question :(
I can use HEX editor to see the difference, unfortunately, some different bytes here and there does not tell me much :(

My Roadmap for now:
2). I could not find "UnRenamed" dat for QL. I'm preparing the collection for upload. Sorting by software type (games, os, software e.t.c), and by publisher/developper. If i can find the year, I rename it properly.
2.1 Create the dat with clrMame. I will finish in ~3-4 hours, then it will be ready for upload.

3). I dounloaded "UnRenamed dat for Japanese Systems: surprise! All my titles were properly recognized. There is a little problem however: almost all x68k dat's entrys are .dim files (floppy images), but in my collection I have all the titles on hdf and xdf (SASI and SCSI hdd images), and two CD images. And as I don't know Japanais, it's quiet hard to say that "DAIMAKAIMURA" and "ghouls'n'ghosts" are the same game (actually they are).  So:
3.1 Firstly I will try to get all this "UnRenamed" for x68k. Actually, there is not so much.
3.2 I will create a script on AutoIt that will list all files from all the TOSEC .dim, and then the files from my hdf, xdf, img and downloaded "UnRenamed".
3.3 Comparing the lists
3.3.1 Eliminate the dupes.
3.4 Ready to upload. Aproximatively in a day or two.

4). Will learn to rip and try to get same crc for already dumped titles.
4.1) Optionally, if I will not success, I'll contact somewone from TOSEC ISO.

1). Just let me know what to do with multiple versions of titles that are not in the dats, and when I can not find any difference.

Correct me when I'm wrong, please.
Right now, i'm sorting, renaming and preparing the Sinclair QL collection for upload.
I return to work :)

Offline Cassiel

  • Administrator
  • Hero Member
  • *****
  • Posts: 1574
    • Email
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #6 on: December 03, 2009, 10:07:16 PM »
Couple of quick suggestions:

VFIC
http://www.vector.co.jp/soft/win95/util/se151106.html
for manipulating and converting a number of floppy disk based images. Remember that just because an image found in the wild has an extension of XYZ, it does not mean it really is XYZ format...

Easy Duplicate Finder
http://www.easyduplicatefinder.com/
for identifying REAL duplicates (does byte-for-byte and CRC32 scanning to determine if image is duplicate).

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1332
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #7 on: December 03, 2009, 10:44:23 PM »
Hi again,
can't help you much, mek1 is able to read japanese i think and TKaos talked with him some time ago, they know better about the formats and what they mean, there was something about it so be careful not to lost anything converting them :P

I suppose unrenamed dat Sinclair has Spectrum and QL and so on, like commodore one has c64 + amiga, etc :P
If there is something you can't identify you can just upload it and maybe one day someone will be able too, eventually these files should go to unrenamed dats until they are added somewhere.
I would say you should scan it all with unrenamed dats and tosec dats. Most of the files on unrenamed dats are available around (some aren't afaik), you could upload them and decide what you want to do, one step at a time, if you plan on doing too much you might get tired or frustrated so just decide what you want to do first: rename some elektronika, dump some isos, check other sets/systems and so on :)


edit: if you want you can also appear on #tosec2.0 irc channel on foreverchat network and talk with some of us (but most of the time we are there just idling ;D)
« Last Edit: December 03, 2009, 10:47:47 PM by PandMonium »

Offline TKaos

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 539
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #8 on: December 03, 2009, 11:07:18 PM »
Couple of quick suggestions:

VFIC
http://www.vector.co.jp/soft/win95/util/se151106.html
for manipulating and converting a number of floppy disk based images. Remember that just because an image found in the wild has an extension of XYZ, it does not mean it really is XYZ format...

Why would you want to convert existing disk images or modify them, or do I get this wrong?
We're using the images that are around and not convert them to a specific file to sort them better.

Also for Sharp x68k, DON'T just convert those because most of them won't work anymore then, I already talked about that with mek1 also some of them were released in a specific format and I really don't see a reason to change that format then.
« Last Edit: December 03, 2009, 11:10:36 PM by TKaos »

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1332
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #9 on: December 04, 2009, 10:49:18 PM »
Hi Xttx, how is it going?
Just a small update to tell you that gorski english is bad (he is from bulgaria) but he said you have a great amount of unknown/unrenamed QL files which he says were good stuff, Spectrum is different than QL (according to him, i don't know :P) and he currently has a baby so he don't know when he can work on them but he said at least these new files will not be lost :P

I'm really looking forward about it, hope someone has the time and will to do the work (if you want and get used to the work you can help us too and do it :)) because currently afaik we only have a Sinclair QL datfile with 33 sets ("Various" datfile) and it would be much better to have all this new stuff correctly organized in more dats instead of one stupid "Various" dat.

Thanks for your files :)

Offline Xttx

  • Newbie
  • *
  • Posts: 10
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #10 on: December 05, 2009, 09:36:44 AM »
because currently afaik we only have a Sinclair QL datfile with 33 sets ("Various" datfile)
Yes, and all of them a homebrew bas files :)
The collection I upload is just to preserve the original files as they were delivered on commercial downloadble sites.
I can not rename them properly because of QL .zip format. It should be disk images, but the emulator I use is comercial, and can not save on disks.
I already ordered this emulator for 60 euro, so, when i'll get it the work continue.
I also ordered 17 games that are not in my collection. Some contribution from my part :)

For tosecISO, i'm in on stand by mode for now. The people contribute something almost averyday there, so, i prefere to finish with QL and BK sets.

For X68K - I will need to convert disk images to files JUST TO COMPARE THEM, not to release! I will do it once I decide to take a pause with BK set.

Now I'm working with BK. There are about 3000 files, that I have to rename. And about 100 are already complete.
3.3% complete.

Offline TKaos

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 539
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #11 on: December 05, 2009, 11:29:07 AM »
I wonder why this wasn't posted here yet, most of your renaming questions should be solved by reading this:
http://tosecdev.org/index.php/downloads/category/5-tosec-naming-convention?download=9%3Atosec-naming-convention-2008-07-20
If there are still questions then just ask here. :)

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1332
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #12 on: December 05, 2009, 01:36:09 PM »
humm interesting, so those zips can't be renamed because they are in QL zip format or something like that right? Or did i got it wrong?
Renaming the zips / rebuilding them in clrmame would make those zips useless in QL and even if we go out to use a zip to old each of those zips (like if the current ql zips were roms) it wouldn't be a great solution i guess and cmp would broke it because it think it would rebuild zips inside those zips too :P

Great to read all that stuff about your progresses and good luck with that :)

Offline Xttx

  • Newbie
  • *
  • Posts: 10
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #13 on: December 05, 2009, 02:47:21 PM »
TKaos
I already begin to read it. There is so much to care about.... But i will try to do my best :)

PandMonium
The thing with QL zip's, that when zipping under QDos or SMSQ/e (with QL zip, in other worlds), the archiver streaps the file headers, and unzipping the files with winzip, winrar or something else under windows will corrupt the files with no possibility to recover them. QDOS unzip is aware of theese headers, and it adds them on the fly while unzipping.
What about renaming - theese zip files can be renamed but only withing 8.3 format, and without ( ) [ ] { } ' ? etc... Otherwise QDOS will not recognize them.
I don't know if clrMame can rezip the zip inside another zip, but if they will be released as they are now, it will breake them for sure.
And i agree: zipping the zip is not the best idea.
IMHO, the best way to do is to unzip them to the floppy images, then zip those images and relase as is.
As usual, correct me if I'm wrong.
« Last Edit: December 05, 2009, 02:59:02 PM by Xttx »

Offline PandMonium

  • Administrator
  • Hero Member
  • *****
  • Posts: 1332
Re: Dumps that are not in tosec yet: May be this could be useful for the teem.
« Reply #14 on: December 05, 2009, 03:28:38 PM »
I agree, your point is the best solution up now, the problem is creating those images but i hope you can do it and save those sets from disappearing :)

About the document, it is a bit extensive than the older ones but i wanted to cover it all (and failed at it, need to redo part of it... one day!), any question just ask and we will be happy to answer :)