CMPro, TOSEC and one huge headahce

[^VoiD]

I've just nipped back from the BBQ to get more beer :| ... checked on this and it appears to have stopped towards the end of BBC (pic attached), everything below that is red. Funnily enough, as you mentioned it, I was also thinking that I wonder if it's because the ROMs are zipped up.

They are zipped up and in .zip files ... and this was how they were in the torrent I joined yonks ago. I've now left me PC unzipping everything that's been moved to the 'ToSort' folder, and that will take an age, so I'll check that when I get home later


Stef

[NLS]

Oh wow. This is what happens with a weekend not checking a forum.


I am gonna reply to things as I read them...


^VoiD
- Definitely 10% for 20 hours is out of the question.
- Seeing your pic, indeed seems it went on finishing a folder and then stopped (froze?). Never seen this before. In your place I would just reclick "Fix ROMs".
Maddog
- I suspect your guide is better.
Maddog/tomse
- Good advice on braking things down to easily fix, but I can verify I have tested with full non-ISO and works fine.
^VoiD
- 10% can reach pretty fast if it goes through small ROMs. Fix percentage seems to not be related to file sizes.
- In the shot where it says it is missing a file in ToSort, seems that something happened that a file that was there during scan got removed without informing RV (possibly when it got stuck). You would need to re-scan ROMs I guess.
- Complete looks ok ("Close" instead of "Cancel"), the missing bar is possibly the file you miss.
- About the variable speed some things happen. Note that RV caches dat and file status, so whatever is already parsed, goes through fast.
(erm Maddog already says much of this... stupid that I reply as I read, instead of reading the whole thread first)
- One thing that I suggest is to NOT have archives in archives. Makes things so much tougher for RV (and everybody).
- BBQ beats managing retro ROMs 9 out of 10 times.
- Seems that indeed you have zips in zips. Which is possibly how things came from PD torrent, right? I really suggest against having them like that... except if you plan to again seed for PD (although the latest TOSEC is not yet there). Thing is, the ROMs are unusable like that, while single-rom-zips are manageable not just by RV, but most emulators too, directly.


Hope all goes well.
Keep us posted.

[^VoiD]

Ok,

After leaving it overnight, I appear to be roughly where I want to be. I have a TOSEC folder which is based on the (TOSEC v2018-04-01) release and it's 134Gb in size. The issue must have been down to .zips inside of .zips, and yes NLS, it was from the PD torrent. I also decided I'd rather have it as just one level of zipping for the reasons you mentioned, that an emulator can read it like that, it is more as it was intended, and it won't cause any other hassles down the road.

The reason I wanted to go this way, apart from the hassles it can cause (as I found out) is that I wanted to have a crack at updating it myself, I have access to newgroups and stuff and maybe in the end, if I wasn't treading on anyone's feet (and once complete of course) I could maybe upload this version to PD. Of course if it is going to tread on peoples feet and/or upset anyone I won't bother, also as I'm still a stack of ROMs missing from a complete set, this possibility it still miles down the road, and also just an idea for now.

I do have another question about RV though, if you look at my attached screenshot, a lot of ROMs are in yellow, according to RV's key these are listed as "This ROM is missing here but has been found somewhere else, and so can be fixed", I've tried pressing "Fix ROMs" a few times, but nothing happens, how can I get RV to fix these?

Also, after trying a couple of times, I'm thinking there is no way to drag ROMs or folders of ROMs into RV for rebuilding "missing's" like you can with CMP? I'm thinking I would have to put any fixes I do get into the 'ToSort' folder and then press "Scan ROMs", "Find Fixes" and lastly "Fix ROMs" again?

Oh, also also, is there a way to get RV to remove the matched source files once rebuilt, like CMP does? ... Or does it do this anyway? I'm unsure? ... I extracted all the "zips inside zips" files, and that's what rebuilt my set to where it currently is, but the "ToSort" folder now has 1938 files in it that amount to 195Gb. These aren't part of the fully rebuilt set as I've moved that back into the original folder it came from, I'll just point RV to that folder in future instead of moving it around). Is it safe to delete these? Or are these files that we're unused in the rebuild? If so, there's an awful lot?


Anyway, thanks all for the help so far, I'm still not fully there, but I'm a damn sight closer than I was a few days ago. I now need to find a way to fill them "missing's", I will have a look through newsgroups, but other than that, does anyone have any suggestions of where I might be able to get them from? Other than waiting for a complete torrent, which of course has the potential to be not in this format again :\


Regards

Stef

[NLS]

1) I can verify that main TOSEC complete is 140GB and 759463 files. Don't take the file count exactly, as some system files and RV ignored files (there is an ignore list) might be in there. Looks you are not far from a complete main TOSEC. Start posting your miss-dats! (generate reports, last button)


2) Seeing your screenshot things are OK, you don't have to press fix again. You just have to learn how to read this. On the LEFT side, green you know, red means you miss ALL roms, yellow means you miss some. NOT that it can be fixed from your current set. On the RIGHT side, you can see that it only shows red and green numbers. THIS is where a yellow would mean "buildable from what you have". Since you only have red and green, you have the best that you can already.


3) No you cannot drag 'n drop. I am not sure about upcoming 3.0 version though (in private beta). You indeed have to put in ToSort (or even in ROMroot if you believe you know where they fit) and rescan (files NOT dats) and fix. It will also fix what is already in ROMroot if something is in wrong position or needs a rename etc.


4) The matched files ARE removed automatically. Both from ToSort and from ROMroot. In ToSort only remain UNMATCHED by any dat files. Also files that are not needed from ROMroot, RV throws them to ToSort to check if you need them in the future. If in ToSort there are no zips-in-zips (that are possibly too deep to check), I would safely delete them. Check if these are files from other TOSEC collections though! (ISO or PIX) If you have the dats, you can check. Not sure of a reason you have so many. Maybe some read-only flag is the problem!? (I always run RV in admin mode, to be sure that I can delete files whatever the NTFS flags and security).


EDIT: Note that if files are protected (and no security level to high enough to delete anyway), the fix step takes WAY longer, since it tries to delete each and fails! This takes place AFTER the actual fixing (the fix bar is all green) when doing the "cleanup" (waiting for "CANCEL" button to turn to "EXIT").

5) For just 6GB missing, I am sure you can get them from here with fixdats.


EDIT #2: Since I re-read it and is not clear, I am joking about here being able to fix you missing. PM me.

[Maddog]

In general, I agree with virtually everything that NLS has posted in the last post and I am also glad that we got ^void on the right track.
A couple of small notes below...

Maddog/tomse
- Good advice on braking things down to easily fix, but I can verify I have tested with full non-ISO and works fine.

I advised that as there were doubts about RomVault working properly, time needed for a full rebuild on ^void's computer and so on. I always do a full rebuild both for the MAIN and for PIX, which is even bigger in size (although less files) and never had a problem with that either.
It should also be made clear that you don't have to put everything unknown or on initial scan inside the "To Sort" folder. RV is clever enough to find the stuff in any of the active paths. For example if you put all Spectrum Games .TZX into Games .TAP folder and press scan, it will detect the wrongly placed roms and when you press fix it will move them to their correct place.
I am mentioning that mostly for ^Void to keep in mind for the next rebuild down the line (since sometime in the future we will have another new release). Just leave everything you have in their folders and scan. Anything moved, renamed or removed will be fixed without you having to move anything manually into To Sort folder.

Regarding torrents, "fixes" etc: please keep all this in other forums. TOSECDEV can't give advice on that. We just rename files found in the wild and do not distribute them in this forum under any circumstances. Please remember that most of those "roms" are still copyrighted.

One thing I would like to mention though is that it's impossible for something as massive as TOSEC-MAIN to fit in a torrent. It is barely possible to make a working .torrent file for the Sinclair ZX Spectrum files only, as it becomes huge and most trackers and clients won't accept it at all. Just imagine trying to do the same over the entire TOSEC. That's why they had to zip the zips over at PD.

[^VoiD]

All,

Still wading through all this, I'm unzipping all the files that were left over in the ToSort folder, as suspected most of these do seem to be .zip files in .zip files. I'm up to 'Nintendo' so not far from the end now.

When this is done, I have read that a lot of the fixes for the latest release of TOSEC can be found in the MAME Software List set, which I have. To save wasting peoples time with fixes when I may be able to fix a lot of it myself, am I able to scan for fixes in RV without it removing said files when it matches them? Or may I have to manually wade through and handpick what RV may be looking for? I cannot find any settings in RV to control this?


Again, thanks for all the help with this...


Stef

[NLS]

There is no setting yet for this. I hear there is in 3.0 beta. Not sure.


I think I would just copy few systems at a time from MAME Software Lists to ToSort and parse them, clean ToSort from remains. Then again and again. Break in 5 parts is enough I think to easily go through everything as MAME SL is around 55GB zipped.

[^VoiD]

Thanks NLS,

I'll set about doing this later



Stef

[Tim2460]

Sorry about the Whole Zip in Zips mess Void ,))

It has upset some people true ... but i can't see other easy solutions to make it manegeable...

For my part i keep several ToseC Tree Directories, zipped and Zipped/Zipped....

[Maddog]

In all reality, TOSEC has grown too massive to be manageable using torrents, so I guess zips within zips was unavoidable for PD distribution. Most torrent trackers and most clients simply can't handle properly the number of files. I guess it never occured to the creators that it might be used for something like that xD.

Given the chance, I'd also like to thank you Tim for the hard work you have done making these torrents over several years and making TOSEC content available more widely.

[NLS]

A couple of systems need two or three torrents each.
A couple of systems needs a torrent each.
All the rest can fit in a single torrent.


I guess 10 torrents for TOSEC (non-ISO) is ok and manageable.
Not sure if this is more work for Tim though than zipping entire huge folders (which I guess is done by some batch).
Of course that said, torrent creation can also be done with a script.
Both things are CPU and disk hungry although having double TOSEC is MORE disk hungry I guess (including for everyone seeding AND using TOSEC).

So if somebody asked my opinion, it is pretty obvious, but since Tim manages these things, it's his call and I understand.


_{Actually I cannot even post in PD (muted for some unknown reason - funny how at least one admin played Pontius Pilate perfectly), so I guess my vote counts even less than normal.}

[Tim2460]

A couple of systems need two or three torrents each.
A couple of systems needs a torrent each.
All the rest can fit in a single torrent.


I guess 10 torrents for TOSEC (non-ISO) is ok and manageable.
Not sure if this is more work for Tim though than zipping entire huge folders (which I guess is done by some batch).
Of course that said, torrent creation can also be done with a script.
Both things are CPU and disk hungry although having double TOSEC is MORE disk hungry I guess (including for everyone seeding AND using TOSEC).

So if somebody asked my opinion, it is pretty obvious, but since Tim manages these things, it's his call and I understand.


_{Actually I cannot even post in PD (muted for some unknown reason - funny how at least one admin played Pontius Pilate perfectly), so I guess my vote counts even less than normal.}

I can't remember well the number of files i used to use on my last split release but it was 18 torrents for just the C64 games part....
Making just one streesed the PD webserver itself as it has to manage over 5meg attachment for the first time...

Sure it can be done by hand and split in 30 torrents ... but not by me... The worst was to split in 20 000 or 50 000 files pack from the c64 games tree... I didntt and still don't really see how to script that easily...

And you have to cope with all the people saying you can't make a torent, or that it crashed their seedbox... or that i forgot the Whole amiga part etc etc etc ...

Zipping/Zipping has cut the crap by at least 90%...

[NLS]

Well the last argument is fair.   Cutting crap is major. I feel you.


Still a smart script could get things automated (it could even handle splitting folders "up to" 20k files). I might look into this.



(and thanks for reminding me were I posted that thing about splitting torrent again )