CMPro, TOSEC and one huge headahce

[^VoiD]

Hi all,

I'm absolutely at my wits end with this, I keep coming back to it from time to but just cannot solve it. I trawled what seems to be endless websites, and although I can find info or posts that touch on this issue, I cannot find anything that solves my particular problem.

In a nutshell, I'm having bother trying to rebuild my entire TOSEC (Main Branch) set from scratch, to a different folder using CMPro, I can do most of it, but it fails right at the end ...


What I 'm trying to do ...

Rebuild the entire TOSEC Main Branch set from scratch using what I currently have (100%), other than to learn it myself for future reference, it's now become something of a vendetta given how much bother it's given me just to do this, also, TorrentCheck is showing I have nearly 70GB of excess files, so it would also be handy to also get that back. Given all this, a full rebuild from scratch seems the best option, and it can't harm to freshen every thing up.


This is what I can do ...

I have created my new directory and used the 2 scripts off the TOSEC website, the first to create all the directories and the sub folders, and the second to move all the .dats to their respective folders, so far so good ...

But that's when the wheels fall off, I drop all of the .dats into CMPro, go to the 'profiler', tag them all, and load them all, I next see the CMPro 'Profile Batchrun' window. I have been over the 3 tabs (Scanner, Rebuilder and Misc) more times than I care to remember, but something just isn't right. Every time I start the process CMPro does starting pulling all the roms from the old folders correctly, but instead of putting them into their respective folders in the new TOSEC directory, it dumps them all into the new TOSEC root folder? As soon as I notice this I stop the process, I then go and make a few changes and try again, but still the same?

What have I missed? What am I not doing right? While I still have some hair left :/

Please no suggestions of RomVault, whilst I appreciate this is also a good ROM manager and some people swear by it, I've used CMPro for many many years, it handles all my other collections just fine and don't really want to confuse myself even more if I can help it, besides I did try to use it not long back, but it just kept crashing on me :\


So that't me, there must be other avid CMPro users out there that can do what I'm trying to achieve with little effort? It just doesn't seem to to work for me.

Can anyone help me please? You really have no idea how grateful and chuffed I would be to finally nail this.

All help, however small is really appreciated


Regards

Stef

[NLS]

Yeah. Me too. I used CMP for many years.

Why really you want to suffer?
"Please no suggestions of RomVault, whilst I appreciate this is also a good ROM manager and some people swear by it, I've used CMPro for many many years, it handles all my other collections just fine and don't really want to confuse myself even more if I can help it, besides I did try to use it not long back, but it just kept crashing on me :\"
Well... USE RV.
You move the whole DAT folder structure you made, in DATroot.
You throw all existing TOSEC in ROMroot OR ToShort.
You click first button. It scans all dats.
You click second button. It scans both ROMroot and ToShort to know what it's got.
You click third button. It finds the matches.
You click fourth button. It makes the fixes, cleans up ROMroot to have a proper TOSEC structure, deletes what is uneeded (actually puts it in ToShort in case you need them for something else).
DONE (and already torrentzipped on the side).

Now if you want to torture yourself, be my guest.
Feel free to use RV for this and then get back to CMP. I am sure CMP will forgive your disloyalty.

[tomse]

It seems the question is more suited for the developer of CMPRO.

My guess is that TOSEC has become larger than what CMPRO can handle thus the crash.

this isn't a solution, but you can try and split the systems into each it's own CMPRO installation
or take a small portion at a time.

[Maddog]

The answer seems to be hiding in our forum actually. I was able to find it only after a few seconds of Google-fu: https://www.tosecdev.org/forum/tosec-tools/clrmame-how-to-'batch-setting'-the-rom-paths/
(check the post by user Cr3tyl)

Note: I have not personally tried this. While I do use ClrMAME Pro for all my other dats, I far prefer RomVault for TOSEC, as it makes updating TOSEC a breeze. On that I totally agree with NLS. Do give it a chance. We all love our CMPro, but admittedly the interface isn't exactly well documented or easy to use, at least for this type of usage.

[NLS]

You are using x64 version of CMP?


Actually I don't use CMP any more (still update it though).


- Gruby. RV works great.
- TOSEC (main, ISO, PIX). We said already. Perfect.
- trurip. Well superdats kind of suck, it worked, but I am moving away from trurip anyway.
- redump. Pending evaluation.
- Amiga firmware. Perfect.
- UnExotica. Fine.
- WHDLoad. Works nice, although it does torrentzip the folders that I want extracted (so I just re-extract after fixing).


ALL the above are branches on the same tree, ticking and unticking on demand (to save some time when scanning files, no other reason).
Collections "auto" help other collections fill (for common files).


For all these, same 4 button process. ONCE.
Can't be much easier than that.
I remember the CMP process.

Ah...

- MAME. I actually don't use any ROM manager for that any more. I have perfect sets from PD, cleaned with torrentcheck and filled to 100% completion afterwards (which in practice -again: PRACTICE- IS the same as building in a ROM manager and then completing the torrents - only WAY faster).

[^VoiD]

Thanks for the replies all,

I've got a couple things mentioned here to try, and if RV really is as easy as mentioned, I best give that another try.

And yes, I'm using the x64 version of CMP


Thanks all

Stef

[^VoiD]

Well I'm beat, I followed your guide NLS, but RV appears to have made things 10 times worse ...

To cut a long and tedious story short that consists of RV, complete exhaustion and literally losing the will to live, I was following your guide, and it was going well, up until the part that attempts to fix the files.

RV got stuck at 10% and it stayed there for nearly 20 hours ... Given I have a capable decent fast PC, I considered that nearly 20 hours with no movement past 10% to be wrong. Bottom line is, after stopping RV, I now have 1 folder with the Acorn, Tandy and Texas Instruments folders that contain most (if not all) of the ROMs for them sets, and I have another folder (RomVault/ToSort) that contains ALL the rest of the TOSEC roms in .zip format and NO folder structure to be found, anywhere.

I've tried reasoning with RV, but it just doesn't seem interested anymore.

Can anyone give me a complete doofus guide to RV and how I can now get all my ROMs back into their respective TOSEC folders please? I don't doubt for one minute that for a lot of you this will be a cake walk and you'll think I'm a complete idiot. I know it's going to be something along the lines of creating all the folder structure again (I can do this easily) and then getting RV (or 1 by 1 with CMP) to move all the files back to their folders, but I just don't use RV so I'm not familiar with it, to be fair it may as well be written in Japanese :\


As per, any help is greatly appreciated ...



Regards

Stef

[^VoiD]

All,

I've (tried to) attach a picture to this post which may make my issue a bit more clearer ...


If you can see the pic, it shows that the fix got as far as, and completed the Acorn Archimedes / Application roms, as shown in green as now complete. But that's it, it's stopped there and got stuck?

It sat like this for just under 20 hours, hence I stopped it.



Stef

[Maddog]

I have previously posted a relatively detailed guide on RomVault for TOSEC usage here. Read this in case it gives you some more insight.
I think it is superior to NLS's version, since it doesn't force you to put every rom into Romroot or ToSort directory. Sorry NLS! 

It is not very clear from your post if you actually got stuck in the "Scan Roms" stage or the "Fix Roms" stage.
If the delay was in the "Scan Roms" stage, it might even be normal as the first pass for that on something as massive as a complete TOSEC takes literally many hours. If the delay was in the "Fix Roms" stage, it probably is abnormal as there's obvious and more continuous activity on this stage. Also, if you have fiddled with the settings page, use Scanning Level 2 and Fixing Level Torrentzip Level 2. I think these are the defaults anyway but mention it just in case, since level 3 is incredibly slow.

One thing I would propose: since you are not familiar with the program, try first will a smaller subset of TOSEC. Pick a part with several roms, but not too many (eg Amstrad) and see if it completes successfully. Or if your roms are now all together, just pick a subset of about 10000 files and put it into To Sort dir, while keeping the rest temporarily in a dir that RomVault will not scan (ie, not inside To Sort or inside your ROM path). If it does complete successfully, you can continue with additional and possibly incrementally larger parts because now you have an idea of how long it takes to process a given number of files on your specific machine. If it doesn't complete, then quite obviously something is wrong with the setup. Let us know of your results.

[tomse]

I can only guess you're trying to collect the whole TOSEC which has a lot of entries.

If you do as I suggested, by adding things to ToSort in portions and not everything at once, you'll have more control
of what's happening.  Suggested is to take one system at a time.

Make note that I recently remade my database for Commodore, ISO/Commodore, PIX/Commodore which took
some 20+ hours (mostly because of all the ISO's, and it's need to extraction, and re-compression).

[^VoiD]

Thanks for the replies ...

Sorry I forgot to say that it got stuck at 10% when trying to fix the ROMs, the other stages prior to his ran fine.

It got to 10% on the fixing stage quite quickly, then stopped there. Had I seen anything happening at all, I would have left it going, but as it had been at this same position for so long and there was no activity from the HD LED on my tower, I felt nothing was actually happening anymore.

Maddog, I'm now going to go through your detailed guide, will let you know what happens. I'm off out soon to a BBQ and won't be back to the wee hours of the morning, this should give it plenty of time to do it's stuff ...


Thanks again

Stef

[^VoiD]

Just hit 'Fix ROMs' and I've got this (screenshot attached)


Stef

[^VoiD]

I pressed 'OK' and it's now showing 'Complete' but at 85% of the Acorn stuff (screenshot below).

As it says 'Complete' instead of 'Fixing' at the top of the window, one assumes it's now stopped?


I'm going to try to run it again ...



Stef

[^VoiD]

Ok, well I *think* it somehow stopped during the original scanning of the .dat folder, I've run the scan again and it says there are 4646 left to process, which would explain this. I also noticed that it seemed to stop in a hurry when it was scanning the .dats the first time around.

Whether it stopped because it fell over or I inadvertently stopped it who knows, but it seems to be back on track again ...


Stef

[Maddog]

I pressed 'OK' and it's now showing 'Complete' but at 85% of the Acorn stuff (screenshot below).

As it says 'Complete' instead of 'Fixing' at the top of the window, one assumes it's now stopped?

Yes, it has stopped. The sure way to tell it has stopped is that the "Cancel" button on the right  becomes "Close".

Just hit 'Fix ROMs' and I've got this (screenshot attached)

For the program to function correctly, it is important to always and only hit the buttons in the expected order. Ie first Scan Roms, second Find Fixes, third Fix Roms. I have never seen this error message, but one possibility that caused this is that you manually removed the file named Acorn Archimedes - Applications (TOSEC v2013-10-16_CM).zip before hitting the Fix Files button but after hitting the previous buttons. RomVault uses a kind of cache for identified files and it probably got confused when not finding a file it previously identified.

Also while we are at it: What are the contents of this file? Is it all the roms packed together in one zip? If that's the case, probably explains the terrible delay. As far as I know, RomVault will move the rom it knows to its' correct place and then try to repack the zip without the single ROM it moved, before going to the next rom. So if the file contains 1000 roms, it will probably cause 1000 repacks of the large container zip file before it has put everything in place. If you have your roms like that, please unpack them. They work much better as individual zips or even as unzipped files (RV will zip them up and torrentzip them too).