Author Topic: Roadmap / WIP - big picture  (Read 959 times)

Offline mictlantecuhtle

  • Global Moderator
  • Full Member
  • *****
  • Posts: 146
Roadmap / WIP - big picture
« on: July 15, 2022, 04:30:48 PM »
Aware that things have been relatively quiet on the forums and that, while I and others have been working away at various projects, we haven't always been great at being transparent about how things are progressing and what we're working towards.

To that end I thought it would be worth doing a kind of brain dump of where I see my current roadmap for the next few years at least (and probably further on for some of the "blue sky" stuff).

This is not intended to be definitive or prescriptive by any means, but rather a picture of where I see some of the work that needs to be done and the future potential for TOSEC as a whole.

(1) Immediate priority cleanup items

Some of the past conventions which drove TOSEC have changed over time, and as a result we have a number of DATs and ROMs which no longer adhere to modern standards.

This can be broken down into two broad categories - "Various" DATs and "ZZZ-UNK-" ROMs. An immediate priority for the project should be to properly split out the DATs, and to substantially reduce (or eliminate where possible) the ZZZ-UNK-s.

Various DATs
Code: [Select]
Advanced Computer Design PDQ-3 - Various (TOSEC-v2017-07-25_CM)
Burroughs B1000 Series - Various (TOSEC-v2017-04-05_CM)
Casio CFX-9850 - Various (TOSEC-v2011-10-06_CM)
Elektronska Industrija Nis PECOM 32 & 64 - Various (TOSEC-v2014-01-23_CM)
Ferguson Big Board II - Various (TOSEC-v2017-04-05_CM)
GCE Vectrex - Demos - Various (TOSEC-v2022-06-08_CM)
Hewlett-Packard HP48 - Various (TOSEC-v2014-01-24_CM)
Hewlett-Packard HP49 - Various (TOSEC-v2014-01-24_CM)
Matsushita JR-200 - Various (TOSEC-v2022-06-08_CM)
MITS Altair 8800 - Various - [BIN] (TOSEC-v2014-04-11_CM)
MITS Altair 8800 - Various - [DSK] (TOSEC-v2014-04-11_CM)
MITS Altair 8800 - Various - [HEX] (TOSEC-v2014-04-11_CM)
Motorola EXORciser - Various (TOSEC-v2017-04-05_CM)
NorthStar Horizon - Various (TOSEC-v2017-04-05_CM
OCC Osborne 1 & Osborne Executive - Various (TOSEC-v2014-06-27_CM)
PEL Varazdin Orao - Various (TOSEC-v2014-04-23_CM)
PTC Sol Terminal Computer SOL-20 - Various - [ASC] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [ASM] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [BAS] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [BS5] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [COM] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [ECB] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [ENT] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [HEX] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [LIB] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [ORIG] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [OPN] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [PL] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [PRN] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [SMU] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [SOL] (TOSEC-v2011-08-31_CM)
PTC Sol Terminal Computer SOL-20 - Various - [SVT] (TOSEC-v2011-08-31_CM)
Radio-86RK Apogej BK-01 - Various (TOSEC-v2014-04-23_CM)
Radio-86RK Mikrosha - Various (TOSEC-v2014-04-23_CM)
Radio-86RK Partner-01.01 - Various (TOSEC-v2014-04-23_CM)
Radio-86RK Radio-86RK - Various (TOSEC-v2014-04-23_CM)
Radio-86RK YuT-88 - Various (TOSEC-v2014-04-23_CM)
Robotron HC900, KC85 2, KC85 3 & KC85 4 - Various (TOSEC-v2014-05-13_CM)
Robotron KC Compact - Various (TOSEC-v2014-05-13_CM))
Robotron Z1013 - Various (TOSEC-v2014-05-13_CM)
Robotron Z9001 & KC85 1 - Various (TOSEC-v2014-05-13_CM)
Sony PocketStation - Various (TOSEC-v2014-05-29_CM)
Spectravideo SVI-318 & SVI-328 - Various - [BIN] (TOSEC-v2014-05-29_CM)
Spectravideo SVI-318 & SVI-328 - Various - [CAS] (TOSEC-v2014-05-29_CM)
Tandy Radio Shack TRS-80 Model 4 - Various (TOSEC-v2014-06-27_CM)
Tandy Radio Shack TRS-80 Model I - Various (TOSEC-v2014-06-27_CM)
Tandy Radio Shack TRS-80 Model III - Various (TOSEC-v2014-06-27_CM)
Tangerine Oric-1 & Oric Atmos - Various - [Multipart] (TOSEC-v2014-06-27_CM)
Texas Instruments TI-99 4A - Various (TOSEC-v2014-06-06_CM)
Xerox 820-II - Various (TOSEC-v2017-04-05_CM)

ZZZ-UNK (in order of most to least)
Code: [Select]
MGT Sam Coupe - Demos - [DSK] (624)
Tangerine Oric-1 & Oric Atmos - Various - [TAP] (592)
Atari Lynx - Games - [LNX] (172)
PTC Sol Terminal Computer SOL-20 - Various - [ENT] (145)
Atari Lynx - Demos (95)
MGT Sam Coupe - Games - [DSK] (86)
Sinclair ZX81 - Games - [P] (56)
Apple II - Compilations - Games - [DSK] (45)
Robotron HC900, KC85 2, KC85 3 & KC85 4 - Various (40)
OCC Osborne 1 & Osborne Executive - Various (39)
Casio CFX-9850 - Various (38)
Radio-86RK Radio-86RK - Various (37)
Sharp X68000 - Games - [DIM] (37)
MGT Sam Coupe - Games - [DSK] (36)
Robotron Z1013 - Various (29)
Radio-86RK Partner-011 - Various (28)
Sinclair ZX Spectrum - Unknown - [Z80] (23)
Apple II - Games - [DSK] (22)
Nascom I & II - Games - [NAS] (20)
Sinclair ZX Spectrum - Unknown - [TZX] (20)
Robotron Z9001 & KC85 1 - Various (18)
MGT Sam Coupe - Applications - [DSK] (17)
Apple IIGS - Applications - [2MG] (12)
Sharp X68000 - Operating Systems (12)
Atari 8bit - Applications - [XEX] (11)
RCA Chip-8 - Games (10)
Thomson TO8, TO8D, TO9, TO9+ - Compilations - Games - [SAP] (9)
Apple II - Applications - [DSK] (7)
Apple II - Compilations - Games - [PO] (7)
Apple II - Compilations - Games - [DO] (6)
MGT Sam Coupe - Demos - [TD0] (6)
Radio-86RK Apogej BK-01 - Various (6)
Elektronika BK-0011-411 - Games (5)
MSX MSX2 - Games - [DSK] (5)
Thomson TO8, TO8D, TO9, TO9+ - Compilations - Various - [SAP] (5)
GCE Vectrex - Demos - Various (4)
Sinclair ZX Spectrum - Unknown - [TAP] (4)
Tandy Radio Shack TRS-80 MC-10 - Various (4)
Apple II - Applications - [DO] (3)
Coleco ColecoVision ADAM - Games - [IMG] (3)
Fujitsu FM Towns - Games - [D88] (3)
MGT Sam Coupe - Demos - [SAD] (3)
Sharp X68000 - Applications (3)
Tesla PMD 85 - Games - [02b] (3)
Apple IIGS - Games - [2MG] (2)
Commodore Amiga - Operating Systems - Workbench (2)
Commodore C64 - Games - [TAP] (2)
DEC PDP-8 - Applications - [RX1] (2)
PTC Sol Terminal Computer SOL-20 - Various - [PL] (2)
Radio-86RK Mikrosha - Various (2)
Sinclair ZX Spectrum - Games - [Z80] (2)
Texas Instruments TI-92 - Games (2)
Commodore VIC20 - Applications - [CRT] (1)
Elektronika BK-0010-0011M - Applications (1)
Elektronika BK-0010-0011M - Games (1)
PTC Sol Terminal Computer SOL-20 - Various - [HEX] (1)
Sharp X68000 - Multimedia (1)
Sinclair ZX Spectrum - Unknown - [SP] (1)
Thomson TO8, TO8D, TO9, TO9+ - Compilations - Applications - [SAP] (1)

(2) Other maintenance tasks
  • Properly audit floppy image DATs to correctly identify formats and amalgamate as appropriate (e.g. IMA/IMG could be different extension but functionally same format)
  • Clean up and decide on a proper approach for multi-file DATs (e.g. Casio PB-1000)
  • Clean up and deduplication of systems where headers are a potential issue - NES, Famicom Disk System, Atari Lynx, others?
  • Clean up and deduplication of systems where multiple, interchangeable formats are a potential issue (N64, others?)
[li]Clean up of DATs where there is discrepancy between set/rom/description
[/li][/list]

List of multi-file DATs:
Code: [Select]
Atari ST - Online Magazines (TOSEC-v2013-12-13)
Casio PB-1000 - Applications (TOSEC-v2013-12-16)
Casio PB-1000 - Games (TOSEC-v2013-12-16)
Commodore Amiga - Games - SPS (TOSEC-v2022-06-11)
Commodore C65 - Applications (TOSEC-v2011-10-06)
Commodore C65 - Demos (TOSEC-v2011-10-06)
Elektronika BK-0010-0011M - Applications (TOSEC-v2014-01-23_CM)
Elektronika BK-0010-0011M - Games (TOSEC-v2014-01-23_CM)
Enterprise 64 & 128 - Demos (TOSEC-v2014-01-23_CM)
Enterprise 64 & 128 - Games - [Multipart] (TOSEC-v2011-10-16_CM)
ETL Mark II - Applications (TOSEC-v2014-01-24_CM)
ETL Mark IV - Applications (TOSEC-v2014-01-24_CM)
ETL Mark IV A - Applications (TOSEC-v2014-01-24_CM)
Fidelity Sensory Chess Challenger - Modules (TOSEC-v2018-07-01_CM)
Fuji Photo Film FUJIC - Applications (TOSEC-v2014-01-24_CM)
Fujitsu FM Towns - Firmware (TOSEC-v2011-02-26_CM)
Fujitsu FM77-AV - Games - [RAW] (TOSEC-v2018-03-08_CM)
Grundy NewBrain - Games (TOSEC-v2021-07-25_CM)
Hewlett-Packard HP9835A-B - Option ROMs (TOSEC-v2018-07-01_CM)
HomeLab BraiLab - Firmware (TOSEC-v2011-09-04_CM)
HomeLab HomeLab - Firmware (TOSEC-v2011-09-04_CM)
IBM PC Compatibles - Applications - [RAW] (TOSEC-v2018-11-15_CM)
IBM PC Compatibles - Compilations - Applications - [RAW] (TOSEC-v2019-09-01_CM)
IBM PC Compatibles - Compilations - Games - [RAW] (TOSEC-v2019-09-01_CM)
IBM PC Compatibles - Coverdisks - [RAW] (TOSEC-v2018-07-01_CM)
IBM PC Compatibles - Drivers - [RAW] (TOSEC-v2019-09-01_CM)
IBM PC Compatibles - Educational - [RAW] (TOSEC-v2018-11-15_CM)
IBM PC Compatibles - Games - [RAW] (TOSEC-v2019-09-01_CM)
IBM PC Compatibles - Miscellaneous - [RAW] (TOSEC-v2018-11-15_CM)
IBM PC Compatibles - Operating Systems - [RAW] (TOSEC-v2018-11-15_CM)
IBM PCjr - Applications - Files (TOSEC-v2014-04-10_CM)
Infocom Z-Machine - Games (TOSEC-v2022-06-08_CM)
Keio University K-1 - Applications (TOSEC-v2014-06-27_CM)
Matsushita JR-200 - Various (TOSEC-v2022-06-08_CM)
Mattel Intellivision - Compilations - [Multipart] (TOSEC-v2018-07-01_CM)
Mattel Intellivision - Demos - [Multipart] (TOSEC-v2018-07-01_CM)
Mattel Intellivision - Firmware - [Multipart] (TOSEC-v2018-07-01_CM)
Mattel Intellivision - Games - [Multipart] (TOSEC-v2018-07-01_CM)
Mattel Intellivision ECS - Educational - [Multipart] (TOSEC-v2021-07-25_CM)
Mattel Intellivision ECS - Games - [Multipart] (TOSEC-v2021-07-25_CM)
Microkey Primo - Applications - [Multipart] (TOSEC-v2011-09-04_CM)
Microkey Primo - Firmware (TOSEC-v2011-09-04_CM)
Microkey Primo - Games - [Multipart] (TOSEC-v2011-09-04_CM)
NEC PC-88VA - Games - [RAW] (TOSEC-v2018-03-08_CM)
NEC PC-9801 - Games - [RAW] (TOSEC-v2018-03-08_CM)
Nintendo 3DS - Games - [3DSX] (TOSEC-v2022-06-08_CM)
RM Nimbus PC - Applications (TOSEC-v2014-06-27_CM)
RM Nimbus PC - Games (TOSEC-v2014-06-27_CM)
Rockwell AIM65 - Applications (TOSEC-v2018-03-08_CM)
Sega Mega Drive & Genesis - Applications - [Multipart] (TOSEC-v2019-09-01_CM)
Sega Mega Drive & Genesis - Games - [Multipart] (TOSEC-v2019-09-01_CM)
Sharp X68000 - Games - [RAW] (TOSEC-v2018-03-08_CM)
Sinclair ZX Spectrum - Applications - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX Spectrum - Compilations - Demos - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX Spectrum - Compilations - Games - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX Spectrum - Compilations - Magazines - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX Spectrum - Demos - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX Spectrum - Documentation - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX Spectrum - Games - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX Spectrum - Magazines - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX81 - Applications - [Multipart] (TOSEC-v2011-09-24_CM)
Sinclair ZX81 - Games - [Multipart] (TOSEC-v2011-09-24_CM)
Sony SMC-777 - Games - [RAW] (TOSEC-v2020-10-26_CM)
Tandy Radio Shack TRS-80 Color Computer - Magazines (TOSEC-v2014-06-27_CM)
Tandy Radio Shack TRS-80 Model 4 - Various (TOSEC-v2014-06-27_CM)
Tandy Radio Shack TRS-80 Model I - Various (TOSEC-v2014-06-27_CM)
Tandy Radio Shack TRS-80 Model III - Various (TOSEC-v2014-06-27_CM)
Tangerine Oric-1 & Oric Atmos - Various - [Multipart] (TOSEC-v2014-06-27_CM)
Tektronix 405x - Utilities (TOSEC-v2017-04-05_CM)
Texas Instruments TI-73 - Applications (TOSEC-v2014-06-30_CM)
Texas Instruments TI-73 - Demos (TOSEC-v2014-06-30_CM)
Texas Instruments TI-73 - Games (TOSEC-v2014-06-30_CM)
Texas Instruments TI-73 - Shells (TOSEC-v2014-06-30_CM)
Texas Instruments TI-73 - Sources (TOSEC-v2014-06-30_CM)
Texas Instruments TI-80 - Applications (TOSEC-v2014-06-30_CM)
Texas Instruments TI-80 - Demos (TOSEC-v2020-07-12_CM)
Texas Instruments TI-80 - Games (TOSEC-v2014-06-30_CM)
Texas Instruments TI-81 - Applications (TOSEC-v2014-06-30_CM)
Texas Instruments TI-81 - Demos (TOSEC-v2014-06-30_CM)
Texas Instruments TI-81 - Games (TOSEC-v2014-06-30_CM)
Texas Instruments TI-82 - Applications (TOSEC-v2014-06-30_CM)
Texas Instruments TI-82 - Compilations - Applications (TOSEC-v2014-06-30_CM)
Texas Instruments TI-82 - Compilations - Games (TOSEC-v2014-06-30_CM)
Texas Instruments TI-82 - Demos (TOSEC-v2014-06-30_CM)
Texas Instruments TI-82 - Games (TOSEC-v2021-07-25_CM)
Texas Instruments TI-82 - Shells (TOSEC-v2014-06-30_CM)
Texas Instruments TI-83 - Applications (TOSEC-v2014-06-30_CM)
Texas Instruments TI-83 - Compilations - Applications (TOSEC-v2014-06-30_CM)
Texas Instruments TI-83 - Compilations - Games (TOSEC-v2014-06-30_CM)
Texas Instruments TI-83 - Demos (TOSEC-v2014-06-30_CM)
Texas Instruments TI-83 - Games (TOSEC-v2022-06-08_CM)
Texas Instruments TI-83 - Modules - Ion (TOSEC-v2014-06-30_CM)
Texas Instruments TI-83 - Shells (TOSEC-v2014-06-30_CM)
University of Tokyo PC-1 - Applications (TOSEC-v2014-06-30_CM)
Videoton TV-Computer - Games - [Multipart] (TOSEC-v2014-07-30_CM)
Visual Technology Visual 1050 - Sources (TOSEC-v2014-01-22_CM)

(3) Integration / cross-referencing with other projects
  • Cross-reference relevant DATs with No-Intro database to correctly identify known good / bad / hacked dumps and ensure info integrated into TOSEC
  • Ongoing programme of integration of dumps included in No-Intro, renamed according to TOSEC Naming Convention
  • Cross-reference with MAME software lists to pick up any relevant information e.g. serials, hardware requirements, etc. and ensure this info is integrated into TOSEC
  • Ongoing programme of integration of dumps included in MAME software lists, renamed according to TOSEC Naming Convention

(4) Cross-referencing with other online databases of information about software releases

Hard to list all of these, but to my mind there there are four broad categories:
  • System-specific (e.g. CSDb, Amiga Demoscene Archive)
  • Publisher-focused (e.g. Atarimania, Sega Retro)
  • Category specific (e.g. Demozoo, pouet.net)
  • Universal / wide-ranging (e.g. Mobygames)

Broadly, what I'd really like to see here is a significant reduction in the number of sets where we don't have any useful information about release dates or publishers. Obviously each of these sites has potential to give much more information - particularly those focused around a particular system.

(5) Longer term ingestion of other large sets / sources of dumps
  • Neo Kobe sets
  • Bitsavers
  • GoodSets
  • Yori Yoshizuki UnRenamed / NonGood

(6) The blue sky stuff
  • Online database / DAT generation tool to (a) provide a searchable frontend for the information contained in TOSEC, (b) allow DAT maintainers to add information on an ad-hoc basis rather than being restricted to specific release cycles and (c) allow end users to split and generate DATs to suit their own needs
  • Proper examination and databasing not just of dumps as globs of binary data but of the information they contain. Thinking in particular here of disk images, cataloguing / hashing files contained on disk for example

I'm sure I've missed plenty and that others have additional priorities for the list, so please consider this the starting point for a discussion and let me know what else we should be doing.

Final point, if you've looked at anything on the above list and thought "that's just the job for me!", then let me know!