[v2.8.8.8[15/12/09]]OpenGenie Framework-opensource SMS/MD/32X/SegaCD rom diggin'

[Conle]

OpenGenie - SMS/MD/32X/SegaCD rom diggin' Opensource framework.



[~CREDITS~]
Coded by : conleon1988 (-see mail in package-)

[~CONTRIBUTORS~]
ChillyWilly(bug reports & sram 68k opcode + data pair patching)

[~THANKS~]
Special thanks : To all SMS/MD and SEGA retro community , to all emu athors ,

and NEOFLASH(http://www.neoflash.com) , first for supporting HOMEBREW all these years,
and second for providing me with their 3in1 MYTH CART for testing
purposes and im able to confirm that all the patches also work great on their cart(NEO 3in1 MD MYTH 512M).
Also a huge thanks to all authors who wrote these awesome awesome hardware docs(see hardware docs directory)!!


[~INFO~]
All the project is coded in C and supports GCC 4.3+ new function allocators.
Also to port this project to another platform all you have to do is to checkout "port.h"
in "framework" directory , add your own directives/code and then make sure that you
have ported the "directory_iterator.c" so that it uses the native functions of your OS.

Please keep the code clean and optimized as it is now. Thx.

What this framework can do:
-Supports .SMD,BIN,GEN,32x roms and SEGA CD ISO
-Is able to patch region code with simple commands
-Is able to patch linked roms with .IPS translated files on-the-fly
-Is able to patch IO device addresses
-Is able to patch all information without writting low level code
-Is able to recognize incorrect checksums and re-calculate & patch them
-Is able to convert roms to BIN(clean dumps)
-Supports bitfields for easy and efficient patching
-Supports a smart directory iterator for direct access to the wanted content
-Supports basic html generator for info dumping
->Is able to patch SRAM 68K opcode  + data pair (ChillyWilly's idea + explanation)
->Countless more patches like copyright hack and more

[~SOURCE LICENCE~]

MIT/X11 - You can do whatever you want with it.

[Conle]

Latest changelog

6 / SEP / 09
->Initial release

7 / SEP / 09
->Improved some code in fmt_smd.c
->Updated framework.h
->Started working on SuperMD Tool
->Fixed ident issues in ips_patcher.c and directory_iterator.c caused by CODEBLOCKS.Sorry about this

8 / SEP / 09
->Some changes in ips_patcher.c
->Improved html_gen.c
->SMDTool now supports patching
->SMDTool can now try all loaders if the proper one fail. (for roms that have wrong extension)
->Improved util.c -> skipWhitespace

9 / SEP / 09
->Improved rom.c
->In rom.c , loadRom() now returns success of failure instead the pointer
->Updated rom.h
->Finished SMDTool commandline parsing(PATCHADDR W-I-P)
->An important fix in checksum_patcher.c

10 / SEP / 09
->SMDTool can now patch custom address
->Finished almost all functions and now SMDTool works
->Added smart versioning in SMDTool
->Added usage screen

11 / SEP / 09
->Improved all code in SMDTool
->Added CRC check in SMDTool(global radius)
->Added 2 new functions for parsing support in util.c

12 / SEP / 09
->ips_patcher.c now supports both "rom.ext.ips" and "rom.ips" patch files!
->SMDTool is almost complete.TODO XXX test all functions XXX

13 / SEP / 09
->Updated port.h with new functions
->Improved directory_iterator.c
->Looked over all source files for possible optimizations. All checks passed.
->SMDTool can now take another argument "PATCH_DIRECTORY" to forward all rom patches to a custom directory
->SMDTool passed all tests. Now it must be compiled in windows too
->fmt_smd.c - chunk buffer is no longer dynamic
->rom_Writter.c - its possible to dump the header to a file
->Updated rom_Writter.h

16 - 23 / SEP / 09
->Improved IPS patcher : Added new mode to detect the end of buffer.
->Fixed a bug in rom.c that sometimes it adds a garbage character the end of the filename [ reported by Chilly Willy - Thanks! ]
->Fixed another bug related with memory length in rom.c
->Fixed a small bug in util.c "getFileType()" and now it supports files with no extensions. [ reported by Chilly Willy - Thanks! ]
->Improved HTML generator : Supports also tables but its disabled at the moment. The default mode is also improved and generates a clean text block.
->Added multiple modes in port.h and more options
->Fixed a small bug in directory iterator that it could cause a memory leak if not used right.
->Basic Gamegenie interface : Supports decoding of streams for now but not hex addresses.
->Added a nice logger which reports the line and the source file when printing text.
The logger is de-activated by default.
->Replaced all debug prints with dumpConsole() define and now we can disable logging
->Updated rom writter
->A small change in fmt_smd.c

24 / SEP / 09
->Added support for 32X roms
->Updated rom.c
->Updated util.c
->Updated framework.h


27 / SEP / 09 [release edition]
->SMDTool updated to 1.0.0.9
->Framework updated to 2.0.0.2
->Updated port.h
->Updated framework.c and the header to add getFrameworkVersionString() function.
->Some updates in game_genie.c
->Some updates in htmlgen regarding pointer access safety
->A small change in logger.c
->Confirmed 32x rom support
->SMDTool now uses the buildin logger
->Added multiPrint methods in SMDtool
->Fixed SMDTool's copyright hack "bug" at line #753 where i had the month with 2 characters instead 3[ reported by Chilly Willy - Thanks! ].
->Added a special section for contributors in the framework -> contributors.txt
->Updated rom_writter to allow dumping of 32x roms

27-28-29 / SEP / 09
This update is focused on last possible optimizations regarding memory copies and allocations. Plus a faster way to detect
if a rom is supported.
This update increases the speed by 40-55% for batch rom conversions and 100% less allocations , and about 3-7% for a single rom.

->Added setBitFieldRange() in bit_field.c and .h To reset fields without recreating them
->Improved bit field handling in rom.c for batch rom conversions/info.Now when resetting a rom , fields just reset to the default state.
->Improved isSupportedRom() and now it does a memory range check before validating information to avoid crash on bad/unknown binary files.
->Improved rom.c and now first it checks if the rom is supported , and then it updates information. isSupportedRom() is also updated
to support this.
->[minor]A small update in smdtool . Updated loadRomEx()

29-30 / SEP / 09
->Added global framework exit func hook
->Added stack_patch.c . We will need this for Sega cd roms
--->supports Any image size , like 200 - 300mb...There is no limit
->Updated stack_patch.c and added sorting
->Updated stack_patch.c and added stack iterator
->stack_patch sorting now is done only once after calling iterator stack.
->Sega CD ISO 96## and Binary/mp3 support improved.
->Sega CD mode now allocates only a few bytes in memory.

30SEP - 1 / OCT / 09
->Stack patch now sorts only the used slots
->Stack patch is now 100% complete
->Added proper SEGA CD image stream handle in rom.c and h
->Rom writter can now dump SegaCD isos synchronized with the sorted stack patch list.This is very fast , no loops are involved.
I have tried it all patches on an Sega CD image and they work as one would expect on Gens emulator. --I don't have sega cd to test it,
but it should be okay.
->IPS Patcher can now write stack patches for SCD images
->Found a little issue in htmlgen with extractFilename when there's no directory -- fixed -- and i have to wrap strrchr sometime...
->Added support for MARS 32x header

1 / OCT / 09
->All patches now fully support Sega CD images
->Added support for SCD headers that need byte padding

2 / OCT / 09
->Dir iterator now allows custom file extension filter callback.
->Framework updated to version 2.0.0.7[2/10/09]

3 / OCT / 09
->Added SMD write support!! That is , now the framework can generate SMD dumps too!

3 - 4 / OCT / 09
->Added a faster hexademical to integer function
->Now if you pass custom directory argument to SMDTool , it will not append "_PATCHED.extension".
->Added my own version of strrchr which is fastest and it will end all extra handling...
->Added MAKESMD option for SuperMd tool's -MODE switch
->Optimized getFileType()
->Added RLE IPS patch support

7 / OCT / 09
->Added C externs for C++ compatibility
->Started working on the GUI version of the tool

7 - 8 / OCT / 09 [release edition][/color]
->Added raw_patchfile.h and raw_patchfile.c to import raw patch files (supports also SEGA CD using stack patch)
->Added applyGameGenieCode (supports also SEGA CD using stack patch)

15  / DEC / 09 [ release edition ]
->Implemented ChillyWilly's SRAM 68K CPU opcode patching(Also supports Sega CD image using stack patch).
->Version codename changed to : v2.8.8.8
->A quick change in contributors.txt

[Hatta]

You don't have a port.h set up for Linux already do you?

[Conle]

You don't have a port.h set up for Linux already do you?

Hi,
Im using linux so of course you can compile it.
Just replace

Code: [Select]

#define WINDOWS_BUILD

with

Code: [Select]

#define UNIX_BUILD

 

An updated version coming soon! I just need to finish the first tool that uses this framework.

[Hatta]

Sounds great.  Can I ask how you flash the cart with linux?  I've been using an XP installation in VirtualBox.  Works great, but a native solution is always preferable.

[Conle]

Sounds great.  Can I ask how you flash the cart with linux?  I've been using an XP installation in VirtualBox.  Works great, but a native solution is always preferable.

There is no way to flash the cart on linux without some drivers & software.
I'm just using our another pc or i just switch to windows xp x64 pro.

And some good news : the tool is finished , i just need to write some basic documentation  .

Here are some examples showing how to region patch a whole directory :

smdtool -MODE REGION -DIR roms -REGION_CODE EUJ -CHECK_CRC ON

or how to patch addresses with custom data :

smdtool -MODE PATCHADDR 0x1f0 datadata -SINGLE_FILE rom.smd -CHECK_CRC ON

 

[Conle]

Game genie decoding progress:
http://www.neoflash.com/forum/index.php/topic,5780.msg40942.html#msg40942

[ChillyWilly]

Okay, got around to testing this: the linux binary included doesn't run; it just gives me this error: bash: ./smdtool: cannot execute binary file. The source compiles fine, and when I run it, it prints the usage message, but when I tried to do the info option, it seg-faulted leaving a 0 byte html file in the directory.

The terminal output is

jlfenton@coreduo-desktop:~/Tools/linux/OpenGenie/SMDToolsrc$ ./app -MODE INFO -DIR roms
----INFO----
SMDTool v1.0.0.6 [13/9/09]

----CREDITS----
SMDTool coded by conleon1988 (conleon1988@gmail.com)
OpenGenie Framework coded by conleon1988 (conleon1988@gmail.com)

----THANKS----
Special thanks to :
1)ALL Emulator author's and to all people who worked hard on the documents
2)To NEOFLASH(www.neoflash.com) for providing me with their
-flashcart(NEO MD MYTH 3in1 512M),and im able to confirm that their flashcart is fully compatible with the patched roms.
Generating SMDTool_Results.hmtl from target(s)
Segmentation fault

There are a mix of bin and smd files in the roms directory - 32 files all total. This is a 32 bit Xubuntu 9.04 system. I'm using gcc 4.3.3 from the Ubuntu repo. If you need more info, let me know.

Doing one rom works - "./app -MODE INFO -SINGLE_FILE roms/wolf32x.bin" worked fine with the html file having the proper info. It's just the DIR option having a problem.

One other thing in the readme - you have the line

-CHECK_SRC : This validates and fixes incorrect checksums. This is global switch.

That should be CHECK_CRC. 

[Conle]

Thank you , that post was very useful.

Okay, got around to testing this: the linux binary included doesn't run; it just gives me this error: bash: ./smdtool: cannot execute binary file.

I compiled the code on a x64 ubuntu system , and i believe it generates x86-x64 code. I couldn't make it to compile x86 instructions, even
with the proper libraries.

The source compiles fine, and when I run it, it prints the usage message, but when I tried to do the info option, it seg-faulted leaving a 0 byte html file in the directory.

With that info i know where it might hang :
Either at line 382 in main.c or between lines 69 - 98 , 154 - 170 in directory_iterator.c.

I noticed something in line 154 of directory_iterator.c .
I have placed a memset after releasing the directory stack if it was previously used.

Can you plaease replace this function in directory_iterator.c->
http://pastebin.com/d3ac29574
-sorry about the ident , pastebin doesn't seem to support well my ident style-

with this one - updated!- :
http://pastebin.com/d37e8001d

Damn i forgot to replace that value.
This :
for(i = 0; i < directoryStackLength; i++)
Should be :
for(i = 0; i < DIRECTORY_STACK_SIZE; i++)

Sorry 
And tell me if it worked?
I can't reproduce this error , but i think that this could be the issue.

That should be CHECK_CRC.  

Good catch! , i will fix this in the next version.


By the way , are you the famous coder Chilly Willy behind all 32x ports and psp doom/wolf ports ?

[Conle]

I'm uploading now the framework folder...Will edit this post in a sec.


Okay , done , try to recompile the code with this updated framework folder.
It also has the gamegenie decoder , and probably that  bug fixed

Attachment removed

[ChillyWilly]

Thank you , that post was very useful.
I compiled the code on a x64 ubuntu system , and i believe it generates x86-x64 code. I couldn't make it to compile x86 instructions, even
with the proper libraries.

That also explains the significant difference in size I saw.  

I know what you mean about compiling 32 bit on 64 bit systems - it's a royal pain. Maybe I can provide the 32 bit binaries.

With that info i know where it might hang :
Either at line 382 in main.c or between lines 69 - 98 , 154 - 170 in directory_iterator.c.

I noticed something in line 154 of directory_iterator.c .
I have placed a memset after releasing the directory stack if it was previously used.

Can you plaease replace this function in directory_iterator.c->
http://pastebin.com/d3ac29574
-sorry about the ident , pastebin doesn't seem to support well my ident style-

It's the line endings - you're using Windows line endings, and it wants unix line endings. Most editors have an option for that. I use Geany as my code editor, and it lets you set the type of line endings to Windows, unix, or Mac.

with this one - updated!- :
http://pastebin.com/d37e8001d

And tell me if it worked?
I can't reproduce this error , but i think that this could be the issue.
Good catch! , i will fix this in the next version.

Still blows up... here's the backtrace

#0  0xb7f4c656 in strcasecmp () from /lib/tls/i686/cmov/libc.so.6
#1  0x0804f2f1 in getFileType ()
#2  0x08048ea4 in scanDirectory ()
#3  0x08049036 in dirIteratorInit ()
#4  0x080508d5 in commandINFO2HTML ()
#5  0x08050a85 in main ()

I'll do a little more looking into it here since you can't reproduce it. I'll post when I find something.

By the way , are you the famous coder Chilly Willy behind all 32x ports and psp doom/wolf ports ?

Yep! I got the Neo Myth 3 in 1 because I needed a second flash cart so I could test networking between two MDs with Doom. Not to that point yet, but I will at some point. I was considering a second MD-Pro, but then the Neo Myth was released and I just had to get it.  

[Conle]

That also explains the significant difference in size I saw.  

I know what you mean about compiling 32 bit on 64 bit systems - it's a royal pain. Maybe I can provide the 32 bit binaries.

Thanks that will really help me out , since i can't install 32bit linux OS on our 2nd pc thanks to my brother .
Its for general use...

It's the line endings - you're using Windows line endings, and it wants unix line endings. Most editors have an option for that. I use Geany as my code editor, and it lets you set the type of line endings to Windows, unix, or Mac.

I always use Gedit , even if my system is dual boot win & linux , i barely switch to windows .
I guess i will have to install an IDE sometime even though i love my gedit ,makefile, and gdb setup  

Still blows up... here's the backtrace

I have uploaded a small fix , and i found another small bug that its not that important at the moment.
I think that it might fix this issue.

And i guess i have to learn to use svn sometime

Yep! I got the Neo Myth 3 in 1 because I needed a second flash cart so I could test networking between two MDs with Doom. Not to that point yet, but I will at some point. I was considering a second MD-Pro, but then the Neo Myth was released and I just had to get it.

Wow , then respect , master .
Even my proffessor has this psp port on his psp . Believe it or not!!!! .

I needed a second flash cart so I could test networking between two MDs with Doom.

This sounds cool no matter how it will play!!

[ChillyWilly]

In util.c getFileType(), you have this

Code: [Select]

if(!ext)
However, the extension may not have existed, which your macro will return as 1, so you really need

Code: [Select]

if(!ext || ext == (char *)1)
That will cause it to return the format as unknown if there's no extension. Once I added that, it worked fine for the directory option.


EDIT: Oh, one thing on the info - the copyright string is one character too short when printed. You're chopping one character off the month.  

[Conle]

In util.c getFileType(), you have this

Code: [Select]

if(!ext)
However, the extension may not have existed, which your macro will return as 1, so you really need

Code: [Select]

if(!ext || ext == (char *)1)
That will cause it to return the format as unknown if there's no extension. Once I added that, it worked fine for the directory option.

I'm attaching my 32 bit linux binary.

Nice one , thank you!, i couldn't have had thought about this : A  rom file without extension   .

I will upload soon the updated code and the binaries.

[ChillyWilly]

Whoops, didn't see this post when I added another bit of info to my last post... you're chopping one character off the end of the copyright string. So the month is printed with two chars instead of three.