How can I implement my own custom subsystem in windows ?
I posted the query on some other forums but it seems that nobody knows much about it.
Should I review the source code of ReactOS to find out more about this ?

Go ahead and look at ReactOS's SkyOS subsystem for an example.

You'll need three things:

1) A subsystem process (such as csrss) that talks to SMSS properly, and is registered in the registry
2) Subsystem DLLs that register with the subsystem process, at the very least. They'll also wrap whatever functionality you want to support
3) A special loader to launch your application by calling your subsystem process -- kernel32's CreateProcess has hard-coded checks for POSIX, OS/2, DOS and Win32, so unless you have source access, you won't be able to launch your app just by double-clicking on it -- a special loader will be required.

Thanks a lot. I'll try to implement one soon.