In my case, N=B80A90BF53C6C979; D=2492F4F0BA859F9; E=10001

c=cryptograph
m=plaintext

c=m^e%n
m=c^d%n

Assumes the initialization of the bignum N,D in the following bignum form of

D dw 02
db 00,01,00,01

N dw 04
db 0b8h,0ah,090h,0bfh,053h,0c6h,0c9h,079h

M (source buffer/plaintext) = 39 ascii(9)

then the RSA_PRIVATE_ENCRYPT will return the correct values 9DC42A4223FD011D
in the bignum form of DB 04,00,9d,c4,2a,42,23,fd,01,1d,00,00

the problem is in the related reversing process :

Assume:

N dw 04
db 0b8h,0ah,090h,0bfh,053h,0c6h,0c9h,079h

D dw 04
db 02h,049h,02fh,04fh,0bh,0a8h,059h,0f9h

then invoke the process of RSA_PRIVATE_ENCRYPT, it won't return the correct values

e.g. 39 in the bignum form of db 01,00,39.

it return a 4*16bit wrong value, like the following
db 04,00,xx,xx,xx,xx,xx,xx,xx,xx

or m=CADE4E832627B4F6 return c=32F9776EF8F0F9E6
but if u calculate the m with c u will get a wrong result
such as
expmod(0x32F9776EF8F0F9E6,0x10001,0xB80A90BF53C6C979)=12D3BDC3D260EB7D


I don't know what caused this obvious wrong result in the procedure of RSA_PRIVATE_ENCRYPT.

Could I ask u how to fix the problem in the mentioned keygenRSA.asm?
If it is prossable Please give me some guides with simple demo/example. Thanks a lot.

In my case, N=B80A90BF53C6C979; D=2492F4F0BA859F9; E=10001According to http://www.alpertron.com.ar/ECM.HTM

N = 13261571204164536697 = 3533507051 * 3753090347 = p * q, so

phi(N) = (p-1)*(q-1) = 3533507050 x 3753090346 = 13261571196877939300

D = E ^ -1 mod phi(N) = 65537 ^ -1 mod phi(N)
= 164714877920238073
= 0x2492F4F0BA859F9

So you got that right. Take the plaintext you're trying to produce, raise it to the D power mod N, and use that as the ciphertext. Then when the serial checker raises it to the E power, you get your plaintext back.

If that's not working for you, it must be a bug in your code or a typo or something.

thanks