I'm trying to set a custom IV for AES encryption. When I try either a 16 or 8 byte IV string it fails with param4 error. But when I let it generate one it creates a 16 byte IV. Are there any special parameters I need to pay attention to when creating an IV

I don't know cryptlib myself so I can't really help you on the question.
But what I do know is that you left out a few interesting details such as:

Which encryption-scheme are you using? Since you need IV's it can't be ECB. My best guess is you use CBC or CTR.

Why did you try 8 byte IV's? It sounds to me you are not really knowing what you are doing?
If you use AES with 128 bit keys and 128 bit blocklength (minimum) you will need a 128 bit IV also. If you choose less then 128 bit IV (you mentioned 64 bit IV's) you won't make use of the full possible keyspace.

Why not include some specific code which demonstrates your problem?
If you're not experienced in programming then maybe it's just a trivial bug. How should anyone be able to guess what param4 error means UNLESS they actually tried to do the exact same thing you're doing?

// CyberHeg

Ok, if you don't know cryptlib, then why even respond? Insulting people that ask for help is not 'leet'. I posted here because there are people on this board that do know the library. It's not incredibly well documented, so I asked.

I actually forgot that AES used the 128 bit blocks, so I did make the mistake of trying a 64 bit IV. When it generates its own, it uses 128, so I tried a 16, which still didn'y work. Param4 is the parameter that states IV length. By default cryptlib uses CBC.

Relax Guys.

Hey, where are the admins?

Having Phun?

I am relaxed, thanks

I did not mean to insult anyone. However it's always easier to help someone when you got something to work with.
Even though I don't know cryptlib itself I have enough experience with other libraries and programming in general so eventually I could help if I had some code to look at.

I never found out where it gives you a param4 error. Is it the compiler?, the runtime? or maybe some other thing?
Posting some code which demonstrates the problem makes it easier.
But If you rather want this hide and seek thing, then it's your call.

// CyberHeg

Sorry, I missed this thread somehow!

First rule of cryptlib is "Use the mailing list". That's what it's there for, and they know a lot more about it than I do.

Param4 is the 4th (1-based indexing) parameter in whatever call you're using. I assume you're creating an encryption context and then setting the iv attribute, right? Under "Working with initialization vectors" in the manual:

cryptSetAttributeString( cryptContext, CRYPT_CTXINFO_IV, iv, ivSize );

ivSize is the 4th parameter, the size of the IV in bytes. So ivSize has to be 16 for AES.

More than that I can't say without you posting code.