Of course I can open it but software like Elcomsoft pdf cracker seems to determine very fast if it's the right key.

Any idea?

In the specs (check the Adobe site) there is a description of how to detect the correct password, with a complete algorithm. Essentially, the password is transformed to a key (using a document specific salt, mostly, via hashing) and then a known plaintext (16 or 32 bytes, depending on the document for recent variants, there are more ways to encrypt) is encrypted with that key, and the result is stored in the encrypted PDF. The check for a password checks that the result is that known/ computed plaintext.

There must be some kind of mechanism with a password protected pdf but we need the (non-'Standard') Security handler protection. For that I haven't found a mechanism for it. You might check if strings are in an allowed threshold but with unicode and chinese characters that doesn't have to work .

Don't most security handlers also have some checkdata like normal password protected (via the standard methods) have? What security handler are you talking about?
I do believe all must have such a check, for the reasons you described.

Otherwise you could decompress a compressed part (after decryption), and then watch for errors.

Of course you can see it if you manually open it but how can you deterministically automate this?

Please provide details of the handler in question. I'm convinced there is an easy check somewhere.

Do you know how to decrypt it (programmatically) when you have the correct password?
If so, you could try to decrypt and then decompress (e.g. using Deflate, using zlib) some object.
If you get an error, you have the wrong password.