In order for them to access your computer using the ssh key, you must add the public key of the person/computer you want to give access to in the authorized_keys file located in ~/.ssh/ (in the home of the user allowing the connection) on your computer.

Remember that the public key helps you to encrypt the communication but not to decrypt it, so sharing your public key is not dangerous (id_rsa.pub), what is very dangerous is to share, lose or have your private key (id_rsa) stolen.

I hope this helps.