Passwords and pass-phrases are an authentication method; they authenticate the user as the only one, at least in theory, who knows the correct phrase. With encryption tools, they serve an additional purpose; they either generate the symmetric key directly through one or more hashes (eg. cryptsetup), or they decrypt an existing stored key (eg. LUKS, GnuPG).
SSH2 public key authentication is purely an authentication method. There is no symmetric key involved. The requester (eg. sshd) generates a challenge, sends it to the client or agent for a signature using the private key, and verifies the signed response using the public key. If the public key successfully decrypts the signature, the private key must have generated it, and thus the user, as the only one with access to the private key, in theory, is authenticated.
I recently worked with someone who was encrypting their home folder on a remote system with pam-encfs, but wanted to use public key authentication for the initial SSH authentication, rather than an interactive password. The first problem with remotely accessing encrypted home folders via SSH is with the authorized_keys file. This file is normally in the user's home directory, and thus inaccessible until the user logs in and decrypts it. There are known ways around that, at least, such as placing the file outside of the user's home directory.
The second problem is generating the decryption key. When using SSH public key authentication, there is no password for pam-encfs to use to generate the symmetric key. Since the authentication task is passed back to the ssh client or agent, the password is never present on the remote system.
Stepping back for a moment, pam-encfs does seem to be the logical place to decrypt and mount the file-system. It has the advantage of PAM's modular design, and knows when the session start and session stop events occur to mount/umount it. Plus, at least with interactive logins, PAM already has access to the user's pass-phrase, and can pass that to encfs without needing a second prompt.
Not so with public key authentication, unfortunately. There is no way to uncover the symmetric key without this pass-phrase or the key material that is generated from it. So the question becomes, while authenticating the user via the SSH agent, how can we access a pass-phrase for encfs? It must be stored somewhere, with agent authentication used to authorize access to it. I've come up with a couple of ideas, neither of which I'm thrilled with.
Hold password in trusted keyring with access authorized by agent
This would require a keyring of some sort on the remote system which maintains the password. If it encrypts the password, it would still need to be able to decrypt it locally upon authorized requests, so has all the problems of a classic DRM system.
Pro: No calls to external systems needed
Con: Passphrase or hash is stored on the system which uses it
Con: Requires a trusted keyring on the system
Retrieve the password from an SSH "key store" using SSH agent
The system holding the private key is the most trusted platform in this scenario. This platform could also host the password, and release it on an authorized SSH connection authenticated using its own private key. This key storing/serving SSH server could run on a non-standard port and be limited to just the one host and key and just return the required passphrase instead of opening a shell.
Encfs has a parameter to specify a command which returns the plaintext password on stdout. With a little patching and scripting, pam-encfs could be modified to make a call back to the originating server for the password.
Pro: Can be compatible with interactive logins by storing same password
Pro: Only requires a pam-encfs patch and a few scripts
Con: Password exists in plaintext on the key store
Con: Additional complexity, with no fallback on failure
Modify encfs to take raw key option, and retrieve from "key store"
Since the only thing the remote system needs the pass-phrase for is to generate the symmetric key, we could avoid exposing the pass-phrase entirely by storing that key instead of the pass-phrase. Then, we retrieve that key and apply it directly in order to mount the encrypted file-system.
Pro: Can be compatible with interactive logins by using same password
Pro: Password is not stored in any format, only its hash
Con: Likely also requires encfs patch to take raw symmetric key as an option
Con: Even more complexity, still no fallback on failure
Either of the above two, but with encrypted store
We could require two things for access to the plaintext pass-phrase/key: the encrypted pass-phrase/key from the keystore, and a symmetric key stored on the system which uses it. This would prevent a compromise of the originating machine from directly leading to the plaintext, at the cost of more complexity.
Pro: Pass-phrase/key not stored in plaintext
Con: Even more complexity
Modify PAM (conf?) to require password after public key
This may turn out to be the best solution. If the existing modules can be configured such that sshd authenticates via public key, but PAM still requires a password, then pam-encfs has what it needs to create the symmetric key, and all is well. If this is not a configuration issue, it may require a patch to one of the PAM modules, sshd, or login. It requires two passwords: one to unlock the private key (or login), and one to decrypt the file system.
Pro: 100% compatible with interactive logins
Pro: Password is not stored in any form, no less secure than interactive login
Pro: At best, a config option. At worst, one of the PAM modules may need a patch
Pro: Increased security, as a key AND a pass-phrase is required for remote access
Con: defeats SSO capabilities of SSH agent authentication
The user's solution
The solution that was implemented is to mount the file-system outside of PAM using the shell's login/logout scripts. This has a few potential problems, such as concurrency issues if the user connects more than once, and the inability to execute commands directly from the ssh client. However, it gives a uniform experience whether connecting remotely or at the console, and in fact allows different passwords for login and decryption.