}}

Unsealing the Secret Store

Enterprise DC/OS Preview Updated: February 15, 2017

About unsealing the Secret Store

The Secret Store can become sealed under the following circumstances.

A sealed Secret Store cannot be accessed from the web interface. Secret values cannot be retrieved using the Secrets API. Services that depend on values provisioned to them via environment variables may fail to deploy.

The procedure for unsealing the Secret Store differs according to the keys used to seal it.

Prerequisites:

Note: In these procedures, we will use two terminal prompt tabs: one to SSH into the master and use GPG; another to execute curl requests and use xxd. The master does not have xxd installed by default at this time. Nor does it have a package manager. If you do not wish to shuttle between terminal prompt tabs, you can run xxd inside a container on the master.

Unsealing a Secret Store sealed with default keys

  1. From a terminal prompt, check the status of the Secret Store via the following command.
    $ curl --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/secrets/v1/seal-status/default
    
  2. The Secret Store service should return the following response.
    {"sealed":true,"threshold":1,"shares":1,"progress":0}
    

    If the value of "sealed" is false, do not complete the rest of this procedure. Your Secret Store is not sealed, so you cannot unseal it.

  3. After confirming that your Secret Store is indeed sealed, open a new terminal prompt tab.

  4. From the new tab, SSH into your master and launch the ZooKeeper command line interface as follows.

    $ /opt/mesosphere/packages/exhibitor--*/usr/zookeeper/bin/zkCli.sh
    
  5. Execute the following ZooKeeper command to gain additional privileges, specifying the user name and password of the ZooKeeper superuser. By default, this is set to super:secret but we recommend changing the default.
    addauth digest super:secret
    
  6. Retrieve the default private GPG key using the following command.
    get /dcos/secrets/keys/bootstrap_user.key
    
  7. Select the first value returned, everything in between the quote marks, and copy it to your clipboard.

  8. Type quit to exit the ZooKeeper command line.

  9. Decode the private GPG key using the following command.

    $ echo <base64-encoded-gpg-key> | base64 -d
    
  10. This will return the decoded private GPG key, which should look as follows.
    -----BEGIN PGP PRIVATE KEY BLOCK-----
    xcZYBFfr8jEBEACoG/RL2hGhwoUYRpWue4nTZYQYna1Hbm0TaPYWjiek/ScXwgIt
    ...
    =Xc0I
    -----END PGP PRIVATE KEY BLOCK-----
    
  11. Select everything in between and including -----BEGIN PGP PRIVATE KEY BLOCK and END PGP PRIVATE KEY BLOCK-----. Copy it to your clipboard and paste it into a new file giving it a name such as gpg-private.key.

  12. Load the decoded GPG key into GPG as follows.

    $ gpg --allow-secret-key-import --import gpg-private.key
    
  13. Delete the file.
    $ rm -rf gpg-private.key
    
  14. Switch back to the original terminal prompt tab.

  15. Use the init endpoint of the Secrets API to retrieve the encrypted unseal key as shown in the curl below.

    $ curl --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/secrets/v1/init/default
    
  16. This command should return a JSON object similar to the following.
    {"initialized":true,"keys":["c1c..."],"pgp_fingerprints":["524c98..."],"root_token":"147de72..."}
    
  17. Copy the value of "keys" to your clipboard. This is your encrypted unseal key in ASCII format.

  18. Transform the encrypted unseal key into binary and save the result into a new file using the following command. Before executing the command, replace c1c04c...d00 with the value of your encrypted unseal key.

    $ echo "c1c04c...d00" | xxd -r -p > binary-unseal.key
    
  19. Use secure copy to transfer the new file to your master, as shown below. Replace <cluster-IP> below with the IP address of your cluster. You can locate this value in the top left of the DC/OS dashboard.
    $ scp binary-unseal.key core@<cluster-IP>:~
    
  20. Return to your secure shell terminal prompt tab.

  21. Confirm that the binary-unseal.key file copied over successfully using the following command.

    $ ls -la
    
  22. Use the following command to decrypt the unseal key with GPG.
    $ gpg -d binary-unseal.key
    
  23. This should return the decrypted unseal key value. Copy this value to your clipboard.

  24. Return to the original terminal prompt tab.

  25. Use the following curl command to unseal the store. Before executing this command, replace c9e...33 with the decrypted unseal key value.

    $ curl -X PUT --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" -d '{"key":"c9e...33"}' $(dcos config show core.dcos_url)/secrets/v1/unseal/default -H 'Content-Type: application/json'
    
  26. The Secret Store service should return the following JSON response, indicating success.
    {"sealed":false,"threshold":1,"shares":1,"progress":0}
    

Unsealing a Secret Store sealed with custom keys

  1. From a terminal prompt, check the status of the Secret Store via the following command.
    $ curl --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/secrets/v1/seal-status/default
    
  2. Check the response to make sure it corresponds to the following.
    {"sealed":true,"threshold":1,"shares":1,"progress":0}
    

    If the value of "sealed" is false, do not complete the rest of this procedure. Your Secret Store is not sealed, so you cannot unseal it.

  3. Use the init endpoint of the Secrets API to retrieve the encrypted unseal key as shown in the curl below.

    $ curl --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/secrets/v1/init/default
    
  4. This command should return a JSON object similar to the following.
    {"initialized":true,"keys":["c1c...0700"],"pgp_fingerprints":["9b25...622b"],"root_token":"3fd...a3d"}
    
  5. Copy the value of "keys" to your clipboard. This is your encrypted unseal key in ASCII format.

  6. Transform the encrypted unseal key into binary and save the result into a new file using the following command. Before executing the command, replace c1c...0700 with the value of your encrypted unseal key.

    $ echo "c1c...0700" | xxd -r -p > binary-unseal.key
    
  7. Use secure copy to transfer the new file to your master, as shown below. Before executing the command, replace <master-IP> with the IP address of the master.
    $ scp binary-unseal.key core@<master-IP>:~
    

    Tip: If you used GPG to generate the custom GPG keypair as described in Reinitializing the Secret Store with a custom GPG keypair and you have multiple masters, use the IP address of the master that you used to generate the keypair.

  8. SSH into your master

  9. Confirm that the binary-unseal.key file copied over successfully using the following command.

    $ ls
    
  10. If your GPG private key is not already loaded into GPG, go ahead and load it now.
    $ gpg --allow-secret-key-import --import gpg.key
    

    Tip: If you recently completed the Reinitializing the Secret Store with your own GPG key procedure, your private key should already be loaded in GPG.

  11. Use the following command to decrypt the unseal key with GPG.

    $ gpg -d binary-unseal.key
    
  12. GPG should return the decrypted unseal key value. Copy this value to your clipboard.

  13. Switch to another terminal prompt tab.

  14. Use the following curl command to unseal the store, replacing bd3e...78c with the decrypted unseal key value.

    $ curl -X PUT --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" -d '{"key":"bd3e...78c"}' $(dcos config show core.dcos_url)/secrets/v1/unseal/default -H 'Content-Type: application/json'
    
  15. It should return the following JSON response, indicating success.
    {"sealed":false,"threshold":1,"shares":1,"progress":0}