Unsealing the Secret Store

ENTERPRISE

About unsealing the Secret Store

The Secret Store can become sealed under the following circumstances.

A sealed Secret Store cannot be accessed from the GUI. Secret values cannot be retrieved using the Secrets API. Services that depend on values provisioned to them via environment variables may fail to deploy.

Prerequisites:

  • DC/OS CLI installed
  • Logged into the DC/OS CLI as a superuser via dcos auth login
  • If your security mode is permissive or strict, you must get the root cert before issuing the curl commands in this section. If your security mode is disabled, you must delete --cacert dcos-ca.crt from the commands before issuing them.

Note: In these procedures, we will use two terminal prompt tabs: one to SSH into the master and use GPG; another to execute curl requests and use xxd. The master does not have xxd installed by default at this time. Nor does it have a package manager. If you do not wish to shuttle between terminal prompt tabs, you can run xxd inside a container on the master.

Unsealing a Secret Store sealed with default keys

  1. From a terminal prompt, check the status of the Secret Store via the following command.

    curl --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/secrets/v1/seal-status/default
    
  2. The Secret Store service should return the following response.

    {"sealed":true,"threshold":1,"shares":1,"progress":0}
    

    If the value of "sealed" is false, do not complete the rest of this procedure. Your Secret Store is not sealed, so you cannot unseal it.

  3. After confirming that your Secret Store is indeed sealed, open a new terminal prompt tab.

  4. From the new tab, SSH into your master and launch the ZooKeeper command line interface as follows.

    /opt/mesosphere/packages/exhibitor--*/usr/zookeeper/bin/zkCli.sh
    
  5. Execute the following ZooKeeper command to gain additional privileges, specifying the user name and password of the ZooKeeper superuser. By default, this is set to super:secret but we recommend changing the default.

    addauth digest super:secret
    
  6. Retrieve the default private GPG key using the following command.

    get /dcos/secrets/keys/bootstrap_user.key
    
  7. Select the first value returned, everything in between the quote marks, and copy it to your clipboard.

  8. Type quit to exit the ZooKeeper command line.

  9. Decode the private GPG key using the following command.

    echo <base64-encoded-gpg-key> | base64 -d
    
  10. This will return the decoded private GPG key, which should look as follows.

    -----BEGIN PGP PRIVATE KEY BLOCK-----
    xcZYBFfr8jEBEACoG/RL2hGhwoUYRpWue4nTZYQYna1Hbm0TaPYWjiek/ScXwgIt
    ...
    =Xc0I
    -----END PGP PRIVATE KEY BLOCK-----
    
  11. Select everything in between and including -----BEGIN PGP PRIVATE KEY BLOCK and END PGP PRIVATE KEY BLOCK-----. Copy it to your clipboard and paste it into a new file giving it a name such as gpg-private.key.

  12. Load the decoded GPG key into GPG as follows.

    gpg --allow-secret-key-import --import gpg-private.key
    
  13. Delete the file.

    rm -rf gpg-private.key
    
  14. Switch back to the original terminal prompt tab.

  15. Use the init endpoint of the Secrets API to retrieve the encrypted unseal key as shown in the curl below.

    curl --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" $(dcos config show core.dcos_url)/secrets/v1/init/default
    
  16. This command should return a JSON object similar to the following.

    {"initialized":true,"keys":["c1c..."],"pgp_fingerprints":["524c98..."],"root_token":"147de72..."}
    
  17. Copy the value of "keys" to your clipboard. This is your encrypted unseal key in ASCII format.

  18. Transform the encrypted unseal key into binary and save the result into a new file using the following command. Before executing the command, replace c1c04c...d00 with the value of your encrypted unseal key.

    echo "c1c04c...d00" | xxd -r -p > binary-unseal.key
    
  19. Use secure copy to transfer the new file to your master, as shown below. Replace <cluster-IP> below with the IP address of your cluster. You can locate this value in the top left of the DC/OS dashboard.

    scp binary-unseal.key core@<cluster-IP>:~
    
  20. Return to your secure shell terminal prompt tab.

  21. Confirm that the binary-unseal.key file copied over successfully using the following command.

    ls -la
    
  22. Use the following command to decrypt the unseal key with GPG.

    gpg -d binary-unseal.key
    
  23. This should return the decrypted unseal key value. Copy this value to your clipboard.

  24. Return to the original terminal prompt tab.

  25. Use the following curl command to unseal the store. Before executing this command, replace c9e...33 with the decrypted unseal key value.

    curl -X PUT --cacert dcos-ca.crt -H "Authorization: token=$(dcos config show core.dcos_acs_token)" -d '{"key":"c9e...33"}' $(dcos config show core.dcos_url)/secrets/v1/unseal/default -H 'Content-Type: application/json'
    
  26. The Secret Store service should return the following JSON response, indicating success.

    {"sealed":false,"threshold":1,"shares":1,"progress":0}