From Dotenv to Dotenvx: Next Generation Config Management

I think it's good advice to not pass secrets through environment variables. Env vars leak a lot. Think php_info, Sentry, java vm dumps, etc. Also, env vars leak into sub-processes if you don't pay extra attention. Instead, read secrets from a vault or from a file-system from _inside_ your process. See also [1] (or [2] which discusses [1]). Dotnet does this pretty good with user secrets [3].

[1] https://blog.diogomonica.com/2017/03/27/why-you-shouldnt-use... [2] https://security.stackexchange.com/questions/197784/is-it-un... [3] https://learn.microsoft.com/en-us/aspnet/core/security/app-s...

I've started using Mise for some stuff at work. Haven't digged in a lot yet, but looks really promising.

https://mise.jdx.dev/

It handles task running (wipe local test db, run linting scripts, etc), environment variables and 'virtual environments', as well as replacing stuff like asdf, nvm, pyenv and rbenv.

Still somewhat early days, tasks are experimental. But looks very promising and the stuff I've tried to far (tasks) works really well.

The only reason I use .env is because it’s dead simple and very obvious as to how it works to anyone.

If now someone has to read docs to figure out how to configure the app, I’d rather have them read docs for some other safer and more powerful configuration scheme.

With leaking secrets being such a big concern, it seems wise to require that secrets be encrypted to use dotenvx. That is, it will only work with encrypted secrets. As others have commented, this doesn't eliminate the risk entirely, but I think having a tool that doesn't support unencrypted secrets at all, although a bit less convenient, is a win.

Seems pretty similar to sops[0], but without the encrypted-by-default feature that makes sops feel significantly safer for secret management.

Sops also integrates easily with AWS and other existing key management solutions, so that you can use your existing IAM controls on keys.

I mentioned in another comment, but I've been using it over five years at two jobs and have found it to be great.

[0]: https://github.com/getsops/sops

I don't think encryption is a good idea, and the reason is forming bad habits. Now developers have a very strong and non-ambiguous habit: never put .env files in version control (except may be for .example.env). However, with this, you'll get accustomed to commit .env in _some_ projects, so you'll easily slip and commit it in another project where the vars are not encrypted.

Encrypting secrets and committing them seems very convenient but I'm paranoid about these sorts of things. Can anyone tell me why this would be a bad idea?

One reason I can think of is that normally with secrets I actually don't keep any copies of them. I just set them in whatever secret manager my cloud environment uses and never touch them again unless I need to rotate them. Meaning there is no way to accidentally expose them other than by the secret vault being hacked or my environment being hacked.

With this approach if someone gets access to the encryption key all secrets are exposed.

Can a kind soul point me some documentation on how to put .env in a vault correctly? Possibly open-source solutions?

If the vault is password protected, aren't you just adding one more indirection and nothing more? How is that helpful, since now I have to write the vault password in clear-text somewhere such that my application can read the env file from the vault?

I want an option to manage all env in a single file using a TOML like format like this.

  [local]
  API_KEY=local-key
  API_SECRET=local-secret
  DB=postgresql://username:password@localhost:5432/database_name
  [production]
  API_KEY=prod-key
  API_SECRET=prod-secret
  DB=postgresql://username:password@prod-db:5432/database_name
  [staging]
  API_KEY=stg-key
  API_SECRET=stg-secret
  DB=$(production.DB)

It makes it easier to update all env at once, compare, and share. It's not much help, but it helps me avoid a few annoyances.

On an unrelated note, I always find it a real headache to keep the naming convention of the environments throughout the project. It always ends up like a mixed bag:

  * Juggling production/prod, staging/stg, and develop/dev,
  * Inconsistent placement of env, e.g. prod-myproject or myproject-stg,
  * Skipping env name sometimes, e.g. myproject-bucket for dev S3 bucket but prod-myproject-bucket for prod (though it's okay to emit env name for user facing places like URL),
  * Inconsistent resource sharing between envs, e.g. same S3 bucket for local and dev but different DB, or same Kubernetes cluster with different labels for dev/stg but different cluster without a label for prod.

These inconsistencies often result from quick decisions without much thought or out of necessities, and everyone is too scared to fix them anyway. But it bothers me a lot and sometimes causes serious bugs in production.

Fix: format

Env vars over-share and files depend on local permissions. We should have a capabilities -like way to send secrets between processes. e.g., decrypt and expose on a Unix socket with a sha filename that can only be read from once, and then gets torn down. Share the file name, target can read it and immediately the secret is now at-rest encrypted. Encryption based on config containing a whitelist of ssh public keys and what they can access, sort of like age.

On my phone so can’t double test, but can’t you get this by adding “export” in front of every line in your env file and then source before running command?

I suppose if you don’t want it to stay after execution i believe you can:

    > $(source .env; my command)

I’m sure there is a fairly straightforward way to encrypt and decrypt a local file

Importing a set of library and dependencies to handle reading a plain text file poses more risks than just leaving the file unencrypted.

You don't need to encrypt your keys, with what keys are you going to do so? Will you encrypt those?

if someone is in your server you are pwned anyways.

It's ok if you identify yourself as a cybersecurity dude and hold a cybersecurity role and you need to justify your livelihood.

But do it in a way where you don't bother people. It's ok if you bother devs, but then you go on and bother users with 4FA, 5 rule passwords, systems that can't answer subpoenas because you have encrypted your sense of self.

When you are improving security at the expense of every other variable, that's annoying, but when you keep "improving security" at the expense even of security, is the point where people will start ignoring and hiding shit from you

We’ve been pushing for committing encrypted secrets for many years now, and have written an open source spec and implementation in multiple languages: https://github.com/neosmart/securestore-rs

Thanks, but I would rather go with the imperfect setup that I understand than an allegedly perfect setup with dozens of third party dependencies that I don't understand.

Doubly the case now that env is natively supported by node now.

dotenv has zero npm dependencies. dotenvx has 21, including a few I have never heard of. Is this really more secure?

This is similar to how Rails handles secrets - https://edgeguides.rubyonrails.org/security.html#environment...

In Rails, the entire file is encrypted unlike here where only the secrets are

Something I have done for secrets is use a syntax in environment variables to tell the process to go a key vault for the secret.

So we can have

FOOPW=pw1

when testing locally, but

FOOPW="{vault1:secret1}"

in production. Env vars are processed simply by running a regex with callback that fetches secrets from vaults. This is quite flexible and has the advantage of being able to inject the secrets in the same place as other configuration, without actually having the secrets in environment variables or git etc (even encrypted)

There's a typo in the article, if the author reads this : > But I think we have a solution to all three today - with dotenvx. In reverse problem order

The URL in `dotenvx` points to https://gitub.com/dotenvx/dotenvx (gitub without the h)

I’ve been using dotenv-cli for a long time now. I appreciate the encryption, but I will keep loading secrets from a vault instead.

I don't really understand why this is a new project. Seems it would have been pretty simple to add these in a backwards compatible way.

It would only break in cases where people's values specifically started with "encrypted:"

I always used foreman [0] which I found to be superior to dotenv in every way. Even superior to this new dotenvx

[0] https://github.com/ddollar/foreman

Your "a single gitignore append" command overwrites existing .gitignore files. It should be `>>`

A lot of people here are saying that a vault is preferable to this, especially for secrets.

Is there a good primer on using vaults? I know how to query and insert into Azure Key Vaults, but architecting around it is unclear to me.

Things that come up for me:

- As (azure) key vaults don't support per secret access rights, where do I store secrets between deployments?

- Should I store connection strings to cloud resources, or just ask the resource for the connection string at deployment time (for Azure, a cloud function pretty much needs a connection string for most basic things. They say they are moving away from this but ...)

- A security warning is send if a key is accessed more then x/times per hour. Does that mean I should pull in the key from vault at deployment? Cache it after first call during runtime?

- Most of our 3rd party vendors gives us 1 and only 1 key. How do I manage that key between development, production and several developers? Right now we mostly forward the e-mail from the vendor with the key ...

I'm not comfortable that this tool is able to make HTTP requests, IMO it ought to be completely offline and not perform network operations. I did a search for requests and found some results pushing to and pulling from a 'hub'. But I couldn't figure out what the hub refers to.

I think for some reason people don't like such simple tools. I think it seems more reasonable to deploy a vault service running on kubernetes with jenkins. In some comments, they didn't even understand what the tool was for. Dear dude, you can travel between dimensions with this tool.

Secrets in env vars in production is not too secure either, ideally you’ll move to your app pulling secrets in-process from your infrastructure at boot-up or upon use. This also gives a nice advantage of not needing to rebuild the app or container or whatever to rotate a secret.

Would be nice to be able to configure which .env file to read as an environment variable. Why? Imagine a package.json with this line:

`start: dotenvx run -f .env.local -f .env -- node index.js`

Instead of the -f flag, which now cannot be overriden, one could invoke it with

`DOTENV=.env.staging npm run start`

This seems to encourage committing encrypted secrets, which is a bad idea. Configuration and code should be in separate repositories. Secrets should be protected elsewhere.

The correct fix for “it’s too easy to accidentally commit .env files with secrets” is to not function (panic/throw) if there isn’t a suitable .gitignore/.dockerignore, not a specialized cryptosystem for .env files. This just creates a different problem.

I simply use an envdir outside of the project and update all my run scripts to use “envdir $CONFIG_PATH <whatever>”. Simpler and safer.

I really wish 1Password would ship an environment manager for their op CLI.

I don't think this is the best approach. I am building https://phase.dev which lets you import secrets (.env), encrypt secrets (end-to-end encrypted with keys you control), sync them to other services/platforms (think AWS, GitHub, Kubernetes), and inject them into applications at runtime (e.g., phase run node index.js).

Source: https://github.com/phasehq/console

Since node v20.06, has built in support for --env-file=.env on load... as for local(ish) encryption and pushing them into source control, I don't like this at all. I'm fine using a vault or secret distribution from either the environment host (k8s) or ci/cd deployment.

I do like to keep a .env.example that you can rename to .env and adjust as desired. I tend to have defaults for running a compose stack locally that close to "just works" as possible.

I doubt I'd ever want to use this in practice.

My beef with it is that it's written in Javascript whereas what it does is so simple that they should just code it up in Rust or Go and be done with it, particularly when it could be packed up as a tiny binary that doesn't require you to install Node, Python, the JVM or any other runtime or library.

Wait, people use the .env file for secrets?! I never even looked into this but just assumed this was a convenience for local dev to hold a local non-secret app config? In production you'd load the env from a secrets store like what kubernetes has or aws has for ec2 instances. You shouldn't ever have those secrets on your local machine... Because they are secret.

We implemented the exact same method for config encryption a year ago or so, using pub/private key auth and the same `encrypted:` prefixes for encrypted config values.

This is a great tradeoff: easy way to share configuration, easy way to edit non-encrypted config values, reasonable security for the private values.

Doesn't solve key rotation of course, but for small teams this is a great solution.

I've recently taken a simpler approach to handling secrets in .env files. Since I use autoenv and conda venvs for everything, I persist secrets outside all projects in appropriately-named files, so including them in a .env becomes similar to `source $HOME/.secrets/work__aws_access`. Also makes them easier to manage across projects.

So with this, I have to prefix every executable with dotenvx in order to utilize the env variables?

Can’t I somehow do this in the script itself so “ruby index.rb” is enough? I know I’m only saving a couple of characters in the command line but I’m asking out of curiosity.

I always thought highly of the approach used by https://www.npmjs.com/package/@strong-config/node

Does dotenvx support secrets managers?

Can someone help enlightening me. In terms of preventing leaks, since we still need to ensure that the .env.keys doesn't leak, wouldn't effort still be the same?

The ability to use arbitrary filename for.env is quite nice though!

How would this work with application-specific Intellij Run/Debug configurations? Would this require switching to a "Shell Script" configuration, thereby losing debugging capability etc?

Based on my skim, this doesn’t really solve anything? The private key to decrypt is still stored on machine in a similar place? All this is doing is obfuscating?

So.. it just swapped ignoring `.env` for `.env.keys`?

> An attacker needs the DOTENV_PRIVATE_KEY

And the attackers will be after this file not the .env anymore.

It looks great nonetheless, especially the cross-language feature.

Personally, I like sops for encrypting my secrets.

Wait, it's also doing command interpolation? I missed that. Where is the code for that? I thought it uses dotenv-expand for interpolation, which can't do commands. dotenv-expand is already a bit meh, because of how it resolves the variables after parsing the dotenv file. Meaning you can't have a verbatim single quoted string and also you can get a stack overflow doing this:

    A=$B
    B=$A

Anyway, I hope they don't do command interpolation on top of that (like Ruby dotenv does), because then you can inject code via environment variables (like in the Ruby version).

I recently looked into various dotenv implementations just for fun. They're all different. No unified syntax at all. A lot don't do proper parsing either, but just use some regular expressions (like this one), which means they just skip over what doesn't matches. I started to document all the quirks I could find and wrote my own dotenv dialect just for fun. Nobody use it! Anyway, here it is: https://github.com/panzi/punktum

Direct link to the quirks of the JavaScript dotenv implementation: https://github.com/panzi/punktum?tab=readme-ov-file#javascri...

I've also tried to write a parser compatible to JavaScript dotenv (no x) in C++: https://github.com/panzi/cpp-dotenv

The whole idea of using environment variables for configuration information is good, but ultimately flawed, and we are way past the point where this should continue to be the status quo.

Environment variables are great for configuration because:

  - you can inherit them from a previous application or application(s)
  - you can override them in each environment you run your app in
  - you can pass them on to other applications
  - they are globals that can be loaded by libraries
  - they're not hardcoded in the code, so easier to change things without rebuilding, easier to reuse in different ways/environments/configurations
  - the OS has primitives for them
  - they're simple

Environment variables are bad for configuration:

  - because (by default) when set in application, they are passed on to all future applications/forks/execs
  - they are often dumped as part of troubleshooting and aren't considered confidential
  - they can often be viewed by external processes/users
  - there are restrictions on key names and values and size depending on the platform
  - typical "dotenv" solution doesn't necessarily handle things like multi-line strings, has no formal specification
  - no types, schemas

What we actually need that environment variables are being used for:

  - configuration information passed at execution time that can change per environment
  - loading or passing secret values
  - development environments
  - production environments

So what would be a good alternative?

  - an application library ("libconfig") that can load configuration of various types from various sources in various ways
  - support for configuration types: key-value, file/blob, integer/float
  - support for confidentiality (require specific function to unseal secret values; in programming languages the intent would be you can't just print a stringified version of the variable without an unseal function)
  - support for schema (application defines schema, throws exception if value does not match)
  - support allowing a configuration to be overloaded by different sources/hierarchies
  - support passing a configuration on to other applications
  - support tracing, verbose logging
  - truly cross-platform and cross-language with one specification, behavior for all

How would it work?

  - devs can create a .env file if they want
  - devs load 'libconfig' into app, use it to load their configuration values during development. library can have default sources, and even set env vars or an object internally, so no code needs to be written to use it
  - in production, same code causes libconfig to look at cloud-native and other sources for configuration
  - when debugging, secret confidentiality is maintained, tracing communicates sources of configuration, what was loaded, from where, etc

GOAL: To be able to commit all your envs to git.

This is the only goal and this tool archives it. In the simplest way. While keeping you as secure as you were before, manually setting envs on heroku, railway, aws, jenkins etc.

GitOps FTW

Huge fan of dotenv, excited to try this out!

It doesn’t have Typescript types!

Written in JS; I think I’ll pass.

dotenvx encryption goes a long way towards solving THE BIGGEST problem with dotenv; using multiple tools to ensure that secrets are protected.

I wonder if dotenvx ensures that .env is in .gitignore and yells loudly if it is not.

I encrypt my dotenvs with gpg, but that's hella esoteric and everyone shouldn't be forced to do that.

Intersting

Yet another problem already solved 25 years ago by the mighty Sun in the J2EE spec.

I _detest_ this kind of encryption. It's literally worse than useless. It makes life much harder during debugging, and it eventually leads to developers just storing the decryption keys locally.

For this kind of encryption to work, you need to supply the decryption key from some outside system (e.g. via env vars, AWS SSM, etc.). And if it can supply the key, then why not just use it for other important secrets directly?

I don't get it. Dotenv is only good for local dev. Otherwise you should put your secrets in environment variables (the "env" in ".env"). That people put .env files in prod is a mistake itself, and the proposed fixes here seem to not really do much about that.

Seems odd to announce new features without calling out to the fact that libraries with the same name in other languages have had those same features for years. (for example, the dotenv gem in Ruby)

"Config Management" means something else to systems folks. Suggest adding "for node projects" or something akin to the title to clarify.