You can enable or disable a directory at any time. If you disable a directory, your configuration details will remain but the application will not recognize the users and groups in that directory.
You have to disable a directory before you can remove it. Removing a directory will remove the details from the database.