bash - How can I exclude all "permission denied" messages from "find"?

I need to hide all permission denied messages from:

find . > files_and_folders

I am experimenting when such message arises. I need to gather all folders and files, to which it does not arise.

Is it possible to direct the permission levels to the files_and_folders file?

How can I hide the errors at the same time?

Answer

^{Note:
* This answer probably goes deeper than the use case warrants, and find 2>/dev/null may be good enough in many situations. It may still be of interest for a cross-platform perspective and for its discussion of some advanced shell techniques in the interest of finding a solution that is as robust as possible, even though the cases guarded against may be largely hypothetical.
* If your system is configured to show localized error messages, prefix the find calls below with LC_ALL=C (LC_ALL=C find ...) to ensure that English messages are reported, so that grep -v 'Permission denied' works as intended. Invariably, however, any error messages that do get displayed will then be in English as well.}

If your shell is bash or zsh, there's a solution that is robust while being reasonably simple, using only POSIX-compliant find features; while bash itself is not part of POSIX, most modern Unix platforms come with it, making this solution widely portable:

find . > files_and_folders 2> >(grep -v 'Permission denied' >&2)

Note: There's a small chance that some of grep's output may arrive after find completes, because the overall command doesn't wait for the command inside >(...) to finish. In bash, you can prevent this by appending | cat to the command.

>(...) is a (rarely used) output process substitution that allows redirecting output (in this case, stderr output (2>) to the stdin of the command inside >(...).
In addition to bash and zsh, ksh supports them as well in principle, but trying to combine them with redirection from stderr, as is done here (2> >(...)), appears to be silently ignored (in ksh 93u+).
- grep -v 'Permission denied' filters out (-v) all lines (from the find command's stderr stream) that contain the phrase Permission denied and outputs the remaining lines to stderr (>&2).

This approach is:

robust: grep is only applied to error messages (and not to a combination of file paths and error messages, potentially leading to false positives), and error messages other than permission-denied ones are passed through, to stderr.

side-effect free: find's exit code is preserved: the inability to access at least one of the filesystem items encountered results in exit code 1 (although that won't tell you whether errors other than permission-denied ones occurred (too)).

POSIX-compliant solutions:

Fully POSIX-compliant solutions either have limitations or require additional work.

If find's output is to be captured in a file anyway (or suppressed altogether), then the pipeline-based solution from Jonathan Leffler's answer is simple, robust, and POSIX-compliant:

find . 2>&1 >files_and_folders | grep -v 'Permission denied' >&2

Note that the order of the redirections matters: 2>&1 must come first.

Capturing stdout output in a file up front allows 2>&1 to send only error messages through the pipeline, which grep can then unambiguously operate on.

The only downside is that the overall exit code will be the grep command's, not find's, which in this case means: if there are no errors at all or only permission-denied errors, the exit code will be 1 (signaling failure), otherwise (errors other than permission-denied ones) 0 - which is the opposite of the intent.
That said, find's exit code is rarely used anyway, as it often conveys little information beyond fundamental failure such as passing a non-existent path.
However, the specific case of even only some of the input paths being inaccessible due to lack of permissions is reflected in find's exit code (in both GNU and BSD find): if a permissions-denied error occurs for any of the files processed, the exit code is set to 1.

The following variation addresses that:

find . 2>&1 >files_and_folders | { grep -v 'Permission denied' >&2; [ $? -eq 1 ]; }

Now, the exit code indicates whether any errors other than Permission denied occurred: 1 if so, 0 otherwise.
In other words: the exit code now reflects the true intent of the command: success (0) is reported, if no errors at all or only permission-denied errors occurred.
This is arguably even better than just passing find's exit code through, as in the solution at the top.

gniourf_gniourf in the comments proposes a (still POSIX-compliant) generalization of this solution using sophisticated redirections, which works even with the default behavior of printing the file paths to stdout:

{ find . 3>&2 2>&1 1>&3 | grep -v 'Permission denied' >&3; } 3>&2 2>&1

In short: Custom file descriptor 3 is used to temporarily swap stdout (1) and stderr (2), so that error messages alone can be piped to grep via stdout.

Without these redirections, both data (file paths) and error messages would be piped to grep via stdout, and grep would then not be able to distinguish between error message Permission denied and a (hypothetical) file whose name happens to contain the phrase Permission denied.

As in the first solution, however, the the exit code reported will be grep's, not find's, but the same fix as above can be applied.

Notes on the existing answers:

There are several points to note about Michael Brux's answer, find . ! -readable -prune -o -print:
- It requires GNU find; notably, it won't work on macOS. Of course, if you only ever need the command to work with GNU find, this won't be a problem for you.
- Some Permission denied errors may still surface: find ! -readable -prune reports such errors for the child items of directories for which the current user does have r permission, but lacks x (executable) permission. The reason is that because the directory itself is readable, -prune is not executed, and the attempt to descend into that directory then triggers the error messages. That said, the typical case is for the r permission to be missing.
- Note: The following point is a matter of philosophy and/or specific use case, and you may decide it is not relevant to you and that the command fits your needs well, especially if simply printing the paths is all you do:
  - If you conceptualize the filtering of the permission-denied error messages a separate task that you want to be able to apply to any find command, then the opposite approach of proactively preventing permission-denied errors requires introducing "noise" into the find command, which also introduces complexity and logical pitfalls.
  - For instance, the most up-voted comment on Michael's answer (as of this writing) attempts to show how to extend the command by including a -name filter, as follows:
    find . ! -readable -prune -o -name '*.txt'
    This, however, does not work as intended, because the trailing -print action is required (an explanation can be found in this answer). Such subtleties can introduce bugs.

The first solution in Jonathan Leffler's answer, find . 2>/dev/null > files_and_folders, as he himself states, blindly silences all error messages (and the workaround is cumbersome and not fully robust, as he also explains). Pragmatically speaking, however, it is the simplest solution, as you may be content to assume that any and all errors would be permission-related.

mist's answer, sudo find . > files_and_folders, is concise and pragmatic, but ill-advised for anything other than merely printing filenames, for security reasons: because you're running as the root user, "you risk having your whole system being messed up by a bug in find or a malicious version, or an incorrect invocation which writes something unexpectedly, which could not happen if you ran this with normal privileges" (from a comment on mist's answer by tripleee).

The 2nd solution in viraptor's answer, find . 2>&1 | grep -v 'Permission denied' > some_file runs the risk of false positives (due to sending a mix of stdout and stderr through the pipeline), and, potentially, instead of reporting non-permission-denied errors via stderr, captures them alongside the output paths in the output file.

Blog

Thursday 14 February 2019