Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Linker error when tensorflow is being built with clang-17 inside anaconda environment #68246

Open
chudur-budur opened this issue May 20, 2024 · 2 comments
Open

Linker error when tensorflow is being built with clang-17 inside anaconda environment #68246

chudur-budur opened this issue May 20, 2024 · 2 comments
Assignees
@sushreebarsa
Labels
stale This label marks the issue/pr stale - to be closed automatically if no activity stat:awaiting response Status - Awaiting response from author subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues TF 2.16 type:build/install Build and install issues

Comments

@chudur-budur
Copy link

Issue type

Build/Install

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

tf 2.16

Custom code

No

OS platform and distribution

Ubuntu 22.04.4 LTS

Mobile device

No response

Python version

3.11

Bazel version

6.5.0

GCC/compiler version

clang 17.0.6

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

I need to compile TensorFlow code using clang-17 compiler provided by anaconda, i.e. clangxx_linux-64==17.

It looks the compiler is setup correctly:

(tfdev) user@host:/localdisk/user/tensorflow|(r2.16)> clang -v
clang version 17.0.6 (https://github.com/conda-forge/clangdev-feedstock 71dc3d053bf53a7cb6294eba448c059a3060dbe6)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /localdisk/user/opt/miniconda3/envs/tfdev/bin
Found candidate GCC installation: /localdisk/user/opt/miniconda3/envs/tfdev/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0
Selected GCC installation: /localdisk/user/opt/miniconda3/envs/tfdev/bin/../lib/gcc/x86_64-conda-linux-gnu/13.2.0
Candidate multilib: .;@m64
Selected multilib: .;@m64

However, when I try to compile, I am getting a linking error saying ld.lld: error: cannot open /lib64/libc.so.6: No such file or directory:

(tfdev) user@host:/localdisk/user/tensorflow|(r2.16)> bazel build //tensorflow/tools/pip_package/v2:wheel --repo_env=WHEEL_NAME=tensorflow_cpu --verbose_failures
Starting local Bazel server and connecting to it...
INFO: Reading 'startup' options from /localdisk/user/tensorflow/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=170
INFO: Reading rc options for 'build' from /localdisk/user/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /localdisk/user/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /localdisk/user/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/bin/python3 --action_env PYTHON_LIB_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/lib/python3.11/site-packages --python_path=/localdisk/user/opt/miniconda3/envs/tfdev/bin/python3 --action_env CLANG_COMPILER_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 --repo_env=CC=/localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 --repo_env=BAZEL_COMPILER=/localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 --copt=-Wno-gnu-offsetof-extensions
INFO: Found applicable config definition build:short_logs in file /localdisk/user/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /localdisk/user/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:linux in file /localdisk/user/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /localdisk/user/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
DEBUG: /localdisk/user/tensorflow/tensorflow/tools/toolchains/python/python_repo.bzl:32:14:
TF_PYTHON_VERSION environment variable was not set correctly; using Python 3.11.

To set Python version, run:
export TF_PYTHON_VERSION=3.11
INFO: Analyzed target //tensorflow/tools/pip_package/v2:wheel (686 packages loaded, 44338 targets configured).
INFO: Found 1 target...
ERROR: /localdisk/user/.cache/bazel/_bazel_user/e4eace97acba9979e5f9a9765d32d25e/external/com_google_protobuf/BUILD.bazel:827:10: Linking external/com_google_protobuf/python/google/protobuf/internal/_api_implementation.so failed: (Exit 1): clang-17 failed: error executing command (from target @com_google_protobuf//:python/google/protobuf/internal/_api_implementation.so)
  (cd /localdisk/user/.cache/bazel/_bazel_user/e4eace97acba9979e5f9a9765d32d25e/execroot/org_tensorflow && \
  exec env - \
    CLANG_COMPILER_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 \
    PATH=/home/user/.cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba252857307/bin:/localdisk/user/opt/miniconda3/envs/tfdev/bin:/localdisk/user/opt/miniconda3/condabin:/usr/sbin:/usr/local/sbin:/usr/local/games:/usr/local/bin:/usr/games:/usr/bin:/snap/bin:/sbin:/localdisk/user/opt/miniconda3/bin:/localdisk/user/opt/gcm:/localdisk/user/opt/bazel/bin:/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/bin/python3 \
    PYTHON_LIB_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/lib/python3.11/site-packages \
    TF2_BEHAVIOR=1 \
  /localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 @bazel-out/k8-opt/bin/external/com_google_protobuf/python/google/protobuf/internal/_api_implementation.so-2.params)
# Configuration: 672fd81c885b04f9d9569be32c92fcdcc1cc6a1eb79d855132a6dbfe682bc7b4
# Execution platform: @local_execution_config_platform//:platform
ld.lld: error: cannot open /lib64/libc.so.6: No such file or directory
clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
Target //tensorflow/tools/pip_package/v2:wheel failed to build
INFO: Elapsed time: 22.861s, Critical Path: 0.58s
INFO: 815 processes: 681 internal, 134 local.
FAILED: Build did NOT complete successfully

How do I fix this?

Standalone code to reproduce the issue

# Steps to reproduce:

# Make a conda environment:
(base) user@host:/localdisk/user/tensorflow|(r2.16)> conda create -n tfdev python=3.11
(base) user@host:/localdisk/user/tensorflow|(r2.16)> conda activate tfdev

# Install clangxx_linux-64:
(tfdev) user@host:/localdisk/user/tensorflow|(r2.16)> conda install -c conda-forge clangxx_linux-64==17

# Configure:
(tfdev) user@host:/localdisk/user/tensorflow|(r2.16)> ./configure
You have bazel 6.5.0 installed.
Please specify the location of python. [Default is /localdisk/user/opt/miniconda3/envs/tfdev/bin/python3]:


Found possible Python library paths:
  /localdisk/user/opt/miniconda3/envs/tfdev/lib/python3.11/site-packages
Please input the desired Python library path to use.  Default is [/localdisk/user/opt/miniconda3/envs/tfdev/lib/python3.11/site-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: n
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: n
No CUDA support will be enabled for TensorFlow.

Do you want to use Clang to build TensorFlow? [Y/n]: y
Clang will be used to compile TensorFlow.

Please specify the path to clang executable. [Default is /usr/lib/llvm-17/bin/clang]: /localdisk/user/opt/miniconda3/envs/tfdev/bin/clang++


You have Clang 17.0.6 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]:


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]:
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
        --config=mkl            # Build with MKL support.
        --config=mkl_aarch64    # Build with oneDNN and Compute Library for the Arm Architecture (ACL).
        --config=monolithic     # Config for mostly static monolithic build.
        --config=numa           # Build with NUMA support.
        --config=dynamic_kernels        # (Experimental) Build kernels into separate shared objects.
        --config=v1             # Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
        --config=nogcp          # Disable GCP support.
        --config=nonccl         # Disable NVIDIA NCCL support.
Configuration finished

# Build
(tfdev) user@host:/localdisk/user/tensorflow|(r2.16)> bazel build //tensorflow/tools/pip_package/v2:wheel --repo_env=WHEEL_NAME=tensorflow_cpu --verbose_failures

Relevant log output

Starting local Bazel server and connecting to it...
INFO: Reading 'startup' options from /localdisk/user/tensorflow/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=170
INFO: Reading rc options for 'build' from /localdisk/user/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /localdisk/user/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /localdisk/user/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/bin/python3 --action_env PYTHON_LIB_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/lib/python3.11/site-packages --python_path=/localdisk/user/opt/miniconda3/envs/tfdev/bin/python3 --action_env CLANG_COMPILER_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 --repo_env=CC=/localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 --repo_env=BAZEL_COMPILER=/localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 --copt=-Wno-gnu-offsetof-extensions
INFO: Found applicable config definition build:short_logs in file /localdisk/user/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /localdisk/user/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:linux in file /localdisk/user/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /localdisk/user/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
DEBUG: /localdisk/user/tensorflow/tensorflow/tools/toolchains/python/python_repo.bzl:32:14:
TF_PYTHON_VERSION environment variable was not set correctly; using Python 3.11.

To set Python version, run:
export TF_PYTHON_VERSION=3.11
INFO: Analyzed target //tensorflow/tools/pip_package/v2:wheel (686 packages loaded, 44338 targets configured).
INFO: Found 1 target...
ERROR: /localdisk/user/.cache/bazel/_bazel_user/e4eace97acba9979e5f9a9765d32d25e/external/com_google_protobuf/BUILD.bazel:827:10: Linking external/com_google_protobuf/python/google/protobuf/internal/_api_implementation.so failed: (Exit 1): clang-17 failed: error executing command (from target @com_google_protobuf//:python/google/protobuf/internal/_api_implementation.so)
  (cd /localdisk/user/.cache/bazel/_bazel_user/e4eace97acba9979e5f9a9765d32d25e/execroot/org_tensorflow && \
  exec env - \
    CLANG_COMPILER_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 \
    PATH=/home/user/.cache/bazelisk/downloads/sha256/a40ac69263440761199fcb8da47ad4e3f328cbe79ffbf4ecc14e5ba252857307/bin:/localdisk/user/opt/miniconda3/envs/tfdev/bin:/localdisk/user/opt/miniconda3/condabin:/usr/sbin:/usr/local/sbin:/usr/local/games:/usr/local/bin:/usr/games:/usr/bin:/snap/bin:/sbin:/localdisk/user/opt/miniconda3/bin:/localdisk/user/opt/gcm:/localdisk/user/opt/bazel/bin:/bin \
    PWD=/proc/self/cwd \
    PYTHON_BIN_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/bin/python3 \
    PYTHON_LIB_PATH=/localdisk/user/opt/miniconda3/envs/tfdev/lib/python3.11/site-packages \
    TF2_BEHAVIOR=1 \
  /localdisk/user/opt/miniconda3/envs/tfdev/bin/clang-17 @bazel-out/k8-opt/bin/external/com_google_protobuf/python/google/protobuf/internal/_api_implementation.so-2.params)
# Configuration: 672fd81c885b04f9d9569be32c92fcdcc1cc6a1eb79d855132a6dbfe682bc7b4
# Execution platform: @local_execution_config_platform//:platform
ld.lld: error: cannot open /lib64/libc.so.6: No such file or directory
clang-17: error: linker command failed with exit code 1 (use -v to see invocation)
Target //tensorflow/tools/pip_package/v2:wheel failed to build
INFO: Elapsed time: 22.861s, Critical Path: 0.58s
INFO: 815 processes: 681 internal, 134 local.
FAILED: Build did NOT complete successfully
@google-ml-butler google-ml-butler bot added the type:build/install Build and install issues label May 20, 2024
@sushreebarsa sushreebarsa added subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues TF 2.16 labels May 20, 2024
@chudur-budur chudur-budur mentioned this issue May 20, 2024
Open
@sushreebarsa
Copy link
Contributor

@chudur-budur Could you ensure the path to system libraries is included in the LIBRARY_PATH environment variable for the build process. You might need to modify the Bazel configuration to achieve this. If the issue persists, please consider rebuilding your Anaconda environment to ensure all libraries are properly installed and configured for the chosen compiler (clang-17) and let us know?

Thank you!

@sushreebarsa sushreebarsa added the stat:awaiting response Status - Awaiting response from author label May 21, 2024
@github-actions GitHub Actions
Copy link

This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you.

@github-actions github-actions bot added the stale This label marks the issue/pr stale - to be closed automatically if no activity label May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sushreebarsa sushreebarsa

Labels
stale This label marks the issue/pr stale - to be closed automatically if no activity stat:awaiting response Status - Awaiting response from author subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues TF 2.16 type:build/install Build and install issues
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@chudur-budur @sushreebarsa