0: Lmod has detected the following error: These module(s) or extension(s) exist
0: but cannot be loaded as requested: "CrayEnv"
0:    Try: "module spider CrayEnv" to see how to load the module(s).
0: 
0: 
0: 
3: Lmod has detected the following error: These module(s) or extension(s) exist
3: but cannot be loaded as requested: "CrayEnv"
3:    Try: "module spider CrayEnv" to see how to load the module(s).
3: 
3: 
3: 
0: Lmod has detected the following error: The following module(s) are unknown:
0: "suse-repo-deps/sam-default"
0: 
0: Please check the spelling or version number. Also try "module spider ..."
0: It is also possible your cache file is out-of-date; it may help to try:
0:   $ module --ignore-cache load "suse-repo-deps/sam-default"
0: 
0: Also make sure that all modulefiles written in TCL start with the string
0: #%Module
0: 
0: 
0: 
6: Lmod has detected the following error: These module(s) or extension(s) exist
6: but cannot be loaded as requested: "CrayEnv"
6:    Try: "module spider CrayEnv" to see how to load the module(s).
6: 
6: 
6: 
0: Lmod has detected the following error: The following module(s) are unknown:
0: "rocm/sam-5.2.3"
0: 
0: Please check the spelling or version number. Also try "module spider ..."
0: It is also possible your cache file is out-of-date; it may help to try:
0:   $ module --ignore-cache load "rocm/sam-5.2.3"
0: 
0: Also make sure that all modulefiles written in TCL start with the string
0: #%Module
0: 
0: 
0: 
7: Lmod has detected the following error: The following module(s) are unknown:
7: "suse-repo-deps/sam-default"
7: 
7: Please check the spelling or version number. Also try "module spider ..."
7: It is also possible your cache file is out-of-date; it may help to try:
7:   $ module --ignore-cache load "suse-repo-deps/sam-default"
7: 
7: Also make sure that all modulefiles written in TCL start with the string
7: #%Module
7: 
7: 
7: 
0: Lmod has detected the following error: The following module(s) are unknown:
0: "rccl/sam-develop"
0: 
0: Please check the spelling or version number. Also try "module spider ..."
0: It is also possible your cache file is out-of-date; it may help to try:
0:   $ module --ignore-cache load "rccl/sam-develop"
0: 
0: Also make sure that all modulefiles written in TCL start with the string
0: #%Module
0: 
0: 
0: 
2: Lmod has detected the following error: The following module(s) are unknown:
2: "suse-repo-deps/sam-default"
2: 
2: Please check the spelling or version number. Also try "module spider ..."
2: It is also possible your cache file is out-of-date; it may help to try:
2:   $ module --ignore-cache load "suse-repo-deps/sam-default"
2: 
2: Also make sure that all modulefiles written in TCL start with the string
2: #%Module
2: 
2: 
2: 
1: Lmod has detected the following error: The following module(s) are unknown:
1: "suse-repo-deps/sam-default"
1: 
1: Please check the spelling or version number. Also try "module spider ..."
1: It is also possible your cache file is out-of-date; it may help to try:
1:   $ module --ignore-cache load "suse-repo-deps/sam-default"
1: 
1: Also make sure that all modulefiles written in TCL start with the string
1: #%Module
1: 
1: 
1: 
3: Lmod has detected the following error: The following module(s) are unknown:
3: "suse-repo-deps/sam-default"
3: 
3: Please check the spelling or version number. Also try "module spider ..."
3: It is also possible your cache file is out-of-date; it may help to try:
3:   $ module --ignore-cache load "suse-repo-deps/sam-default"
3: 
3: Also make sure that all modulefiles written in TCL start with the string
3: #%Module
3: 
3: 
3: 
0: Lmod has detected the following error: The following module(s) are unknown:
0: "aws-ofi-rccl/sam-default"
0: 
0: Please check the spelling or version number. Also try "module spider ..."
0: It is also possible your cache file is out-of-date; it may help to try:
0:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
0: 
0: Also make sure that all modulefiles written in TCL start with the string
0: #%Module
0: 
0: 
0: 
7: Lmod has detected the following error: The following module(s) are unknown:
7: "rocm/sam-5.2.3"
7: 
7: Please check the spelling or version number. Also try "module spider ..."
7: It is also possible your cache file is out-of-date; it may help to try:
7:   $ module --ignore-cache load "rocm/sam-5.2.3"
7: 
7: Also make sure that all modulefiles written in TCL start with the string
7: #%Module
7: 
7: 
7: 
2: Lmod has detected the following error: The following module(s) are unknown:
2: "rocm/sam-5.2.3"
2: 
2: Please check the spelling or version number. Also try "module spider ..."
2: It is also possible your cache file is out-of-date; it may help to try:
2:   $ module --ignore-cache load "rocm/sam-5.2.3"
2: 
2: Also make sure that all modulefiles written in TCL start with the string
2: #%Module
2: 
2: 
2: 
5: Lmod has detected the following error: The following module(s) are unknown:
5: "suse-repo-deps/sam-default"
5: 
5: Please check the spelling or version number. Also try "module spider ..."
5: It is also possible your cache file is out-of-date; it may help to try:
5:   $ module --ignore-cache load "suse-repo-deps/sam-default"
5: 
5: Also make sure that all modulefiles written in TCL start with the string
5: #%Module
5: 
5: 
5: 
4: Lmod has detected the following error: The following module(s) are unknown:
4: "suse-repo-deps/sam-default"
4: 
4: Please check the spelling or version number. Also try "module spider ..."
4: It is also possible your cache file is out-of-date; it may help to try:
4:   $ module --ignore-cache load "suse-repo-deps/sam-default"
4: 
4: Also make sure that all modulefiles written in TCL start with the string
4: #%Module
4: 
4: 
4: 
3: Lmod has detected the following error: The following module(s) are unknown:
3: "rocm/sam-5.2.3"
3: 
3: Please check the spelling or version number. Also try "module spider ..."
3: It is also possible your cache file is out-of-date; it may help to try:
3:   $ module --ignore-cache load "rocm/sam-5.2.3"
3: 
3: Also make sure that all modulefiles written in TCL start with the string
3: #%Module
3: 
3: 
3: 
1: Lmod has detected the following error: The following module(s) are unknown:
1: "rocm/sam-5.2.3"
1: 
1: Please check the spelling or version number. Also try "module spider ..."
1: It is also possible your cache file is out-of-date; it may help to try:
1:   $ module --ignore-cache load "rocm/sam-5.2.3"
1: 
1: Also make sure that all modulefiles written in TCL start with the string
1: #%Module
1: 
1: 
1: 
6: Lmod has detected the following error: The following module(s) are unknown:
6: "suse-repo-deps/sam-default"
6: 
6: Please check the spelling or version number. Also try "module spider ..."
6: It is also possible your cache file is out-of-date; it may help to try:
6:   $ module --ignore-cache load "suse-repo-deps/sam-default"
6: 
6: Also make sure that all modulefiles written in TCL start with the string
6: #%Module
6: 
6: 
6: 
7: Lmod has detected the following error: The following module(s) are unknown:
7: "rccl/sam-develop"
7: 
7: Please check the spelling or version number. Also try "module spider ..."
7: It is also possible your cache file is out-of-date; it may help to try:
7:   $ module --ignore-cache load "rccl/sam-develop"
7: 
7: Also make sure that all modulefiles written in TCL start with the string
7: #%Module
7: 
7: 
7: 
2: Lmod has detected the following error: The following module(s) are unknown:
2: "rccl/sam-develop"
2: 
2: Please check the spelling or version number. Also try "module spider ..."
2: It is also possible your cache file is out-of-date; it may help to try:
2:   $ module --ignore-cache load "rccl/sam-develop"
2: 
2: Also make sure that all modulefiles written in TCL start with the string
2: #%Module
2: 
2: 
2: 
5: Lmod has detected the following error: The following module(s) are unknown:
5: "rocm/sam-5.2.3"
5: 
5: Please check the spelling or version number. Also try "module spider ..."
5: It is also possible your cache file is out-of-date; it may help to try:
5:   $ module --ignore-cache load "rocm/sam-5.2.3"
5: 
5: Also make sure that all modulefiles written in TCL start with the string
5: #%Module
5: 
5: 
5: 
4: Lmod has detected the following error: The following module(s) are unknown:
4: "rocm/sam-5.2.3"
4: 
4: Please check the spelling or version number. Also try "module spider ..."
4: It is also possible your cache file is out-of-date; it may help to try:
4:   $ module --ignore-cache load "rocm/sam-5.2.3"
4: 
4: Also make sure that all modulefiles written in TCL start with the string
4: #%Module
4: 
4: 
4: 
3: Lmod has detected the following error: The following module(s) are unknown:
3: "rccl/sam-develop"
3: 
3: Please check the spelling or version number. Also try "module spider ..."
3: It is also possible your cache file is out-of-date; it may help to try:
3:   $ module --ignore-cache load "rccl/sam-develop"
3: 
3: Also make sure that all modulefiles written in TCL start with the string
3: #%Module
3: 
3: 
3: 
6: Lmod has detected the following error: The following module(s) are unknown:
6: "rocm/sam-5.2.3"
6: 
6: Please check the spelling or version number. Also try "module spider ..."
6: It is also possible your cache file is out-of-date; it may help to try:
6:   $ module --ignore-cache load "rocm/sam-5.2.3"
6: 
6: Also make sure that all modulefiles written in TCL start with the string
6: #%Module
6: 
6: 
6: 
1: Lmod has detected the following error: The following module(s) are unknown:
1: "rccl/sam-develop"
1: 
1: Please check the spelling or version number. Also try "module spider ..."
1: It is also possible your cache file is out-of-date; it may help to try:
1:   $ module --ignore-cache load "rccl/sam-develop"
1: 
1: Also make sure that all modulefiles written in TCL start with the string
1: #%Module
1: 
1: 
1: 
7: Lmod has detected the following error: The following module(s) are unknown:
7: "aws-ofi-rccl/sam-default"
7: 
7: Please check the spelling or version number. Also try "module spider ..."
7: It is also possible your cache file is out-of-date; it may help to try:
7:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
7: 
7: Also make sure that all modulefiles written in TCL start with the string
7: #%Module
7: 
7: 
7: 
3: Lmod has detected the following error: The following module(s) are unknown:
3: "aws-ofi-rccl/sam-default"
3: 
3: Please check the spelling or version number. Also try "module spider ..."
3: It is also possible your cache file is out-of-date; it may help to try:
3:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
3: 
3: Also make sure that all modulefiles written in TCL start with the string
3: #%Module
3: 
3: 
3: 
2: Lmod has detected the following error: The following module(s) are unknown:
2: "aws-ofi-rccl/sam-default"
2: 
2: Please check the spelling or version number. Also try "module spider ..."
2: It is also possible your cache file is out-of-date; it may help to try:
2:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
2: 
2: Also make sure that all modulefiles written in TCL start with the string
2: #%Module
2: 
2: 
2: 
5: Lmod has detected the following error: The following module(s) are unknown:
5: "rccl/sam-develop"
5: 
5: Please check the spelling or version number. Also try "module spider ..."
5: It is also possible your cache file is out-of-date; it may help to try:
5:   $ module --ignore-cache load "rccl/sam-develop"
5: 
5: Also make sure that all modulefiles written in TCL start with the string
5: #%Module
5: 
5: 
5: 
4: Lmod has detected the following error: The following module(s) are unknown:
4: "rccl/sam-develop"
4: 
4: Please check the spelling or version number. Also try "module spider ..."
4: It is also possible your cache file is out-of-date; it may help to try:
4:   $ module --ignore-cache load "rccl/sam-develop"
4: 
4: Also make sure that all modulefiles written in TCL start with the string
4: #%Module
4: 
4: 
4: 
6: Lmod has detected the following error: The following module(s) are unknown:
6: "rccl/sam-develop"
6: 
6: Please check the spelling or version number. Also try "module spider ..."
6: It is also possible your cache file is out-of-date; it may help to try:
6:   $ module --ignore-cache load "rccl/sam-develop"
6: 
6: Also make sure that all modulefiles written in TCL start with the string
6: #%Module
6: 
6: 
6: 
1: Lmod has detected the following error: The following module(s) are unknown:
1: "aws-ofi-rccl/sam-default"
1: 
1: Please check the spelling or version number. Also try "module spider ..."
1: It is also possible your cache file is out-of-date; it may help to try:
1:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
1: 
1: Also make sure that all modulefiles written in TCL start with the string
1: #%Module
1: 
1: 
1: 
6: Lmod has detected the following error: The following module(s) are unknown:
6: "aws-ofi-rccl/sam-default"
6: 
6: Please check the spelling or version number. Also try "module spider ..."
6: It is also possible your cache file is out-of-date; it may help to try:
6:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
6: 
6: Also make sure that all modulefiles written in TCL start with the string
6: #%Module
6: 
6: 
6: 
5: Lmod has detected the following error: The following module(s) are unknown:
5: "aws-ofi-rccl/sam-default"
5: 
5: Please check the spelling or version number. Also try "module spider ..."
5: It is also possible your cache file is out-of-date; it may help to try:
5:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
5: 
5: Also make sure that all modulefiles written in TCL start with the string
5: #%Module
5: 
5: 
5: 
4: Lmod has detected the following error: The following module(s) are unknown:
4: "aws-ofi-rccl/sam-default"
4: 
4: Please check the spelling or version number. Also try "module spider ..."
4: It is also possible your cache file is out-of-date; it may help to try:
4:   $ module --ignore-cache load "aws-ofi-rccl/sam-default"
4: 
4: Also make sure that all modulefiles written in TCL start with the string
4: #%Module
4: 
4: 
4: 
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: 2023-03-17 13:19:59.455061: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
0: 2023-03-17 13:19:59.455095: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
0: 2023-03-17 13:19:59.455122: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
0: 2023-03-17 13:19:59.455119: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
0: 2023-03-17 13:19:59.455140: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
0: 2023-03-17 13:19:59.455152: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
0: 2023-03-17 13:19:59.455170: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
0: 2023-03-17 13:19:59.455208: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
0: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
5: 2023-03-17 13:19:59.455621: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
5: 2023-03-17 13:19:59.455651: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
5: 2023-03-17 13:19:59.455621: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
5: 2023-03-17 13:19:59.455674: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
5: 2023-03-17 13:19:59.455720: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
5: 2023-03-17 13:19:59.455742: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
5: 2023-03-17 13:19:59.455793: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
5: 2023-03-17 13:19:59.455792: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
5: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
4: 2023-03-17 13:19:59.456022: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
4: 2023-03-17 13:19:59.456062: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
4: 2023-03-17 13:19:59.456087: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
4: 2023-03-17 13:19:59.456138: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
4: 2023-03-17 13:19:59.456163: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
4: 2023-03-17 13:19:59.456179: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
4: 2023-03-17 13:19:59.456181: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
4: 2023-03-17 13:19:59.456238: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
4: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
6: 2023-03-17 13:19:59.456099: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
6: 2023-03-17 13:19:59.456112: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
6: 2023-03-17 13:19:59.456114: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:19:59.456817: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:19:59.456835: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:19:59.456838: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: 2023-03-17 13:19:59.456146: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:19:59.456852: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
6: 2023-03-17 13:19:59.456177: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:19:59.456887: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:19:59.456883: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:19:59.456897: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: 2023-03-17 13:19:59.456241: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
6: 2023-03-17 13:19:59.456234: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
6: 2023-03-17 13:19:59.456258: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:19:59.456907: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
7: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
6: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1: 2023-03-17 13:19:59.456976: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1: 2023-03-17 13:19:59.456995: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1: 2023-03-17 13:19:59.457012: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1: 2023-03-17 13:19:59.457041: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1: 2023-03-17 13:19:59.457038: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1: 2023-03-17 13:19:59.457072: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1: 2023-03-17 13:19:59.457051: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
1: 2023-03-17 13:19:59.457053: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
1: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2: 2023-03-17 13:19:59.458343: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2: 2023-03-17 13:19:59.458334: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2: 2023-03-17 13:19:59.458357: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2: 2023-03-17 13:19:59.458358: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2: 2023-03-17 13:19:59.458342: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2: 2023-03-17 13:19:59.458375: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2: 2023-03-17 13:19:59.458389: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2: 2023-03-17 13:19:59.458363: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
2: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
3: 2023-03-17 13:19:59.461757: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
3: 2023-03-17 13:19:59.461760: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
3: 2023-03-17 13:19:59.461775: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
3: 2023-03-17 13:19:59.461772: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
3: 2023-03-17 13:19:59.461790: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
3: 2023-03-17 13:19:59.461802: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
3: 2023-03-17 13:19:59.461809: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
3: 2023-03-17 13:19:59.461814: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
3: To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
7: 2023-03-17 13:20:14.995629: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:20:14.995657: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:20:14.995683: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:20:14.995684: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:20:14.995715: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:20:14.995675: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:20:14.995716: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:20:14.995728: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:20:14.996314: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
7: 2023-03-17 13:20:14.996333: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
7: 2023-03-17 13:20:14.996343: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
7: 2023-03-17 13:20:14.996362: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
7: 2023-03-17 13:20:14.996366: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
7: 2023-03-17 13:20:14.996363: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
7: 2023-03-17 13:20:14.996377: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
7: 2023-03-17 13:20:14.996388: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
4: 2023-03-17 13:20:14.996400: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:20:14.996430: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:20:14.996442: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:20:14.996728: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
4: 2023-03-17 13:20:14.996459: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:20:14.996753: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
4: 2023-03-17 13:20:14.996468: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:20:14.996762: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
4: 2023-03-17 13:20:14.996485: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:20:14.996488: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:20:14.996776: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
4: 2023-03-17 13:20:14.996783: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
4: 2023-03-17 13:20:14.996794: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
4: 2023-03-17 13:20:14.996495: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:20:14.996808: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
4: 2023-03-17 13:20:14.996815: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
5: 2023-03-17 13:20:14.996708: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.996738: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.996797: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.996834: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.996844: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.996856: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.996773: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.996876: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.997497: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
5: 2023-03-17 13:20:14.997536: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
5: 2023-03-17 13:20:14.997546: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
5: 2023-03-17 13:20:14.997564: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.996798: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:20:14.997576: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
5: 2023-03-17 13:20:14.997587: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
5: 2023-03-17 13:20:14.997591: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
5: 2023-03-17 13:20:14.997596: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.996830: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:20:14.996849: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:20:14.996884: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:20:14.996871: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:20:14.996880: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:20:14.996886: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:20:14.996903: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.996916: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.996945: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.996982: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.997400: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
1: 2023-03-17 13:20:14.997408: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
1: 2023-03-17 13:20:14.997037: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.997012: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.997034: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.997423: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.997511: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
1: 2023-03-17 13:20:14.997049: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.997044: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:20:14.997434: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
1: 2023-03-17 13:20:14.997436: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
1: 2023-03-17 13:20:14.997442: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
1: 2023-03-17 13:20:14.997444: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
1: 2023-03-17 13:20:14.997447: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.997531: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.997540: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.997564: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.997563: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.997568: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.997595: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
3: 2023-03-17 13:20:14.997602: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
0: 2023-03-17 13:20:14.998468: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.998500: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.998512: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.998510: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.998516: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.998525: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.998675: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.998529: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.999296: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
0: 2023-03-17 13:20:14.999326: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
0: 2023-03-17 13:20:14.999327: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.998596: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:20:14.998632: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:20:14.998995: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.998657: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:20:14.999363: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
0: 2023-03-17 13:20:14.999372: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
0: 2023-03-17 13:20:14.999374: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
0: 2023-03-17 13:20:14.999373: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.998676: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:20:14.999018: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
0: 2023-03-17 13:20:14.999384: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.998697: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:20:14.998705: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:20:14.999042: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.998717: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:20:14.999052: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.998726: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:20:14.999060: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.999066: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.999080: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:20:14.999091: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
6: 2023-03-17 13:20:14.998316: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:20:14.998344: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:20:14.998768: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
6: 2023-03-17 13:20:14.998403: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:20:14.998784: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
6: 2023-03-17 13:20:14.998434: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:20:14.998444: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:20:14.998361: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:20:14.998808: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
6: 2023-03-17 13:20:14.998447: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:20:14.998812: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
6: 2023-03-17 13:20:14.998463: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:20:14.998823: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
6: 2023-03-17 13:20:14.998830: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
6: 2023-03-17 13:20:14.998831: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
6: 2023-03-17 13:20:14.998842: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2: 2023-03-17 13:21:03.391855: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.391889: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.391910: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.391928: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.391944: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.391949: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.391964: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.391967: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.392979: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.392998: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.393669: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.393676: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.393680: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.393707: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.393710: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.393718: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.394160: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.394192: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.394203: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.394234: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.394238: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.394247: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.394267: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.394464: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.417352: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.417379: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.417400: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.417409: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.417424: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.417421: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.417435: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.417448: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.417971: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.418011: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.418002: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.418034: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.418046: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.418068: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.418070: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.418246: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.419901: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.419936: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.419950: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.419962: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.419972: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.419981: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.419989: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.420000: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517638: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517641: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517643: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517645: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517649: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517645: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517645: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517651: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
1: 2023-03-17 13:21:03.517669: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
1: 2023-03-17 13:21:03.517674: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
1: 2023-03-17 13:21:03.517676: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
1: 2023-03-17 13:21:03.517674: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
1: 2023-03-17 13:21:03.517681: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
1: 2023-03-17 13:21:03.517683: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
1: 2023-03-17 13:21:03.517682: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
1: 2023-03-17 13:21:03.517684: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
3: 2023-03-17 13:21:03.518133: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.518136: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.518157: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
3: 2023-03-17 13:21:03.518157: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
3: 2023-03-17 13:21:03.518153: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.518154: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.518157: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.518160: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.518162: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.518162: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
3: 2023-03-17 13:21:03.518190: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
3: 2023-03-17 13:21:03.518191: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
3: 2023-03-17 13:21:03.518192: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
3: 2023-03-17 13:21:03.518193: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
3: 2023-03-17 13:21:03.518195: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
3: 2023-03-17 13:21:03.518194: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2: 2023-03-17 13:21:03.518964: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.518965: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.518966: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.518967: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.518968: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.518972: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.518975: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.518983: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
2: 2023-03-17 13:21:03.519009: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2: 2023-03-17 13:21:03.519011: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2: 2023-03-17 13:21:03.519012: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2: 2023-03-17 13:21:03.519013: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2: 2023-03-17 13:21:03.519015: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2: 2023-03-17 13:21:03.519016: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2: 2023-03-17 13:21:03.519016: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2: 2023-03-17 13:21:03.519019: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
6: 2023-03-17 13:21:03.518774: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.518775: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.519302: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.519306: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.519788: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.519797: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.519800: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.519801: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.519813: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
6: 2023-03-17 13:21:03.518773: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.518776: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.519308: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.519312: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.519804: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.518780: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.518780: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.519316: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.519315: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.519822: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
0: 2023-03-17 13:21:03.519825: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
0: 2023-03-17 13:21:03.519826: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
6: 2023-03-17 13:21:03.518786: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.518782: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.518799: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
5: 2023-03-17 13:21:03.519321: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.519325: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
5: 2023-03-17 13:21:03.519347: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
0: 2023-03-17 13:21:03.519828: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
0: 2023-03-17 13:21:03.519826: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
0: 2023-03-17 13:21:03.519827: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.518798: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
6: 2023-03-17 13:21:03.518799: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
6: 2023-03-17 13:21:03.518798: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
5: 2023-03-17 13:21:03.519348: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
5: 2023-03-17 13:21:03.519349: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
5: 2023-03-17 13:21:03.519353: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
0: 2023-03-17 13:21:03.519829: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
6: 2023-03-17 13:21:03.518808: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
6: 2023-03-17 13:21:03.518809: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
6: 2023-03-17 13:21:03.518810: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
5: 2023-03-17 13:21:03.519355: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
5: 2023-03-17 13:21:03.519356: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
5: 2023-03-17 13:21:03.519357: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
0: 2023-03-17 13:21:03.519850: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
0: 2023-03-17 13:21:03.519853: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
0: 2023-03-17 13:21:03.519856: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
6: 2023-03-17 13:21:03.518810: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
5: 2023-03-17 13:21:03.519359: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
4: 2023-03-17 13:21:04.513675: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.513689: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.513684: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.513686: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.513695: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.513691: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.513690: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.513710: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.513778: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.513785: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.513793: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.513789: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.513793: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.513793: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514516: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514518: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.513788: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.513798: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514522: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514523: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514523: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514525: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514527: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514529: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
4: 2023-03-17 13:21:04.514528: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
4: 2023-03-17 13:21:04.514533: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
4: 2023-03-17 13:21:04.514538: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
4: 2023-03-17 13:21:04.514537: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
4: 2023-03-17 13:21:04.514539: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
4: 2023-03-17 13:21:04.514539: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
4: 2023-03-17 13:21:04.514582: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
4: 2023-03-17 13:21:04.514593: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
7: 2023-03-17 13:21:04.514615: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.514618: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.514619: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.514625: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.514623: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.514628: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.514633: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
7: 2023-03-17 13:21:04.514634: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
7: 2023-03-17 13:21:04.514635: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
7: 2023-03-17 13:21:04.514637: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
7: 2023-03-17 13:21:04.514637: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
7: 2023-03-17 13:21:04.514639: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
7: 2023-03-17 13:21:04.514635: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.514649: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
7: 2023-03-17 13:21:04.514696: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/cray/pe/python/3.9.12.1/lib:/opt/cray/pe/gcc-libs:/opt/cray/libfabric/1.15.0.0/lib64
7: 2023-03-17 13:21:04.514709: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
2: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
1: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
3: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
7: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
6: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
5: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
4: /opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
0: Loading extension module scaled_upper_triang_masked_softmax_cuda...
0: Loading extension module scaled_masked_softmax_cuda...
0: Loading extension module fused_mix_prec_layer_norm_cuda...
0: Successfully preprocessed all matching files.
0: Successfully preprocessed all matching files.
7: Successfully preprocessed all matching files.
0: Traceback (most recent call last):
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
0: Traceback (most recent call last):
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
0:     subprocess.run(
0:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
0:     raise CalledProcessError(retcode, process.args,
0: subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
0: 
0: The above exception was the direct cause of the following exception:
0: 
0: Traceback (most recent call last):
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235, in <module>
0:     main()
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
0:     subprocess.run(
0:     return f(*args, **kwargs)  File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
0: 
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
0:     pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
0:     raise CalledProcessError(retcode, process.args,
0: subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
0: 
0: The above exception was the direct cause of the following exception:
0: 
0: Traceback (most recent call last):
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
0:     initialize_megatron(extra_args_provider=extra_args_provider,
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
0:     _compile_dependencies()
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
0:     fused_kernels.load(args)
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
0:     _write_ninja_file_and_build_library(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
0:     scaled_masked_softmax_cuda = _cpp_extention_load_helper(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
0:     return cpp_extension.load(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
0:     _run_ninja_build(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
0:     return _jit_compile(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
0:     raise RuntimeError(message) from e
0: RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-be
0: ttercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
0: FAILED: scaled_masked_softmax_hip.cuda.o 
0: /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/ro
0: cm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
0: In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
0: #include <hipsparse/hipsparse.h>
0:          ^~~~~~~~~~~~~~~~~~~~~~~
0: 1 error generated when compiling for gfx1030.
0: ninja: build stopped: subcommand failed.
0: 
0:     
0: During handling of the above exception, another exception occurred:
0: 
0: _write_ninja_file_and_build_library(Traceback (most recent call last):
0: 
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235, in <module>
0:     main()
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
0:     return f(*args, **kwargs)
0:       File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
0: _run_ninja_build(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
0:     pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
0:     initialize_megatron(extra_args_provider=extra_args_provider,
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
0:     _compile_dependencies()
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
0:     raise RuntimeError(message) from e
0: RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-be
0: ttercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
0: FAILED: scaled_masked_softmax_hip.cuda.o 
0: /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/ro
0: cm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
0: In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
0: #include <hipsparse/hipsparse.h>
0:          ^~~~~~~~~~~~~~~~~~~~~~~
0: 1 error generated when compiling for gfx1030.
0: ninja: build stopped: subcommand failed.
0: 
0:     fused_kernels.load(args)
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
0:     scaled_masked_softmax_cuda = _cpp_extention_load_helper(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
0:     return cpp_extension.load(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
0:     return _jit_compile(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1520, in _jit_compile
0:     baton.release()
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/file_baton.py", line 49, in release
0:     os.remove(self.lock_file_path)
0: FileNotFoundError: [Errno 2] No such file or directory: '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/build/lock'
7: Traceback (most recent call last):
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
7:     subprocess.run(
7:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
7:     raise CalledProcessError(retcode, process.args,
7: subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
7: 
7: The above exception was the direct cause of the following exception:
7: 
7: Traceback (most recent call last):
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
7:     _write_ninja_file_and_build_library(
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
7:     _run_ninja_build(
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
7:     raise RuntimeError(message) from e
7: RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-be
7: ttercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
7: FAILED: scaled_masked_softmax_hip.cuda.o 
7: /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/ro
7: cm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
7: In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
7: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
7: #include <hipsparse/hipsparse.h>
7:          ^~~~~~~~~~~~~~~~~~~~~~~
7: 1 error generated when compiling for gfx1030.
7: ninja: build stopped: subcommand failed.
7: 
7: 
7: During handling of the above exception, another exception occurred:
7: 
7: Traceback (most recent call last):
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235, in <module>
7:     main()
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
7:     return f(*args, **kwargs)
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
7:     pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
7:     initialize_megatron(extra_args_provider=extra_args_provider,
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
7:     _compile_dependencies()
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
7:     fused_kernels.load(args)
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
7:     scaled_masked_softmax_cuda = _cpp_extention_load_helper(
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
7:     return cpp_extension.load(
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
7:     return _jit_compile(
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1520, in _jit_compile
7:     baton.release()
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/file_baton.py", line 49, in release
7:     os.remove(self.lock_file_path)
7: FileNotFoundError: [Errno 2] No such file or directory: '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/build/lock'
0: Successfully preprocessed all matching files.
7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90799 closing signal SIGTERM
7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90800 closing signal SIGTERM
7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90802 closing signal SIGTERM
7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90803 closing signal SIGTERM
7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90804 closing signal SIGTERM
7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90805 closing signal SIGTERM
7: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 90806 closing signal SIGTERM
0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106009 closing signal SIGTERM
0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106011 closing signal SIGTERM
0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106013 closing signal SIGTERM
0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106014 closing signal SIGTERM
0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106015 closing signal SIGTERM
0: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106016 closing signal SIGTERM
0: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 106010) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
7: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 2 (pid: 90801) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
6: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
6: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
6: terminate called after throwing an instance of 'std::runtime_error'
6:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
6: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
6: Fatal Python error: Aborted
6: 
6: Thread 0x000014c813ac7b80 (most recent call first):
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
4: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
4: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
4: terminate called after throwing an instance of 'std::runtime_error'
4:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
4: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
4: Fatal Python error: Aborted
4: 
4: Thread 0x000014f1f5ae0b80 (most recent call first):
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102637 closing signal SIGTERM
4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102638 closing signal SIGTERM
4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102639 closing signal SIGTERM
4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102640 closing signal SIGTERM
4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102641 closing signal SIGTERM
4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102642 closing signal SIGTERM
4: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 102643 closing signal SIGTERM
6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7574 closing signal SIGTERM
6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7575 closing signal SIGTERM
6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7576 closing signal SIGTERM
6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7577 closing signal SIGTERM
6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7578 closing signal SIGTERM
6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7579 closing signal SIGTERM
6: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 7580 closing signal SIGTERM
4: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 102636) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
6: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 7573) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
5: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
5: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
5: terminate called after throwing an instance of 'std::runtime_error'
5:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
5: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
5: Fatal Python error: Aborted
5: 
5: Thread 0x000014eb231bab80 (most recent call first):
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
2: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
2: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
2: terminate called after throwing an instance of 'std::runtime_error'
2:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
2: Fatal Python error: Aborted
2: 
2: Thread 0x00001534ede95b80 (most recent call first):
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
3: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
3: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
3: terminate called after throwing an instance of 'std::runtime_error'
3:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
3: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
3: Fatal Python error: Aborted
3: 
3: Thread 0x00001470142fbb80 (most recent call first):
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106549 closing signal SIGTERM
3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106550 closing signal SIGTERM
3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106551 closing signal SIGTERM
3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106552 closing signal SIGTERM
3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106553 closing signal SIGTERM
3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106554 closing signal SIGTERM
3: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 106555 closing signal SIGTERM
2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23541 closing signal SIGTERM
2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23542 closing signal SIGTERM
2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23543 closing signal SIGTERM
2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23544 closing signal SIGTERM
2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23545 closing signal SIGTERM
2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23546 closing signal SIGTERM
2: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 23547 closing signal SIGTERM
5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96928 closing signal SIGTERM
5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96929 closing signal SIGTERM
5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96930 closing signal SIGTERM
5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96931 closing signal SIGTERM
5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96932 closing signal SIGTERM
5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96933 closing signal SIGTERM
5: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 96934 closing signal SIGTERM
3: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 106548) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
2: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 23540) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
5: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 96927) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
1: [E ProcessGroupNCCL.cpp:456] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.
1: [E ProcessGroupNCCL.cpp:461] To avoid data inconsistency, we are taking the entire process down.
1: terminate called after throwing an instance of 'std::runtime_error'
1:   what():  NCCL error: unhandled system error, NCCL version 2.11.4
1: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote peer.
1: Fatal Python error: Aborted
1: 
1: Thread 0x000014a0ec61eb80 (most recent call first):
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3152 in barrier
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 228 in _compile_dependencies
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164 in initialize_megatron
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99 in pretrain
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231 in main
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346 in wrapper
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 235 in <module>
1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110370 closing signal SIGTERM
1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110371 closing signal SIGTERM
1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110372 closing signal SIGTERM
1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110375 closing signal SIGTERM
1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110376 closing signal SIGTERM
1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110377 closing signal SIGTERM
1: WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 110378 closing signal SIGTERM
1: ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -6) local_rank: 0 (pid: 110369) of binary: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/bin/python
4: Traceback (most recent call last):
4:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
4:     return _run_code(code, main_globals, None,
4:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
4:     exec(code, run_globals)
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
7: ERROR:torch.distributed.elastic.multiprocessing.errors.error_handler:no error file defined for parent, to copy child error file (/tmp/torchelastic_nqvzuk6t/none_soqb0ebb/attempt_0/2/error.json)
6: Traceback (most recent call last):
6:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
6:     return _run_code(code, main_globals, None,
6:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
7: Traceback (most recent call last):
7:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
6:     exec(code, run_globals)
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
7:     return _run_code(code, main_globals, None,
7:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
7:     exec(code, run_globals)
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
4:     main()
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
4:     return f(*args, **kwargs)
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
3: Traceback (most recent call last):
3:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
5: Traceback (most recent call last):
5:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
3:     return _run_code(code, main_globals, None,
3:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
5:     return _run_code(code, main_globals, None,
5:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
3:     exec(code, run_globals)
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
5:     exec(code, run_globals)
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
6:     main()
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
4:     run(args)
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
7:     main()
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
6:     return f(*args, **kwargs)
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
7:     return f(*args, **kwargs)
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
4:     elastic_launch(
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
6:     run(args)
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
5:     main()
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
4:     return launch_agent(self._config, self._entrypoint, list(args))
4:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
3:     main()
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
7:     run(args)
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
5:     return f(*args, **kwargs)
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
3:     return f(*args, **kwargs)
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
6:     elastic_launch(
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
4:     raise ChildFailedError(
7:     elastic_launch(
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
4: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
4: =======================================================
4: Megatron-DeepSpeed/pretrain_gpt.py FAILED
4: -------------------------------------------------------
4: Failures:
4:   <NO_OTHER_FAILURES>
4: -------------------------------------------------------
4: Root Cause (first observed failure):
4: [0]:
4:   time      : 2023-03-17_13:50:34
4:   host      : nid005624
4:   rank      : 32 (local_rank: 0)
4:   exitcode  : -6 (pid: 102636)
4:   error_file: <N/A>
4:   traceback : Signal 6 (SIGABRT) received by PID 102636
4: =======================================================
6:     return launch_agent(self._config, self._entrypoint, list(args))
6:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
5:     run(args)
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
3:     run(args)
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
7:     return launch_agent(self._config, self._entrypoint, list(args))
7:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
6:     raise ChildFailedError(
5:     elastic_launch(
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
2: Traceback (most recent call last):
2:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
7:     raise ChildFailedError(
6: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
6: =====================================================
6: Megatron-DeepSpeed/pretrain_gpt.py FAILED
6: -----------------------------------------------------
6: Failures:
6:   <NO_OTHER_FAILURES>
6: -----------------------------------------------------
6: Root Cause (first observed failure):
6: [0]:
6:   time      : 2023-03-17_13:50:34
6:   host      : nid005626
6:   rank      : 48 (local_rank: 0)
6:   exitcode  : -6 (pid: 7573)
6:   error_file: <N/A>
6:   traceback : Signal 6 (SIGABRT) received by PID 7573
6: =====================================================
3:     elastic_launch(
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
0: ERROR:torch.distributed.elastic.multiprocessing.errors.error_handler:no error file defined for parent, to copy child error file (/tmp/torchelastic__af85h8z/none_zcro5z6y/attempt_0/1/error.json)
7: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
7: ============================================================
7: Megatron-DeepSpeed/pretrain_gpt.py FAILED
7: ------------------------------------------------------------
7: Failures:
7:   <NO_OTHER_FAILURES>
7: ------------------------------------------------------------
7: Root Cause (first observed failure):
7: [0]:
7:   time      : 2023-03-17_13:50:24
7:   host      : nid005627
7:   rank      : 58 (local_rank: 2)
7:   exitcode  : 1 (pid: 90801)
7:   error_file: /tmp/torchelastic_nqvzuk6t/none_soqb0ebb/attempt_0/2/error.json
7:   traceback : Traceback (most recent call last):
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
7:       subprocess.run(
7:     File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
7:       raise CalledProcessError(retcode, process.args,
2:     return _run_code(code, main_globals, None,
2:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
0: Traceback (most recent call last):
0:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
7:   subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
7:   
7:   The above exception was the direct cause of the following exception:
7:   
7:   Traceback (most recent call last):
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
7:       _write_ninja_file_and_build_library(
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
7:       _run_ninja_build(
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
7:       raise RuntimeError(message) from e
2:     exec(code, run_globals)
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
7:   RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-
0:     return _run_code(code, main_globals, None,
0:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
7: bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
7:   FAILED: scaled_masked_softmax_hip.cuda.o 
5:     return launch_agent(self._config, self._entrypoint, list(args))
5:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
0:     exec(code, run_globals)
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
7:   /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/
7: rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
7:   In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
7:   /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
7:   #include <hipsparse/hipsparse.h>
7:            ^~~~~~~~~~~~~~~~~~~~~~~
7:   1 error generated when compiling for gfx1030.
7:   ninja: build stopped: subcommand failed.
7:   
7:   
7:   During handling of the above exception, another exception occurred:
7:   
7:   Traceback (most recent call last):
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
7:       return f(*args, **kwargs)
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
7:       pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
7:       initialize_megatron(extra_args_provider=extra_args_provider,
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
7:       _compile_dependencies()
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
7:       fused_kernels.load(args)
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
7:       scaled_masked_softmax_cuda = _cpp_extention_load_helper(
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
7:       return cpp_extension.load(
3:     return launch_agent(self._config, self._entrypoint, list(args))
3:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
7:       return _jit_compile(
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1520, in _jit_compile
7:       baton.release()
7:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/file_baton.py", line 49, in release
7:       os.remove(self.lock_file_path)
7:   FileNotFoundError: [Errno 2] No such file or directory: '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/build/lock'
7:   
7: ============================================================
1: Traceback (most recent call last):
1:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 197, in _run_module_as_main
5:     raise ChildFailedError(
1:     return _run_code(code, main_globals, None,
1:   File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/runpy.py", line 87, in _run_code
5: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
5: ======================================================
5: Megatron-DeepSpeed/pretrain_gpt.py FAILED
5: ------------------------------------------------------
5: Failures:
5:   <NO_OTHER_FAILURES>
5: ------------------------------------------------------
5: Root Cause (first observed failure):
5: [0]:
5:   time      : 2023-03-17_13:50:39
5:   host      : nid005625
5:   rank      : 40 (local_rank: 0)
5:   exitcode  : -6 (pid: 96927)
5:   error_file: <N/A>
5:   traceback : Signal 6 (SIGABRT) received by PID 96927
5: ======================================================
3:     raise ChildFailedError(
1:     exec(code, run_globals)
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 766, in <module>
3: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
3: =======================================================
3: Megatron-DeepSpeed/pretrain_gpt.py FAILED
3: -------------------------------------------------------
3: Failures:
3:   <NO_OTHER_FAILURES>
3: -------------------------------------------------------
3: Root Cause (first observed failure):
3: [0]:
3:   time      : 2023-03-17_13:50:39
3:   host      : nid005623
3:   rank      : 24 (local_rank: 0)
3:   exitcode  : -6 (pid: 106548)
3:   error_file: <N/A>
3:   traceback : Signal 6 (SIGABRT) received by PID 106548
3: =======================================================
0:     main()
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
2:     main()
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
0:     return f(*args, **kwargs)
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
2:     return f(*args, **kwargs)
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
0:     run(args)
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
2:     run(args)
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
0:     elastic_launch(
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
2:     elastic_launch(
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
0:     return launch_agent(self._config, self._entrypoint, list(args))
0:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
1:     main()
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
1:     return f(*args, **kwargs)
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 762, in main
0:     raise ChildFailedError(
2:     return launch_agent(self._config, self._entrypoint, list(args))
2:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
0: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
0: ============================================================
0: Megatron-DeepSpeed/pretrain_gpt.py FAILED
0: ------------------------------------------------------------
0: Failures:
0: [1]:
0:   time      : 2023-03-17_13:50:24
0:   host      : nid005620
0:   rank      : 3 (local_rank: 3)
0:   exitcode  : 1 (pid: 106012)
0:   error_file: /tmp/torchelastic__af85h8z/none_zcro5z6y/attempt_0/3/error.json
0:   traceback : Traceback (most recent call last):
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
0:       subprocess.run(
0:     File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
0:       raise CalledProcessError(retcode, process.args,
0:   subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
0:   
0:   The above exception was the direct cause of the following exception:
0:   
0:   Traceback (most recent call last):
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
0:       _write_ninja_file_and_build_library(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
0:       _run_ninja_build(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
0:       raise RuntimeError(message) from e
0:   RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-
0: bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
0:   FAILED: scaled_masked_softmax_hip.cuda.o 
2:     raise ChildFailedError(
2: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
2: ======================================================
2: Megatron-DeepSpeed/pretrain_gpt.py FAILED
2: ------------------------------------------------------
2: Failures:
2:   <NO_OTHER_FAILURES>
2: ------------------------------------------------------
2: Root Cause (first observed failure):
2: [0]:
2:   time      : 2023-03-17_13:50:39
2:   host      : nid005622
2:   rank      : 16 (local_rank: 0)
2:   exitcode  : -6 (pid: 23540)
2:   error_file: <N/A>
2:   traceback : Signal 6 (SIGABRT) received by PID 23540
2: ======================================================
0:   /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/
0: rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
0:   In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
0:   /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
0:   #include <hipsparse/hipsparse.h>
0:            ^~~~~~~~~~~~~~~~~~~~~~~
0:   1 error generated when compiling for gfx1030.
0:   ninja: build stopped: subcommand failed.
0:   
0:   
0:   During handling of the above exception, another exception occurred:
0:   
0:   Traceback (most recent call last):
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
0:       return f(*args, **kwargs)
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
0:       pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
1:     run(args)
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
0:       initialize_megatron(extra_args_provider=extra_args_provider,
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
0:       _compile_dependencies()
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
0:       fused_kernels.load(args)
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
0:       scaled_masked_softmax_cuda = _cpp_extention_load_helper(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
0:       return cpp_extension.load(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
0:       return _jit_compile(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1520, in _jit_compile
0:       baton.release()
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/file_baton.py", line 49, in release
0:       os.remove(self.lock_file_path)
0:   FileNotFoundError: [Errno 2] No such file or directory: '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/build/lock'
0:   
0: ------------------------------------------------------------
0: Root Cause (first observed failure):
0: [0]:
0:   time      : 2023-03-17_13:50:24
0:   host      : nid005620
0:   rank      : 1 (local_rank: 1)
0:   exitcode  : 1 (pid: 106010)
0:   error_file: /tmp/torchelastic__af85h8z/none_zcro5z6y/attempt_0/1/error.json
0:   traceback : Traceback (most recent call last):
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/run.py", line 753, in run
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1900, in _run_ninja_build
0:       subprocess.run(
0:     File "/opt/cray/pe/python/3.9.12.1/lib/python3.9/subprocess.py", line 528, in run
0:       raise CalledProcessError(retcode, process.args,
0:   subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
0:   
0:   The above exception was the direct cause of the following exception:
0:   
0:   Traceback (most recent call last):
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
0:       return f(*args, **kwargs)
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/pretrain_gpt.py", line 231, in main
0:       pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/training.py", line 99, in pretrain
0:       initialize_megatron(extra_args_provider=extra_args_provider,
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 164, in initialize_megatron
0:       _compile_dependencies()
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/initialize.py", line 222, in _compile_dependencies
0:       fused_kernels.load(args)
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 95, in load
0:       scaled_masked_softmax_cuda = _cpp_extention_load_helper(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/__init__.py", line 56, in _cpp_extention_load_helper
0:       return cpp_extension.load(
1:     elastic_launch(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1284, in load
0:       return _jit_compile(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1508, in _jit_compile
0:       _write_ninja_file_and_build_library(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1623, in _write_ninja_file_and_build_library
0:       _run_ninja_build(
0:     File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build
0:       raise RuntimeError(message) from e
0:   RuntimeError: Error building extension 'scaled_masked_softmax_cuda': [1/2] /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
0: bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
0:   FAILED: scaled_masked_softmax_hip.cuda.o 
0:   /opt/rocm-5.1.0/bin/hipcc  -DWITH_HIP -DTORCH_EXTENSION_NAME=scaled_masked_softmax_cuda -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -I/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/torch/csrc/api/include -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/TH -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THC -isystem /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/THH -isystem /opt/
0: rocm-5.1.0/include -isystem /opt/rocm-5.1.0/miopen/include -isystem /opt/rocm-5.1.0/hip/include -isystem /opt/cray/pe/python/3.9.12.1/include/python3.9 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -O3 -fPIC -D__HIP_PLATFORM_HCC__=1 -DUSE_ROCM=1 -DCUDA_HAS_FP16=1 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 -O3 -D__HIP_NO_HALF_OPERATORS__=1 -D__HIP_NO_HALF_CONVERSIONS__=1 --amdgpu-target=gfx900 --amdgpu-target=gfx906 --amdgpu-target=gfx908 --amdgpu-target=gfx90a --amdgpu-target=gfx1030 -fno-gpu-rdc -c /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip -o scaled_masked_softmax_hip.cuda.o 
0:   In file included from /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip:25:
0:   /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/include/ATen/hip/HIPContext.h:7:10: fatal error: 'hipsparse/hipsparse.h' file not found
0:   #include <hipsparse/hipsparse.h>
0:            ^~~~~~~~~~~~~~~~~~~~~~~
0:   1 error generated when compiling for gfx1030.
0:   ninja: build stopped: subcommand failed.
0:   
0:   
0: ============================================================
1:     return launch_agent(self._config, self._entrypoint, list(args))
1:   File "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 246, in launch_agent
1:     raise ChildFailedError(
1: torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
1: =======================================================
1: Megatron-DeepSpeed/pretrain_gpt.py FAILED
1: -------------------------------------------------------
1: Failures:
1:   <NO_OTHER_FAILURES>
1: -------------------------------------------------------
1: Root Cause (first observed failure):
1: [0]:
1:   time      : 2023-03-17_13:50:44
1:   host      : nid005621
1:   rank      : 8 (local_rank: 0)
1:   exitcode  : -6 (pid: 110369)
1:   error_file: <N/A>
1:   traceback : Signal 6 (SIGABRT) received by PID 110369
1: =======================================================
srun: error: nid005620: task 0: Exited with exit code 1
srun: launch/slurm: _step_signal: Terminating StepId=3326770.0
srun: error: nid005625: task 5: Exited with exit code 1
srun: error: nid005627: task 7: Exited with exit code 1
srun: error: nid005623: task 3: Exited with exit code 1
srun: error: nid005626: task 6: Exited with exit code 1
srun: error: nid005624: task 4: Exited with exit code 1
srun: error: nid005622: task 2: Exited with exit code 1
srun: error: nid005621: task 1: Exited with exit code 1