Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[need help] error compiling 32x hdfs native client #2

Open
cdarlint opened this issue Oct 9, 2019 · 1 comment
Open

[need help] error compiling 32x hdfs native client #2

cdarlint opened this issue Oct 9, 2019 · 1 comment

Comments

@cdarlint
Copy link
Owner

cdarlint commented Oct 9, 2019 •
edited
Loading

There are errors when compiling hadoop 3.2.0 and 3.2.1 on windows
I believe there might be configuration problems or some glitches of windows-related problems
I'll post error messages here and please help

This is a dedicated virtual machine for compiling hadoop
I've successfully compiled multiple versions, up to 3.1.x

as my machine is Chinese language and locale, I translated and hand picked most related messages, as below

near the end of the output, there are mesasges like:
APACHE Hadoop HDFS Native Client ................... FAILURE [ 9.908 s]
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-hdfs-native-client: An Ant BuildException has occured: exec returned 1
[ERROR] around Ant part ...(exec failonerror="true" dir="C:\h3\hadoop-hdfs-project\hadoop-hdfs-native-client\target/native" executable="msbuild">... @ 9:121 in c:\h3\hadoop-hdfs-project\hadoop-hdfs-native-client\target\antrun\build-main.xml

above this, there are error messages during compiling native client
there are basically two problems
1 cannot resolve reference
2 missing semicolon

problem 1:
test_libhdfs_ops.obj : error LNK2019: unresolved external symbol hdfsFileDisableDirectRead referenced in main
test_libhdfs_ops.obj : error LNK2019: unresolved external symbol hdfsFileUsesDirectRead referenced in main

problem 2. error messages got too long so I opened this solution in VS2010 on the same VM and collected error messages there:
ClCompile:
test_libhdfs_threaded.c
..........\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(163): error C2143: grammar error : missing ";" (before 'type')
..........\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(164): error C2065: "invalid_path": undeclared symbol
@tang2087
Copy link

tang2087 commented Jan 19, 2020 •
edited
Loading

Hey @cdarlint

I also got same error when building 3.2.1 on Windows 10 and I've fixed it temporarily in my local code.

Refer to my post Compile and Build Hadoop 3.2.1 on Windows 10 Guide Section II Issue Fixes for more details.

The error occurred because in my environment VS2010 C/C++ compiler doesn't support doesn't support declare variables in the middle of the function or in the loop for C code.

[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:32 min]
[INFO] Apache Hadoop HDFS Native Client ................... FAILURE [ 47.921 s]

     [exec] "C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj" (default target) (9) ->
     [exec] (ClCompile target) ->
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(163): error C2143: syntax error : missing ';' before 'type' [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(164): error C2065: 'invalid_path' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(210): error C2143: syntax error : missing ';' before 'type' [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(211): error C2065: 'dirList' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(216): error C2065: 'dirList' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(219): error C2143: syntax error : missing ';' before 'type' [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(220): error C2065: 'listDirTest' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(221): error C2065: 'listDirTest' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(222): error C2065: 'listDirTest' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(223): error C2143: syntax error : missing ';' before 'type' [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(224): error C2065: 'nFile' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(224): error C2065: 'nFile' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(224): error C2065: 'nFile' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(226): error C2065: 'listDirTest' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(226): error C2065: 'nFile' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(231): error C2065: 'dirList' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(231): error C2065: 'listDirTest' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(232): error C2065: 'dirList' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(233): error C2065: 'dirList' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(312): error C2065: 'invalid_path' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(317): error C2143: syntax error : missing ';' before 'type' [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(318): error C2065: 'fs2' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(319): error C2065: 'fs2' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]   ..\..\..\..\..\src\main\native\libhdfs-tests\test_libhdfs_threaded.c(320): error C2065: 'fs2' : undeclared identifier [C:\hdp\hadoop\hadoop-hdfs-project\hadoop-hdfs-native-client\target\native\main\native\libhdfs\test_libhdfs_threaded_hdfs_static.vcxproj]
     [exec]
     [exec]     29 Warning(s)
     [exec]     24 Error(s)
     [exec]
     [exec] Time Elapsed 00:00:11.01

Two examples of test_libhdfs_threaded.c that causes errors:

char invalid_path[] = "/some_invalid/path";
EXPECT_NULL_WITH_ERRNO(hdfsListDirectory(fs, invalid_path, &numEntries), ENOENT);

    /* There should be 1 entry in the directory. */
    hdfsFileInfo * dirList = hdfsListDirectory(fs, paths->prefix, &numEntries);
    EXPECT_NONNULL(dirList);
    if (numEntries != 1) {
        fprintf(stderr, "hdfsListDirectory set numEntries to "
                "%d on directory containing 1 file.", numEntries);
    }
    hdfsFreeFileInfo(dirList, numEntries);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@cdarlint @tang2087