Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IllegalArgumentException: Wrong FS when running hadoop wordcount job example. #137

Open
septianw opened this issue Sep 27, 2017 · 0 comments
Open

IllegalArgumentException: Wrong FS when running hadoop wordcount job example. #137

septianw opened this issue Sep 27, 2017 · 0 comments

Comments

@septianw
Copy link

I don't know if this is issue in the glusterfs hadoop plugin or on my configuration, but every time i try to run a wordcount example job, always return error like this:
java.lang.IllegalArgumentException: Wrong FS: glusterfs:/mapred/system, expected: file:///

Complete configuration below.

Hadoop version is 2.8.1
glusterfs version is 3.8.15-2.el7
run on Centos 7

Here is the complete command and stdout

[hadoop@gluster1 hadoop]$ bin/hadoop jar /tmp/hadoop-0.20.2-examples.jar wordcount /hadoop/yarn-hadoop-resourcemanager-gluster1.out /hadoop/out/
17/09/27 10:50:03 INFO glusterfs.GlusterVolume: Initializing gluster volume..
17/09/27 10:50:03 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS
17/09/27 10:50:03 INFO glusterfs.GlusterFileSystem: Initializing GlusterFS,  CRC disabled.
17/09/27 10:50:03 INFO glusterfs.GlusterFileSystem: GIT INFO={git.commit.id.abbrev=f0fee73, [email protected], git.commit.message.full=Merge pull request #122 from c
hildsb/getfattrparse

Refactor and cleanup the BlockLocation parsing code, git.commit.id=f0fee73c336ac19461d5b5bb91a77e05cff73361, git.commit.message.short=Merge pull request #122 from childsb/getfattrparse, gi
t.commit.user.name=bradley childs, git.build.user.name=Unknown, git.commit.id.describe=GA-12-gf0fee73, git.build.user.email=Unknown, git.branch=master, git.commit.time=30.03.2015 @ 20:06:4
6 UTC, git.build.time=20.09.2017 @ 10:02:14 UTC}
17/09/27 10:50:03 INFO glusterfs.GlusterFileSystem: GIT_TAG=GA
17/09/27 10:50:03 INFO glusterfs.GlusterFileSystem: Configuring GlusterFS
17/09/27 10:50:03 INFO glusterfs.GlusterVolume: Initializing gluster volume..
17/09/27 10:50:03 INFO glusterfs.GlusterVolume: Gluster volume: gv0 at : /mnt
17/09/27 10:50:03 WARN glusterfs.GlusterVolume: mapred.system.dir/mapreduce.jobtracker.system.dir does not exist: glusterfs:/mapred/system
17/09/27 10:50:03 WARN glusterfs.GlusterVolume: working directory does not exist: glusterfs:/user/hadoop
17/09/27 10:50:03 INFO glusterfs.GlusterVolume: Working directory is : glusterfs:/user/hadoop
17/09/27 10:50:03 INFO glusterfs.GlusterVolume: Write buffer size : 131072
17/09/27 10:50:03 INFO glusterfs.GlusterVolume: Default block size : 67108864
17/09/27 10:50:03 INFO glusterfs.GlusterVolume: Directory list order : fs ordering
17/09/27 10:50:03 INFO glusterfs.GlusterVolume: File timestamp lease significant digits removed : 0
17/09/27 10:50:03 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/09/27 10:50:03 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
java.lang.IllegalArgumentException: Wrong FS: glusterfs:/mapred/system, expected: file:///
        at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:665)
        at org.apache.hadoop.fs.FileSystem.makeQualified(FileSystem.java:484)
        at org.apache.hadoop.fs.FilterFileSystem.makeQualified(FilterFileSystem.java:120)
        at org.apache.hadoop.mapred.LocalJobRunner.getSystemDir(LocalJobRunner.java:864)
        at org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:187)
        at org.apache.hadoop.mapreduce.Cluster$1.run(Cluster.java:185)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:421)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
        at org.apache.hadoop.mapreduce.Cluster.getFileSystem(Cluster.java:185)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1336)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359)
        at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
        at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:148)

core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
  <name>fs.glusterfs.impl</name>
  <value>org.apache.hadoop.fs.glusterfs.GlusterFileSystem</value>
</property>

<property>
  <name>fs.AbstractFileSystem.glusterfs.impl</name>
  <value>org.apache.hadoop.fs.local.GlusterFs</value>
</property>

<property>
    <name>fs.default.name</name>
    <value>glusterfs:///</value>
</property>

<property>
    <name>fs.glusterfs.volumes</name>
    <value>gv0</value>
</property>

<property>
    <name>fs.glusterfs.volume.fuse.gv0</name>
    <value>/mnt</value>
</property>

<property>
    <name>dfs.permissions</name>
    <value>false</value>
</property>
</configuration>

mapred-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
  <property>
    <name>yarn.app.mapreduce.am.staging-dir</name>
    <value>glusterfs:///tmp/hadoop-yarn/staging/mapred/.staging</value>
  </property>
  <property>
    <name>mapred.healthChecker.script.path</name>
    <value>glusterfs:///mapred/jobstatus</value>
  </property>
  <property>
    <name>mapred.job.tracker.history.completed.location</name>
    <value>glusterfs:///mapred/history/done</value>
  </property>
  <property>
    <name>mapred.system.dir</name>
    <value>glusterfs:///mapred/system</value>
  </property>
  <property>
    <name>mapreduce.jobhistory.done-dir</name>
    <value>glusterfs:///job-history/done</value>
  </property>

  <property>
    <name>mapreduce.jobhistory.intermediate-done-dir</name>
    <value>glusterfs:///job-history/intermediate-done</value>
  </property>

  <property>
    <name>mapreduce.jobtracker.staging.root.dir</name>
    <value>glusterfs:///user</value>
  </property>
  <property>
        <name>mapred.job.tracker</name>
        <value>hadoop-master:9001</value>
  </property>
</configuration>

hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
<property>
        <name>dfs.data.dir</name>
        <value>/opt/hadoop/hadoop/dfs/name/data</value>
        <final>true</final>
</property>
<property>
        <name>dfs.name.dir</name>
        <value>/opt/hadoop/hadoop/dfs/name</value>
        <final>true</final>
</property>
<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>
</configuration>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@septianw