[Gluster-devel] Hadoop-glusterfs Error in configuring object

ZG Niu niu.zglinux at gmail.com
Wed Aug 1 07:34:51 UTC 2012 

The problem was occurring because the mapred.job.tracker property in
mapred-site.xml file was not set correctly, I changed it and the problem
disappeared in a jiffy.

On Wed, Aug 1, 2012 at 1:14 PM, Venky Shankar <yknev.shankar at gmail.com>wrote:

> Can you provide the configuration files: conf/core-site.xml and
> conf/mapred-site.xml ?
>
> -Venky
>
> On Tue, Jul 31, 2012 at 2:55 PM, ZG Niu <niu.zglinux at gmail.com> wrote:
>
>> Hi list,
>>
>> I see the following exception on release-3.3 configured with hadoop
>> compatible storage,
>> The problem was occurring when I run the Terasort benchmarks   "hadoop
>> jar hadoop-0.20.2-examples.jar terasort output sorted"
>>
>>
>> 12/07/31 01:08:36 INFO terasort.TeraSort: starting
>> Initializing GlusterFS
>> 12/07/31 01:08:36 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 12/07/31 01:08:36 WARN snappy.LoadSnappy: Snappy native library is
>> available
>> 12/07/31 01:08:36 INFO util.NativeCodeLoader: Loaded the native-hadoop
>> library
>> 12/07/31 01:08:36 INFO snappy.LoadSnappy: Snappy native library loaded
>> 12/07/31 01:08:36 INFO zlib.ZlibFactory: Successfully loaded &
>> initialized native-zlib library
>> 12/07/31 01:08:36 INFO compress.CodecPool: Got brand-new compressor
>> Making 1 from 100000 records
>> Step size is 100000.0
>> 12/07/31 01:08:36 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>> processName=JobTracker, sessionId=
>> 12/07/31 01:08:36 INFO mapred.FileInputFormat: Total input paths to
>> process : 1
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> _partition.lst in
>> /data/mapred/local/archive/3689403515442034031_-1440303010_1461729625/Namenode1/mnt/output-work-2689044872239508516
>> with rwxr-xr-x
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Cached
>> glusterfs://Namenode1:9000/mnt/output/_partition.lst#_partition.lst as
>> /data/mapred/local/archive/3689403515442034031_-1440303010_1461729625/Namenode1/mnt/output/_partition.lst
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Cached
>> glusterfs://Namenode1:9000/mnt/output/_partition.lst#_partition.lst as
>> /data/mapred/local/archive/3689403515442034031_-1440303010_1461729625/Namenode1/mnt/output/_partition.lst
>> 12/07/31 01:08:37 WARN mapred.LocalJobRunner: LocalJobRunner does not
>> support symlinking into current working dir.
>> 12/07/31 01:08:37 INFO mapred.TaskRunner: Creating symlink:
>> /data/mapred/local/archive/3689403515442034031_-1440303010_1461729625/Namenode1/mnt/output/_partition.lst
>> <- /data/mapred/local/localRunner/_partition.lst
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /tmp/hadoop-root/mapred/staging/root-434405171/.staging/job_local_0001/job.jar
>> <- /data/mapred/local/localRunner/job.jar
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /tmp/hadoop-root/mapred/staging/root-434405171/.staging/job_local_0001/.job.xml.crc
>> <- /data/mapred/local/localRunner/.job.xml.crc
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /tmp/hadoop-root/mapred/staging/root-434405171/.staging/job_local_0001/job.split
>> <- /data/mapred/local/localRunner/job.split
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /tmp/hadoop-root/mapred/staging/root-434405171/.staging/job_local_0001/job.splitmetainfo
>> <- /data/mapred/local/localRunner/job.splitmetainfo
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /tmp/hadoop-root/mapred/staging/root-434405171/.staging/job_local_0001/.job.split.crc
>> <- /data/mapred/local/localRunner/.job.split.crc
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /tmp/hadoop-root/mapred/staging/root-434405171/.staging/job_local_0001/.job.splitmetainfo.crc
>> <- /data/mapred/local/localRunner/.job.splitmetainfo.crc
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /tmp/hadoop-root/mapred/staging/root-434405171/.staging/job_local_0001/job.xml
>> <- /data/mapred/local/localRunner/job.xml
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Creating
>> symlink:
>> /tmp/hadoop-root/mapred/staging/root-434405171/.staging/job_local_0001/.job.jar.crc
>> <- /data/mapred/local/localRunner/.job.jar.crc
>> 12/07/31 01:08:37 INFO mapred.JobClient: Running job: job_local_0001
>> 12/07/31 01:08:37 INFO mapred.MapTask: numReduceTasks: 1
>> 12/07/31 01:08:37 INFO mapred.MapTask: io.sort.mb = 100
>> 12/07/31 01:08:37 INFO mapred.MapTask: data buffer = 79691776/99614720
>> 12/07/31 01:08:37 INFO mapred.MapTask: record buffer = 262144/327680
>> 12/07/31 01:08:37 WARN mapred.LocalJobRunner: job_local_0001
>> java.lang.RuntimeException: Error in configuring object
>>     at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
>>     at
>> org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
>>     at
>> org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
>>     at
>> org.apache.hadoop.mapred.MapTask$OldOutputCollector.<init>(MapTask.java:481)
>>     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391)
>>     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325)
>>     at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)
>> Caused by: java.lang.reflect.InvocationTargetException
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
>>     ... 6 more
>> Caused by: java.lang.IllegalArgumentException: can't read paritions file
>>     at
>> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.configure(TeraSort.java:213)
>>     ... 11 more
>> Caused by: java.io.FileNotFoundException: File _partition.lst does not
>> exist.
>>     at
>> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383)
>>     at
>> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
>>     at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776)
>>     at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424)
>>     at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419)
>>     at
>> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:153)
>>     at
>> org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.configure(TeraSort.java:210)
>>     ... 11 more
>> 12/07/31 01:08:37 INFO filecache.TrackerDistributedCacheManager: Deleted
>> path
>> /data/mapred/local/archive/3689403515442034031_-1440303010_1461729625/Namenode1/mnt/output/_partition.lst
>> 12/07/31 01:08:38 INFO mapred.JobClient:  map 0% reduce 0%
>> 12/07/31 01:08:38 INFO mapred.JobClient: Job complete: job_local_0001
>> 12/07/31 01:08:38 INFO mapred.JobClient: Counters: 0
>> 12/07/31 01:08:38 INFO mapred.JobClient: Job Failed: NA
>> java.io.IOException: Job failed!
>>     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1246)
>>     at org.apache.hadoop.examples.terasort.TeraSort.run(TeraSort.java:248)
>>     at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>>     at
>> org.apache.hadoop.examples.terasort.TeraSort.main(TeraSort.java:257)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at
>> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
>>     at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
>>     at
>> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>     at java.lang.reflect.Method.invoke(Method.java:597)
>>     at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
>>
>>
>> --
>> Best Regards,
>> NiuZG
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> https://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>>
>
>
> --
> Astalavista Baby!!!
>



-- 
Best Regards,
NiuZG
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20120801/b570cdb6/attachment-0003.html>

More information about the Gluster-devel mailing list