Fully distributed Mode 에서 예제 실행
fully distributed mode 에서 하둡 예제를 실행 해보고 클러스터 상태를 모니터링 하는데 사용되는 하둡의 웹 인터페이스를 살펴보겠습니다.
예제 실행을 위해서 입력파일을 hdfs 안에 올려놔야 한다.
start-all.sh 모든 데몬들을 실행시키고 시작한다.
[root@master hadoop-1.0.3]# hadoop fs -put input input // local 에서 input파일을 동일한 이름으로 hdfs 안에 저장
[root@master hadoop-1.0.3]# hadoop fs -ls // 검색해보면 파일이 올라갔음을 볼수 있다.
Found 1 items
-rw-r--r-- 1 root supergroup 25 2012-08-31 17:16 /user/root/input
[root@master hadoop-1.0.3]# hadoop jar hadoop-examples-1.0.3.jar wordcount input output
12/08/31 17:26:13 INFO input.FileInputFormat: Total input paths to process : 1
12/08/31 17:26:13 INFO util.NativeCodeLoader: Loaded the native-hadoop library
12/08/31 17:26:13 WARN snappy.LoadSnappy: Snappy native library not loaded
12/08/31 17:26:13 INFO mapred.JobClient: Running job: job_201208311703_0001
12/08/31 17:26:14 INFO mapred.JobClient: map 0% reduce 0%
12/08/31 17:26:29 INFO mapred.JobClient: map 100% reduce 0%
12/08/31 17:26:41 INFO mapred.JobClient: map 100% reduce 100%
12/08/31 17:26:46 INFO mapred.JobClient: Job complete: job_201208311703_0001 // 작업완료
12/08/31 17:26:46 INFO mapred.JobClient: Counters: 29
12/08/31 17:26:46 INFO mapred.JobClient: Job Counters
12/08/31 17:26:46 INFO mapred.JobClient: Launched reduce tasks=1
12/08/31 17:26:46 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=13171
12/08/31 17:26:46 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
12/08/31 17:26:46 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
12/08/31 17:26:46 INFO mapred.JobClient: Rack-local map tasks=1
12/08/31 17:26:46 INFO mapred.JobClient: Launched map tasks=1
12/08/31 17:26:46 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=10384
12/08/31 17:26:46 INFO mapred.JobClient: File Output Format Counters
12/08/31 17:26:46 INFO mapred.JobClient: Bytes Written=8
12/08/31 17:26:46 INFO mapred.JobClient: FileSystemCounters
12/08/31 17:26:46 INFO mapred.JobClient: FILE_BYTES_READ=18
12/08/31 17:26:46 INFO mapred.JobClient: HDFS_BYTES_READ=124
12/08/31 17:26:46 INFO mapred.JobClient: FILE_BYTES_WRITTEN=43087
12/08/31 17:26:46 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=8
12/08/31 17:26:46 INFO mapred.JobClient: File Input Format Counters
12/08/31 17:26:46 INFO mapred.JobClient: Bytes Read=25
12/08/31 17:26:46 INFO mapred.JobClient: Map-Reduce Framework
12/08/31 17:26:46 INFO mapred.JobClient: Map output materialized bytes=18
12/08/31 17:26:46 INFO mapred.JobClient: Map input records=5
12/08/31 17:26:46 INFO mapred.JobClient: Reduce shuffle bytes=18
12/08/31 17:26:46 INFO mapred.JobClient: Spilled Records=2
12/08/31 17:26:46 INFO mapred.JobClient: Map output bytes=40
12/08/31 17:26:46 INFO mapred.JobClient: Total committed heap usage (bytes)=176623616
12/08/31 17:26:46 INFO mapred.JobClient: CPU time spent (ms)=1660
12/08/31 17:26:46 INFO mapred.JobClient: Combine input records=4
12/08/31 17:26:46 INFO mapred.JobClient: SPLIT_RAW_BYTES=99
12/08/31 17:26:46 INFO mapred.JobClient: Reduce input records=1
12/08/31 17:26:46 INFO mapred.JobClient: Reduce input groups=1
12/08/31 17:26:46 INFO mapred.JobClient: Combine output records=1
12/08/31 17:26:46 INFO mapred.JobClient: Physical memory (bytes) snapshot=179654656
12/08/31 17:26:46 INFO mapred.JobClient: Reduce output records=1
12/08/31 17:26:46 INFO mapred.JobClient: Virtual memory (bytes) snapshot=692523008
12/08/31 17:26:46 INFO mapred.JobClient: Map output records=4
[root@master hadoop-1.0.3]# hadoop fs -ls //output 디렉토리가 생성되어 있음
Found 2 items
-rw-r--r-- 1 root supergroup 25 2012-08-31 17:16 /user/root/input
drwxr-xr-x - root supergroup 0 2012-08-31 17:26 /user/root/output
[root@master result]# hadoop fs -ls output/ //디렉토리 안에 보면 part-r-00000 이 생성되어 잇음 (오류가 있다면 이 파일이 생기지 않음)
Found 3 items
-rw-r--r-- 1 root supergroup 0 2012-08-31 17:26 /user/root/output/_SUCCESS
drwxr-xr-x - root supergroup 0 2012-08-31 17:26 /user/root/output/_logs
-rw-r--r-- 1 root supergroup 8 2012-08-31 17:26 /user/root/output/part-r-00000
[root@master result]# hadoop fs -cat output/part-r-00000 //확인해 보면 정확하게 카운팅이 되어 있음
hello 4