Bryan

  BlogJava :: 首页 :: 联系 :: 聚合  :: 管理
  37 Posts :: 3 Stories :: 24 Comments :: 0 Trackbacks
Please refer to hadoop-2.7.0-src\BUILDING.txt and install all the required components.

1. install jdk1.7.0_25 and configure the JAVA_HOME and PATH, CLASSPATH, install apache-maven-3.3.3  and config MAVEN_HOME and also the PATH
use java -version, javac -version, mvn -version to validate the installation.

2. install windows sdk with the following URL : http://www.microsoft.com/en-us/download/details.aspx?id=8279
Windows SDK 7.1 Command Prompt will appear

3. download and install windows git package Git-1.9.5-preview20150319.exe from site http://git-scm.com/download/win,and add 
C:\Program Files (x86)\Git\bin to the path so that shell command(shell command in pom.xml) can be used.

4. download and install protoc-2.5.0-win32.zip, place in system32 folder or added to the path.
use command protoc --version to test

5.download cmake and install http://www.cmake.org/download/
cmake -version

6. get hadoop source hadoop 2.7.0 and 
open Windows SDK 7.1 Command Prompt and build hadoop with the following command
set Platform=x64
cd C:\hadoop\hadoop-2.7.0-src
mvn package -Pdist,native-win -DskipTests -Dtar

The following are the files I have listed which need to install, some of them is for resolving issues of hadoop maven build  and if you are not facing them, you can ignore.

hadoop-2.7.0-src.tar.gz  --->required
Git-1.9.5-preview20150319.exe  --->required
apache-maven-3.3.3  --->required
cmake-3.2.2-win32-x86.exe  --->required
winsdk_web.exe  --->required
protoc-2.5.0-win32.zip  --->required
hadoop-2.7.0-src  --->required
VC-Compiler-KB2519277.exe -----> sp1
ammintrin.h
MicrosoftFixit.ProgramInstallUninstall.RNP.6635582324812150.4.1.Run.exe  ------> tool to uninstall the newer version of visual c++ if it exists

The following are the issues encountered when I compile Hadoop 2.7.0

Installation Blockers:
A newer version of Microsoft Visual C++ 2010 Redistributable has been detected on the machine.
Final Result: Installation failed with error code: (0x000013EC), "A StopBlock was hit or a System Requirement was not met." (Elapsed time: 0 00:00:01).

Please refer to the link(
http://stackoverflow.com/questions/19366006/error-when-installing-windows-sdk-7-1
) when errors are found. Try to uninstall the newer version first. It does have a newer version in my laptop,but  I could not find it in windows uninstaller, but this tool (
http://support.microsoft.com/mats/Program_Install_and_Uninstall
)can.


LINK : fatal error LNK1123: failure during conversion to COFF: file invalid or c
orrupt [D:\hadoop\hadoop-2.7.0-src\hadoop-common-project\hadoop-common\src\main\
winutils\winutils.vcxproj]


Answer:
download and install Microsoft Visual C++ 2010 Service Pack 1 Compiler Update for the Windows SDK 7.1
https://www.microsoft.com/en-us/download/details.aspx?id=4422

C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\intrin.h(26): f
atal error C1083: Cannot open include file: 'ammintrin.h': No such file or direc
tory [D:\hadoop\hadoop-2.7.0-src\hadoop-common-project\hadoop-common\src\main\na
tive\native.vcxproj]
  C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include\intrin.h(26): f
atal error C1083: Cannot open include file: 'ammintrin.h': No such file or direc
tory [D:\hadoop\hadoop-2.7.0-src\hadoop-common-project\hadoop-common\src\main\na
tive\native.vcxproj]

    1 Warning(s)
    2 Error(s)

download file ammintrin.h from the internet and place it in folder C:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\include


posted on 2015-05-23 17:00 Life is no respector of any genius. 阅读(2574) 评论(8)  编辑  收藏 所属分类: hadoop

Feedback

# re: Hadoop2.7.0 build on win7 64 2015-07-10 22:14 Life is no respector of any genius.
hadoop2.7.1
http://pan.baidu.com/s/1dDwF4op  回复  更多评论
  

# re: Hadoop2.7.0 build on win7 64 2015-07-12 07:16 Bryan
When runing hadoop2.7.1 on SunOS, the following error happens, and after one day of analysis, It is due to that "df -k -P" fails on "SunOS plmdev 5.10 Generic_141444-09 sun4v sparc SUNW,SPARC-Enterprise-T5220"

2015-07-11 22:27:11,205 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for active state
2015-07-11 22:27:11,205 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Ending log segment 1
2015-07-11 22:27:11,217 INFO org.apache.hadoop.hdfs.server.namenode.FSEditLog: Number of transactions: 2 Total time for transactions(ms): 5 Number of transactions batched in Syncs: 0 Number of syncs: 3 SyncTimes(ms): 163
2015-07-11 22:27:11,230 INFO org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Finalizing edits file /data/ALMR/hadoop/data/dfs/namenode/current/edits_inprogress_0000000000000000001 -> /data/ALMR/hadoop/data/dfs/namenode/current/edits_0000000000000000001-0000000000000000002
2015-07-11 22:27:11,241 INFO org.apache.hadoop.ipc.Server: Stopping server on 9000
2015-07-11 22:27:11,338 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for active state
2015-07-11 22:27:11,338 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Stopping services started for standby state
2015-07-11 22:27:11,354 INFO org.mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:50070
2015-07-11 22:27:11,366 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode metrics system...
2015-07-11 22:27:11,369 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system stopped.
2015-07-11 22:27:11,370 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
2015-07-11 22:27:11,409 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: Failed to start namenode.
ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:554)
at org.apache.hadoop.util.Shell.run(Shell.java:456)
at org.apache.hadoop.fs.DF.getFilesystem(DF.java:76)
at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker$CheckedVolume.<init>(NameNodeResourceChecker.java:69)
at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.addDirToCheck(NameNodeResourceChecker.java:165)
at org.apache.hadoop.hdfs.server.namenode.NameNodeResourceChecker.<init>(NameNodeResourceChecker.java:134)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1058)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:678)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:664)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
2015-07-11 22:27:11,424 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2015-07-11 22:27:11,430 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at plmdev/135.251.35.155
************************************************************/
  回复  更多评论
  

# re: Hadoop2.7.0 build on win7 64 2015-08-01 22:23 Life is no respector of any genius.
continue with the above message:

It seems that I am making a mistake, there are different versions of du df and id command, and finally, I am using /usr/local/bin, and this problem disappears.


SYNOPSIS
/usr/bin/du [-adr] [-k | -h] [-H | -L] [-o | -s] [file
...]

/usr/xpg4/bin/du [-a | -s] [-k | -h] [-H | -L] [-rx]
[file ...]

Also, map reduce jobs get stuck in accepted state and could not get it run, finally I found that the disk space is greater than 90% or /tmp reaches 90% threshold.  回复  更多评论
  

# re: Hadoop2.7.0 build on win7 64 2015-08-03 05:58 Life is no respector of any genius.
http://git-scm.com/download/linux

unix package tool
http://www.opencsw.org/packages/CSWbashcmplt/  回复  更多评论
  

# re: Hadoop2.7.0 build on win7 64 2015-10-17 20:45 Life is no respector of any genius.
hadoop 2.6.1
http://yun.baidu.com/s/1o6zRwgy

hadoop-eclipse-plugin-2.6.1.jar
http://pan.baidu.com/s/1qMSJO  回复  更多评论
  

# re: Hadoop2.7.0 build on win7 64 2015-12-20 11:15 Life is no respector of any genius.
windows 7 build of hadoop
hadoop 2.6.2
http://pan.baidu.com/s/1n0vYQ

hadopop 2.6.3
http://pan.baidu.com/s/1c1y9OaO
  回复  更多评论
  

# re: Hadoop2.7.0 build on win7 64 2016-01-02 09:53 Life is no respector of any genius.
http://javadev.org/sitemap/  回复  更多评论
  

# re: Hadoop2.7.0 build on win7 64 2016-02-01 08:38 tongdayong
hadoop 2.7.2
http://pan.baidu.com/s/1i4plIfF  回复  更多评论
  


只有注册用户登录后才能发表评论。


网站导航: