Bryan
  BlogJava :: 首页 :: 联系 :: 聚合  :: 管理
  37 Posts :: 3 Stories :: 24 Comments :: 0 Trackbacks
Hadoop2.5.0 build on win7 64

Please refer to hadoop-2.5.0-src\BUILDING.txt and install all the required components.

1. install jdk1.6.0_45 and configure the JAVA_HOME and PATH, CLASSPATH, install apache-maven-3.2.3  and config MAVEN_HOME and also the PATH
use java -version, javac -version, mvn -version to validate the installation.

2. install windows sdk with the following URL : http://www.microsoft.com/en-us/download/details.aspx?id=8279
Windows SDK 7.1 Command Prompt will appear

3. download and install  cygwin64 setup-x86_64.exe,and add C:\cygwin64\bin to the path so that shell command(shell command in pom.xml) can be used.
shell command , eg sh command can be used

4. download and install protoc-2.5.0-win32.zip, place in system32 folder or added to the path.
use command protoc --version to test 

5. download and install zlib(firstly,I installed and then uninstall, but it also can run, I am not sure whether It is required.)

6. get hadoop source hadoop-2.5.0-src.tar.gz and 
open Windows SDK 7.1 Command Prompt and build hadoop with the following command
set Platform=x64
cd C:\hadoop\hadoop-2.5.0-src
mvn package -Pdist,native-win -DskipTests -Dtar

The following is the path for my win7 system and I manually add the jdk ,maven and cygwin64 to User PATH variable , others are automatically generated in System PATH variable.

C:\hadoop\hadoop-2.5.0-src>path
PATH=C:\Windows\Microsoft.NET\Framework64\v4.0.30319;C:\Windows\Microsoft.NET\Fr
amework\v4.0.30319;C:\Windows\Microsoft.NET\Framework64\v3.5;C:\Windows\Microsof
t.NET\Framework\v3.5;;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common
7\IDE;C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\Tools;;C:\Prog
ram Files (x86)\Microsoft Visual Studio 10.0\VC\Bin\amd64;C:\Program Files (x86)
\Microsoft Visual Studio 10.0\VC\Bin\VCPackages;;C:\Program Files\Microsoft SDKs
\Windows\v7.1\Bin\NETFX 4.0 Tools\x64;C:\Program Files\Microsoft SDKs\Windows\v7
.1\Bin\x64;C:\Program Files\Microsoft SDKs\Windows\v7.1\Bin;;C:\Windows\system32
;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\
;C:\Program Files\Microsoft Windows Performance Toolkit\;C:\Java\jdk1.7.0_67\bin
;C:\hadoop\apache-maven-3.2.3\bin;C:\cygwin64\bin;

I did not install findbugs ,but It also can buid successfully.

The following are errors I encounter while compiling hadoop 2.5.0.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:
run (pre-dist) on project hadoop-project-dist: An Ant BuildException has occured
: Execute failed: java.io.IOException: Cannot run program "sh" (in directory "C:
\hadoop\hadoop-2.5.0-src\hadoop-project-dist\target"): CreateProcess error=2, ??
???????
[ERROR] around Ant part ...<exec dir="C:\hadoop\hadoop-2.5.0-src\hadoop-project-
dist\target" executable="sh" failonerror="true">... @ 31:104 in C:\hadoop\hadoop
-2.5.0-src\hadoop-project-dist\target\antrun\build-main.xml
[ERROR] -> [Help 1]

cygwin64 should be installed and add C:\cygwin64\bin to the path 

[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec (com
pile-ms-winutils) on project hadoop-common: Command execution failed. Process ex
ited with an error: 1(Exit value: 1) -> [Help 1]
 
Windows SDK 7.1 Command Prompt run the following cmd first before mvn build.  
set Platform=x64

[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:2.5.0:prot
oc (compile-protoc) on project hadoop-common: org.apache.maven.plugin.MojoExecut
ionException: 'protoc --version' did not return a version -> [Help 1]

add protoc.exe to the PATH or just place it in WINDOWS/system32 folder

I have uploaded different versions of hadoop build and they can be found in the location. they are compiled with the following requirements
jdk 1.6.0_45
apache-maven-3.2.3 
Cygwin64
Windows SDK 7.1
protoc 2.5.0

hadoop eclipse plugins build

I have also compiled the hadoop eclipse plugins and you can find them  with URL hadoop eclipse plugin.It includes versions of 2.2.0 2.3.0 2.4.0 2.4.1 and 2.5.0.
I tested and sometimes the following errors happen and I extract the eclipse again from eclipse build and they works, so I think the jar has no problem. as I have no idea of eclipse plugin development, so I am not sure about actual reason.

java.io.IOException: Unable to resolve plug-in "platform:/plugin/org.apache.hadoop.eclipse/resources/hadoop-logo-16x16.png".
    at org.eclipse.core.internal.runtime.PlatformURLPluginConnection.parse(PlatformURLPluginConnection.java:64)
    at org.eclipse.core.internal.runtime.PlatformURLPluginConnection.resolve(PlatformURLPluginConnection.java:75)
    at org.eclipse.core.internal.boot.PlatformURLHandler.openConnection(PlatformURLHandler.java:67)
    at org.eclipse.osgi.internal.url.URLStreamHandlerProxy.openConnection(URLStreamHandlerProxy.java:114)
    at java.net.URL.openConnection(URL.java:971)
    at org.eclipse.core.internal.runtime.PlatformURLConverter.toFileURL(PlatformURLConverter.java:33)
    at org.eclipse.core.runtime.FileLocator.toFileURL(FileLocator.java:206)
    at org.eclipse.jface.resource.URLImageDescriptor.getFilePath(URLImageDescriptor.java:141)
    at org.eclipse.jface.resource.URLImageDescriptor.createImage(URLImageDescriptor.java:162)
    at org.eclipse.jface.resource.ImageDescriptor.createImage(ImageDescriptor.java:230)
    at org.eclipse.jface.resource.ImageDescriptor.createImage(ImageDescriptor.java:208)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.getImageFromURI(SWTPartRenderer.java:221)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.getImage(SWTPartRenderer.java:233)
    at org.eclipse.e4.ui.workbench.renderers.swt.StackRenderer.createTab(StackRenderer.java:829)
    at org.eclipse.e4.ui.workbench.renderers.swt.LazyStackRenderer.processContents(LazyStackRenderer.java:134)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.SashRenderer.processContents(SashRenderer.java:185)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.SashRenderer.processContents(SashRenderer.java:185)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.PerspectiveRenderer.processContents(PerspectiveRenderer.java:49)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.PerspectiveStackRenderer.showTab(PerspectiveStackRenderer.java:103)
    at org.eclipse.e4.ui.workbench.renderers.swt.LazyStackRenderer.postProcess(LazyStackRenderer.java:98)
    at org.eclipse.e4.ui.workbench.renderers.swt.PerspectiveStackRenderer.postProcess(PerspectiveStackRenderer.java:77)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:678)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.SashRenderer.processContents(SashRenderer.java:185)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.workbench.renderers.swt.SWTPartRenderer.processContents(SWTPartRenderer.java:68)
    at org.eclipse.e4.ui.workbench.renderers.swt.WBWRenderer.processContents(WBWRenderer.java:664)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:674)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.safeCreateGui(PartRenderingEngine.java:766)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.access$2(PartRenderingEngine.java:737)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$7.run(PartRenderingEngine.java:731)
    at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.createGui(PartRenderingEngine.java:715)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1078)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032)
    at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148)
    at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636)
    at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332)
    at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579)
    at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150)
    at org.eclipse.ui.internal.ide.application.IDEApplication.start(IDEApplication.java:135)
    at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
    at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:382)
    at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:236)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:601)
    at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648)
    at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603)
    at org.eclipse.equinox.launcher.Main.run(Main.java:1465)
    at org.eclipse.equinox.launcher.Main.main(Main.java:1438)

Ref    http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os?utm_source=tuicool                        
         http://www.codeproject.com/Articles/757934/Apache-Hadoop-for-Windows-Platform
posted on 2014-08-22 21:00 Life is no respector of any genius. 阅读(8944) 评论(6)  编辑  收藏 所属分类: hadoop

Feedback

# re: Hadoop2.5.0 build on win7 64 2014-11-01 01:23 lakshmi
I am getting this error and I dont know how to resolve it. I have cygwin64's path in my environment variables.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.6:
run (pre-dist) on project hadoop-project-dist: An Ant BuildException has occured
: exec returned: 2 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal o
rg.apache.maven.plugins:maven-antrun-plugin:1.6:run (pre-dist) on project hadoop
-project-dist: An Ant BuildException has occured: exec returned: 2
at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor
.java:216)......

.......

Caused by: C:\Users\xxxx\Documents\hdfs\hadoop-project-dist\target\antrun\build-main.xml:31: exec returned: 2


Can you help me with this?

I am trying to build hadoop 2.2.0 but even 2.5.0 is giving the same error. Os: Windows 7 64 bit  回复  更多评论
  

# re: Hadoop2.5.0 build on win7 64 2014-11-01 12:00 Life is no respector of any genius.
You have to test the sh command first and see whether It works on windows. From the error output(exec returned: 2), It seems the exec command can not be completed successfully, please check why It resulted in the error.

you can copy the shell script to one file and replace the variables and test to see what are the problems.

<exec executable="sh" dir="${project.build.directory}" failonerror="true">
<arg line="./dist-copynativelibs.sh"/>
</exec>  回复  更多评论
  

# re: Hadoop2.5.0 build on win7 64 2014-11-22 15:35 bryan
All the hadoop windows build and eclipse plugin can be found
http://yun.baidu.com/share/home?uk=3976278079&view=share#category/type=0  回复  更多评论
  

# re: Hadoop2.5.0 build on win7 64 2014-12-21 08:45 Life is no respector of any genius.
In order to use hadoop eclipse plugins2.6.0(not include previous versions) in windows eclipse for hadoop 2.6.0, we need to do the following things.

1. use hadoop2.6.0 and config the hadoop_home and path and ensure It points to hadoop2.6.0 env.If this step is ignored, NativeCrc32 will has Unsatisfied Linkage error.

2. the plugin jar file needs to include one more jar file htrace-core-3.0.4.jar compared with previous versions.

  回复  更多评论
  

# re: Hadoop2.5.0 build on win7 64 2015-05-09 19:54 Life is no respector of any genius.

hadoop-eclipse-plugin-2.7.0.jar

1. update the following jar filename(htrace-core-3.1.0-incubating.jar) and add servlet-api-2.5.jar file to classpath. 2.Tested with eclipse-jee-luna-SR2-win32-x86_64.

http://download.csdn.net/detail/tondayong1981/8680589


hadoop-eclipse-plugin-2.6.0.jar

It takes me two days to find the root cause for the exceptions and finally generate this jar file for hadoop 2.6.0.

1.use hadoop 2.6.0.
2.change the hadoop_home and path to point to hadoop 2.6.0
3.include one more jar file htrace-core-3.0.4.jar compared with previous versions.

Tested with eclipse-jee-luna-R-win32-x86_64.
Any problem,please let me know.
http://download.csdn.net/detail/tondayong1981/8280995  回复  更多评论
  

# re: Hadoop2.5.0 build on win7 64 2015-07-12 07:46 Life is no respector of any genius.
hadoop 2.7.1 eclipse plugin
http://download.csdn.net/detail/tondayong1981/8889075

hadoop 2.7.1 win7
http://pan.baidu.com/s/1dDwF4op  回复  更多评论
  


只有注册用户登录后才能发表评论。
网站导航: