from:http://pfmiles.github.io/blog/recently-hessian-deserialize-problem-and-thread-pool-executor-experience/

最近工作中遇到一个诡异的问题:别人远程调用我们的系统暴露的服务,同步调用,底层使用hessian协议做序列化;
调用方系统报空指针,反序列化失败:

2013-04-18 16:52:10,308 [AvatarRuleChargeService.java:74] [com.alibaba.itbu.billing.biz.adaptor.avatar.AvatarRuleChargeService] ERROR com.alibaba.itbu.billing.biz.adaptor.crm.ChargeProxy :: avatar charge sys error com.alibaba.dubbo.rpc.RpcException: Failed to invoke the method match in the service com.alibaba.china.ruleservice.RuleService. Tried 3 times of the providers [172.22.6.83:20980, 172.22.6.80:20980, 172.22.9.76:20980] (3/3) from the registry dubbo-reg1.hst.xyi.cn.alidc.net:9090 on the consumer 172.30.118.26 using the dubbo version 2.4.9. Last error is: Failed to invoke remote method: match, provider: dubbo://172.22.6.83:20980/com.alibaba.china.ruleservice.RuleService?anyhost=true&application=billing&check=false&default.reference.filter=dragoon&dubbo=2.4.9&interface=com.alibaba.china.ruleservice.RuleService&methods=match&pid=18616&revision=1.0-SNAPSHOT&side=consumer&timeout=5000&timestamp=1366275108588&version=1.0.0, cause: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:275)     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:155)     at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:396)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2070)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2005)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1990)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1538)     at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:94)     at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:99)     at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:83)     at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:109)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:97)     at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:128)     at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:87)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:49)     at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:135)     at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)     at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)     at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)     at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)     at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)     at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)     at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)     at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)     at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:271)     ... 28 more Caused by: java.lang.NullPointerException     at com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult.<init>(RuleServiceImpl.java:163)     ... 33 more      at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:101)     at com.alibaba.dubbo.rpc.cluster.support.AbstractClusterInvoker.invoke(AbstractClusterInvoker.java:226)     at com.alibaba.dubbo.rpc.cluster.support.wrapper.MockClusterInvoker.invoke(MockClusterInvoker.java:72)     at com.alibaba.dubbo.rpc.proxy.InvokerInvocationHandler.invoke(InvokerInvocationHandler.java:52)     at com.alibaba.dubbo.common.bytecode.proxy1.match(proxy1.java)     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)     at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)     at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)     at com.alibaba.itbu.billing.framework.aop.OpenApiLogAspect.logExecuteTime(OpenApiLogAspect.java:38)     at sun.reflect.GeneratedMethodAccessor199.invoke(Unknown Source)     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)     at java.lang.reflect.Method.invoke(Method.java:597)     at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:627)     at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:616)     at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:64)     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)     at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)     at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)     at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)     at $Proxy107.match(Unknown Source)     at com.alibaba.itbu.billing.biz.adaptor.avatar.AvatarRuleChargeService.chargeByFactor(AvatarRuleChargeService.java:72)     at com.alibaba.itbu.billing.biz.charge.times.RuleChargeByTimesProcessor.getChargeResult(RuleChargeByTimesProcessor.java:62)     at com.alibaba.itbu.billing.biz.charge.times.ChargeByTimesProcessor.charge(ChargeByTimesProcessor.java:117)     at com.alibaba.itbu.billing.biz.task.ChargeByIncInstantTimesTask.charge(ChargeByIncInstantTimesTask.java:174)     at com.alibaba.itbu.billing.biz.task.ChargeByIncInstantTimesTask$1.run(ChargeByIncInstantTimesTask.java:109)     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)     at java.lang.Thread.run(Thread.java:662) Caused by: com.alibaba.dubbo.remoting.RemotingException: com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated com.alibaba.com.caucho.hessian.io.HessianProtocolException: 'com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult' could not be instantiated     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:275)     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:155)     at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:396)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2070)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2005)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1990)     at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:1538)     at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:94)     at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:99)     at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:83)     at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:109)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:97)     at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:128)     at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:87)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:49)     at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:135)     at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)     at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)     at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)     at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)     at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)     at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)     at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)     at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)     at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)     at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.reflect.InvocationTargetException     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)     at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:271)     ... 28 more Caused by: java.lang.NullPointerException     at com.alibaba.china.ruleservice.RuleServiceImpl$DynamicPluginInvocationMatchedResult.<init>(RuleServiceImpl.java:163)     ... 33 more      at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.returnFromResponse(DefaultFuture.java:190)     at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:110)     at com.alibaba.dubbo.remoting.exchange.support.DefaultFuture.get(DefaultFuture.java:84)     at com.alibaba.dubbo.rpc.protocol.dubbo.DubboInvoker.doInvoke(DubboInvoker.java:96)     at com.alibaba.dubbo.rpc.protocol.AbstractInvoker.invoke(AbstractInvoker.java:144)     at com.alibaba.dubbo.rpc.listener.ListenerInvokerWrapper.invoke(ListenerInvokerWrapper.java:74)     at com.alibaba.dubbo.monitor.dragoon.filter.DragoonFilter.invoke0(DragoonFilter.java:82)     at com.alibaba.dubbo.monitor.dragoon.filter.DragoonFilter.invoke(DragoonFilter.java:34)     at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)     at com.alibaba.dubbo.monitor.support.MonitorFilter.invoke(MonitorFilter.java:75)     at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)     at com.alibaba.dubbo.rpc.protocol.dubbo.filter.FutureFilter.invoke(FutureFilter.java:53)     at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)     at com.alibaba.dubbo.rpc.filter.ConsumerContextFilter.invoke(ConsumerContextFilter.java:48)     at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)     at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)     at com.alibaba.dubbo.rpc.cluster.support.FailoverClusterInvoker.doInvoke(FailoverClusterInvoker.java:77)     ... 32 more 

看到这个日志第一反映是觉得RuleServiceImpl.java:163数据错误,抛空指针,但检查那块代码发现那个地方根本不可能抛空指针 —— 所有用到的变量都是new出来的;
而且,依据调用方提供的错误日志的抛出时间,我在被调用方系统的所有机器的日志里查找了一遍,没有发现对应的服务端日志;照理说服务端抛空指针,服务端也会有对应日志,但没有任何线索…

因此只好翻开了JavaDeserializer.java:275作检查,发现有这么一段:

  public JavaDeserializer(Class cl)   {     _type = cl;     _fieldMap = getFieldMap(cl);      _readResolve = getReadResolve(cl);      if (_readResolve != null) {       _readResolve.setAccessible(true);     }      Constructor []constructors = cl.getDeclaredConstructors();     long bestCost = Long.MAX_VALUE;      for (int i = 0; i < constructors.length; i++) {       Class []param = constructors[i].getParameterTypes();       long cost = 0;        for (int j = 0; j < param.length; j++) {     cost = 4 * cost;      if (Object.class.equals(param[j]))       cost += 1;     else if (String.class.equals(param[j]))       cost += 2;     else if (int.class.equals(param[j]))       cost += 3;     else if (long.class.equals(param[j]))       cost += 4;     else if (param[j].isPrimitive())       cost += 5;     else       cost += 6;       }        if (cost < 0 || cost > (1 << 48))     cost = 1 << 48;        cost += param.length << 48;        if (cost < bestCost) {         _constructor = constructors[i];         bestCost = cost;       }     }      if (_constructor != null) {       _constructor.setAccessible(true);       Class []params = _constructor.getParameterTypes();       _constructorArgs = new Object[params.length];       for (int i = 0; i < params.length; i++) {         _constructorArgs[i] = getParamArg(params[i]);       }     }   } 

看完这段后,再结合远程调用的返回结果类后恍然大悟:
上面这段代码,是hessian在反序列化的时候,用于在被反序列化的类里面找一个“得分最低”的构造函数,反序列化时会加以调用;
构造函数的“得分”规则大致是:参数越少得分越低;参数个数相同时,参数类型越接近JDK内置类得分越低
而我们的调用返回的类只有一个构造函数,当然只有这个构造函数会被选中
但是,hessian反序列化调用被选中的构造函数时,是这样来创造该构造函数需要的参数的:

protected static Object getParamArg(Class cl)   {     if (! cl.isPrimitive())       return null;     else if (boolean.class.equals(cl))       return Boolean.FALSE;     else if (byte.class.equals(cl))       return new Byte((byte) 0);     else if (short.class.equals(cl))       return new Short((short) 0);     else if (char.class.equals(cl))       return new Character((char) 0);     else if (int.class.equals(cl))       return new Integer(0);     else if (long.class.equals(cl))       return new Long(0);     else if (float.class.equals(cl))       return new Float(0);     else if (double.class.equals(cl))       return new Double(0);     else       throw new UnsupportedOperationException();   } 

可以看到,如果参数不是primitive类型,会被null代替;但不巧的是我们的构造函数内部对该参数调用了一个方法,因此抛出了空指针…
也就是说这个空指针是抛在客户端反序列化的时候而不是服务端内部,因此服务端当然找不到对应的错误日志了;
解决的办法也很简单:可以新增一个无参构造函数(无参构造函数肯定“得分”最低一定会被选中);或者修改代码保证任意参数为null的时候都不会出问题


下面这个问题更有意思,说的是ThreadPoolExecutorRejectedExecutionHandler的使用:

threadPool = new ThreadPoolExecutor(5, maxThreadNum, 5, TimeUnit.MINUTES, new SynchronousQueue<Runnable>(), tf,             new RejectedExecutionHandler() {                 public void rejectedExecution(Runnable r, ThreadPoolExecutor e) {                     logger.error("Ep thread pool exhausted, epId: " + id + "!!");                     r.run();                 }             }); 

这个代码是说,当我这个ThreadPool不够用,又不能再新增线程数的时候,由调用方线程自己来执行这个Runnable任务…
本来这看上去没什么问题,问题出在这个Runnable本身的实现上 —— 它内部将执行它的线程block到了一个blocking queue上面,当调用方主线程亲自来执行它时,使得主线程再也回不去做它自己该做的事情了,因此会出大问题…
所以这么看来,“当线程池不够用就让调用方线程自己来干”的这个策略在实际使用时要非常谨慎

这个问题是通过一个方便的thread dump分析工具: tda来查找的,因为出问题的这个应用是个多线程程序,有1600多个常驻线程,thread dump非常之大,肉眼直接看很不方便,但tda能将thread dump变得更友好易读,方便排查问题,在此推荐一下

posted @ 2016-05-17 15:22 小马歌 阅读(863) | 评论 (0)编辑 收藏
 
from:http://www.infoq.com/cn/articles/Kubernetes-system-architecture-introduction

1. 前言

Together we will ensure that Kubernetes is a strong and open container management framework for any application and in any environment, whether in a private, public or hybrid cloud.

Urs Hölzle, Google

Kubernetes作为Docker生态圈中重要一员,是Google多年大规模容器管理技术的开源版本,是产线实践经验的最佳表现[G1] 。如Urs Hölzle所说,无论是公有云还是私有云甚至混合云,Kubernetes将作为一个为任何应用,任何环境的容器管理框架无处不在。正因为如此, 目前受到各大巨头及初创公司的青睐,如Microsoft、VMWare、Red Hat、CoreOS、Mesos等,纷纷加入给Kubernetes贡献代码。随着Kubernetes社区及各大厂商的不断改进、发展,Kuberentes将成为容器管理领域的领导者。

接下来我们会用一系列文章逐一探索Kubernetes是什么、能做什么以及怎么做。

2. 什么是Kubernetes

Kubernetes是Google开源的容器集群管理系统,其提供应用部署、维护、 扩展机制等功能,利用Kubernetes能方便地管理跨机器运行容器化的应用,其主要功能如下:

1) 使用Docker对应用程序包装(package)、实例化(instantiate)、运行(run)。

2) 以集群的方式运行、管理跨机器的容器。

3) 解决Docker跨机器容器之间的通讯问题。

4) Kubernetes的自我修复机制使得容器集群总是运行在用户期望的状态。

当前Kubernetes支持GCE、vShpere、CoreOS、OpenShift、Azure等平台,除此之外,也可以直接运行在物理机上。

接下来本文主要从以下几方面阐述Kubernetes:

1) Kubernetes的主要概念。

2) Kubernetes的构件,包括Master组件、Kubelet、Proxy的详细介绍。

3. Kubernetes主要概念

3.1. Pods

Pod是Kubernetes的基本操作单元,把相关的一个或多个容器构成一个Pod,通常Pod里的容器运行相同的应用。Pod包含的容器运行在同一个Minion(Host)上,看作一个统一管理单元,共享相同的volumes和network namespace/IP和Port空间。

3.2. Services

Services也是Kubernetes的基本操作单元,是真实应用服务的抽象,每一个服务后面都有很多对应的容器来支持,通过Proxy的port和服务selector决定服务请求传递给后端提供服务的容器,对外表现为一个单一访问接口,外部不需要了解后端如何运行,这给扩展或维护后端带来很大的好处。

3.3. Replication Controllers

Replication Controller确保任何时候Kubernetes集群中有指定数量的pod副本(replicas)在运行, 如果少于指定数量的pod副本(replicas),Replication Controller会启动新的Container,反之会杀死多余的以保证数量不变。Replication Controller使用预先定义的pod模板创建pods,一旦创建成功,pod 模板和创建的pods没有任何关联,可以修改pod 模板而不会对已创建pods有任何影响,也可以直接更新通过Replication Controller创建的pods。对于利用pod 模板创建的pods,Replication Controller根据label selector来关联,通过修改pods的label可以删除对应的pods。Replication Controller主要有如下用法:

1) Rescheduling

如上所述,Replication Controller会确保Kubernetes集群中指定的pod副本(replicas)在运行, 即使在节点出错时。

2) Scaling

通过修改Replication Controller的副本(replicas)数量来水平扩展或者缩小运行的pods。

3) Rolling updates

Replication Controller的设计原则使得可以一个一个地替换pods来rolling updates服务。

4) Multiple release tracks

如果需要在系统中运行multiple release的服务,Replication Controller使用labels来区分multiple release tracks。

3.4. Labels

Labels是用于区分Pod、Service、Replication Controller的key/value键值对,Pod、Service、 Replication Controller可以有多个label,但是每个label的key只能对应一个value。Labels是Service和Replication Controller运行的基础,为了将访问Service的请求转发给后端提供服务的多个容器,正是通过标识容器的labels来选择正确的容器。同样,Replication Controller也使用labels来管理通过pod 模板创建的一组容器,这样Replication Controller可以更加容易,方便地管理多个容器,无论有多少容器。

4. Kubernetes构件

Kubenetes整体框架如下图3-1,主要包括kubecfg、Master API Server、Kubelet、Minion(Host)以及Proxy。

图3-1 Kubernetes High Level构件

4.1. Master

Master定义了Kubernetes 集群Master/API Server的主要声明,包括Pod Registry、Controller Registry、Service Registry、Endpoint Registry、Minion Registry、Binding Registry、RESTStorage以及Client, 是client(Kubecfg)调用Kubernetes API,管理Kubernetes主要构件Pods、Services、Minions、容器的入口。Master由API Server、Scheduler以及Registry等组成。从下图3-2可知Master的工作流主要分以下步骤:

1) Kubecfg将特定的请求,比如创建Pod,发送给Kubernetes Client。

2) Kubernetes Client将请求发送给API server。

3) API Server根据请求的类型,比如创建Pod时storage类型是pods,然后依此选择何种REST Storage API对请求作出处理。

4) REST Storage API对的请求作相应的处理。

5) 将处理的结果存入高可用键值存储系统Etcd中。

6) 在API Server响应Kubecfg的请求后,Scheduler会根据Kubernetes Client获取集群中运行Pod及Minion信息。

7) 依据从Kubernetes Client获取的信息,Scheduler将未分发的Pod分发到可用的Minion节点上。

下面是Master的主要构件的详细介绍:

图3-2 Master主要构件及工作流

3.1.1. Minion Registry

Minion Registry负责跟踪Kubernetes 集群中有多少Minion(Host)。Kubernetes封装Minion Registry成实现Kubernetes API Server的RESTful API接口REST,通过这些API,我们可以对Minion Registry做Create、Get、List、Delete操作,由于Minon只能被创建或删除,所以不支持Update操作,并把Minion的相关配置信息存储到etcd。除此之外,Scheduler算法根据Minion的资源容量来确定是否将新建Pod分发到该Minion节点。

3.1.2. Pod Registry

Pod Registry负责跟踪Kubernetes集群中有多少Pod在运行,以及这些Pod跟Minion是如何的映射关系。将Pod Registry和Cloud Provider信息及其他相关信息封装成实现Kubernetes API Server的RESTful API接口REST。通过这些API,我们可以对Pod进行Create、Get、List、Update、Delete操作,并将Pod的信息存储到etcd中,而且可以通过Watch接口监视Pod的变化情况,比如一个Pod被新建、删除或者更新。

3.1.3. Service Registry

Service Registry负责跟踪Kubernetes集群中运行的所有服务。根据提供的Cloud Provider及Minion Registry信息把Service Registry封装成实现Kubernetes API Server需要的RESTful API接口REST。利用这些接口,我们可以对Service进行Create、Get、List、Update、Delete操作,以及监视Service变化情况的watch操作,并把Service信息存储到etcd。

3.1.4. Controller Registry

Controller Registry负责跟踪Kubernetes集群中所有的Replication Controller,Replication Controller维护着指定数量的pod 副本(replicas)拷贝,如果其中的一个容器死掉,Replication Controller会自动启动一个新的容器,如果死掉的容器恢复,其会杀死多出的容器以保证指定的拷贝不变。通过封装Controller Registry为实现Kubernetes API Server的RESTful API接口REST, 利用这些接口,我们可以对Replication Controller进行Create、Get、List、Update、Delete操作,以及监视Replication Controller变化情况的watch操作,并把Replication Controller信息存储到etcd。

3.1.5. Endpoints Registry

Endpoints Registry负责收集Service的endpoint,比如Name:"mysql",Endpoints: ["10.10.1.1:1909","10.10.2.2:8834"],同Pod Registry,Controller Registry也实现了Kubernetes API Server的RESTful API接口,可以做Create、Get、List、Update、Delete以及watch操作。

3.1.6. Binding Registry

Binding包括一个需要绑定Pod的ID和Pod被绑定的Host,Scheduler写Binding Registry后,需绑定的Pod被绑定到一个host。Binding Registry也实现了Kubernetes API Server的RESTful API接口,但Binding Registry是一个write-only对象,所有只有Create操作可以使用, 否则会引起错误。

3.1.7. Scheduler

Scheduler收集和分析当前Kubernetes集群中所有Minion节点的资源(内存、CPU)负载情况,然后依此分发新建的Pod到Kubernetes集群中可用的节点。由于一旦Minion节点的资源被分配给Pod,那这些资源就不能再分配给其他Pod, 除非这些Pod被删除或者退出, 因此,Kubernetes需要分析集群中所有Minion的资源使用情况,保证分发的工作负载不会超出当前该Minion节点的可用资源范围。具体来说,Scheduler做以下工作:

1) 实时监测Kubernetes集群中未分发的Pod。

2) 实时监测Kubernetes集群中所有运行的Pod,Scheduler需要根据这些Pod的资源状况安全地将未分发的Pod分发到指定的Minion节点上。

3) Scheduler也监测Minion节点信息,由于会频繁查找Minion节点,Scheduler会缓存一份最新的信息在本地。

4) 最后,Scheduler在分发Pod到指定的Minion节点后,会把Pod相关的信息Binding写回API Server。

4.2. Kubelet

图3-3 Kubernetes详细构件

根据上图3-3可知Kubelet是Kubernetes集群中每个Minion和Master API Server的连接点,Kubelet运行在每个Minion上,是Master API Server和Minion之间的桥梁,接收Master API Server分配给它的commands和work,与持久性键值存储etcd、file、server和http进行交互,读取配置信息。Kubelet的主要工作是管理Pod和容器的生命周期,其包括Docker Client、Root Directory、Pod Workers、Etcd Client、Cadvisor Client以及Health Checker组件,具体工作如下:

1) 通过Worker给Pod异步运行特定的Action。

2) 设置容器的环境变量。

3) 给容器绑定Volume。

4) 给容器绑定Port。

5) 根据指定的Pod运行一个单一容器。

6) 杀死容器。

7) 给指定的Pod创建network 容器。

8) 删除Pod的所有容器。

9) 同步Pod的状态。

10) 从Cadvisor获取container info、 pod info、root info、machine info。

11) 检测Pod的容器健康状态信息。

12) 在容器中运行命令。

4.3. Proxy

Proxy是为了解决外部网络能够访问跨机器集群中容器提供的应用服务而设计的,从上图3-3可知Proxy服务也运行在每个Minion上。Proxy提供TCP/UDP sockets的proxy,每创建一种Service,Proxy主要从etcd获取Services和Endpoints的配置信息,或者也可以从file获取,然后根据配置信息在Minion上启动一个Proxy的进程并监听相应的服务端口,当外部请求发生时,Proxy会根据Load Balancer将请求分发到后端正确的容器处理。

5. 下篇主题

下篇讲述在CentOS7上用Kubernetes来管理容器。

6. 个人简介

杨章显,现就职于Cisco,主要从事WebEx SaaS服务运维,系统性能分析等工作。特别关注云计算,自动化运维,部署等技术,尤其是Go、OpenvSwitch、Docker及其生态圈技术,如Kubernetes、Flocker等Docker相关开源项目。Email: yangzhangxian@gmail.com

7. 参考资料

  1. https://github.com/GoogleCloudPlatform/kubernetes/tree/master/docs
  2. http://www.slideshare.net/rajdeep
  3. http://www.docker.com

感谢郭蕾对本文的策划和审校。

给InfoQ中文站投稿或者参与内容翻译工作,请邮件至editors@cn.infoq.com。也欢迎大家通过新浪微博(@InfoQ)或者腾讯微博(@InfoQ)关注我们,并与我们的编辑和其他读者朋友交流。

QCon北京2016】近日,极客邦科技经常被热门大片儿《QCon的后裔》攻陷,不少Q迷们都直呼膜拜美国硅谷 Airbnb 美女工程师朱赟的“撩汉”技能,女神已放话要美美的来QCon啦!男神阿里巴巴资深总监庄卓然也将在此邂逅。4月21~23日,3天的约会,等你来见证。再不行动,神都帮不了你了。约约约
posted @ 2016-04-23 11:14 小马歌 阅读(329) | 评论 (0)编辑 收藏
 

Java Collections Cheat Sheet

Java Collections cheat sheet

Every Java program tends to have one thing in common. They’ll all use Java collections! They’re so fundamental, we could not even avoid thinking to omit them from out RebelLabs cheat sheet collection. This is a tough challenge, since there’s so much you need to know about the collections framework, the implementation details, correct use cases, how to choose the right collection type, what can they do and when to turn to the third party libraries as opposed to using the built in collections in the JDK.

However, we never take the easy path, remember, we’ve tackled:

Anyway, no topic as broad as Java collections framework can be fully explained in a single A4 page, but we’ve tried to incorporate the most essential information you will need to reference again and again. The corresponding explanations and details behind the decisions are right here, in this blogpost.

GET ME A JAVA COLLECTIONS CHEAT SHEET!

We’ve also created an interactive flow of questions that you can go through to determine which Java collection you should consider using for your data. Hit Enter to get going or Esc to go back a step:

Continue reading to get a better understanding of the collection classes available in Java and when to use them.

Collections interfaces and implementations in JDK

You might think that the collections framework was always a part of the JDK. Surprise, surprise, that’s not true. Collections are one of the most fundamentals parts of the JDK, or any programming language for that matter, but it’s also one of the hardest aspects of programming to get right. On top of that it’s really challenging to make the collections library simple, coherent, and easy to use (I’m looking at you, Scala). So the Java collections library was introduced to Java in the Java 1.2 in 1998 and has stuck with us since then. Since backwards compatibility is one of the core values of the Java platform, collections haven’t changed a lot since then. However, recently in the Java 8 release, the language evolved enough to provide tools to enhance the collection interfaces without breaking backwards compatibility.

All in all, Java collections is a comprehensive library that contains the utilities to tackle almost any container-like data structure you might need. Let’s dive into which types of collections are available and outline what makes them special and why you might want to use them.

Short description of the most used collections

There are two main approaches to choosing the collection type you need in your code. The first is simple enough, but not always the most effective:

if it fits into an ArrayList, use ArrayList, otherwise you probably need a HashMap.

The other includes having an understanding of the operations you will be needing from the collection, the performance you would expect from them and what kind of data you intend to put into the collection. While the advice above can take you quite far, it’s much better to understand the options available to help you make an informed decision.

Collection interfaces you need to master

There are 4 main types of Java collections available. Naturally, the implementations for each type can be found under the corresponding interfaces. Before we start digging into these, there are 2 main umbrella interfaces you want to know about: Iterable and Collection. Typically you rarely need to think about these, but it’s important to at least understand them. The iterable interface allows one to obtain an iterator and traverse the sequence of elements by calling the next() method. It also makes it possible to iterate through elements with the syntactic sugar of the “for-each” loop:

for(E e: iterable)  

The collection interface extends the Iterable and represents an iterable group of elements which can be added, removed, or checked for presence in the collection. However, the implementations usually implement the more specific interfaces that are designed with the collection implementation in mind. Here are the most used collection interfaces.

List

List is an ordered collection of elements. Some languages call that a sequence, or you can think of if it as an array of varying length. You can add elements to it, in the middle of it, access and replace the elements using an index.

Not surprisingly, the list is one of the most common collections used. However, there’s almost no variety in the used implementations required. Yeah, in school you learned about the dreaded linked lists and how to implement them and how awesome it is to have access to the head and the tail of it. In practice, you are better using the ArrayList 99% of the time.

The reason for that is quite simple, it consumes less memory, it is friendlier to the caches, and it is in general faster than the LinkedList from the JDK. In fact, you should try not to use a LinkedList.

If you foresee that you’ll need to establish concurrent access to a list, you’ll need to worry about the synchronization yourself. Typically, that means that either you will use a CopyOnWriteArrayListwhich is thread-safe and immutable. If your data can be mutated by another thread, but the performance of the reading accesses is much more important. CopyOnWriteArrayList is your list of choice.

Set

A collection that contains no duplicate elements is a set. Just like the mathematical set, the interface denotes a collection that holds elements and basically answers a single question: is a given element contained in the set.

Set doesn’t necessarily specify the order of elements when you iterate them, but all set implementations are created with the performance of the contains method in mind.

There are two main Set implementations that you should know about: HashSet and TreeSet. HashSet hashes the elements and distributes them into buckets by the hash value. Tree set is backed by a balanced tree, which makes it ordered and navigable (so you can ask what for the previous and next elements by value). TreeSet operations will have worse complexity compared to the HashSet as a result, but bear in mind that the operations still take sublinear time of the set size, which means that for realistic values of the set sizes, it’s still quite a short time.

However, typically use-cases for sets do not require navigating from element to element, so the HashSet is the goto implementation you’ll tend to use. And it’s a good one. Just remember to correctly implement the hashCode() and equals() methods for your elements.

Map

Perhaps the most used collection type of all time — an object that maps keys to values, an associative array, a table: the map. Maps are the most versatile collection type because the association of keys to values is really what computers are all about: a dictionary, mapping object properties to their values just like javascript does, mapping filenames to their contents and metadata, mapping usernames to password hashes, session attributes, product cart items, game high scores. Wherever you look you’ll find a use case for a map.

Naturally there are different implementations that are tweaked to provide different performance trade-offs. But the default goto implementation of the map interface is undoubtedly the infamousHashMap.

The Hashmap implementation depends on the Key objects implementing the hashCode() and equals() methods correctly, so always take care of these. In return the HashMap promises you the almost constant time performance which is as amazing as you can get out of any data structure and scales amazingly well.

If you need the navigation between elements or you’d need to deal with unhashable element types, you can use a TreeMap. Using a balanced tree, just like the sets above, treemap scales appropriately and gives you a chance to iterate the contents in a sorted order.

Now we don’t want to dive into all the API methods for maps. You can put and get values, as you’d expect. But we would love to point out one super amazing method that has been available to us since Java 8: computeIfAbsent.

default V computeIfAbsent(K key,                           Function<? super K,? extends V> mappingFunction) 

What computeIfAbsent does is give you an opportunity to specify how to obtain the value for the key, if it’s not in the collection yet. Essentially it checks if the given key is contained in the map. If it happens to be so, you get the corresponding value back. If not, the mapping function is executed and the resulting values is put into the map and returned to you.

That’s more or less how caches work. Amazing, right?

Queue

queue is a collection designed to hold elements prior to processing, the elements wait in line to be consumed and can be added to the tail of the queue. Queues are those fundamental pieces of software that tie components together. When components are communicating in a system, there’s typically a queue of messages sitting between them.

Queues are a special sort of collection, because are mostly operated using the Queue interface, rather than methods inherited from the Collection interface. To add an element to a queue, use:

E e;  Queue<E> q = …  q.offer(e); // try to put the element into the queue, if the queue is full, do nothing q.poll(); // remove the head of the queue  q.peek(); // return the head of the queue without modifying it. 

There are also corresponding methods that throw exceptions if operations don’t succeed. But all in all, there are three main operations one performs on a queue: enqueue, deque and peek.
Note that you can get an iterator from the queue and process the elements similar to any other collection. It would even work for some implementations of the queue interface, but not for all.

These are the typical classes that you might use that implement the queue interface:

  • ArrayDeque – a good general purpose battle-tested implementation of a queue. ArrayDeque is backed by a resizeable array that can grow to accommodate additional elements. Most ArrayDeque operations run in amortized constant time.
  • PriorityQueue – a queue of sorted elements. The sorting is achieved either naturally or determined by a provided comparator, but the head of the queue is always the minimal element. Note that a priority queue won’t change its order if you mutate the elements inside it. If you’re curious about the performance, the mutating operations run in logarithmic time of the queue size, peeking and such — in constant, and the search is linear.

Collection utilities

Now we’ve talked about all the most common collection types you’ll encounter, it’s time to dig into the other parts of the Java collections framework. There’s an amazing class, calledCollections, that provides handy utility methods that you should be aware of. First of all let’s talk about the collection wrappers. Any collection type can have additional requirements to its functionality, these orthogonal qualities are implemented with the wrappers. The original collection is wrapped into another one that delegates the element handling logic to it.

The prime example of some functionality that is suitable for a wrapper is making a collection immutable. Indeed, you just need to ignore all the mutating operations like: put, insert, and so on. And all the non-mutating operations can be easily delegated to the actual collection implementations.

The Collections class provides a set of methods that give you just that: unmodifiable collections:

Java-collections-wrappers

Another useful type of wrapper available is the synchronized collections wrapper. Indeed, most of the collection implementations do not implement the synchronization themselves. What would you do if you need to use HashSet concurrently from multiple threads? That’s right, you call the corresponding wrapper method and use the returned set.

Collections.synchronizedSet(new HashSet<E>()); 

It’s important to remember that the wrapper does not change the implementation of the underlying collection. That is if you obtain a reference to the wrapped hash set and use that concurrently there are no guarantees.

Here’s a short list of the useful methods in the Collections class that typically are used more than others:

  • binarySearch – a fast way to find an element in a sorted List.
  • frequency – tells you how many times an element is encountered in a collection
  • min / max – returns the smallest / largest element of the collection
  • reverseOrder – provides you with a Comparator to sort elements in the descending order
  • singleton – wraps an object into the Set containing that object, the similar methods are available for the List and Map classes.

Other libraries

There are more questions that you can ask of your collection implementation and obviously there are more answers than just the stock implementations you can find in the JDK. Here are some libraries that we think are amazing and give you more functionality for the collection creation and manipulation and more collection types.

  • Guava – Google Core Libraries for Java 6+. Perhaps the default third party collection library for Java projects. Contains a magnitude of convenient methods for creating collection, like fluent builders, as well as advanced collection types.
  • Eclipse Collections – Features you want with the collections you need. Previously known as gs-collections, this library includes almost any collection you might need: primitive type collections, multimaps, bidirectional maps and so on.
  • Fastutil – Fast & compact type-specific collections for Java. Great default choice for collections of primitive types, like int or long. Also handles big collections with more than 2^31 elements.
  • JCTools – Java Concurrency Tools for the JVM. If you work on high throughput concurrent applications and need a way to increase your performance, check out JCTools. It contains lots of queues for all kinds of single / multiple producer / consumer environments and other useful classes.

Sure, this is not a definitive list, but these are the essential libraries that you should know about. You’re might be using Guava in your project already; at least once you have thought about how awesome would it be to have the collections for primitives, and so on. So check these out!

Improving JDK collections

It’s not easy to change something as fundamental as the collections library in the JDK. However, the Java 8 release had implemented a major enhancements to the collections library and more than that, with the inclusion of the default methods it made it possible to evolve the collection interfaces without breaking all of the implementations.

The progress doesn’t stop there. The upcoming Java 9 release contains the JEP 269: Convenience Factory Methods for Collections, the proposal to include the methods for easier collection creation. Finally, we would be able to specify the collections with a simpler syntax like:

  List.of(a, b, c); Set.of(d, e, f, g); Map.ofEntries(     entry(k1, v1),     entry(k2, v2),     entry(k3, v3)); 

That would be really sweet.

Conclusion

We’ve talked a lot about Java collections and different implementations of the collection interfaces you can find in the java.util package. Sets, Lists and Maps are your friends and you’ll need to apply quite a bit of imagination to come up with a program that doesn’t need the collections API. We touched on the subject of collection wrappers and how to make collections immutable and synchronize them for concurrent access.

Be sure to check out those libraries above with the alternative collections implementations. Hopefully you liked this post and the cheat sheet that tries to give you the information about Java collections on a single printable A4 sized piece of paper.

SHOW ME JAVA COLLECTIONS CHEAT SHEET!

    posted @ 2016-04-22 14:46 小马歌 阅读(484) | 评论 (0)编辑 收藏
     
         摘要: 时间 2013-04-16 09:08:44  Script Ahead, Code Behind原文  http://rednaxelafx.iteye.com/blog/1847971主题 JVM(未经许可请勿转载。希望转载请与我联系。) (如果打开此页面时浏览器有点卡住的话请耐心等待片刻。大概是ItEye的代码高亮太耗时了...  阅读全文
    posted @ 2016-04-22 09:59 小马歌 阅读(575) | 评论 (0)编辑 收藏
     
    作者:曹旭东
    链接:https://www.zhihu.com/question/24401191/answer/37601385
    来源:知乎
    著作权归作者所有。商业转载请联系作者获得授权,非商业转载请注明出处。

    下文所使用的java版本信息

    $ java -version java version "1.8.0_20" Java(TM) SE Runtime Environment (build 1.8.0_20-b26) Java HotSpot(TM) 64-Bit Server VM (build 25.20-b23, mixed mode) 


    java中的注解是一种继承自接口`java.lang.annotation.Annotation`的特殊接口。参见下面的[JLS][2]和JDK文档。

    An annotation type declaration specifies a new annotation type,  a special kind of interface type. To distinguish an annotation  type declaration from a normal interface declaration, the keyword  interface is preceded by an at-sign (@).  ...  The direct superinterface of every annotation type is java.lang.annotation.Annotation. 

    /**  * The common interface extended by all annotation types.  Note that an  * interface that manually extends this one does <i>not</i> define  * an annotation type.  Also note that this interface does not itself  * define an annotation type.  *  * More information about annotation types can be found in section 9.6 of  * <cite>The Java&trade; Language Specification</cite>.  *  * The {@link java.lang.reflect.AnnotatedElement} interface discusses  * compatibility concerns when evolving an annotation type from being  * non-repeatable to being repeatable.  *  * @author  Josh Bloch  * @since   1.5  */ public interface Annotation {     ... } 

    下面看一下具体示例。

    定义一个注解:
    import java.lang.annotation.ElementType; import java.lang.annotation.Retention; import java.lang.annotation.RetentionPolicy; import java.lang.annotation.Target;  /**  * Created by Administrator on 2015/1/18.  */ @Target(ElementType.TYPE) @Retention(RetentionPolicy.RUNTIME) public @interface TestAnnotation {      int count() default 1;  } 

    经过编译之后,注解`TestAnnotation`的字节码是这样的:
    Classfile /e:/workspace/intellij/SpringTest/target/classes/TestAnnotation.class   Last modified 2015-1-18; size 379 bytes   MD5 checksum 200dc3a75216b7a88ae17873d5dffd4f   Compiled from "TestAnnotation.java" public interface TestAnnotation extends java.lang.annotation.Annotation   minor version: 0   major version: 52   flags: ACC_PUBLIC, ACC_INTERFACE, ACC_ABSTRACT, ACC_ANNOTATION Constant pool:    #1 = Class              #14            // TestAnnotation    #2 = Class              #15            // java/lang/Object    #3 = Class              #16            // java/lang/annotation/Annotation    #4 = Utf8               SourceFile    #5 = Utf8               TestAnnotation.java    #6 = Utf8               RuntimeVisibleAnnotations    #7 = Utf8               Ljava/lang/annotation/Target;    #8 = Utf8               value    #9 = Utf8               Ljava/lang/annotation/ElementType;   #10 = Utf8               TYPE   #11 = Utf8               Ljava/lang/annotation/Retention;   #12 = Utf8               Ljava/lang/annotation/RetentionPolicy;   #13 = Utf8               RUNTIME   #14 = Utf8               TestAnnotation   #15 = Utf8               java/lang/Object   #16 = Utf8               java/lang/annotation/Annotation { } SourceFile: "TestAnnotation.java" RuntimeVisibleAnnotations:   0: #7(#8=[e#9.#10])   1: #11(#8=e#12.#13) 

    从反编译后的信息中可以看出,注解就是一个继承自`java.lang.annotation.Annotation`的接口。

    那么接口怎么能够设置属性呢?简单来说就是java通过动态代理的方式为你生成了一个实现了"接口"`TestAnnotation`的实例(对于当前的实体来说,例如类、方法、属性域等,这个代理对象是单例的),然后对该代理实例的属性赋值,这样就可以在程序运行时(如果将注解设置为运行时可见的话)通过反射获取到注解的配置信息。

    具体来说是怎么实现的呢?

    写一个使用该注解的类:
    import java.io.IOException;  /**  * Created by Administrator on 2015/1/18.  */ @TestAnnotation(count = 0x7fffffff) public class TestMain {      public static void main(String[] args) throws InterruptedException, NoSuchFieldException, IllegalAccessException, IOException {         TestAnnotation annotation = TestMain.class.getAnnotation(TestAnnotation.class);         System.out.println(annotation.count());         System.in.read();     }  } 

    反编译一下这段代码:
    Classfile /e:/workspace/intellij/SpringTest/target/classes/TestMain.class   Last modified 2015-1-20; size 1006 bytes   MD5 checksum a2d5367ea568240f078d5fb1de917550   Compiled from "TestMain.java" public class TestMain   minor version: 0   major version: 52   flags: ACC_PUBLIC, ACC_SUPER Constant pool:    #1 = Methodref          #10.#34        // java/lang/Object."<init>":()V    #2 = Class              #35            // TestMain    #3 = Class              #36            // TestAnnotation    #4 = Methodref          #37.#38        // java/lang/Class.getAnnotation:(Ljava/lang/Class;)Ljava/lang/annotation/Annotation;    #5 = Fieldref           #39.#40        // java/lang/System.out:Ljava/io/PrintStream;    #6 = InterfaceMethodref #3.#41         // TestAnnotation.count:()I    #7 = Methodref          #42.#43        // java/io/PrintStream.println:(I)V    #8 = Fieldref           #39.#44        // java/lang/System.in:Ljava/io/InputStream;    #9 = Methodref          #45.#46        // java/io/InputStream.read:()I   #10 = Class              #47            // java/lang/Object   #11 = Utf8               <init>   #12 = Utf8               ()V   #13 = Utf8               Code   #14 = Utf8               LineNumberTable   #15 = Utf8               LocalVariableTable   #16 = Utf8               this   #17 = Utf8               LTestMain;   #18 = Utf8               main   #19 = Utf8               ([Ljava/lang/String;)V   #20 = Utf8               args   #21 = Utf8               [Ljava/lang/String;   #22 = Utf8               annotation   #23 = Utf8               LTestAnnotation;   #24 = Utf8               Exceptions   #25 = Class              #48            // java/lang/InterruptedException   #26 = Class              #49            // java/lang/NoSuchFieldException   #27 = Class              #50            // java/lang/IllegalAccessException   #28 = Class              #51            // java/io/IOException   #29 = Utf8               SourceFile   #30 = Utf8               TestMain.java   #31 = Utf8               RuntimeVisibleAnnotations   #32 = Utf8               count   #33 = Integer            2147483647   #34 = NameAndType        #11:#12        // "<init>":()V   #35 = Utf8               TestMain   #36 = Utf8               TestAnnotation   #37 = Class              #52            // java/lang/Class   #38 = NameAndType        #53:#54        // getAnnotation:(Ljava/lang/Class;)Ljava/lang/annotation/Annotation;   #39 = Class              #55            // java/lang/System   #40 = NameAndType        #56:#57        // out:Ljava/io/PrintStream;   #41 = NameAndType        #32:#58        // count:()I   #42 = Class              #59            // java/io/PrintStream   #43 = NameAndType        #60:#61        // println:(I)V   #44 = NameAndType        #62:#63        // in:Ljava/io/InputStream;   #45 = Class              #64            // java/io/InputStream   #46 = NameAndType        #65:#58        // read:()I   #47 = Utf8               java/lang/Object   #48 = Utf8               java/lang/InterruptedException   #49 = Utf8               java/lang/NoSuchFieldException   #50 = Utf8               java/lang/IllegalAccessException   #51 = Utf8               java/io/IOException   #52 = Utf8               java/lang/Class   #53 = Utf8               getAnnotation   #54 = Utf8               (Ljava/lang/Class;)Ljava/lang/annotation/Annotation;   #55 = Utf8               java/lang/System   #56 = Utf8               out   #57 = Utf8               Ljava/io/PrintStream;   #58 = Utf8               ()I   #59 = Utf8               java/io/PrintStream   #60 = Utf8               println   #61 = Utf8               (I)V   #62 = Utf8               in   #63 = Utf8               Ljava/io/InputStream;   #64 = Utf8               java/io/InputStream   #65 = Utf8               read {   public TestMain();     descriptor: ()V     flags: ACC_PUBLIC     Code:       stack=1, locals=1, args_size=1          0: aload_0          1: invokespecial #1                  // Method java/lang/Object."<init>":()V          4: return       LineNumberTable:         line 7: 0       LocalVariableTable:         Start  Length  Slot  Name   Signature             0       5     0  this   LTestMain;    public static void main(java.lang.String[]) throws java.lang.InterruptedException, java.lang.NoSuchFieldException, java.lang.IllegalAccessException, java.io.IOException;     descriptor: ([Ljava/lang/String;)V     flags: ACC_PUBLIC, ACC_STATIC     Code:       stack=2, locals=2, args_size=1          0: ldc           #2                  // class TestMain          2: ldc           #3                  // class TestAnnotation          4: invokevirtual #4                  // Method java/lang/Class.getAnnotation:(Ljava/lang/Class;)Ljava/lang/annotation/Annotation;          7: checkcast     #3                  // class TestAnnotation         10: astore_1         11: getstatic     #5                  // Field java/lang/System.out:Ljava/io/PrintStream;         14: aload_1         15: invokeinterface #6,  1            // InterfaceMethod TestAnnotation.count:()I         20: invokevirtual #7                  // Method java/io/PrintStream.println:(I)V         23: getstatic     #8                  // Field java/lang/System.in:Ljava/io/InputStream;         26: invokevirtual #9                  // Method java/io/InputStream.read:()I         29: pop         30: return       LineNumberTable:         line 10: 0         line 11: 11         line 12: 23         line 13: 30       LocalVariableTable:         Start  Length  Slot  Name   Signature             0      31     0  args   [Ljava/lang/String;            11      20     1 annotation   LTestAnnotation;     Exceptions:       throws java.lang.InterruptedException, java.lang.NoSuchFieldException, java.lang.IllegalAccessException, java.io.IOException } SourceFile: "TestMain.java" RuntimeVisibleAnnotations:   0: #23(#32=I#33) 

    最后一行的代码说明,注解`TestAnnotation`的属性设置是在编译时就确定了的。(对属性的说明在[这里][1])。

    然后,运行上面的程序,通过CLHSDB在eden区找到注解实例,

    hsdb> scanoops 0x00000000e1b80000 0x00000000e3300000 TestAnnotation 0x00000000e1d6c360 com/sun/proxy/$Proxy1 

    类型`com/sun/proxy/$Proxy1`是jdk动态代理生成对象时的默认类型,其中`com.sun.proxy`是默认的包名,定义于`ReflectUtil`类的`PROXY_PACKAGE`字段中。代理类名`$PROXY1`包含两部分,其中前缀`$PROXY`是jdk种默认的代理类类名前缀(参见`java.lang.reflect.Proxy`类的javadoc),后的1是自增的结果。

    下面看一下这个代理类的内容。运行java程序时添加参数`-Dsun.misc.ProxyGenerator.saveGeneratedFiles=true`可以将转储出jdk动态代理类的class文件。若是项目较大或是使用了各种框架的话,慎用此参数。

    Classfile /e:/workspace/intellij/SpringTest/target/classes/com/sun/proxy/$Proxy1.class   Last modified 2015-1-19; size 2062 bytes   MD5 checksum 7321e44402258ba9e061275e313c5c9f public final class com.sun.proxy.$Proxy1 extends java.lang.reflect.Proxy implements TestAnnotation   minor version: 0   major version: 49   flags: ACC_PUBLIC, ACC_FINAL ... 

    太长了,只截取一部分。从中可以看到,这个代理类实现了继承自`java.lang.reflect.Proxy`类,又实现了“接口”TestAnnotation。

    接下来查看一下代理对象的内容:

    hsdb> inspect 0x00000000e1d6c360 instance of Oop for com/sun/proxy/$Proxy1 @ 0x00000000e1d6c360 @ 0x00000000e1d6c360 (size = 16) _mark: 1 _metadata._compressed_klass: InstanceKlass for com/sun/proxy/$Proxy1 h: Oop for sun/reflect/annotation/AnnotationInvocationHandler @ 0x00000000e1ce7670 Oop for sun/reflect/annotation/Annota tionInvocationHandler @ 0x00000000e1ce7670 
    其中,0xe1ce74e0是成员变量h的地址(h是定义在`java.lang.reflect.Proxy`类中的),通过查看类`AnnotationInvocationHandler`的源码可以知道注解的代理实例的值就存储在它的成员变量`memberValues`中,然后继续向下挖就好了:

    hsdb> inspect 0x00000000e1ce7670 instance of Oop for sun/reflect/annotation/AnnotationInvocationHandler @ 0x00000000e1ce7670 @ 0x00000000e1ce7670 (size =  24) _mark: 1 _metadata._compressed_klass: InstanceKlass for sun/reflect/annotation/AnnotationInvocationHandler type: Oop for java/lang/Class @ 0x00000000e1ccc5f8 Oop for java/lang/Class @ 0x00000000e1ccc5f8 memberValues: Oop for java/util/LinkedHashMap @ 0x00000000e1ce7548 Oop for java/util/LinkedHashMap @ 0x00000000e1ce7548 memberMethods: null null hsdb> inspect 0x00000000e1ce7548 instance of Oop for java/util/LinkedHashMap @ 0x00000000e1ce7548 @ 0x00000000e1ce7548 (size = 56) _mark: 1 _metadata._compressed_klass: InstanceKlass for java/util/LinkedHashMap keySet: null null values: null null table: ObjArray @ 0x00000000e1ce75b8 Oop for [Ljava/util/HashMap$Node; @ 0x00000000e1ce75b8 entrySet: null null size: 1 modCount: 1 threshold: 1 loadFactor: 0.75 head: Oop for java/util/LinkedHashMap$Entry @ 0x00000000e1ce75d0 Oop for java/util/LinkedHashMap$Entry @ 0x00000000e1ce7 5d0 tail: Oop for java/util/LinkedHashMap$Entry @ 0x00000000e1ce75d0 Oop for java/util/LinkedHashMap$Entry @ 0x00000000e1ce75d0 accessOrder: false hsdb> inspect 0x00000000e1ce75d0 instance of Oop for java/util/LinkedHashMap$Entry @ 0x00000000e1ce75d0 @ 0x00000000e1ce75d0 (size = 40) _mark: 1 _metadata._compressed_klass: InstanceKlass for java/util/LinkedHashMap$Entry hash: 94852264 key: "count" @ 0x00000000e1bd7c90 Oop for java/lang/String @ 0x00000000e1bd7c90 value: Oop for java/lang/Integer @ 0x00000000e1ce7630 Oop for java/lang/Integer @ 0x00000000e1ce7630 next: null null before: null null after: null null hsdb> inspect 0x00000000e1ce7630 instance of Oop for java/lang/Integer @ 0x00000000e1ce7630 @ 0x00000000e1ce7630 (size = 16) _mark: 1 _metadata._compressed_klass: InstanceKlass for java/lang/Integer value: 2147483647 

    最后可以看到,key=“count”, value=Integer(2147483647 = 0x7fffffff),正是在TestMain中设置的值.

    嗯,就这样吧。


    [1]: Chapter 4. The class File Format
    [2]: Chapter 9. Interfaces
    posted @ 2016-04-22 09:58 小马歌 阅读(343) | 评论 (0)编辑 收藏
     
         摘要: from:http://my.oschina.net/pangyangyang/blog/361753ElasticSearch的安装http://www.elasticsearch.org/下载最新的ElastiSearch版本。解压下载文件。cd到${esroot}/bin/,执行elasticsearch启动。使用curl -XPOST localhost:9200/_shutdown关闭E...  阅读全文
    posted @ 2016-04-15 14:03 小马歌 阅读(474) | 评论 (0)编辑 收藏
     

    我安装的MySQL版本是5.7.10 。 官网最新版本下载地址是:MySQL下载地址

    1、选择一个DMG 后  下载->安装

    安装完后会提示一句话 如下

    A temporary password is generated for root@localhost: dwstkti5xJ<5

    If you lose this password, please consult the section How to Reset the Root Password in the MySQL reference manual.

    把root@localhost: dwstkti5xJ<5 复制到一个地方 后面要用。

    mysql 默认是安装到了  usr/local/mysql  下面

    2、安装完后 在偏好设置里启动MySQL服务,点击"Start MySQL Server"



    3、添加MySQL的快捷 命令 方式 。 为什么要弄这个? 为了方便的直接在终端默认打开的目录下使用mysql xxx 命令 而不用麻烦的进入mysql的安装目录下 进行操作。 有如下两个:

    alias mysql=/usr/local/mysql/bin/mysql

    alias mysqladmin=/usr/local/mysql/bin/mysqladmin

    4、重置(修改)MySQL的root密码。为什么要重置呢?MySQL安装完后会给一个临时的密码 也就是上文中的 dwstkti5xJ<5   ,如果你不修改这个密码 使用临时密码登陆mysql 后 各种命令是不能用的,会一直提示你  需要重置密码。

    mysqladmin -u root -p password hahaha      #hahaha是我要修改为的密码  回车后 输入临时密码dwstkti5xJ<5   就算是修改成功了

    5、使用 步骤4里设置的新密码来登陆。 为什么要登陆?登陆后才能在mysql里创建数据库和各种表 等等。

    mysql -u root -p

    输入新密码 后 回车

    6、设置配置文件。为什么要弄配置文件 ?  其实到第五步 就算是安装完成了,但是正常情况下mysql需要一个配置文件,里面存放了 许多属性 比如字符编码啦  连接数啦 什么的。这个配置文件默认是在 /usr/local/mysql/support-files/  下面  叫做 my-default.cnf 。 mysql启动时默认会从下面四个位置寻找my.cnf 然后使用 , 大家都使用第一种情况,所以咱也使用。

     /etc/my.cnf      /etc/mysql/my.cnf      /usr/local/mysql/etc/my.cnf       ~/.my.cnf

    所以现在要做的就是 把my-default.cnf 复制到 /etc 的下面,并且修改名字为my.cnf 。 命令如下:

    sudo cp -rv /usr/local/mysql/support-files/my-default.cnf  /etc  #复制 配置文件 到etc下面

    sudo mv my-default.cnf my.cnf    #修改名字为my.cnf

    配置文件就算是弄好了  要想使之生效 必须重启mysql ,还是到偏好设置里 先停止 再开启。

    7、修改配置文件my.cnf 。为什么要修改呢?可以不改  本步骤只是根据一个例子 说明如何修改配置文件。本步骤修改的是字符的编码。  使用新密码登陆mysql后 

    mysql> show variables like '%char%';    #回车后  会 看到如下文字

    | Variable_name            | Value                                                  |

    | character_set_client    | utf8                                                  |

    | character_set_connection | utf8                                                  |

    | character_set_database  | latin1                                                |

    | character_set_filesystem | binary                                                |

    | character_set_results    | utf8                                                  |

    | character_set_server    | latin1                                                |

    | character_set_system    | utf8                                                  |

    | character_sets_dir      | /usr/local/mysql-5.7.10-osx10.9-x86_64/share/charsets/

    可以看到character_set_server和character_set_database的字符编码是latin1  我现在就是要把它改为utf8格式的。

    sudo chmod a+w /etc/my.cnf   #修改权限为可写  因为复制过来的这个my-default.cnf文件(现在改名为my.cnf了) 是只读权限的 要想修改里面的内容当然要改为可写权限的了。

    vi  /etc/my.cnf  #进去后开始修改  找到 [mysqld]  这个标示后在它的下面粘贴上需要配置的参数  最终效果如下


    修改完后 保存退出。

    sudo chmod a-w /etc/my.cnf   #取消my.cnf 的可写权限  因为如果不取消这个配置文件的可写权限  mysql启动时就不理你修改好的这个配置文件 就是给你忽略掉了,意思就是说 这个配置文件必须是只读的。

    修改完后再在偏好设置里重启一下mysql。

    各种字符编码的意义如下 (网上抄的)

    character_set_client  为客户端使用的字符集;

    character_set_connection 为连接数据库的字符集设置类型 如果程序没有指明连接数据库使用的字符集类型 则按照服务器端默认的字符集设置。

    character_set_database  为数据库服务器中某个库使用的字符集设定,如果建库时没有指明 将使用服务器安装时指定的字符集设置。

    character_set_results  为数据库 给客户端返回时使用的字符集设定 如果没指明 使用服务端默认的字符集。

    character_set_server  为服务器安装时指定的默认字符集设定

    character_set_system    为数据库系统使用的字符集设定


    结束。



    文/大象飞(简书作者)
    原文链接:http://www.jianshu.com/p/65ee08a4a0d0
    著作权归作者所有,转载请联系作者获得授权,并标注“简书作者”。
    posted @ 2016-04-15 10:14 小马歌 阅读(321) | 评论 (0)编辑 收藏
     
         摘要: from:http://blog.csdn.net/smartsmile2012/article/details/17316351版权声明:本文为博主原创文章,未经博主允许不得转载。[html] view plain copy print?<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML&n...  阅读全文
    posted @ 2016-04-13 13:47 小马歌 阅读(329) | 评论 (0)编辑 收藏
     
         摘要: from:http://blog.csdn.net/isea533/article/details/50412212Spring Boot 静态资源处理Spring Boot 默认的处理方式就已经足够了,默认情况下Spring Boot 使用WebMvcAutoConfiguration中配置的各种属性。建议使用Spring Boot 默认处理方式,需要自己配置的地方可以通过配置文件修改。但是如果...  阅读全文
    posted @ 2016-04-12 14:21 小马歌 阅读(1024) | 评论 (0)编辑 收藏
     

    from:http://zeroturnaround.com/rebellabs/5-unexpectedly-useful-command-line-tools-you-might-overlook/

    My previous RebelLabs post showed us that there still is a nice amount of interest among the developers in neat command line tools. This is great news as it means there are still lots of geeks reaching for their maximum l33t potential! This post will, as the title suggests, jump into 5 more tools that will hopefully tickle your fancy.

    Whilst it is hard to find a set of tools relevant for every different command line professional or enthusiast, I can still wholly recommend glancing over these tools briefly. You never know when you’ll get stuck in a terminal only environment or will need to impress a friend with your exceptional command line skills. So without further ado, here are 5 more command line tools you should consider using!

    z

    1. Install it once
    2. Forget it’s even there
    3. Profit from its productivity: z.

    OK, but what’s it all about? Well, z allows you to quickly jump from folder to folder, without having to bother writing the full absolute or relative paths. To achieve this, it simply records all the folders you visit, and then ranks them based on a combination of the frequency and time of last use. Now all you need to do is type z part/of/path and hit enter. With this, z will automagically guide you to the highest ranking match.

    To install z, download z.sh and source it in ~/.zshrc or ~/.bashrc, if you still haven’t made the inevitable jump to ZSH. Now go about your regular flow, cd-ing all over the place, and you are done!

    Alternatively, if you use the oh-my-zsh framework, simply add it in the plugins listing of ~/.zshrc.

     $ vim ~/.zshrc  plugins=(brew git mercurial mvn osx sbt scala vi-mode z) 

    By default, z stores its data in ~/.z, below are my contents after I cleared them and moved around a bit. After printing the data, you can see I moved to 2 different folders by specifying only a small part of the path, irrespective of the current folder. Take note that the directory name was matched mid-word in a case insensitive way — very handy!

    All in all I highly recommend this great tool to anyone that spends any amount of time at the command line, whatever your background, interest or use case may be.

    z

    youtube-dl

    The second tool of this post is the excellent youtube-dl. A cross-platform tool allowing you to, unsuprisingly, download Youtube videos. The easiest way to install is either via Homebrew orpip as shown below. Other download options can be found via the link above.

     $ brew install youtube-dl  $ sudo pip install --upgrade youtube_dl 

    At times, for whatever reason, you may want to download a presentation for archiving’s purposes. Or better yet, imagine you spend the weekend in the Estonian countryside, devoid of internet, yet you really wanted to see the latest Virtual JUG sessions. A simple youtube-dl https://youtu.be/PQPvZkA-6bg suffices at this point.

    youtube-dl

    Other nice things to take note of:

    • You are not limited to downloading from Youtube only, Vimeo for example is also supported.
    • Simply point to a Youtube playlist and youtube-dl will be smart enough to automatically download the full list.
    • As shown in the example image, youtube-dl will happily follow shortened URLs for you.
    • By default the highest quality video will be downloaded, however a simple youtube-dl -f $FORMAT $LINK will override this. You find available formats with the -F flag.
    • A veritable host of additional options are nicely documented and easily accessible via man youtube-dl.

    shellcheck

    I will assume that anyone spending significant some time at the command line will want to automate tasks using shell scripts. Bash itself is renowned for its many pitfalls, and even advanced scripters will from time to time bump into something unexpected. Unfortunately for us, a shell script has no undo button, and “unexpected” results may well be synonymous with “catastrophic” results.

    Long intro short: we have a valuable tool on our side to guard ourselves against exactly this,shellcheck! In essence, it is a static analyser that will tell you where your script goes wrong.

    Install shellcheck via your package manager of choice: Homebrew, Pacman, APT; build it from source, or run it inside your browser.

     $ brew install shellcheck  # pacman -S shellcheck  # apt-get install shellcheck 

    This list may be of particular interest to you as it describes code samples that shellcheck can protect you against. To run, simply execute shellcheck my-script, it will read the shebang (#!) directive to decide whether to analyse as shbash or ksh.

    shellcheck

    As shown in the example output above, there is one syntax error, it’s marked in red: spaces surrounding the assignment. However, shellcheck goes beyond that. In yellow markings is a warning of a potentially catastrophic event: what if toDelete is ever empty? Granted, the example is somewhat contrived, without –no-preserve-root there will be no damage should the statement resolve to /, but my point still stands! Finally, in green you will find general warnings on potential future mishaps that are best not to be ignored.

    As a closing remark on shellcheck: use the neat Syntastic Vim plugin to integrate the tool inside Vim itself, configuring when it should run, how it should behave, etc. The same script that generated the ouput above now looks like the following, from our favorite text editor.

    shellcheck-vim

    multitail

    For the sys-admins and dev-opses amongst us: stop using tail -f, and start using, multitail, tail on steroids! This ultimate log-viewer allows you to do a couple of really cool things that make it worth mentioning. Alternatively you could always either use tmux or screen to get in-shell multiplexing, or just use a modern terminal emulator allowing you to do the splitting like that, such as iTerm2 or Terminator, however in these cases you may miss some of the features below.

    • Show multiple windows at the same time, in the same shell using ncurses.
    • Merge multiple logs in the same window, for example the stdout and the stderr log of the same application.
    • Perform filtering dynamically via in-menu editable regexes.
    • Use the predefined color highlighters to make the logs more legible, or define your own, again with regular expressions.

    You know the drill how to install:

     $ brew install multitail  # pacman -S multitail  # apt-get install multitail 

    For an actual example, here is a screenshot of a window where I’m developing JRebel by attaching it to Tomcat, running the infamous petclinic project. At that time I needed both Tomcat’s own output and the JRebel one. Take note of the -CS flag, signifying “use this color scheme for all subsequent files”, followed by the name of the scheme.
    As both logs color nicely with the Apache rules, they are set, finally followed by the actual paths of the logfiles.

     $ multitail -CS apache "$TOMCAT_HOME/logs/catalina.out" "$HOME/.jrebel/jrebel.log" 

    multitail

    tree

    Our final tool in this post is the essential tree utility. Tree prints a nice, structured, tree-view of your directories, allowing you to instantly get an idea about the structure of your data, without having to lscd or z all over the place. If it is not yet pre-installed in your favourite *NIX, then grab it via your favourite package manager.

     $ brew install tree  # pacman -S tree  # apt-get install tree 

    In its most basic form, you simply type $ tree to print the current folder’s structure.

    tree

    Personally, I prefer to spice it up just a little bit, adding some flags to print human readable filesizes, hidden files, and a nice total sizecount: $ tree -ah --du.

    tree-fancy

    And that is about all I have to tell you about tree. It’s really convenient to grep its output, it gives you an awesome representation of the filesystem, and I’m sure you’ll love it from the first moment.

    That concludes my list of command line tools that you won’t think will change you live, but sure enough after some time you won’t imagine yourself not using them. In fact, you’ll likely curse every time you use a command-line-muggle’s computer without these great tools installed. What are the hidden gems of your command-line-fu? Share your favorite command line utilities in the comments below, I really would like to learn new tricks.

    posted @ 2016-04-06 15:08 小马歌 阅读(269) | 评论 (0)编辑 收藏
    仅列出标题
    共95页: First 上一页 4 5 6 7 8 9 10 11 12 下一页 Last