{"id":44258,"date":"2024-10-06T14:13:19","date_gmt":"2024-10-06T06:13:19","guid":{"rendered":"https:\/\/server.hk\/cnblog\/44258\/"},"modified":"2024-10-06T14:13:19","modified_gmt":"2024-10-06T06:13:19","slug":"centos-%e5%84%aa%e5%8c%96-hadoop-%e6%96%b9%e6%a1%88","status":"publish","type":"post","link":"https:\/\/server.hk\/cnblog\/44258\/","title":{"rendered":"CentOS \u512a\u5316 Hadoop \u65b9\u6848"},"content":{"rendered":"<h1 id=\"centos-%e5%84%aa%e5%8c%96-hadoop-%e6%96%b9%e6%a1%88-kPtOmVuNli\">CentOS \u512a\u5316 Hadoop \u65b9\u6848<\/h1>\n<p>\u5728\u5927\u6578\u64da\u6642\u4ee3\uff0cHadoop \u4f5c\u70ba\u4e00\u500b\u958b\u6e90\u7684\u5206\u4f48\u5f0f\u8a08\u7b97\u6846\u67b6\uff0c\u5df2\u7d93\u6210\u70ba\u6578\u64da\u8655\u7406\u7684\u4e3b\u6d41\u9078\u64c7\u3002Hadoop \u7684\u6027\u80fd\u512a\u5316\u5c0d\u65bc\u63d0\u5347\u6578\u64da\u8655\u7406\u6548\u7387\u81f3\u95dc\u91cd\u8981\uff0c\u800c CentOS \u4f5c\u70ba\u4e00\u500b\u7a69\u5b9a\u7684 Linux \u767c\u884c\u7248\uff0c\u5e38\u88ab\u7528\u4f5c Hadoop \u7684\u904b\u884c\u74b0\u5883\u3002\u672c\u6587\u5c07\u63a2\u8a0e\u5982\u4f55\u5728 CentOS \u4e0a\u512a\u5316 Hadoop \u7684\u6027\u80fd\uff0c\u5e6b\u52a9\u7528\u6236\u66f4\u6709\u6548\u5730\u5229\u7528\u5176\u8cc7\u6e90\u3002<\/p>\n<h2 id=\"1-%e7%a1%ac%e9%ab%94%e9%85%8d%e7%bd%ae-kPtOmVuNli\">1. \u786c\u9ad4\u914d\u7f6e<\/h2>\n<p>\u5728\u512a\u5316 Hadoop \u6027\u80fd\u4e4b\u524d\uff0c\u9996\u5148\u9700\u8981\u78ba\u4fdd\u786c\u9ad4\u914d\u7f6e\u7b26\u5408\u8981\u6c42\u3002\u4ee5\u4e0b\u662f\u4e00\u4e9b\u5efa\u8b70\uff1a<\/p>\n<ul>\n<li><strong>CPU\uff1a<\/strong>\u9078\u64c7\u591a\u6838\u5fc3\u7684\u8655\u7406\u5668\uff0c\u4ee5\u652f\u6301\u4e26\u884c\u8655\u7406\u3002<\/li>\n<li><strong>\u5167\u5b58\uff1a<\/strong>\u81f3\u5c11 16GB \u7684 RAM\uff0c\u5c0d\u65bc\u5927\u6578\u64da\u8655\u7406\uff0c\u5efa\u8b70\u4f7f\u7528 32GB \u6216\u66f4\u9ad8\u3002<\/li>\n<li><strong>\u5b58\u5132\uff1a<\/strong>\u4f7f\u7528 SSD \u786c\u789f\u4ee5\u63d0\u9ad8\u8b80\u5beb\u901f\u5ea6\uff0c\u4e26\u78ba\u4fdd\u6709\u8db3\u5920\u7684\u5b58\u5132\u7a7a\u9593\u3002<\/li>\n<\/ul>\n<h2 id=\"2-%e7%b3%bb%e7%b5%b1%e8%aa%bf%e6%95%b4-kPtOmVuNli\">2. \u7cfb\u7d71\u8abf\u6574<\/h2>\n<p>\u5728 CentOS \u4e0a\u9032\u884c\u4e00\u4e9b\u7cfb\u7d71\u5c64\u9762\u7684\u8abf\u6574\uff0c\u53ef\u4ee5\u986f\u8457\u63d0\u5347 Hadoop \u7684\u6027\u80fd\uff1a<\/p>\n<h3 id=\"2-1-%e8%aa%bf%e6%95%b4%e5%85%a7%e6%a0%b8%e5%8f%83%e6%95%b8-kPtOmVuNli\">2.1 \u8abf\u6574\u5167\u6838\u53c3\u6578<\/h3>\n<p>\u7de8\u8f2f \/etc\/sysctl.conf \u6587\u4ef6\uff0c\u589e\u52a0\u4ee5\u4e0b\u53c3\u6578\uff1a<\/p>\n<pre><code>vm.swappiness = 10\nfs.file-max = 100000\nnet.core.somaxconn = 1024\nnet.ipv4.tcp_max_syn_backlog = 2048\n<\/code><\/pre>\n<p>\u9019\u4e9b\u53c3\u6578\u53ef\u4ee5\u5e6b\u52a9\u7cfb\u7d71\u66f4\u6709\u6548\u5730\u7ba1\u7406\u5167\u5b58\u548c\u7db2\u7d61\u9023\u63a5\u3002<\/p>\n<h3 id=\"2-2-%e8%aa%bf%e6%95%b4-jvm-%e5%8f%83%e6%95%b8-kPtOmVuNli\">2.2 \u8abf\u6574 JVM \u53c3\u6578<\/h3>\n<p>Hadoop \u662f\u57fa\u65bc Java \u7684\uff0c\u56e0\u6b64\u8abf\u6574 JVM \u53c3\u6578\u4e5f\u975e\u5e38\u91cd\u8981\u3002\u53ef\u4ee5\u5728 Hadoop \u7684\u914d\u7f6e\u6587\u4ef6\u4e2d\u8a2d\u7f6e\uff1a<\/p>\n<pre><code>export HADOOP_HEAPSIZE=2048\nexport HADOOP_OPTS=\"-Djava.net.preferIPv4Stack=true\"\n<\/code><\/pre>\n<p>\u9019\u6a23\u53ef\u4ee5\u78ba\u4fdd Hadoop \u5728\u904b\u884c\u6642\u7372\u5f97\u8db3\u5920\u7684\u5167\u5b58\u3002<\/p>\n<h2 id=\"3-hadoop-%e9%85%8d%e7%bd%ae%e5%84%aa%e5%8c%96-kPtOmVuNli\">3. Hadoop \u914d\u7f6e\u512a\u5316<\/h2>\n<p>\u5728 Hadoop \u7684\u914d\u7f6e\u6587\u4ef6\u4e2d\u9032\u884c\u4e00\u4e9b\u8abf\u6574\uff0c\u53ef\u4ee5\u9032\u4e00\u6b65\u63d0\u5347\u6027\u80fd\uff1a<\/p>\n<h3 id=\"3-1-%e8%aa%bf%e6%95%b4-mapreduce-%e5%8f%83%e6%95%b8-kPtOmVuNli\">3.1 \u8abf\u6574 MapReduce \u53c3\u6578<\/h3>\n<p>\u5728 mapred-site.xml \u4e2d\uff0c\u53ef\u4ee5\u8a2d\u7f6e\u4ee5\u4e0b\u53c3\u6578\uff1a<\/p>\n<pre><code>&lt;property&gt;\n    &lt;name&gt;mapreduce.map.memory.mb&lt;\/name&gt;\n    &lt;value&gt;2048&lt;\/value&gt;\n&lt;\/property&gt;\n\n&lt;property&gt;\n    &lt;name&gt;mapreduce.reduce.memory.mb&lt;\/name&gt;\n    &lt;value&gt;2048&lt;\/value&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<p>\u9019\u6a23\u53ef\u4ee5\u70ba\u6bcf\u500b Map \u548c Reduce \u4efb\u52d9\u5206\u914d\u66f4\u591a\u7684\u5167\u5b58\u3002<\/p>\n<h3 id=\"3-2-%e8%aa%bf%e6%95%b4-hdfs-%e5%8f%83%e6%95%b8-kPtOmVuNli\">3.2 \u8abf\u6574 HDFS \u53c3\u6578<\/h3>\n<p>\u5728 hdfs-site.xml \u4e2d\uff0c\u53ef\u4ee5\u8a2d\u7f6e\u4ee5\u4e0b\u53c3\u6578\uff1a<\/p>\n<pre><code>&lt;property&gt;\n    &lt;name&gt;dfs.replication&lt;\/name&gt;\n    &lt;value&gt;3&lt;\/value&gt;\n&lt;\/property&gt;\n<\/code><\/pre>\n<p>\u9019\u6a23\u53ef\u4ee5\u78ba\u4fdd\u6578\u64da\u7684\u9ad8\u53ef\u7528\u6027\uff0c\u4f46\u4e5f\u8981\u6839\u64da\u5be6\u969b\u60c5\u6cc1\u8abf\u6574\u526f\u672c\u6578\u91cf\uff0c\u4ee5\u907f\u514d\u4e0d\u5fc5\u8981\u7684\u8cc7\u6e90\u6d6a\u8cbb\u3002<\/p>\n<h2 id=\"4-%e7%9b%a3%e6%8e%a7%e8%88%87%e8%aa%bf%e8%a9%a6-kPtOmVuNli\">4. \u76e3\u63a7\u8207\u8abf\u8a66<\/h2>\n<p>\u512a\u5316 Hadoop \u6027\u80fd\u7684\u904e\u7a0b\u4e2d\uff0c\u76e3\u63a7\u7cfb\u7d71\u7684\u904b\u884c\u72c0\u614b\u81f3\u95dc\u91cd\u8981\u3002\u53ef\u4ee5\u4f7f\u7528\u4ee5\u4e0b\u5de5\u5177\u9032\u884c\u76e3\u63a7\uff1a<\/p>\n<ul>\n<li><strong>Ganglia\uff1a<\/strong>\u4e00\u500b\u958b\u6e90\u7684\u96c6\u7fa4\u76e3\u63a7\u7cfb\u7d71\uff0c\u53ef\u4ee5\u5be6\u6642\u76e3\u63a7\u96c6\u7fa4\u7684\u6027\u80fd\u3002<\/li>\n<li><strong>Ambari\uff1a<\/strong>\u63d0\u4f9b Hadoop \u96c6\u7fa4\u7684\u7ba1\u7406\u548c\u76e3\u63a7\u529f\u80fd\uff0c\u65b9\u4fbf\u7528\u6236\u9032\u884c\u6027\u80fd\u8abf\u6574\u3002<\/li>\n<\/ul>\n<h2 id=\"%e7%b8%bd%e7%b5%90-kPtOmVuNli\">\u7e3d\u7d50<\/h2>\n<p>\u5728 CentOS \u4e0a\u512a\u5316 Hadoop \u7684\u6027\u80fd\u9700\u8981\u5f9e\u786c\u9ad4\u914d\u7f6e\u3001\u7cfb\u7d71\u8abf\u6574\u3001Hadoop \u914d\u7f6e\u4ee5\u53ca\u76e3\u63a7\u7b49\u591a\u65b9\u9762\u5165\u624b\u3002\u901a\u904e\u5408\u7406\u7684\u914d\u7f6e\u548c\u8abf\u6574\uff0c\u53ef\u4ee5\u986f\u8457\u63d0\u5347 Hadoop \u7684\u6578\u64da\u8655\u7406\u6548\u7387\uff0c\u5f9e\u800c\u66f4\u597d\u5730\u652f\u6301\u5927\u6578\u64da\u61c9\u7528\u7684\u9700\u6c42\u3002\u5982\u679c\u60a8\u9700\u8981\u9032\u4e00\u6b65\u7684\u652f\u6301\u6216\u5c0b\u627e\u5408\u9069\u7684 <a href=\"https:\/\/server.hk\">VPS<\/a> \u89e3\u6c7a\u65b9\u6848\uff0c\u8acb\u8a2a\u554f\u6211\u5011\u7684\u7db2\u7ad9\u4ee5\u7372\u53d6\u66f4\u591a\u4fe1\u606f\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u63a2\u7d22CentOS\u512a\u5316Hadoop\u65b9\u6848\uff0c\u63d0\u5347\u6027\u80fd\u8207\u6548\u7387\uff0c\u5be6\u73fe\u5927\u6578\u64da\u8655\u7406\u7684\u6700\u4f73\u89e3\u6c7a\u65b9\u6848\u3002<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4924],"tags":[],"class_list":["post-44258","post","type-post","status-publish","format-standard","hentry","category-setup-tutorials"],"_links":{"self":[{"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/posts\/44258","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/comments?post=44258"}],"version-history":[{"count":1,"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/posts\/44258\/revisions"}],"predecessor-version":[{"id":44259,"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/posts\/44258\/revisions\/44259"}],"wp:attachment":[{"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/media?parent=44258"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/categories?post=44258"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/server.hk\/cnblog\/wp-json\/wp\/v2\/tags?post=44258"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}