{"id":164,"date":"2012-04-03T18:20:24","date_gmt":"2012-04-04T01:20:24","guid":{"rendered":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/?p=164"},"modified":"2014-06-09T00:04:30","modified_gmt":"2014-06-08T23:04:30","slug":"problems-with-rubys-httpclient-and-netntlm","status":"publish","type":"post","link":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/problems-with-rubys-httpclient-and-netntlm\/","title":{"rendered":"Problems with Ruby&#8217;s httpclient and Net::NTLM"},"content":{"rendered":"<p>I&#8217;ve been trying to use Ruby libraries to access Sharepoint and am having trouble getting to the starting line.\u00a0 NTLM authentication from Ruby doesn&#8217;t seem to be in a good state.\u00a0 I&#8217;m using httpi-0.9.6 as my HTTP library, which by default uses httpclient-2.2.4, and for NTLM support I&#8217;m using httpi-ntlm-0.9.6.\u00a0 All this is running under Ruby 1.8.7 which incorporates Net::NTLM 0.1.1 which http-ntlm relies on.<\/p>\n<p>In my enterprise environment, I need to use NTLM authentication to access the Sharepoint server.\u00a0 So, I format the username string supplied to httpi as &#8220;USERDOMAIN\\username&#8221;.\u00a0 It happens that the Sharepoint machine I want to access is in a different NTLM domain than the user account that I want to access it with.\u00a0 When httpclient does the NTLM authentication phase, it uses facilities from Net::NTLM to format a <a title=\"Type 1 Message\" href=\"http:\/\/davenport.sourceforge.net\/ntlm.html#theType1Message\">Type 1 Message<\/a>.\u00a0 This is where the problems start.\u00a0 Firstly, httpclient doesn&#8217;t do the right (undocumented) things necessary to get Net::NTLM to properly include the user&#8217;s domain into the request.<\/p>\n<pre class=\"brush: ruby; first-line: 472; highlight: [475]; title: httpclient-2.2.4\/lib\/httpclient\/auth.rb; notranslate\" title=\"httpclient-2.2.4\/lib\/httpclient\/auth.rb\">\r\ncase state\r\n  when :init\r\n    t1 = Net::NTLM::Message::Type1.new\r\n    t1.domain = domain if domain\r\n    return t1.encode64\r\nend\r\n<\/pre>\n<p>The trouble here is that this code doesn&#8217;t properly tell Net::NTLM that this domain information should be incorporated into the request.\u00a0 As written, HTTPI generates a corrupt request, as can be seen from a Wireshark trace.\u00a0 It&#8217;s additionally necessary to enable the resulting SecurityBuffer and to set a flag in the request indicating that the domain is being signalled:<\/p>\n<pre class=\"brush: ruby; first-line: 472; highlight: [475,476,477,478,479]; title: httpclient-2.2.4bis\/lib\/httpclient\/auth.rb; notranslate\" title=\"httpclient-2.2.4bis\/lib\/httpclient\/auth.rb\">\r\ncase state\r\n  when :init\r\n    t1 = Net::NTLM::Message::Type1.new\r\n    if domain\r\n      t1.domain = domain\r\n      t1.enable(:domain)\r\n      t1.set_flag(:DOMAIN_SUPPLIED)\r\n    end\r\n    return t1.encode64\r\n<\/pre>\n<p>But when you do this, you find that the packet is still corrupt, because the offset to the domain name data in the resulting request is incorrect.\u00a0 This is because of what appears to be a bug in Net::NTLM.\u00a0 That&#8217;s not so easy to fix and play about with, because it&#8217;s incorporated into Ruby itself.\u00a0 The problem is that the offsets are calculated by adding the sizes of the SecurityBuffers defined for the message type, <strong>regardless of whether a particular SecurityBuffer is active or not.<\/strong> For my testing purposes, I forced the additional &#8220;workstation&#8221; SecurityBuffer to be active (with an arbitrary non-null value) in the same way as the Domain, and by doing so, was able to generate a valid request.<\/p>\n<p>But imagine my horror when my requests were <strong>still<\/strong> denied by the server.\u00a0 Further investigation revealed that Net::NTLM ignores the user-specified domain when generating the subsequent <a title=\"Type 3 Message\" href=\"http:\/\/davenport.sourceforge.net\/ntlm.html#theType3Message\">Type 3 Message<\/a>, substituting instead the Target value returned in the Challenge of the <a title=\"Type 3 Message\" href=\"http:\/\/davenport.sourceforge.net\/ntlm.html#theType3Message\">Type 2 Message<\/a>.<\/p>\n<p>Further hackery will be required to fix that, but another approach might be to use Curb instead of httpclient.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I&#8217;ve been trying to use Ruby libraries to access Sharepoint and am having trouble getting to the starting line.\u00a0 NTLM authentication from Ruby doesn&#8217;t seem to be in a good state.\u00a0 I&#8217;m using httpi-0.9.6 as my HTTP library, which by default uses httpclient-2.2.4, and for NTLM support I&#8217;m using httpi-ntlm-0.9.6.\u00a0 All this is running under [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[25],"tags":[27,26],"class_list":["post-164","post","type-post","status-publish","format-standard","hentry","category-software","tag-ruby","tag-sharepoint"],"_links":{"self":[{"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/posts\/164","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/comments?post=164"}],"version-history":[{"count":7,"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/posts\/164\/revisions"}],"predecessor-version":[{"id":187,"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/posts\/164\/revisions\/187"}],"wp:attachment":[{"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/media?parent=164"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/categories?post=164"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.lemongrass.org.uk\/lemongrass\/wp-json\/wp\/v2\/tags?post=164"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}