使用BrowserSession和HtmlAgilityPack通過.NET登錄到Facebook

Question

我試圖使用Rohit Agarwal的BrowserSession類與HtmlAgilityPack一起登錄並隨後在Facebook上導航。

我以前通過編寫我自己的HttpWebRequest來管理。然而，它只在我從我的瀏覽器手動獲取cookie並在每次進行新的「會話」時向請求插入新的cookie字符串時才起作用。現在我試圖使用BrowserSession來獲得更智能的導航。

下面是當前的代碼：

BrowserSession b = new BrowserSession(); 

b.Get(@"http://www.facebook.com/login.php"); 
b.FormElements["email"] = "some@email.com"; 
b.FormElements["pass"] = "xxxxxxxx"; 
b.FormElements["lsd"] = "qDhIH"; 
b.FormElements["trynum"] = "1"; 
b.FormElements["persistent_inputcheckbox"] = "1"; 

var response = b.Post(@"https://login.facebook.com/login.php?login_attempt=1"); 

上述工作正常。當我嘗試再次使用此BrowserSession來獲取另一個頁面時出現問題。我這樣做是因爲BrowserSession從最後一個響應中保存了Cookie並將它們插入到下一個請求中，因此我不必手動輸入從我的瀏覽器中獲取的cookiedata。

然而，當我嘗試做這樣的事情：

var profilePage = b.Get(@"https://m.facebook.com/profile.php?id=1111111111"); 

我回來的文檔是空的。我將不勝感激任何關於我做錯的投入。

Answer 1

對不起，我不太瞭解你提到的HTML敏捷包或BrowserSession類。但我確實嘗試了與HtmlUnit相同的場景，它工作得很好。我使用的是.NET包裝（它的源代碼可以發現here並解釋多一點here），這裏是我使用的代碼（一些細節去除，以保護無辜者）：

var driver = new HtmlUnitDriver(true); 
driver.Url = @"http://www.facebook.com/login.php"; 

var email = driver.FindElement(By.Name("email")); 
email.SendKeys("some@email.com"); 

var pass = driver.FindElement(By.Name("pass")); 
pass.SendKeys("xxxxxxxx"); 

var inputs = driver.FindElements(By.TagName("input")); 
var loginButton = (from input in inputs 
        where input.GetAttribute("value").ToLower() == "login" 
        && input.GetAttribute("type").ToLower() == "submit" 
        select input).First(); 
loginButton.Click(); 

driver.Url = @"https://m.facebook.com/profile.php?id=1111111111"; 
Assert.That(driver.Title, Is.StringContaining("Title of page goes here")); 

希望這有助於。

Answer 2

你檢查了他們的新API嗎？ http://developers.facebook.com/docs/authentication/

可以調用一個簡單的URL來獲得OAuth2.0用戶訪問令牌，並附上您的要求休息...

https://graph.facebook.com/oauth/authorize? 
    client_id=...& 
    redirect_uri=http://www.example.com/oauth_redirect 

更改REDIRECT_URI任何你想要的網址，它會使用名爲「access_token」的參數對它進行回調。獲取它並進行任何你想要的自動化SDK調用。

Answer 3

您可能需要使用WatiN (Web Application Testing In .Net)或Selenium來驅動您的瀏覽器。這將有助於確保您不必擺弄cookies並進行任何自定義工作以使後續請求能夠工作，因爲您正在模擬實際用戶。

Answer 4

今天我面臨同樣的問題。我還和HtmlAgilityPack一起合作了Rohit Agarwal的BrowserSession課。 經過一整天的試錯編程後，我發現問題是由於在後續請求中沒有設置正確的cookie而導致的。 我沒有改變初始的BrowserSession代碼來正常工作，但我添加了以下功能並略微修改了SameCookieFrom函數。最後它對我很好。

的添加/修改功能如下：

class BrowserSession{ 
    private bool _isPost; 
    private HtmlDocument _htmlDoc; 
    public CookieContainer cookiePot; //<- This is the new CookieContainer 

... 

    public string Get2(string url) 
    { 
     HtmlWeb web = new HtmlWeb(); 
     web.UseCookies = true; 
     web.PreRequest = new HtmlWeb.PreRequestHandler(OnPreRequest2); 
     web.PostResponse = new HtmlWeb.PostResponseHandler(OnAfterResponse2); 
     HtmlDocument doc = web.Load(url); 
     return doc.DocumentNode.InnerHtml; 
    } 
    public bool OnPreRequest2(HttpWebRequest request) 
    { 
     request.CookieContainer = cookiePot; 
     return true; 
    } 
    protected void OnAfterResponse2(HttpWebRequest request, HttpWebResponse response) 
    { 
     //do nothing 
    } 
    private void SaveCookiesFrom(HttpWebResponse response) 
    { 
     if ((response.Cookies.Count > 0)) 
     { 
      if (Cookies == null) 
      { 
       Cookies = new CookieCollection(); 
      }  
      Cookies.Add(response.Cookies); 
      cookiePot.Add(Cookies);  //-> add the Cookies to the cookiePot 
     } 
    } 

做些什麼：它基本上保存的cookies，從最初的「後反應」，並增加了相同的CookieContainer到後來叫請求。我不完全理解爲什麼它在初始版本中不起作用，因爲它在AddCookiesTo-function中以某種方式執行相同的操作。 （如果（Cookies！= null & & Cookies.Count> 0）request.CookieContainer.Add（Cookies）;） 無論如何，使用這些附加功能它現在應該可以正常工作。

它可以像這樣使用：

//initial "Login-procedure" 
BrowserSession b = new BrowserSession(); 
b.Get("http://www.blablubb/login.php"); 
b.FormElements["username"] = "yourusername"; 
b.FormElements["password"] = "yourpass"; 
string response = b.Post("http://www.blablubb/login.php"); 

所有後續調用應該使用：

response = b.Get2("http://www.blablubb/secondpageyouwannabrowseto"); 
response = b.Get2("http://www.blablubb/thirdpageyouwannabrowseto"); 
... 

我希望它可以幫助很多人都面臨着同樣的問題！

Answer 5

我解決了這個問題的根本原因，如果任何人關心。事實證明，Cookie正被保存在REQUEST對象的CookieContainer中，而不是響應對象。我還添加了下載文件的功能（前提是該文件是基於字符串的）。代碼絕對不是線程安全的，但對象不是線程安全的，首先：

public class BrowserSession 
{ 
    private bool _isPost; 
    private bool _isDownload; 
    private HtmlDocument _htmlDoc; 
    private string _download; 

    /// <summary> 
    /// System.Net.CookieCollection. Provides a collection container for instances of Cookie class 
    /// </summary> 
    public CookieCollection Cookies { get; set; } 

    /// <summary> 
    /// Provide a key-value-pair collection of form elements 
    /// </summary> 
    public FormElementCollection FormElements { get; set; } 

    /// <summary> 
    /// Makes a HTTP GET request to the given URL 
    /// </summary> 
    public string Get(string url) 
    { 
     _isPost = false; 
     CreateWebRequestObject().Load(url); 
     return _htmlDoc.DocumentNode.InnerHtml; 
    } 

    /// <summary> 
    /// Makes a HTTP POST request to the given URL 
    /// </summary> 
    public string Post(string url) 
    { 
     _isPost = true; 
     CreateWebRequestObject().Load(url, "POST"); 
     return _htmlDoc.DocumentNode.InnerHtml; 
    } 

    public string GetDownload(string url) 
    { 
     _isPost = false; 
     _isDownload = true; 
     CreateWebRequestObject().Load(url); 
     return _download; 
    } 

    /// <summary> 
    /// Creates the HtmlWeb object and initializes all event handlers. 
    /// </summary> 
    private HtmlWeb CreateWebRequestObject() 
    { 
     HtmlWeb web = new HtmlWeb(); 
     web.UseCookies = true; 
     web.PreRequest = new HtmlWeb.PreRequestHandler(OnPreRequest); 
     web.PostResponse = new HtmlWeb.PostResponseHandler(OnAfterResponse); 
     web.PreHandleDocument = new HtmlWeb.PreHandleDocumentHandler(OnPreHandleDocument); 
     return web; 
    } 

    /// <summary> 
    /// Event handler for HtmlWeb.PreRequestHandler. Occurs before an HTTP request is executed. 
    /// </summary> 
    protected bool OnPreRequest(HttpWebRequest request) 
    { 
     AddCookiesTo(request);    // Add cookies that were saved from previous requests 
     if (_isPost) AddPostDataTo(request); // We only need to add post data on a POST request 
     return true; 
    } 

    /// <summary> 
    /// Event handler for HtmlWeb.PostResponseHandler. Occurs after a HTTP response is received 
    /// </summary> 
    protected void OnAfterResponse(HttpWebRequest request, HttpWebResponse response) 
    { 
     SaveCookiesFrom(request, response); // Save cookies for subsequent requests 

     if (response != null && _isDownload) 
     { 
      Stream remoteStream = response.GetResponseStream(); 
      var sr = new StreamReader(remoteStream); 
      _download = sr.ReadToEnd(); 
     } 
    } 

    /// <summary> 
    /// Event handler for HtmlWeb.PreHandleDocumentHandler. Occurs before a HTML document is handled 
    /// </summary> 
    protected void OnPreHandleDocument(HtmlDocument document) 
    { 
     SaveHtmlDocument(document); 
    } 

    /// <summary> 
    /// Assembles the Post data and attaches to the request object 
    /// </summary> 
    private void AddPostDataTo(HttpWebRequest request) 
    { 
     string payload = FormElements.AssemblePostPayload(); 
     byte[] buff = Encoding.UTF8.GetBytes(payload.ToCharArray()); 
     request.ContentLength = buff.Length; 
     request.ContentType = "application/x-www-form-urlencoded"; 
     System.IO.Stream reqStream = request.GetRequestStream(); 
     reqStream.Write(buff, 0, buff.Length); 
    } 

    /// <summary> 
    /// Add cookies to the request object 
    /// </summary> 
    private void AddCookiesTo(HttpWebRequest request) 
    { 
     if (Cookies != null && Cookies.Count > 0) 
     { 
      request.CookieContainer.Add(Cookies); 
     } 
    } 

    /// <summary> 
    /// Saves cookies from the response object to the local CookieCollection object 
    /// </summary> 
    private void SaveCookiesFrom(HttpWebRequest request, HttpWebResponse response) 
    { 
     //save the cookies ;) 
     if (request.CookieContainer.Count > 0 || response.Cookies.Count > 0) 
     { 
      if (Cookies == null) 
      { 
       Cookies = new CookieCollection(); 
      } 

      Cookies.Add(request.CookieContainer.GetCookies(request.RequestUri)); 
      Cookies.Add(response.Cookies); 
     } 
    } 

    /// <summary> 
    /// Saves the form elements collection by parsing the HTML document 
    /// </summary> 
    private void SaveHtmlDocument(HtmlDocument document) 
    { 
     _htmlDoc = document; 
     FormElements = new FormElementCollection(_htmlDoc); 
    } 
} 

/// <summary> 
/// Represents a combined list and collection of Form Elements. 
/// </summary> 
public class FormElementCollection : Dictionary<string, string> 
{ 
    /// <summary> 
    /// Constructor. Parses the HtmlDocument to get all form input elements. 
    /// </summary> 
    public FormElementCollection(HtmlDocument htmlDoc) 
    { 
     var inputs = htmlDoc.DocumentNode.Descendants("input"); 
     foreach (var element in inputs) 
     { 
      string name = element.GetAttributeValue("name", "undefined"); 
      string value = element.GetAttributeValue("value", ""); 

      if (!this.ContainsKey(name)) 
      { 
       if (!name.Equals("undefined")) 
       { 
        Add(name, value); 
       } 
      } 
     } 
    } 

    /// <summary> 
    /// Assembles all form elements and values to POST. Also html encodes the values. 
    /// </summary> 
    public string AssemblePostPayload() 
    { 
     StringBuilder sb = new StringBuilder(); 
     foreach (var element in this) 
     { 
      string value = System.Web.HttpUtility.UrlEncode(element.Value); 
      sb.Append("&" + element.Key + "=" + value); 
     } 
     return sb.ToString().Substring(1); 
    } 
} 

Answer 6

我有類似的症狀 - 登錄工作，但身份驗證cookie是不存在的cookie容器，所以它不隨後的請求發送。我發現這是因爲web請求在內部處理Location：標頭，在幕後重定向到新頁面，在進程中丟失了Cookie。我通過加入以下內容來解決這個問題：

request.AllowAutoRedirect = false; // Location header messing up cookie handling! 

...到OnPreRequest（）函數。它現在看起來像這樣：

protected bool OnPreRequest(HttpWebRequest request) 
    { 
     request.AllowAutoRedirect = false; // Location header messing up cookie handling! 

     AddCookiesTo(request);    // Add cookies that were saved from previous requests 
     if (_isPost) AddPostDataTo(request); // We only need to add post data on a POST request 
     return true; 
    } 

我希望這可以幫助有人遇到同樣的問題。