VB.net Extract Links From Google-search Using HtmlAgilityPack

I have now updated my code as a test I want to list all URLs that has the word index.php but it also displays other things. Here is my working code: Private Sub Button1_Click(send

Solution 1:

I would use Html Agility Pack to extract the links as below

Dim links As New List(Of String)()
Dim htmlDoc As New HtmlAgilityPack.HtmlDocument()
htmlDoc.LoadHtml(WebSource)
For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//a[@href]")
    Dim att As HtmlAttribute = link.Attributes("href")
    If att.Value.Contains("/forums/") Then
        links.Add(att.Value)
    End If
Next

if it is google search result try something like below

For Each link As HtmlNode In htmlDoc.DocumentNode.SelectNodes("//cite")
    If link.InnerText.Contains("index.php") Then
        links.Add(link.InnerText)
    End If
Next