How to Create a Proxy Grabber/Scraper in Visual Basic

Submitted by GeePee on Sunday, April 26, 2015 - 23:38.

Introduction: Welcome to my tutorial on how to create a proxy ip:port grabber. Steps of Creation: Step 1: First create a form with one button, this will allow the user to select a save location and begin the process. We also want to import a few things and create a global string of the link in which we want to extract the proxy information from...

Imports System.Text.RegularExpressions
Imports System.Net
Imports System.IO

Dim link As String = "http://free-proxy-list.net/uk-proxy.html"

Step 2: Now we want to make a couple of function, we will use these to extract strings from bigger strings later on.

Private Function GetBetween(ByVal Source As String, ByVal Str1 As String, ByVal Str2 As String, Optional ByVal Index As Integer = 0) As String
    Return Regex.Split(Regex.Split(Source, Str1)(Index + 1), Str2)(0)
End Function
Private Function GetBetweenAll(ByVal Source As String, ByVal Str1 As String, ByVal Str2 As String) As String()
    Dim Results, T As New List(Of String)
    T.AddRange(Regex.Split(Source, Str1))
    T.RemoveAt(0)
    For Each I As String In T
        Results.Add(Regex.Split(I, Str2)(0))
    Next
    Return Results.ToArray
End Function

Step 3: For the button click event we want to first let the user select a saving text file directory. Then check if the path is not nothing/null/empty.

Dim fo As New SaveFileDialog()
fo.Filter = "Text Files|*.txt"
fo.FilterIndex = 1
fo.ShowDialog()
If (Not fo.FileName = Nothing) Then
End If

Step 4: Within the if statement we want to get the source code of the given page link and extract the information. Once we have extracted the information we write it line by line in to the text file location.

Dim r As HttpWebRequest = HttpWebRequest.Create(link)
r.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36"
r.KeepAlive = True
Dim re As HttpWebResponse = r.GetResponse()
Dim src As String = New StreamReader(re.GetResponseStream()).ReadToEnd()
Dim rows As String() = GetBetweenAll(src, "<tr>", "</tr>")
Dim tds As New List(Of String)
Dim dones As New List(Of String)
For Each s As String In rows
	If (Not s = rows(0) And s.Contains("<td>") And s.Contains("</td>")) Then
		Dim td As String() = GetBetweenAll(s, "<td>", "</td>")
		Dim ip As String = td(0)
           Dim port As String = td(1)
		dones.Add(ip & ":" & port)
	End If
Next
Using sw As New StreamWriter(fo.FileName)
	For Each s As String In dones
		sw.WriteLine(s)
	Next
End Using
MsgBox("Finished and wrote!")

Project Complete! Below is the full source code and download to the project files.

Imports System.Text.RegularExpressions
Imports System.Net
Imports System.IO
Public Class Form1
    Dim link As String = "http://free-proxy-list.net/uk-proxy.html"
    Private Function GetBetween(ByVal Source As String, ByVal Str1 As String, ByVal Str2 As String, Optional ByVal Index As Integer = 0) As String
        Return Regex.Split(Regex.Split(Source, Str1)(Index + 1), Str2)(0)
    End Function
    Private Function GetBetweenAll(ByVal Source As String, ByVal Str1 As String, ByVal Str2 As String) As String()
        Dim Results, T As New List(Of String)
        T.AddRange(Regex.Split(Source, Str1))
        T.RemoveAt(0)
        For Each I As String In T
            Results.Add(Regex.Split(I, Str2)(0))
        Next
        Return Results.ToArray
    End Function
    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Dim fo As New SaveFileDialog()
        fo.Filter = "Text Files|*.txt"
        fo.FilterIndex = 1
        fo.ShowDialog()
        If (Not fo.FileName = Nothing) Then
            Dim r As HttpWebRequest = HttpWebRequest.Create(link)
            r.UserAgent = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/30.0.1599.69 Safari/537.36"
            r.KeepAlive = True
            Dim re As HttpWebResponse = r.GetResponse()
            Dim src As String = New StreamReader(re.GetResponseStream()).ReadToEnd()
            Dim rows As String() = GetBetweenAll(src, "<tr>", "</tr>")
            Dim tds As New List(Of String)
            Dim dones As New List(Of String)
            For Each s As String In rows
                If (Not s = rows(0) And s.Contains("<td>") And s.Contains("</td>")) Then
                    Dim td As String() = GetBetweenAll(s, "<td>", "</td>")
                    Dim ip As String = td(0)
                    Dim port As String = td(1)
                    dones.Add(ip & ":" & port)
                End If
            Next
            Using sw As New StreamWriter(fo.FileName)
                For Each s As String In dones
                    sw.WriteLine(s)
                Next
            End Using
            MsgBox("Finished and wrote!")
        End If
    End Sub
End Class

Add new comment

277 views

How to Create a Proxy Grabber/Scraper in Visual Basic

Tags

Add new comment