c#抓取网页数据(使用C#登陆带用户名和密码的网站,并获得网页源码 )
优采云 发布时间: 2021-11-06 08:20c#抓取网页数据(使用C#登陆带用户名和密码的网站,并获得网页源码
)
诚恳请教C#登录网站获取网页和链接网页数据。使用C#以用户名密码登录网站,获取网页源代码;这个问题已经解决了!部分代码如下:
关键是这个登录网站有很多链接,我想得到这些链接页面(即子页面)的源码,怎么做?
感谢您的帮助!我哥有急事!
//获取数据时调用
privatevoidgetPage(Stringurl,StringparamList)
{
HttpWebRequestreq=(HttpWebRequest)WebRequest.Create(url);
req.Method="POST";
req.KeepAlive=true;
req.ContentType="application/x-www-form-urlencoded";
CookieContainercookieCon=newCookieContainer();
req.CookieContainer=cookieCon;
stringcookieheader=req.CookieContainer.GetCookieHeader(newUri(url));
req.CookieContainer.SetCookies(newUri(url),cookieheader);
byte[]SomeBytes=Encoding.UTF8.GetBytes(paramList);
req.ContentLength=SomeBytes.Length;
StreamnewStream=req.GetRequestStream();
newStream.Write(SomeBytes,0,SomeBytes.Length);
newStream.Close();
HttpWebResponseres=(HttpWebResponse)req.GetResponse();
StreamReceiveStream=res.GetResponseStream();
byte[]buffer=newbyte[1024];
stringfilename=@"c:\1\source.html";
StreamoutStream=File.Create(filename);
国际;
做
{
l=ReceiveStream.Read(buffer,0,buffer.Length);
if(l>0)
outStream.Write(buffer,0,l);
}while(l>0);
outStream.Close();
}
--------------------编程问答--------------------请求子页面时,附加 Cookie
....
CookieContainercookieCon=newCookieContainer();
cookieCon.SetCookies(newUri(url),"与 Url 关联的 Cookie 字符串");
req.CookieContainer=cookieCon;
--------------------编程问答--------------------
HttpWebResponse res = (HttpWebResponse)req.GetResponse(); <br />
Stream ReceiveStream = res.GetResponseStream(); <br />
StreamReader reader=new StreamReader(ReceiveStream);//套接StreamReader <br />
String source=reader.ReadToEnd();//一次读取所有相应内容<br />
............<br />
<br />
Stream outStream = File.Create(filename); <br />
StreamWriter writer=new StreamWriter(outStream);//套接StreamWriter<br />
writer.Write(source);//将读取到的内容一次写入文件<br />
outStream.Close();<br />
writer.Close();
这似乎比你的更有效率。
--------------------编程问答--------------------关注。帮顶一次! --------------------编程问答--------------------叮叮更健康---- - ---------------编程问答--------------------引用3楼gisfarmer的回复:注意。一次!
补充:.NET技术 , C#