c httpclient抓取网页(.net获取网页的源码下载网络图片要下载好 )

　　c httpclient抓取网页(.net获取网页的源码下载网络图片要下载好

)

　　C# .net 可以使用 HttpWebRequest 来请求网页，如果要获取网页的内容，可以获取网页的源代码。这样，您就可以使用.Net 来实现爬虫和下载网页文件。

　　获取网页源代码

　　//请求的URL链接

String URL = "https://baike.baidu.com/item/%E5%B7%A5%E6%97%B6%E8%AE%B0%E5%BD%95/19028114?fr=aladdin";

//创建HttpWebRequest

HttpWebRequest myrq = (HttpWebRequest)WebRequest.Create(URL);

myrq.KeepAlive = false;

myrq.Timeout = 30 * 1000; //超时时间

myrq.Method = "Get"; //请求方式

myrq.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";

myrq.Host = "baike.baidu.com"; //来源

//定义请求请求Referer

myrq.Referer = "https://www.baidu.com/link?url=krnoB2YHt94yzV5ewGRncTo8ayAJETxd_Yv2VXwmkO6wN9K401boggwFVgiPulgwix76akOoMOt72D6UBXb1WtxZoXFok4wW_BADpdDbcQk8U114CohHj0j-JPr0epo1&wd=&eqid=c0dedaf300022d3f000000025d4a87cd";

//定义浏览器代理

myrq.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4098.3 Safari/537.36";

//请求网页

HttpWebResponse myrp = (HttpWebResponse)myrq.GetResponse();

//判断请求状态

if (myrp.StatusCode != HttpStatusCode.OK)

{

return;

}

//打印网页源码

using (StreamReader sr = new StreamReader(myrp.GetResponseStream()))

{

Console.Write(sr.ReadToEnd());

}

Console.ReadKey();

　　成功获取网页源代码

　　下载网络图片

　　下载网络图片只需要对上面的代码做一些简单的改动，把网页改成图片链接，最后写入文件保存即可。

　　//图片链接

String URL = "https://gss1.bdstatic.com/-vo3dSag_xI4khGkpoWK1HF6hhy/baike/c0%3Dbaike80%2C5%2C5%2C80%2C26/sign=ee44352cdf58ccbf0fb1bd6878b1d75b/8ad4b31c8701a18bda4c8ed1902f07082938fe8d.jpg";

HttpWebRequest myrq = (HttpWebRequest)WebRequest.Create(URL);

myrq.KeepAlive = false;

myrq.Timeout = 30 * 1000; //超时时间

myrq.Method = "Get"; //请求方式

myrq.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8";

myrq.Host = "baike.baidu.com"; //来源

myrq.Referer = "https://www.baidu.com/link?url=krnoB2YHt94yzV5ewGRncTo8ayAJETxd_Yv2VXwmkO6wN9K401boggwFVgiPulgwix76akOoMOt72D6UBXb1WtxZoXFok4wW_BADpdDbcQk8U114CohHj0j-JPr0epo1&wd=&eqid=c0dedaf300022d3f000000025d4a87cd";

myrq.UserAgent = "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 UBrowser/6.2.4098.3 Safari/537.36";

HttpWebResponse myrp = (HttpWebResponse)myrq.GetResponse();

if (myrp.StatusCode != HttpStatusCode.OK)

{

return;

}

//保存图片

using (FileStream fs = new FileStream("1.jpg", FileMode.Create))

{

myrp.GetResponseStream().CopyTo(fs);

}

　　请求完成后，打开项目文件夹，找到下载的图片