Summary of Getting Data From The Web – Arduino + Ethernet
This article explains how to use an Arduino Ethernet Shield as a client to fetch data from a web page. It details parsing specific information by ignoring headers and looking for content between "" delimiters. The guide notes that DNS is not supported, requiring direct IP address access instead of URLs. It references a previous project, Twixie, and demonstrates the concept using a simple PHP file generating random strings.
Parts used in the Arduino Ethernet Client Project:
- Arduino board
- Ethernet Shield
- PHP file (web accessible server)
- Twitter API (referenced in Twixie project)
Yesterday we covered how you would go about controlling pins of your arduino over the internet using the Arduino Ethernet Shield set up as a server. Today we are going to take a look at using the shield as a client to get information off of a web page, and report back. I used this method a few months ago when I made the Nixie Twitter follower counter dubbed Twixie.
The ethernet shield can be used to access any non-password protected site with ease, but getting the information back to you is the hard part. For Twixie I created a special php page that queried the twitter API and displayed only the twitter count. This made it so I didn’t have to tell the arduino what to look for, or scour endless lines of HTML looking for a single number. For our example im going simplier. I created a PHP file that just outputs a random alphanumeric string. I did this because getting everyone setup with an API account somewhere is beyond the scope of this article, and unneeded to prove the concept and get you started. But, the idea is that you could easily take that PHP file (any web accessible file) and taylor it to display whatever you need.
In the client mode, the ethernet shield is able to access a webpage and return what is read. But the reading is done one byte at a time, and it reads the entire thing. So it can be like a needle in a haystack on large pages. Even if the page we are reading contains only the information we need, there is extra information at the beginning that is sent to the arduino. You never see it, but a web server actually sends extra information know as a “header” that tells the browser various information about that page (this is different than HTML’s tag).
Because of this, we need a way to tell the arduino what is junk, and what is the good stuff. To do this we are going to surround the information in < >. When the Arduino starts to read the page we tell it to ignore everything until it sees “<“. From this point on we tell the arduino to record each following character until it sees our ending character “>“. At this point the arduino has everything it needs, and disconnects from the server then reports back with the data it found.
The Ethernet Shield library does not come with DNS support out of the box meaning we can not simply access the website we need by its simple URL (like http://arduino.cc). No, sadly, we need to have access the site through an IP address. For instance, bildr’s IP address is 174.123.231.247 and you can actually access bildr like so http://174.123.231.247/~bildr/ – Not every web-server allows you to do this, but when you can it is typically IPADDRESS/~ACCOUNT_USERNAME – So you can see the PHP file I created for you here http://174.123.231.247/~bildr/examples/ethernet/
For more detail: Getting Data From The Web – Arduino + Ethernet
- How does the Ethernet Shield function as a client?
The shield accesses a webpage and returns what is read byte by byte. - What challenge exists when reading large web pages?
The Arduino reads the entire page including extra header information sent by the server. - How can you filter out unwanted header data?
Surround the desired information with characters so the Arduino ignores text until it finds the start delimiter. - Does the Ethernet Shield library support DNS natively?
No, the library does not come with DNS support out of the box. - How do you access a website without DNS support?
You must access the site through its IP address rather than a simple URL. - What format typically works for accessing files via IP address?
The typical format is IPADDRESS/~ACCOUNT_USERNAME. - Can this method be used for any web accessible file?
Yes, you can tailor the PHP file to display whatever information you need. - What happened after the Arduino found the ending character?
The Arduino disconnected from the server and reported back with the found data.
