- [Tim] So, I ended the last video
by saying you can do a lot more
with the URL connects than just reading a webpage.
You can also get header information from the webpage
without having to read it in its entirety.
Let's have a go at doing that now.
So I'm gonna make some changes to the code.
So what I'm going to do for the time being
is comment out this code here,
relating to reading the webpage,
and just below the inputStream,
we're going to retrieve the headers.
We're gonna start with a Map, creating a Map. Map.
It's gonna be a string.
That's the key, and the values are going
to be a list of String objects.
Import java.util.List and java.util.Map,
and we'll call this one headerFields,
and it's gonna be reconnection.getHeaderFields
So that actually retrieves all the headers
for the webpage automatically, which is pretty cool
and let's now print those out.
So we're gonna do a for, Map.Entry
and it's String, same definition as the previous line
so it's String then List of String objects,
entry, and it'd be headerFields.entrySet
Then retrieve the key for each one,
so String key is equal to entry.getKey.
Then we wanna retrieve the individual headers for that key
so a List of String object we're creating,
we'll call it value equals entry.getValue
and print out a message here.
Key, just so it's stands out a little bit, equals plus key.
Then we need to go through, loop through,
and get all the header values from the particular key.
So for string colon value,
and we're gonna print out the value equals plus value.
Alright so there's our code.
So you can see that we're actually
retrieving the headers on line 24
and once we've retrieved that,
we're going through and iterating through the Map
and we're printing at each key and value.
And if we actually run this,
and you can see there's quite a few key values there.
Scroll through the list there.
So, we've got the version the HTTP connection is using,
when the page was last modified.
Last modified down here.
Expiry date, we've got the content length
and so on and so forth.
Now, you can use the reconnection.getHeaderField
as opposed to getHeaderFields here, that we used on line 24.
So use the .getHeaderField method
to retrieve each of these values individually,
if you're only interested in one or two.
Now, I probably didn't make it clear in the last video,
but probably you've realised this is the case anyway,
but we can't actually write to the example.org page.
The reason I am saying that is I set
this option here on line 18 to setDoOutput to true.
So there's no forms on the page that accept input.
When you want to write to URL,
you could use the reconnection class,
but as I've mentioned the reconnection
is a generic connection class.
When reading webpages or content created in feeds,
essentially anything that can be reached
over the internet using a URL,
then you'll want to use the HttpURLConnection class
which is a subclass of reconnection.
Now, the reason that you want to use that
is that it contains, this is the HttpURLConnection class,
support for HTTP specific features.
Let's just go over quickly,
what happens when your browser
requests a webpage or when you press submit,
the submit button on a form.
I'm gonna speak of this only at a very high level,
because I wanna to give you just enough information
to help you understand how
to use the HttpURLConnection class.
So when your browser asks for a webpage,
it does what's called a GET request.
Now the server that hosts the page will then respond,
and the response will include a code.
For example, the code 200 means okay.
And that means the server found the webpage,
and was able to return it to the browser with no problems.
And a code 404 is returned,
when the server can't find the requested webpage.
And you may have seen this one yourself,
the web, your browser returns a 404 error,
and in fact some sites will return a nice error page,
but others don't handle 404 errors very well,
and you just see a sort of standard generic message
in the browser telling you that the page can't be found.
Moving on, a code of 500 means that something
critically went wrong at the server side.
For example, the database the webpage
needs to access could be down.
So all of these are examples of a return code,
and we refer to that as the response code.
Now when we want to provide information to a URL,
we use a POST operation.
Now I'm not gonna be demonstrating POST here,
because the most common use of POST
is to post values to a form,
and we don't have a form to practise on.
In other words, we haven't got the server side
setup for this, and it wouldn't really
be very nice to use a form on an existing site.
In any event, you'll see some sample code about how to POST,
but you won't be able to run an application
that does so unless you've got a website to put the code on.
Alright, so with that really brief,
and simplified overview out of the way, let's actually
start working with the HttpURLConnection class.
So what I'm going to do
is comment out all this code now.
Or actually, what I'll do is I'll just comment out
everything other than the first two lines there.
So we've still got our URL example.org,
and we've got our connection that we've setup on line 13.
I'll just change the name of that to connection,
make it a bit easier.
Alright.
So by now you're familiar with how to create
a URL instance, because we've done that in the course.
After doing that, we call the url.open Connection method
to open a connection.
And in addition to that,
instead of using the URL as I mentioned,
we're gonna be using the HttpURLConnection.
So, HttpURLConnection.
Then we need to then cast that as a HttpURLConnection.
That's the call to the url.open Connection.
So cast that like that, and we're good to go.
So you can see that we've revised that a little bit.
Now for our connection, we're returning
a HttpURLConnection object this time
with a call to the url.open Connection.
And basically, the method is smart enough
as you can see, to return a HttpURLConnection.
It was actually doing the same thing
with the previous code, we just didn't use
any of the methods that were specific
to the HttpURLConnection class.
In fact, we only used methods from
the parent reconnection class.
And out of interest, another type of URL
that the method recognises, is a jar URL.
So if you provide a URL for a jar file,
the method will return a JarURLConnection instance,
and you do this when you wanna read or write
to a jar file.
And because the method recognises the HTTP protocol
and returns the appropriate connection instance,
it's safe to cast the return value
to a HttpURLConnection object in this case.
Alright, and if you've been paying attention
in previous videos I mentioned
that calling the open Connection method
doesn't actually open the connection to the destination.
You can configure the connection
before calling the connect method to actually connect.
Now we wanna read the webpage,
so we're gonna be using a GET connection.
So we've made the call to open the connection,
next we wanna do a connection.setRequestMethod,
with GET, and GET should be in uppercase as well.
Now because the default method is GET
you don't actually have to call this method
when you want to do a GET operation.
But to make it clear, we've done this here
so it's really abundantly clear with your code
and I recommend that you do add that
even though it's potentially an optional thing.
And in addition to GET and POST,
you can also set the request method
to HEAD, OPTIONS, PUT, DELETE and TRACE.
And if you're interested in what they are
and what they actually mean,
I'm gonna give you a link to find out
more information about those, because
we won't be covering many of those.
Let's paste that link in.
So you can actually read up on those
to see what these other options are.
Options GET, HEAD...
Well GET we're obviously gonna be using,
but POST, PUT, DELETE, so on and so forth.
So refer to that for specific information.
Now it's important to note here,
that each HttpURLConnection instance
can only be used to make one request.
However, the underlying connection
may actually persist across connection objects.
Remember that Java is doing a lot of work
under the covers here.
Now, when working with a high level API,
you're dealing with abstractions.
So when you call the connect method for the first time,
Java has to create a socket, and do
all the handshaking, et cetera,
that we saw in earlier videos,
when we're dealing with a low level API.
But because we're dealing with the high level API,
this is handled for us automatically.
And because the application you're creating here
might wanna perform more requests,
the underlying connection may persist,
even if you close the HttpURLConnection connection.
So having to create the socket for each request
would actually be inefficient,
which is the reason that Java does that under the covers.
So it keeps the connection open.
Now at this point, you wanna set some request properties.
I'm not gonna be able to go through everything you can set,
but what you're doing is setting the header fields,
which contain information about a request or a response.
Now as you saw earlier when getting a webpage,
the header fields contain information about that webpage.
And the good link here that I'm gonna show you,
that you can use to see a list of header fields.
So let's just quickly go to that page as well.
So you can actually read up on what the various
header field names are, and what they actually do
and an example of usage of them as well,
what they'll come back as.
And you've probably seen some of those
when we printed out the header fields earlier in this video.
Now in this case, we're setting the header fields
for the request we're making, which will provide
the server with information about the request.
So we're gonna set the user agent property as an example.
Now this tells the server what browser
or script that you're using.
We're just gonna set it to a random value of Chrome.
So I'm gonna show you how to do that,
so connection.setRequestProperty
we're gonna type in User-Agent,
making sure you get your capitalization correct,
User-Agent, and we're gonna set it to the value of Chrome.
Now don't worry if that's not what you're using by the way.
It's not the string that Chrome would use,
and some webpages would reject the request,
because hackers and spammers sometimes
use questionable user agents.
But the server hosting the example.org page doesn't baulk,
and if for some reason it does, you can just comment out
this particular line of code on line 15,
but it should accept it quite nicely.
So we set the user agent here to Chrome.
Now the setRequestProperty method
is in the reconnection parent class,
the first parameter is the key,
and the second parameter is the value.
Now if the property you're setting
can accept multiple values,
you can only use one setRequestProperty call.
You use commas to separate the values
you pass as a second parameter.
Now if you wanna do a POST on the other hand
at this point you'd wanna call connection.setDoOutput true,
as you saw previously I set to
so that you can actually write to the connection.
But obviously we're not doing that in this case.
Alright, so moving on.
We're gonna call the HttpURLConnection.getResponseCode
method, and that's gonna tell us whether
we'll be able to successfully read the page.
So we'd type something like, int responseCode
is equal to connection.getResponseCode
And let's output that.
We'll put response code plus responseCode
just to see what that code was.
And just as with sockets,
you can also set a timeout value.
So if a website is down,
or the server is slow for some reason,
your application won't hang waiting for a response.
So let's actually set a timeout.
I'm gonna do connection.setReadTimeout
We'll set that to 30,000 here, which is 30 seconds.
And obviously that's far too generous,
normally you wouldn't want to wait 30 seconds,
but in this case it's probably okay,
and you'd certainly expect a server
to respond more quickly than within 30 seconds,
but don't make the timeout too small though,
or you'll constantly time out.
So 10 or 15 seconds would probably
be a reasonable number there.
Alright, and as you can see that...
I've got that in the wrong place,
I'm gonna move that back,
and put that just before the initial call
to connection.getResponseCode.
And what we can actually do is check
to see whether the response code we've got
is the correct value.
So we can put, if responseCode is not equal to 200,
then you can put something like,
error reading web page, return.
So we could sort of exit from that point on.
So let's actually continue while I'm here,
and we're going to uncomment the code we had previously
so that we can actually read from the connection.
So I'm gonna come back here and comment that out.
So I'm going to just get rid of some of that code there.
Actually, what I'll do is I'll take a copy
of that line first,
uncomment that,
then what we're going to do is
get this code here that actually retrieves the data.
Paste that in, uncomment that.
And we'll just fix this up a little bit.
So what we're gonna do is,
we've got a BufferedReader inputReader,
we're gonna change this here, switch to an inputReader.
It goes, new BufferedReader, new InputStreamReader
which is still correct, but in this case
it's going to be connection, not reconnection
connection.getInputStream.
Then we're gonna start by assigning an empty string,
and we'll do a check while,
we're gonna change that a little bit, the while loop,
it's gonna be while, two left parentheses, line
is equal to
inputReader
.readLine,
not equal to null.
And I'm just gonna print out the line there.
Then we're going to do an inputReader.close.
And we needed another parentheses there,
to fix that up so it's happy.
And we'll get rid of that there,
we can just start with an empty string
that's defaulting to null.
So all we're really doing here
is we're setting the response codes,
as you can see there, for Chrome,
and we're setting a timeout,
we're checking to see what the response code is,
if it's not equal to 200 we're printing out
a message that there's an error reading the webpage,
and we're gonna return.
Otherwise, what we're gonna do here is we're
then going to go through and read
the contents of the actual website.
And this line here, line 33, may not be
something that's familiar to you.
What we're doing is, we're actually doing a test there,
we're both assigning the value
of whatever we've read from
the inputReader.readLine method into line,
then we're also checking to see whether that value is null.
While it's not equal to null,
we're gonna continue outputting it,
and we'll loop through to the next line and so on
until we eventually find that line is set to null
in which case it'll exit the while loop
and move on to the next line,
which will close the BufferedReader.
Alright, so let's actually try running that.
And you can see there that we've
successfully read the page okay.
We got a response code of 200 there as well,
and that's obviously why in this case
we didn't get this block of code executing,
because you actually got the response code of 200.
Now you might be thinking at this point,
"wait a minute Tim, I didn't call the connect method,
so how is this working?"
Well the thing to keep in mind here is that
some methods implicitly perform the connection step.
Now when we actually call,
the getResponseCode method down here on line 20,
it performed the connection.
In fact, the HttpURLConnection.getInputStream method
also implicitly connects.
So all the calls to connect have been unnecessary.
Essentially, if an operation depends
on being connected to work, it'll actually
perform the connection where necessary.
Now if you call connect when a connection
has already been established, the method
doesn't do anything, so there's no performance hit.
So in other words, explicitly calling connect
can make the code clearer for anyone reading it,
but feel free to leave it out when the code calls
another method that implicitly performs a connection.
Alright, so let's end the video here.
In the next video, we'll continue on
and we'll actually see what happens
when we try reading a webpage that doesn't exist,
to see whether we get the 404 response code back or not.
So I'll see you in the next video.